BAB 03

03 - Machine Learning Fundamentals: Scikit-learn, XGBoost, Classical ML

Estimasi Waktu: 50 menit Level: Beginner → Intermediate

---

Kenapa Classical ML Masih Penting?

Sebelum lo lompat ke LLM dan deep learning, lo harus paham classical ML. Kenapa?

1. Fondasi konsep — supervised, unsupervised, evaluation, overfitting, cross-validation: semua konsep ini tetap dipakai di deep learning
2. Practical wins — banyak problem bisnis bisa diselesaikan dengan XGBoost, gak perlu neural network
3. Interview — 80% pertanyaan ML interview adalah classical ML concepts
4. Data understanding — classical ML memaksa lo paham data, bukan cuma “lempar ke neural network”

---

Supervised Learning

Model belajar dari data yang punya label (input → output yang diketahui).

1. Regression (Prediksi Angka)

Problem: Prediksi harga rumah, prediksi revenue, prediksi temperature.

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Linear Regression
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(f"R²: {r2_score(y_test, predictions):.3f}")

Algorithms: Linear Regression, Ridge, Lasso, Random Forest Regressor, XGBoost Regressor

2. Classification (Prediksi Kategori)

Problem: Spam detection, sentiment analysis, fraud detection, churn prediction.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(classification_report(y_test, predictions))

Algorithms: Logistic Regression, Decision Trees, Random Forest, XGBoost, SVM

---

Unsupervised Learning

Model belajar dari data tanpa label — menemukan pola tersembunyi.

1. Clustering (Pengelompokan)

Problem: Customer segmentation, anomaly detection, document grouping.

from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

# Reduce dimensions untuk visualisasi
pca = PCA(n_components=2)
reduced = pca.fit_transform(data)

# Clustering
kmeans = KMeans(n_clusters=5, random_state=42)
clusters = kmeans.fit_predict(data)

2. Dimensionality Reduction

Problem: Data terlalu banyak kolom, visualisasi, noise reduction.

from sklearn.decomposition import PCA

pca = PCA(n_components=50)  # reduce 1000 cols → 50
reduced = pca.fit_transform(data)
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.2%}")

---

The ML Workflow

DATA → CLEAN → SPLIT → TRAIN → EVALUATE → TUNE → DEPLOY

1. Data Preparation

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder

# Handle missing values
df = df.fillna(df.median())

# Encode categorical
le = LabelEncoder()
df["category"] = le.fit_transform(df["category"])

# Scale numeric features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

2. Model Training & Evaluation

from sklearn.model_selection import cross_val_score

# Cross-validation
scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"CV scores: {scores}")
print(f"Mean: {scores.mean():.3f} ± {scores.std():.3f}")

3. Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 5, 10, None],
    "learning_rate": [0.01, 0.1, 0.3],
}

grid = GridSearchCV(
    XGBClassifier(),
    param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")

---

XGBoost: The King of Tabular Data

XGBoost adalah algoritma yang mendominasi kompetisi Kaggle dan production ML untuk data tabular.

Kenapa XGBoost:

• Performance — state-of-the-art untuk tabular data
• Speed — optimized C++ backend, GPU support
• Robust — handle missing values, outliers, imbalanced data
• Interpretable — feature importance, SHAP values

import xgboost as xgb

# Training
model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    early_stopping_rounds=10,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False,
)

# Feature importance
importances = model.feature_importances_
for i, imp in enumerate(importances):
    print(f"Feature {i}: {imp:.4f}")

---

Critical ML Concepts

Overfitting vs Underfitting

• Overfitting: Model terlalu “menghafal” training data, performa buruk di test data
• Underfitting: Model terlalu sederhana, gak bisa capture pola
• Solusi: Cross-validation, regularisasi, early stopping

Bias-Variance Tradeoff

• High bias = underfitting (model terlalu sederhana)
• High variance = overfitting (model terlalu kompleks)
• Goal: Sweet spot di tengah

Feature Engineering

• Ini adalah 80% pekerjaan ML engineer
• Buat features yang informatif dari raw data
• Domain knowledge > fancy algorithms

Evaluation Metrics

Problem	Metric	Kapan Pakai
Classification	Accuracy	Balanced classes
Classification	Precision	False positive mahal
Classification	Recall	False negative mahal
Classification	F1	Balance precision + recall
Classification	ROC-AUC	Ranking quality
Regression	MSE/MAE	Standard
Regression	R²	Variance explained

---

From Classical ML to Deep Learning

CLASSICAL ML                         DEEP LEARNING
─────────────                        ─────────────
Manual feature engineering     →     Automatic feature learning
Work well with small data      →     Need lots of data
Interpretable                  →     Black box
Fast to train                  →     Slow to train
CPU is enough                  →     GPU needed
Great for tabular data         →     Great for images, text, audio

---

Latihan

1. Download dataset dari Kaggle (Titanic, House Prices, atau Iris)
2. Full ML pipeline: load → clean → feature engineering → train → evaluate → tune
3. Compare 3 algorithms (Logistic Regression, Random Forest, XGBoost)
4. Tulis laporan singkat: mana yang terbaik, kenapa?
5. Simpan model dengan joblib atau pickle

Target: 1 end-to-end ML project, paham full workflow dari data ke model.