Model Evaluation

1. Classification Metrics

The Confusion Matrix

$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Precision-Recall Tradeoff

Increasing recall (catching more true positives) typically lowers precision (more false positives creep in). The right balance is a business decision:

• High Recall priority: fraud detection, medical diagnosis — missing a real case is very costly (high cost of FN).
• High Precision priority: content moderation, loan approval — a false alarm is very costly (high cost of FP).

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.metrics import precision_recall_curve, average_precision_score

y_pred  = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print("ROC-AUC: ", roc_auc_score(y_test, y_proba))
print("PR-AUC:  ", average_precision_score(y_test, y_proba))

2. ROC-AUC vs. PR-AUC for Imbalanced Data

Why ROC-AUC is optimistic on imbalanced data: FPR = $FP / (FP + TN)$. When the negative class is enormous (e.g., 99.9% of data), even thousands of false positives produce a tiny FPR — making the ROC curve appear excellent despite poor real-world precision. PR-AUC uses Precision (which is sensitive to FP growth) and is therefore more informative.

import matplotlib.pyplot as plt
from sklearn.metrics import RocCurveDisplay, PrecisionRecallDisplay

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

RocCurveDisplay.from_predictions(y_test, y_proba, ax=axes[0])
axes[0].set_title("ROC Curve")

PrecisionRecallDisplay.from_predictions(y_test, y_proba, ax=axes[1])
axes[1].set_title("Precision-Recall Curve")

plt.tight_layout()
plt.show()

3. Handling Imbalanced Data

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier

# CORRECT order: split first, then apply SMOTE to train only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)

smote = SMOTE(random_state=42)
X_train_res, y_train_res = smote.fit_resample(X_train, y_train)   # train only

# Class weights alternative (often cleaner)
model = XGBClassifier(scale_pos_weight=99, eval_metric='aucpr')   # 99:1 imbalance ratio
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

4. Validation Strategies & Data Leakage

Cross-Validation Methods

from sklearn.model_selection import TimeSeriesSplit, cross_val_score

# TimeSeriesSplit: always trains on past, validates on future
tscv = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(model, X, y, cv=tscv, scoring='average_precision')
print(f"PR-AUC (time-series CV): {scores.mean():.4f} ± {scores.std():.4f}")

Common Data Leakage Sources

Decision Threshold Tuning

Default threshold is 0.5 — but this is rarely optimal. The right threshold is determined by the business cost of FP vs. FN:

import numpy as np
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

# Find threshold that maximizes F1
f1_scores = 2 * precision * recall / (precision + recall + 1e-8)
best_idx   = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]
print(f"Best threshold: {best_threshold:.3f}")
print(f"Precision: {precision[best_idx]:.3f}, Recall: {recall[best_idx]:.3f}")

# Apply custom threshold
y_pred_custom = (y_proba >= best_threshold).astype(int)

5. Multi-Class Metrics

Binary classification metrics extend naturally to multiple classes, but the averaging strategy matters: choose based on whether class imbalance and per-class performance both matter to your use case.

Averaging Strategies

from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns, matplotlib.pyplot as plt

y_true = [0, 1, 2, 0, 1, 2, 2, 0, 1]
y_pred = [0, 2, 2, 0, 1, 1, 2, 0, 1]

# Full report with all averaging strategies
print(classification_report(y_true, y_pred, target_names=['class_0', 'class_1', 'class_2']))
# Shows per-class precision/recall/F1 + macro/micro/weighted averages

# Confusion matrix
cm = confusion_matrix(y_true, y_pred)
sns.heatmap(cm, annot=True, fmt='d', xticklabels=['c0','c1','c2'], yticklabels=['c0','c1','c2'])
plt.ylabel('True'); plt.xlabel('Predicted')

Multi-Class ROC and OvR Strategy

Extend ROC-AUC to multiple classes using One-vs-Rest (OvR): train a binary classifier for each class vs. all others, compute AUC for each, then average (macro or weighted). sklearn.metrics.roc_auc_score(y_true, y_score, multi_class='ovr').

Per-Class Inspection

Always inspect per-class metrics, not just the aggregate. A weighted F1 of 0.92 can hide a class with F1 = 0.20 if that class is rare. Common pattern: report macro F1 for business stakeholders ("we perform equally well across all categories") and weighted F1 for technical comparison against baselines.

6. Model Calibration

A calibrated model produces probability outputs that match empirical frequencies. If a model says 70% for 100 instances, roughly 70 of them should be positive. Calibration is critical when predicted probabilities drive decisions (threshold selection, expected value calculations, risk scoring).

Brier Score

Mean squared error of probability predictions:

$$\text{BS} = \frac{1}{N}\sum_{i=1}^{N}(f_i - o_i)^2$$

where $f_i$ is the predicted probability and $o_i \in \{0,1\}$ is the outcome. Range: 0 (perfect) to 1 (worst). A naive model that always predicts the base rate $\bar{p}$ gives $\text{BS} = \bar{p}(1-\bar{p})$. Lower is better.

Expected Calibration Error (ECE)

Groups predictions into $M$ bins by predicted probability; computes weighted average gap between mean predicted probability and actual positive rate:

$$\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$$

A reliability diagram plots confidence on the x-axis vs. accuracy on the y-axis. A perfectly calibrated model lies on the diagonal.

from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt

# Brier score
bs = brier_score_loss(y_true, y_prob)
print(f"Brier Score: {bs:.4f}")

# Reliability diagram
fraction_of_positives, mean_predicted_value = calibration_curve(y_true, y_prob, n_bins=10)
plt.plot(mean_predicted_value, fraction_of_positives, 's-', label='Model')
plt.plot([0, 1], [0, 1], '--', label='Perfect calibration')
plt.xlabel('Mean predicted probability'); plt.ylabel('Fraction of positives')

# Platt scaling (post-hoc calibration via logistic regression)
calibrated = CalibratedClassifierCV(base_model, method='sigmoid', cv=5)
calibrated.fit(X_train, y_train)
y_prob_cal = calibrated.predict_proba(X_test)[:, 1]
print(f"Calibrated Brier: {brier_score_loss(y_test, y_prob_cal):.4f}")

Calibration Methods

Important: Calibration and discrimination (AUC-ROC) are orthogonal. A model can be well-discriminating but poorly calibrated (most tree models and neural networks without calibration). Always calibrate before using probabilities for decisions.

7. Quantile Regression and Prediction Intervals

Standard regression predicts a conditional mean $E[y|X]$. Quantile regression predicts a conditional quantile $Q_\tau(y|X)$ — the value below which a fraction $\tau$ of outcomes fall. This lets you model uncertainty and produce prediction intervals.

Quantile Loss (Pinball Loss)

$$L_\tau(y, \hat{y}) = \begin{cases} \tau(y - \hat{y}) & \text{if } y \geq \hat{y} \\ (1 - \tau)(\hat{y} - y) & \text{if } y < \hat{y} \end{cases}$$

Minimizing this over a training set gives the $\tau$-th conditional quantile. For $\tau = 0.5$, this is equivalent to MAE (median regression).

from sklearn.ensemble import GradientBoostingRegressor
import numpy as np

X_train, y_train = ...

# Train separate models for lower (5th), median (50th), upper (95th) quantiles
models = {}
for q in [0.05, 0.50, 0.95]:
    gbr = GradientBoostingRegressor(loss='quantile', alpha=q,
                                     n_estimators=200, max_depth=4)
    gbr.fit(X_train, y_train)
    models[q] = gbr

# 90% prediction interval
lower = models[0.05].predict(X_test)
median = models[0.50].predict(X_test)
upper = models[0.95].predict(X_test)

# Coverage: what fraction of true values fall within the interval?
coverage = np.mean((y_test >= lower) & (y_test <= upper))
print(f"90% PI coverage: {coverage:.3f}")  # should be ~0.90 if well-calibrated

When to Use Quantile Regression

• When the variance of $y$ is not constant (heteroskedasticity) — e.g., credit loss given default varies hugely by borrower
• When decisions are asymmetric — it may be worse to underestimate a risk than to overestimate it; use a high quantile for conservative planning
• When stakeholders need uncertainty ranges, not just point predictions
• For demand forecasting: use the 90th percentile to set inventory levels