ML Fundamentals

1. Algorithm Mechanics

Linear & Logistic Regression

Linear Regression assumptions (LINE): Linearity, Independence of errors, Normality of errors, Equal variance (Homoscedasticity).

Logistic Regression models the log-odds of the outcome:

$$\log\frac{p}{1-p} = \beta_0 + \beta_1 x_1 + \cdots + \beta_n x_n$$

A coefficient $\beta_i = 0.5$ means a 1-unit increase in $x_i$ multiplies the odds by $e^{0.5} \approx 1.65$. Logistic regression is the regulatory gold standard for credit scoring because it is fully explainable.

Tree-Based Models

Splitting criterion: Gini Impurity = $1 - \sum_i p_i^2$. Information Gain uses Entropy = $-\sum_i p_i \log_2 p_i$. Both produce similar results in practice.

Clustering

• K-Means: Minimizes within-cluster variance. Requires scaling. Assumes spherical clusters. Choose $k$ with elbow method (inertia) or silhouette score.
• DBSCAN: Density-based. No need to specify $k$. Handles non-spherical clusters and identifies outliers as noise. Sensitive to eps and min_samples.

2. The Bias-Variance Tradeoff

$$\text{Expected Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}$$

3. Regularization & Optimization

L1 vs. L2

Elastic Net combines both: $\lambda_1 \sum |\beta_i| + \lambda_2 \sum \beta_i^2$.

Gradient Descent

4. Neural Networks

Neural networks are function approximators composed of layers of learnable linear transformations followed by non-linear activations. They excel when feature interactions are complex and unstructured data (images, text, audio) is involved.

Architecture Fundamentals

• Neuron: computes $z = \mathbf{w}^T\mathbf{x} + b$, then applies activation $a = f(z)$
• Layer: a set of neurons applied in parallel to the same input
• Depth (layers) vs. Width (neurons per layer): depth captures hierarchical features; width captures breadth at each level
• Universal approximation theorem: a single hidden layer with enough neurons can approximate any continuous function — but depth is more parameter-efficient for complex functions

Activation Functions

Backpropagation

Backpropagation is the chain rule of calculus applied to compute gradients of the loss with respect to every weight. Forward pass: compute predictions and loss. Backward pass: propagate gradients layer by layer from output to input.

$$\frac{\partial L}{\partial \mathbf{w}^{(l)}} = \frac{\partial L}{\partial \mathbf{a}^{(l)}} \cdot \frac{\partial \mathbf{a}^{(l)}}{\partial z^{(l)}} \cdot \frac{\partial z^{(l)}}{\partial \mathbf{w}^{(l)}}$$

Vanishing gradients: early layers receive near-zero gradients when sigmoid/tanh is used in deep networks. Solutions: ReLU activations, batch normalization, residual connections (skip connections in ResNets).

Key Regularization Techniques

• Dropout: randomly zero out neurons during training with probability $p$. Forces the network to learn redundant representations. At inference, scale weights by $(1-p)$.
• Batch Normalization: normalize activations within each mini-batch; reduces internal covariate shift; allows higher learning rates; acts as mild regularizer.
• Weight Decay (L2): add $\lambda \|\mathbf{w}\|^2$ to loss; equivalent to L2 regularization on weights.
• Early stopping: stop training when validation loss stops improving; simple and effective.

Weight Initialization

• Xavier/Glorot: $W \sim \mathcal{U}\!\left(-\sqrt{\frac{6}{n_{in}+n_{out}}}, \sqrt{\frac{6}{n_{in}+n_{out}}}\right)$ — for sigmoid/tanh activations
• He initialization: $W \sim \mathcal{N}\!\left(0, \sqrt{\frac{2}{n_{in}}}\right)$ — for ReLU activations (default in PyTorch)

Neural Networks vs. Gradient Boosted Trees

import torch
import torch.nn as nn

class FeedForwardNet(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim, dropout=0.3):
        super().__init__()
        layers = []
        prev_dim = input_dim
        for h in hidden_dims:
            layers += [nn.Linear(prev_dim, h), nn.BatchNorm1d(h), nn.ReLU(), nn.Dropout(dropout)]
            prev_dim = h
        layers.append(nn.Linear(prev_dim, output_dim))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

model = FeedForwardNet(input_dim=20, hidden_dims=[128, 64], output_dim=1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
criterion = nn.BCEWithLogitsLoss()

5. Advanced Boosting: LightGBM, CatBoost, and Stacking

LightGBM

LightGBM grows trees leaf-wise (best-first) rather than level-wise, achieving lower loss with fewer leaves. Key optimizations: Gradient-based One-Side Sampling (GOSS) ignores small-gradient instances; Exclusive Feature Bundling (EFB) merges mutually exclusive features. Result: typically 5–10× faster than XGBoost with comparable or better accuracy.

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    num_leaves=63,          # 2^max_depth - 1; key LightGBM parameter
    min_child_samples=20,   # regularization: min samples per leaf
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,          # L1
    reg_lambda=0.1,         # L2
    early_stopping_rounds=50
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], callbacks=[lgb.log_evaluation(100)])

CatBoost

Handles categorical features natively using ordered target encoding — avoids target leakage by encoding each sample using only prior samples in the permuted order. Also uses symmetric trees for fast inference. Best choice when the dataset has many high-cardinality categorical features.

from catboost import CatBoostClassifier

cat_features = ['city', 'product_category', 'customer_segment']  # pass indices or names
model = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    cat_features=cat_features,   # no manual encoding required
    eval_metric='AUC',
    early_stopping_rounds=50,
    verbose=100
)
model.fit(X_train, y_train, eval_set=(X_val, y_val))

XGBoost vs. LightGBM vs. CatBoost

Model Stacking

Stacking (stacked generalization) trains a meta-learner on the out-of-fold predictions of base models. Base models see the training data; the meta-learner sees only base model predictions. Reduces variance and captures complementary model strengths.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import lightgbm as lgb

base_models = [
    ('rf', RandomForestClassifier(n_estimators=200, random_state=42)),
    ('lgbm', lgb.LGBMClassifier(n_estimators=300, random_state=42)),
]

stack = StackingClassifier(
    estimators=base_models,
    final_estimator=LogisticRegression(),  # meta-learner: simple, regularized
    cv=5,                                   # k-fold for OOF predictions
    passthrough=False                        # meta-learner sees only base preds
)
stack.fit(X_train, y_train)

Key rule: the meta-learner should be simpler than the base models (logistic regression or ridge) to avoid overfitting on the OOF predictions. Stacking helps most when base models are diverse (different algorithms, different feature subsets).