Probability & Statistics

1. Bayesian Inference & Probability

Bayes' Theorem

$$P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$$

Use the Law of Total Probability to expand the denominator when $P(B)$ is not given directly:

$$P(B) = P(B|A)P(A) + P(B|\neg A)P(\neg A)$$

Capital One example: A fraud model has 99% specificity (true negative rate) and 80% sensitivity (true positive rate). 1 in 1,000 transactions is fraudulent. What fraction of flagged transactions are actually fraud?

$$P(\text{fraud} | \text{flagged}) = \frac{0.80 \times 0.001}{0.80 \times 0.001 + 0.01 \times 0.999} \approx 7.4\%$$

This illustrates the base rate fallacy: even a highly accurate model flags mostly non-fraud when the event is rare.

Expected Value

$$E[X] = \sum_{x} x \cdot P(X = x) \quad \text{(discrete)}$$

$$E[X] = \int_{-\infty}^{\infty} x \cdot f(x)\, dx \quad \text{(continuous)}$$

For risk/reward matrices: approve a loan when $E[\text{approve}] > E[\text{decline}]$. See Module 9 for the full expected value framework.

Combinatorics

Permutations (order matters): $P(n, k) = \frac{n!}{(n-k)!}$

Combinations (order doesn't matter): $\binom{n}{k} = \frac{n!}{k!(n-k)!}$

2. Statistical Distributions

Binomial → Poisson approximation: valid when $n$ is large and $p$ is small, with $\lambda = np$.

3. Hypothesis Testing & A/B Testing

Test Selection

Error Types & Power

• Type I Error ($\alpha$): False Positive — reject a true $H_0$. Set by significance level (e.g., 0.05).
• Type II Error ($\beta$): False Negative — fail to reject a false $H_0$.
• Statistical Power ($1 - \beta$): Probability of detecting a real effect. Target ≥ 80%.

Factors that increase power: larger sample size, larger true effect size, lower metric variance, higher $\alpha$.

Confidence Intervals

A 95% CI does not mean "95% probability the parameter is in this interval." It means: if you repeated the experiment many times, 95% of the constructed intervals would contain the true parameter.

4. Non-Parametric Tests

Use non-parametric tests when data is non-normal, ordinal, or sample sizes are small (n < 30). They make no distributional assumptions but generally have lower power than their parametric counterparts.

When to Use Non-Parametric Tests

• Distribution is clearly skewed or has heavy tails
• Data is ordinal (Likert scales, rankings)
• Small samples where normality can't be verified
• Presence of outliers that can't be removed

Key Tests

Mann-Whitney U Test

Tests whether one distribution stochastically dominates another. Null hypothesis: the probability that a random draw from group A exceeds a random draw from group B equals 0.5.

from scipy import stats

group_a = [12.1, 14.3, 11.8, 13.5, 15.2]
group_b = [10.5, 11.2, 12.0, 9.8, 11.5]

stat, p_value = stats.mannwhitneyu(group_a, group_b, alternative='two-sided')
print(f"U statistic: {stat}, p-value: {p_value:.4f}")
# If p < 0.05: reject H0 that distributions are equal

Kruskal-Wallis Test

Extension of Mann-Whitney U to 3+ groups. Ranks all observations together, then tests whether groups differ in their mean rank. If significant, use Dunn's test for pairwise post-hoc comparisons.

from scipy import stats
import scikit_posthocs as sp

group1 = [23, 25, 28, 22, 27]
group2 = [31, 29, 35, 30, 33]
group3 = [19, 21, 18, 22, 20]

stat, p = stats.kruskal(group1, group2, group3)
print(f"H statistic: {stat:.3f}, p-value: {p:.4f}")

# Post-hoc if p < 0.05
data = group1 + group2 + group3
groups = [1]*5 + [2]*5 + [3]*5
print(sp.posthoc_dunn([group1, group2, group3], p_adjust='bonferroni'))

5. Effect Size

A statistically significant result tells you the effect is real. Effect size tells you how big it is. With large samples, even trivial differences become significant — always report both.

Cohen's d (Two-Group Comparison)

$$d = \frac{\mu_1 - \mu_2}{s_{\text{pooled}}}$$

where $s_{\text{pooled}} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}}$

import numpy as np
from scipy import stats

def cohens_d(group1, group2):
    n1, n2 = len(group1), len(group2)
    s_pooled = np.sqrt(((n1-1)*np.var(group1, ddof=1) + (n2-1)*np.var(group2, ddof=1)) / (n1+n2-2))
    return (np.mean(group1) - np.mean(group2)) / s_pooled

control = [50, 52, 48, 53, 49]
treatment = [55, 58, 54, 57, 56]

t_stat, p = stats.ttest_ind(control, treatment)
d = cohens_d(control, treatment)
print(f"p-value: {p:.4f}, Cohen's d: {d:.3f}")
# p may be significant but d tells you if it matters

Other Effect Size Measures

• Hedges' g — bias-corrected Cohen's d for small samples ($n < 20$)
• r (rank-biserial) — effect size for Mann-Whitney U: $r = 1 - \frac{2U}{n_1 n_2}$
• Eta-squared ($\eta^2$) — proportion of variance explained in ANOVA: $\eta^2 = SS_{\text{between}} / SS_{\text{total}}$
• Odds ratio — effect size for binary outcomes in logistic models

6. Bayesian vs. Frequentist Inference

Core Philosophical Difference

Bayesian Update

$$P(\theta | \text{data}) = \frac{P(\text{data} | \theta) \cdot P(\theta)}{P(\text{data})}$$

Posterior $\propto$ Likelihood $\times$ Prior. As data grows, the posterior is dominated by the likelihood and the prior matters less.

Practical Implications

• Confidence interval vs. credible interval: A 95% frequentist CI does not mean "95% probability the parameter is inside." A 95% Bayesian credible interval does — it contains the true parameter with 95% posterior probability.
• When to prefer Bayesian: Small samples, strong domain knowledge to encode as a prior, sequential updating as data arrives, need for probability statements about parameters.
• When to prefer frequentist: Large samples, well-understood test procedures, regulatory contexts that require p-values, computational simplicity.
• Bayesian A/B testing: Compute $P(\theta_B > \theta_A | \text{data})$ directly — no p-value threshold required, and you can stop early without inflating Type I error.