Experiment Design & Causal Inference

1. Was the hypothesis pre-registered, or did you look at the data first? Post-hoc hypothesis selection inflates false positive risk dramatically.
2. How many metrics did you test? Multiple comparisons inflate false positives — 20 metrics at α=0.05 expects 1 false positive even if nothing changed.
3. What is the effect size? Is the difference practically meaningful, or just statistically significant? A 0.01% lift with p=0.04 may not justify the engineering cost.
4. Did the experiment run long enough to clear novelty effects and capture a full business cycle (e.g., a full week including weekends, or a full month)?
5. Are there SUTVA violations? Did treatment group members interact with control group members (referrals, shared households, social feeds)?

Using the two-proportion z-test formula with $p_1 = 0.03$, $p_2 = 0.035$, $z_{\alpha/2} = 1.96$, $z_\beta = 0.84$:

from scipy import stats
import numpy as np

p1, p2 = 0.03, 0.035
z_alpha = stats.norm.ppf(0.975)   # 1.96
z_beta  = stats.norm.ppf(0.80)    # 0.84
p_bar   = (p1 + p2) / 2

num = (z_alpha * np.sqrt(2 * p_bar * (1 - p_bar)) +
       z_beta  * np.sqrt(p1*(1-p1) + p2*(1-p2))) ** 2
n   = int(np.ceil(num / (p2 - p1)**2))
print(f"n per group: {n}")   # ~17,000–20,000

The small MDE (0.5pp absolute) requires roughly 17,000–20,000 users per group. This is a key insight: small effects require large samples. When traffic is limited, you may need to either widen the MDE (accept only detecting larger lifts) or extend the experiment duration.

The parallel trends assumption states that, in the absence of treatment, the treatment and control groups would have followed the same trend over time. The DiD estimator is only valid as a causal estimate if this assumption holds.

You cannot directly test it (the counterfactual does not exist), but you can check its plausibility:

1. Plot pre-treatment time series for both groups. If their trends were parallel before treatment began, the assumption is more credible.
2. Placebo test: Apply the DiD estimator using only pre-treatment periods. Assign a fake "treatment" date in the middle of the pre-period. If DiD ≈ 0, that supports parallel trends.
3. Covariate balance check: Verify that treatment and control groups are similar on observable characteristics before the intervention.

Peeking means checking the p-value repeatedly during the experiment and stopping as soon as p < 0.05. This inflates the false positive rate far above α, because each interim look is an additional opportunity to observe a false positive by chance.

Simulation: with a fixed α = 0.05 and daily peeking over 4 weeks, the true false positive rate can exceed 20–30%.

Sequential testing solutions:

• O'Brien-Fleming alpha spending: Distributes the total α budget across planned interim looks — earlier looks require a more extreme p-value to reject, preserving the overall Type I error.
• Always-valid inference (anytime-valid p-values): Constructs confidence sequences that are valid at any stopping time, making continuous monitoring mathematically sound.
• Pre-registered stopping rules: Decide before launch exactly when and how many interim analyses will occur. Ad hoc stopping after peeking is not correctable post-hoc.