Business Framing & Stakeholder Communication

1. Framing the Business Problem

Before writing a single line of code, align with stakeholders on: what decision this model supports, what success looks like in dollar or operational terms, and what constraints must be satisfied (approval rate floors, fair lending compliance, latency requirements).

The SCQA Framework

Situation → Complication → Question → Answer. A structured way to present data science work to executives without leading with methodology.

Defining Success Metrics Before Building

The order matters: define business KPIs and guardrails with stakeholders, then select model metrics that align with them. Working backwards from a model's AUC to a business rationale is backwards — and invites post-hoc rationalization.

2. Decision Thresholds and Expected Value

The default 0.5 threshold is almost never the right business threshold. The optimal threshold is determined by the relative cost of false positives and false negatives.

Expected Value Framework

For a binary credit approval decision, approve when $EV(\text{approve}) > 0$:

Setting $p$ = model's predicted probability of default, the break-even threshold $p^*$ satisfies $EV = 0$:

With $300 revenue and $2,000 expected loss: $p^* = 300 / 2300 \approx 0.13$. Approve all applicants with predicted default probability below 13%.

import numpy as np

revenue_per_account  = 300    # expected revenue if no default
loss_per_default     = 2000   # expected loss if default

# Break-even threshold
p_star = revenue_per_account / (revenue_per_account + loss_per_default)
print(f"Break-even threshold: {p_star:.3f}")  # 0.130

# Evaluate a range of thresholds on held-out data
from sklearn.metrics import confusion_matrix

def business_value(y_true, y_prob, threshold, revenue, loss):
    y_pred = (y_prob >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    # tp = correctly predicted defaults (we declined, saving loss)
    # fp = wrongly declined good customers (lost revenue)
    # fn = missed defaults (approved; we incur loss)
    # tn = correctly approved good customers (earned revenue)
    value = tn * revenue - fn * loss - fp * revenue
    return value

thresholds = np.arange(0.05, 0.50, 0.01)
values = [business_value(y_true, y_prob, t, revenue_per_account, loss_per_default)
          for t in thresholds]
best_threshold = thresholds[np.argmax(values)]
print(f"Optimal business threshold: {best_threshold:.2f}")

Lift and Gains Charts

Lift measures how much better a model is than random at identifying positives within a scored population segment. "4x lift in the top decile" means that of all accounts in the top 10% of model scores, the model captures 4× more true positives than you would expect by randomly selecting 10% of the population.

import pandas as pd
import numpy as np

def compute_lift(y_true, y_prob, n_bins=10):
    df = pd.DataFrame({'y_true': y_true, 'y_prob': y_prob})
    df['decile'] = pd.qcut(df['y_prob'], q=n_bins, labels=False, duplicates='drop')
    # Decile 9 = top scores (highest predicted probability)
    base_rate = y_true.mean()
    result = (df.groupby('decile')['y_true']
                .agg(['mean', 'count'])
                .rename(columns={'mean': 'positive_rate', 'count': 'n'}))
    result['lift'] = result['positive_rate'] / base_rate
    return result.sort_index(ascending=False)

lift_table = compute_lift(y_true, y_prob)
print(lift_table)

3. When Not to Use ML

A Senior Manager DS must be able to argue against a complex model when simpler alternatives are more appropriate. Over-engineering creates governance burden, reduces explainability, and often delivers negligible performance improvement over well-designed logistic scorecards.

SR 11-7 and Model Risk Management

OCC/Fed Supervisory Guidance SR 11-7 requires that Capital One models are: (1) documented — conceptual soundness, data lineage, limitations; (2) independently validated by a team separate from the developer; (3) monitored on an ongoing basis for performance degradation. More complex models require more validation effort, which extends timelines and increases risk of governance failure. Factor this into model selection decisions.

4. Communicating Results to Executives

Technical metrics do not resonate with non-technical stakeholders. Translate every model improvement into dollars, customer outcomes, or operational efficiency.

Guardrail Communication

Always present results alongside constraints. A model that improves AUC but violates the approval rate floor or introduces disparate impact is not deployable — communicate this proactively rather than after stakeholders have anchored on the headline improvement.

# Structured results summary for stakeholder presentation
results = {
    'model':           'XGBoost v2',
    'auc_champion':    0.81,
    'auc_challenger':  0.84,
    'business_lift':   '$2.3M annual charge-off reduction (estimate)',
    'approval_rate':   '62.4%',   # guardrail: must stay above 60%
    'approval_status': 'PASS',
    'dir_black':       0.83,      # disparate impact ratio; must be >= 0.80
    'dir_status':      'PASS',
    'validation':      'Pending independent model review (SR 11-7)',
    'deployment':      'Champion/Challenger — 10% traffic',
}

for k, v in results.items():
    print(f"{k:20s}: {v}")