ML & Data Foundations · 3 to 4 hours
Design time-aware evaluation and drift response
Design rolling-origin backtests, capacity-bound metrics, label-delay handling, and drift monitoring for a synthetic card-fraud model whose chargebacks arrive weeks late, then decide whether to promote, shadow, or hold a retrained candidate.
System design and evaluation decision
4 h
1
Step 1: The question
Using the fraud scenario, design how the team should evaluate and monitor this model over time: the backtest, metrics tied to the 400-a-day review capacity, handling of labels that are still arriving, drift monitoring, and a response policy for retraining, shadowing, promotion, and rollback. Compute the fold and drift figures, then decide whether to promote, shadow, or hold the candidate today, and say what would change your decision.
Constraints
- Use the team's maturity rule, arrival curve, PSI rule, and capacity of 400 reviews a day as given
- The current promotion gate was written before the backtest; you may extend it, but state any extension before you apply it
- No model promotions in weeks 48 to 52; rollbacks are allowed
- Chargeback labels for payment weeks younger than 8 weeks are incomplete and cannot be completed early
Deliverables
- Backtest design: a fold calendar with training cutoff, left-out weeks, and test weeks, what is retrained and rebuilt at each origin, and which periods the folds cover
- Capacity-bound metrics for every final fold, with means, fold wins, base-rate and capacity context, and the slices you would add
- A label-delay plan: the share of labels arrived for weeks 37 to 44, the newest fold read on known and projected labels, and the early signals you report before labels mature
- Drift monitoring and response: PSI per feature against both baselines, alert volume against capacity now and at the peak, and the response to each kind of drift
- A decision memo: promote, shadow, or hold, with the gate you applied, the re-evaluation date or condition, rollback triggers, and what would change the decision
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.