ML & Data Foundations · 3 to 4 hours
Design time-aware evaluation and drift response
Design rolling-origin backtests, capacity-bound metrics, label-delay handling, and drift monitoring for a synthetic card-fraud model whose chargebacks arrive weeks late, then decide whether to promote, shadow, or hold a retrained candidate.
System design and evaluation decision
4 h
Overview
Evaluate and operate a model whose labels arrive weeks late and whose inputs move with the calendar. The level is practiced on one synthetic card-fraud scenario: design the backtest, the metrics, label-delay handling, drift monitoring, and a response policy, then decide what to do with a retrained candidate. LLM-specific evaluation belongs to Evaluation & Observability.
Study time: 3 to 4 hours
Interview format: System design discussion with backtest, label-maturity, and drift calculations
By the end you can
- Design a rolling-origin backtest whose gap matches the label-maturity window, retraining each compared recipe at every origin
- Measure a ranking model at a fixed daily review capacity, with base rates, capacity ceilings, and slices
- Quantify label maturity and keep immature comparisons, projections, and early signals apart from matured metrics
- Compute PSI against a reference and against the same weeks last year, and tell seasonal, pipeline, and performance drift apart
- Set a response policy for retraining, shadowing, promotion, and rollback, and defend a decision on a candidate whose evidence conflicts
Prerequisites
Helpful background: Comfort with precision and recall, proportions, and natural logarithms helps.
This topic covers data splits, leakage, misleading metrics, and evaluation over time for any model. Evaluation & Observability covers evaluation specific to LLM applications.
In this guide
- Why time changes the evaluation
- Rolling-origin backtests with a label gap
- Metrics at a fixed review capacity
- Labels that arrive late
- Scenario: the fraud model retrain
- Work the backtest evidence
- Read the newest fold and the early signals
- Detect drift and tell it from seasonality
- Work the drift and capacity checks
- Respond: retrain, shadow, promote, roll back
- Write the design and decision memo
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
System Design
Covers data and concept drift, monitoring with delayed outcomes, and champion-challenger retraining and rollout.
System Design
Event time, watermarks, and per-card sliding windows, close relatives of this model's clocks and velocity feature.
AI Engineering
Works a ship decision when offline scores, early metrics, and matured outcomes disagree, with intervals.
Practice problem
Practice computing precision@k and recall@k from a ranked list, the metrics behind a fixed review capacity.
Behavioral Interview
Shows how to state calibrated confidence and the evidence that would reverse a call made on partial data.
Practice the interview question
Answer it in your own words before you look at the reference answer.