ML & Data Foundations
Split data correctly, spot leakage, and respond to drift.
Use training, validation, and test splits correctly, recognize leakage and misleading metrics, and evaluate models over time.
3 questions
Questions
Easy
1.5 h
Explain what training, validation, and test splits are each for, then repair a fictional marketplace's evaluation plan by choosing a split that matches how its model will be used.
Medium
3 h
Implement an audit that flags leaky features, preprocessing, time splits, selection, and mismatched reports, predict its findings on five fixture experiments, and diagnose why a churn model that looked excellent offline failed after launch.
Hard
4 h
Design rolling-origin backtests, capacity-bound metrics, label-delay handling, and drift monitoring for a synthetic card-fraud model whose chargebacks arrive weeks late, then decide whether to promote, shadow, or hold a retrained candidate.
What you will be able to do
- Explain what the training, validation, and test splits are each for, and which decisions each one may inform
- Explain why a test set stops giving an unbiased estimate once it informs any decision, and why every preprocessing step is fitted on training data only
- Choose a random, stratified, grouped, or time-based split from how the model will be used, and handle rows whose label is not final yet
- Use k-fold cross-validation when data is small, and read the average and spread of the fold scores
- State what a single test score does and does not establish
- Explain what leakage means for a model and how it differs from data leakage in security
- Check that every feature is computable at the prediction cutoff and that fitted preprocessing sees only training rows
- Place evaluation splits after the last training label is final, and keep model and threshold choices on validation
- Report accuracy against the majority-class baseline, with precision, recall, and their denominators, on the split's own population
Hands-on exercises
Interview rounds that use this topic
Topic boundaries
This topic covers data splits, leakage, misleading metrics, and evaluation over time for any model. Evaluation & Observability covers evaluation specific to LLM applications.
Related on CodeSnatch
System Design
Shows point-in-time joins, training-serving skew, drift monitoring, and safe rollouts in a production ML system.
AI Engineering
Applies split checks and leakage hunting to a fine-tuning dataset, with a validator for groups that cross splits.
System Design
Covers backfills, late-arriving data, data quality tests, and lineage, the pipeline side of honest training data.
Behavioral Interview
Frames stories about leakage checks and catching your own analytical error, which ML loops grade closely.
System Design
Explains event time versus processing time and late events, the clock reasoning behind delayed labels.
AI Engineering
Takes evaluation online: an A/B test contract with named denominators and guardrails, the step after offline scores.
Practice problem
Practice the as-of lookup by timestamp that a point-in-time feature join performs for each training row.
Community question bundle
Practitioner drills with code for a PSI skew check, a time-safe feature join, and a retraining trigger.