Explain train, validation, and test splits

A model scored on the rows it was fitted on can score well by memorizing them (overfitting), and that score says little about new rows. So an evaluation holds some rows out: the model is never fitted on them, and they stand in for the data it will meet in use. They do that job only if they resemble that data.

Most evaluations divide the labeled rows into three disjoint splits, each with one job.

The option that wins many comparisons on validation carries an optimistic validation score. The test split gives a score that no choice has touched.

Proportions vary (The Elements of Statistical Learning calls 50/25/25 typical); each split must be large enough for the decisions made on it. Teams often refit the chosen setup on training plus validation before the single test run.

Any choice made by looking at a test score, such as which model to keep, which threshold to use, or "one more try", makes that score optimistic. The Elements of Statistical Learning says to keep the test set "in a vault" until the end: choosing a model by its test error underestimates the true error.

Illustration (constructed numbers). Five models would each score 0.80 on new data. Chance gives them test scores of 0.78, 0.79, 0.80, 0.81, and 0.83. Keeping the best reports 0.83, 0.03 above what the chosen model will score on new data.

Google's Machine Learning Crash Course notes that test sets "wear out" with repeated use. Once one has informed a decision, treat it as development data and get a fresh test set, such as newly labeled rows.

These rules exist because of data leakage: using information that would not be available at prediction time to build the model (scikit-learn's definition), which makes offline scores look better than production will.

One example is a scaler whose mean and spread are computed over all rows, test rows included. Split first, fit each preprocessing step on the training split only, and apply the fitted step unchanged to validation and test rows. A pipeline, such as scikit-learn's Pipeline, does this, even inside cross-validation.

The Medium question in this topic covers other forms of leakage and how to audit for them.

Most libraries split rows at random by default. Start instead from one question: when the model is used, what will be new to it?

The choices can combine. scikit-learn's StratifiedKFold, GroupKFold, and TimeSeriesSplit are examples of the last three.

Separately, an overall score can hide a subgroup the model handles badly. Also report the score for each subgroup that matters as a slice, such as entities absent from training, with its row count.

For a model that predicts future events, fit on earlier data, validate on later data, and test on the latest. Google's Rules of Machine Learning (Rule #33) recommends measuring on data gathered after the training data, because that better reflects production; shuffling trains on the future and scores on the past. Let training cover a full seasonal cycle when you can, since a test period covers only its own season.

Some labels take time to become final. A label such as "repaid within 30 days" is unknown until the row's window closes: leave such rows out, and never count them as negatives.

A row near the end of the training period gets its label from events in the next period. Leave a gap at least as long as the label window before the later period starts (TimeSeriesSplit has a gap parameter). The gap applies to every row the final model is fitted on, including rows added in a refit. The Medium question checks this boundary.

With little data, one validation split is noisy. k-fold cross-validation divides the development data (everything except the test set) into k folds, then fits on k minus 1 and validates on the remaining fold, k times, so every development row is used for validation once. Report the average of the k scores and look at their spread.

The test set is still held out and scored once (in scikit-learn's guide, cross-validation replaces the validation set, not the test set). Variants: stratified, grouped, and time-ordered folds, where training folds always come before the validation fold. The cost is k fits instead of one.

A test score estimates how the frozen model and threshold perform on data like the test split, drawn the way that split was drawn. It is coarse and noisy: one row moves an accuracy-style score by 1/n, so 0.5 percentage points on 200 rows (1/200 = 0.005) and 0.01 percentage points on 10,000 rows (1/10,000 = 0.0001), and chance alone moves it by many more rows than one. Report each slice score with its row count.

A test score does not show:

• Performance in periods, seasons, or populations the test split lacks.
• Performance after the data changes: monitor after launch (the Hard question covers drift).
• Whether the feature helps users: that takes an online experiment (see AI Product Experimentation).
• Anything at all once the test set has informed a decision.

Use "Three splits, three jobs" and "Choose the split from how the model will be used". Each row is a separate task at the same furniture marketplace.