ML & Data Foundations · 90 minutes
Explain train, validation, and test splits
Explain what training, validation, and test splits are each for, then repair a fictional marketplace's evaluation plan by choosing a split that matches how its model will be used.
Conceptual scenario
1.5 h
Used in interview rounds:
Technical Screen (Easy)1
Step 1: The question
Using the furniture marketplace scenario, explain what the training, validation, and test splits are each for, and what is wrong with the draft evaluation plan. Then propose a split for this model: which listings go in each split, how many, and which you leave out, and where the scaler and the tip threshold are fitted. Say how you would answer the product manager's question about first-time sellers, and what the final test score will and will not tell the launch review.
Scenario: a sell-within-14-days model
Scenario (fictional). Suppose a secondhand-furniture marketplace wants a model that scores each listing at the moment it is posted and predicts whether it will sell within 14 days. When the predicted chance is below a threshold, the listing form shows the seller a pricing tip. The model uses details known at posting time, such as category, asking price, photo count, and city. The team plans to launch the tip in August 2026, the start of the late-summer peak. Every figure below is synthetic.
Label rule. A listing's label is 1 if it sold within the 14 days after the day it was posted, and 0 otherwise. The label becomes final at the end of the 14th day after posting. The data snapshot was taken at the end of 14 July 2026.
| Listings posted | Listings | Sold within 14 days |
|---|---|---|
| 2025 Q1 (January to March) | 7,200 | 2,160 (30%) |
| 2025 Q2 (April to June) | 9,000 | 3,240 (36%) |
| 2025 Q3 (July to September) | 12,600 | 5,670 (45%) |
| 2025 Q4 (October to December) | 8,400 | 2,352 (28%) |
| 2026 Q1 (January to March) | 7,800 | 2,262 (29%) |
| 2026 Q2 (April to June) | 9,000 | 3,060 (34%) |
| 1 to 14 July 2026 | 1,900 | Not final at the snapshot |
Sellers. About 6,000 sellers posted the 54,000 listings from 2025 Q1 to 2026 Q2. 150 dealer accounts posted 40 or more listings each, 13,500 listings in total. Of the 9,000 listings posted in 2026 Q2, 6,300 came from sellers who had posted before April 2026, and 2,700 from sellers whose first listing was in 2026 Q2.
The product manager asks: "Will the tip work for first-time sellers too?"
A teammate's draft evaluation plan
- Use all 55,900 listings, counting the 1,900 July listings as not sold.
- Fit the scaler for asking price and photo count on all 55,900 listings.
- Shuffle the listings and split them 80/20: 44,720 to train and 11,180 to test.
- Train three candidate models, keep the one with the best test score, then move the tip threshold until that test score is highest.
- Report that test score in the launch review as the score the model will reach all year.
Constraints
- Use only the scenario's figures; all of them are fictional
- Treat the team's metric as fixed: this question is about the splits, not the choice of metric
- Keep the answer to about three minutes spoken
Deliverables
- The job of each split and the decisions it may inform
- Each problem in the draft plan, and why it makes the reported score misleading
- A split for this model: which listings go where, how many, and which rows you leave out
- How you would answer the product manager's question about first-time sellers
- What the final test score does and does not establish
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: Names the decisions each split may inform and says the test split is scored once after everything is frozen.
What earns a 2: Checks each step of the draft plan for rows that both inform a decision and judge the model, and explains the effect on the reported score.
What earns a 2: Derives the split order and unit from when and how the model scores listings, gives counts from the scenario table, and handles rows without a final label.
What earns a 2: Compares the sellers in the test split with production traffic and says how the first-time-seller figure would be measured and reported.
What earns a 2: States what the single score estimates and lists, with reasons from the scenario, what it cannot establish.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Test set kept independent or Split design from the use case
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can say which decisions each split may inform, design a split from the way the model will be used, and state what a single test score can and cannot claim.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
What may the validation split be used for that the test split may not?
What is data leakage?
When should you split by group instead of by row?
Why use k-fold cross-validation?
How should you split data for a model that predicts future events, and why?
7