Model Adaptation · 2 to 3 hours

Validate a fine-tuning dataset and plan SFT

Implement a split validator for a fine-tuning dataset, predict its results on fixture cases, find the leaks it cannot catch, and plan a supervised fine-tuning experiment against a prompted baseline.

Medium

Hands-on

Implementation and experiment design

3 h

Used in interview rounds:

Live Coding (Medium)

Step 1: The question

Implement the split validator for the hint dataset contract and predict its result for each fixture case. Explain three kinds of leakage it cannot detect and a control for each. Then propose an SFT experiment that could show better hints than the prompted baseline without exercise-family leakage: the tokenization checks, the comparison and decision rule, and how you would verify the exported model.

The hint dataset

Suppose a coding-practice platform shows a hint when a learner's submission fails its tests. The hints come from an open-weight model the platform hosts, driven by a prompt; this prompted model is the baseline. Reviewers report hints that are too vague or give away the solution, and the team wants to know whether supervised fine-tuning (SFT) on reviewed examples does better.

Item	Value (synthetic)
Reviewed examples	1,200 conversations
Exercise families	40; a family is one exercise plus its variants and paraphrases
Split plan	By family: train 28 families (840 conversations), validation 6 (180), test 6 (180)
Record	`id`, `group_id` (the exercise family), and `messages`
Messages	An optional system message, a user message (exercise, failing submission, test output, requested help level), and a final assistant hint written or approved by a reviewer
Release concerns	Hint usefulness, correctness, and answer disclosure

The validator contract

The validator checks a narrow contract: system, user, and assistant messages ending in one target assistant response, in splits the caller has already assigned.

validateSplits(splits) in TypeScript or JavaScript, and validate_splits(splits) in Python, checks the input in this order and stops at the first failure with the message shown (an Error in JavaScript, a ValueError in Python):

splits is an object (a dict in Python, never an array or list) whose keys are exactly train, validation, and test: "Expected train, validation, and test".
The splits are processed in the order train, validation, test. Each split is checked completely, row by row in list order, before the next split starts.
The split is a non-empty list: "Every split must be a nonempty list".
The row is an object: "Record must be an object". Its id and group_id are strings that are not empty or whitespace-only: "Missing identifier".
The id has not appeared earlier in any split: "Duplicate ID".
The group_id has not appeared in a different split: "Group crosses splits".
messages is a list of at least 2 items: "Expected conversation and target". Each message, in order, is an object ("Message must be an object") whose role is system, user, or assistant ("Unsupported role") and whose content is a string that is not empty or whitespace-only ("Empty message").
The last message has role assistant: "Final message must be the target". At least one earlier message has role user: "Missing user input".
The normalized conversation has not appeared in a different split: "Conversation crosses splits". To normalize, keep each message's role, split its content on whitespace, and rejoin the pieces with single spaces. The conversation key is the canonical JSON string of the resulting list of [role, content] pairs. Every message counts, including system messages. Nothing else changes: case and punctuation are kept.

Rule 9 compares splits with each other only. Deduplicating conversations inside one split is a separate step before training, not part of this contract.

A valid dataset returns the row count of each split as { train, validation, test }.

The base dataset below is valid. Its user messages are shortened; real ones would carry the exercise, the failing submission, and its test output. In TypeScript and JavaScript the row field is groupId; Python keeps group_id.

Split	`id`	`group_id`	Messages, in order
train	tr-1	family-loops	user: "Why does my loop never end?" / assistant: "Check the loop condition."
train	tr-2	family-recursion	system: "Give one hint, never the full solution." / user: "My recursive function overflows the stack." / assistant: "Look for the case that stops the recursion."
validation	va-1	family-sorting	user: "My sort returns the list unchanged." / assistant: "Check whether each swap writes back to the list."
test	te-1	family-graphs	user: "My BFS visits some nodes twice." / assistant: "Check when you mark a node as visited."

Constraints

Keep the validator's rule order and error messages exactly as stated
Use only the scenario figures and fixtures given; label any other figure as an assumption
Select only on validation; run the test split only for the selected candidate (an export parity rerun is not a new selection).

Deliverables

A validator that handles every fixture case, with each case's result and the rule that decides it
Three leaks the validator cannot detect, each with a control and where it runs
Tokenization checks: template rendering, loss mask, and truncation
An experiment plan: baseline, metrics, blinded paired comparison, unit of independence, and a predefined decision rule
A release check for the exported model and a rollback path

Step 2: Hints

Optional. Each hint gives a little more away.

Step 3: Work through it

Code exercise

Split validator

Implement validateSplits (Python: validate_splits) for the contract above. Apply the rules in the stated order, fail with the first broken rule's exact message, and return the row count of each split when the dataset is valid.

Expected

The linked practice problem calls validateSplits(splits) in JavaScript and validate_splits(splits) in Python. A valid input returns an object (a dict in Python) with the keys train, validation, and test, each holding that split's row count. A failure throws an Error (Python: raises ValueError) whose message contains the rule's text exactly. JavaScript rows use groupId; Python rows use group_id. The tests cover a valid dataset, one failing input per rule, whitespace-only identifiers and content, whitespace-equivalent conversations in different splits, a conversation repeated within one split, and inputs where the order of checks decides the message.

Open coding workspace

Limitations

The validator reads one in-memory dataset against a narrow contract: system, user, and assistant messages ending in one target. Tool-call datasets need a richer schema. Whitespace normalization suits prose, but whitespace can carry meaning in code, so a production code dataset needs field-aware duplicate detection. Passing the validator says nothing about whether the targets are correct, safe to train on, or free of private data.

Predict the outcome

Predict the validator's results

Each case starts from the base dataset and applies the listed change (F12 applies two). Using only the rules, predict whether the validator returns counts or fails, and with which message. Then name the rule that decides it.

Case	Change to the base dataset	Result	Deciding rule
F1	None: the base dataset as shown	?	?
F2	Append `tr-3` to train, with `group_id` `family-loops` and the same two messages as `tr-1`	?	?
F3	Add a fourth top-level key, `dev`, holding an empty list	?	?
F4	Replace the test split with an empty list	?	?
F5	Set `tr-2`'s `id` to `" "` (three spaces)	?	?
F6	Change `te-1`'s `id` to `tr-1`	?	?
F7	Change `va-1`'s `group_id` to `family-loops`	?	?
F8	Insert a message with role `tool` and content `Expected [1, 2, 3]` between `va-1`'s user and assistant messages	?	?
F9	Set the content of `tr-2`'s assistant message to `" "` (two spaces)	?	?
F10	Replace `va-1`'s messages with `tr-2`'s system message followed by `va-1`'s assistant message	?	?
F11	Append `te-2` to test, with `group_id` `family-imported`, user content `"Why does my loop never end?"`, and assistant content `"Check the\tloop condition."` (two spaces after Check, a tab after the)	?	?
F12	Replace the test split with an empty list and change `va-1`'s `id` to `tr-1`	?	?

Diagnose

Find what the validator misses

The full hint dataset (1,200 conversations from 40 families) passes the validator. In blind pairwise ratings on the test split, reviewers prefer the fine-tuned candidate's hints by a wide margin, more than the validation pilot suggested. The team's notes on how the dataset was assembled:

Reviewers wrote or approved a hint for each learner submission. Each row's group_id is its exercise family.
To add volume, a model paraphrased reviewed user messages. An import script gave each paraphrase a group_id taken from its import batch name, not from the original family.
Several families were written from one instructor walkthrough, and different excerpts of it appear in user messages across those families.
The test split's reference hints were written first. The same reviewers then wrote training targets in a matching style, sometimes adapting a test reference hint to a similar exercise.

For each note, decide whether it can let test information reach training, and explain why the validator still accepts the dataset.
For each leak, propose a control and say where it runs: the manifest, the split assignment, review, or evaluation.
Say what the test result can and cannot support until the controls are in place, and what you would rerun.

Answer draft

Kept in this browser. Saving your score below keeps a copy with it.

0 / 5000

Step 4: Reference answer

Step 6: Follow-up questions

An interviewer may ask these next.

Why mask prompt tokens?

Is a lower validation loss enough?

Why check truncation before launching a run?

Why test an exported model?

Why split by exercise family instead of by row?

Step 7: Mark complete

40 of 66

Previous: Decide whether to fine-tune Back to all questions

Next: Design a model specialization program