Validate a fine-tuning dataset and plan SFT

Define the behavior, baseline, population, metrics, and decision rule before training. For the hint writer, a useful objective is expert-rated hint quality without giving away solutions. Track correctness and answer disclosure separately; one blended score can hide a serious regression.

Record the baseline model revision, prompt, inference configuration, dataset manifest, split assignment, and evaluation rubric. Use identical evaluation cases across candidates. Define how ambiguous cases and reviewer disagreements will be adjudicated.

Each example needs a stable ID, a grouping key, provenance, and review status. Group by the smallest unit that must not cross partitions, such as an exercise family including paraphrases and variants. If examples connect through multiple sources, form connected groups before splitting.

Deduplicate before assignment, then freeze the assignment. Exact normalized text checks catch only simple duplication. Similarity search, source tracing, and manual inspection are needed for near duplicates and semantic leakage.

A split proportion is not a quality guarantee. Small or imbalanced slices may require targeted collection. A deterministic hash split is convenient, but it does not ensure representative distributions or adequate sample counts.

The validator checks a narrow contract: system, user, and assistant messages ending in one target assistant response, in splits the caller has already assigned.

validateSplits(splits) in TypeScript or JavaScript, and validate_splits(splits) in Python, checks the input in this order and stops at the first failure with the message shown (an Error in JavaScript, a ValueError in Python):

1. splits is an object (a dict in Python, never an array or list) whose keys are exactly train, validation, and test: "Expected train, validation, and test".
2. The splits are processed in the order train, validation, test. Each split is checked completely, row by row in list order, before the next split starts.
3. The split is a non-empty list: "Every split must be a nonempty list".
4. The row is an object: "Record must be an object". Its id and group_id are strings that are not empty or whitespace-only: "Missing identifier".
5. The id has not appeared earlier in any split: "Duplicate ID".
6. The group_id has not appeared in a different split: "Group crosses splits".
7. messages is a list of at least 2 items: "Expected conversation and target". Each message, in order, is an object ("Message must be an object") whose role is system, user, or assistant ("Unsupported role") and whose content is a string that is not empty or whitespace-only ("Empty message").
8. The last message has role assistant: "Final message must be the target". At least one earlier message has role user: "Missing user input".
9. The normalized conversation has not appeared in a different split: "Conversation crosses splits". To normalize, keep each message's role, split its content on whitespace, and rejoin the pieces with single spaces. The conversation key is the canonical JSON string of the resulting list of [role, content] pairs. Every message counts, including system messages. Nothing else changes: case and punctuation are kept.

Rule 9 compares splits with each other only. Deduplicating conversations inside one split is a separate step before training, not part of this contract.

A valid dataset returns the row count of each split as { train, validation, test }.

The base dataset below is valid. Its user messages are shortened; real ones would carry the exercise, the failing submission, and its test output. In TypeScript and JavaScript the row field is groupId; Python keeps group_id.

A dataset can pass every rule above and still leak. The exercise below applies the manifest section to how the full hint dataset was assembled.

Render examples with the selected model's chat template and inspect the decoded tokens. Confirm role boundaries, end-of-turn tokens, and the absence of duplicated special tokens. Make sure the inference prompt ends where the model should begin its answer.

Decide which tokens contribute to loss. For response learning, prompt tokens may be masked while assistant targets contribute. In multi-turn conversations, decide whether all assistant turns or only selected targets should train. Verify the mask against actual token positions rather than relying on a configuration name: framework support varies with the template, and some assistant-only options work only when the template marks the assistant spans.

Check truncation before launching a run. A sequence limit that removes the answer creates unusable supervision; one that removes the problem statement changes the task. Report how many examples were truncated, discarded, or shortened.

For the set of target token positions T, a common objective is the mean negative log-likelihood of the targets:

Mask padding and any context tokens the objective excludes. Compare token-weighted and example-weighted reporting, because long responses can dominate a token average. Suppose a batch holds a 20-token hint and a 180-token explanation. A token-weighted mean gives the explanation 180 / 200 = 90% of the weight; an example-weighted mean gives each example 50%. If one task produces longer answers, a mixed dataset may overweight it without anyone choosing that.

The objective measures how well the model predicts the supplied targets. It cannot establish that those targets are factually correct or appropriate for learners.

Choose a method your serving stack can reproduce. An experiment that only works in the training framework is not yet a deployable improvement.

1. Freeze the baseline and split manifest.
2. Inspect rendered examples, masks, token lengths, and sensitive fields.
3. Train briefly on a small subset to detect data and optimization failures.
4. Evaluate checkpoints on validation cases, including regressions.
5. Change one interpretable factor at a time when diagnosing problems.
6. Select the candidate using the predefined rule, then run the held-out evaluation.

Track learning rate, batch token counts, accumulation, effective update count, seed, precision, optimizer, adapter configuration, and checkpoint. Effective batch size can differ when examples have unequal lengths; record token throughput as well as example counts.

Use the same inputs for baseline and candidate. Blind the model identity and randomize response order for human comparisons. Separate ties, both-wrong cases, and missing outputs. For automated code evaluation, isolate execution and distinguish test coverage from general correctness.

Calculate uncertainty at the unit of independence. If many cases come from one exercise family, resample families rather than treating every paraphrase as independent. Keep per-slice results alongside aggregate scores.

An illustrative gate might require improved hint usefulness, no material correctness regression, and no increase in answer disclosure beyond a specified tolerance. Choose tolerances from product needs and sample uncertainty; this guide does not prescribe universal numeric thresholds.

Save the base model and adapter identifiers, tokenizer, template, inference settings, dataset and code hashes, evaluation report, and deployment dependencies. Record which checkpoint was selected and why. Verify the exported artifact on the serving path using the same evaluation cases.

Maintain a known-good release and a rollback path. A successful training job is an intermediate artifact, not an automatic production promotion.