Fine-Tuning Dataset Validator

Fine-Tuning Dataset Validator

A fictional coding-practice platform plans to fine-tune a model that writes hints without giving away solutions. Reviewers have collected example conversations and assigned each example to a split: train fits the model, validation picks a checkpoint, and test gives the final held-out score. If one exercise family, or the same conversation, sits in two splits, the held-out score rewards memory instead of skill.

Before any training run, a validator checks a narrow dataset contract. Implement validateSplits(splits). It returns the number of rows in each split as { train, validation, test }, or throws an Error whose message is exactly the message of the first check that fails.

Input

splits maps each split name to an array of rows. One row is one training example:

{
    id: 'h-101',          // unique across the whole dataset
    groupId: 'loops',     // exercise family: variants and paraphrases share it
    messages: [
        { role: 'system', content: 'Give a hint, not the answer.' },
        { role: 'user', content: 'My while loop never ends.' },
        { role: 'assistant', content: 'Check the loop' }, // the target the model learns
    ],
}

Nothing has been validated yet, so any value can have the wrong type.

Checks, in order

First check the container:

Then visit the splits in the fixed order train, validation, test, whatever the order of the keys, and the rows of each split in array order. Finish every check on one row before you move to the next row, and finish a split before you check the next one.

Step 8 runs its three checks on one message before it moves to the next message. In JavaScript, null and arrays do not count as objects for steps 1, 3, and 8.

Normalized conversation

Turn each message into the pair (role, text), where text is the content split on runs of whitespace (spaces, tabs, newlines) and rejoined with single spaces. Leading and trailing whitespace disappears. Every message counts, including system messages. Two conversations are the same when they have the same number of messages and every pair matches exactly. Keep letter case and punctuation: in code, Count and count are different names.

"Check the\tloop\n" and "Check the loop" normalize to the same text. "check the loop" and "Check the loop." do not.

What may repeat

• An id never repeats, not even inside one split.
• A groupId or a normalized conversation may repeat inside one split. Variants of one exercise family belong together, and same-split duplicates are left to a separate deduplication step. They fail only when they reach a second split.
• Do not modify the input.

JavaScript and Python names

Fixture

The base dataset:

Each case changes the base dataset in one way:

Before you run the tests, predict each result: the counts, or the exact message of the first check that fails. The first test checks all ten cases.

Example

validateSplits({
    train: [
        { id: 'a', groupId: 'g1', messages: [{ role: 'user', content: 'Why?' }, { role: 'assistant', content: 'Look  again.' }] },
        { id: 'b', groupId: 'g2', messages: [{ role: 'user', content: 'Why?' }, { role: 'assistant', content: 'Look again.' }] },
    ],
    validation: [{ id: 'c', groupId: 'g3', messages: [{ role: 'user', content: 'How?' }, { role: 'assistant', content: 'Step through it.' }] }],
    test: [{ id: 'd', groupId: 'g1', messages: [{ role: 'user', content: 'When?' }, { role: 'assistant', content: 'Print the index.' }] }],
});
// throws Error('Group crosses splits'): group g1 is in train and in test.
// Rows a and b normalize to the same conversation, which is allowed inside train.

Constraints

• At most 10,000 rows in total, at most 50 messages per row, and at most 10,000 characters per content.
• Test inputs use only spaces, tabs, and newlines as whitespace.

Expected complexity

O(R + C) time and O(R + C) extra space, where R is the number of rows and C the total number of characters across all message contents.

In Python, implement validate_splits(splits) with dicts and lists, use the Python names from the table, and raise ValueError with the same messages.