Evaluation Scoring Harness

Evaluation Scoring Harness

A policy assistant must either answer a supported question with the right fact or abstain when its source lacks the answer. An evaluation runs every case once and records at most one structured outcome per case. Write the harness that scores those outcomes.

Implement summarize(dataset, results).

Inputs

• dataset: an array of cases { id, slice, expected, fact }. expected is "answer" or "abstain". fact is the expected fact key for an answer case and null for an abstain case.
• results: a Map from case ID to the recorded outcome (a dict in Python). An outcome is { kind: "answer", fact }, { kind: "abstain" }, or { kind: "error" }. A case with no entry in results has a missing output.

Scoring rules

A case passes when:

• it expects "abstain" and its outcome is an abstention, or
• it expects "answer" and its outcome is an answer whose fact equals the case's fact.

A missing output never passes. It is not an abstention and not an error.

Report

Return an object (a dict in Python) with exactly these keys:

• total counts every case, including errors and missing outputs, so a missing output stays in the denominator of task success and answer coverage.
• A rate whose denominator is 0 is null (None in Python), never 0 or NaN.
• Every slice with at least one case appears in slices, even when none of its cases pass.

Errors

Check before scoring:

• Two cases share an ID: throw an Error whose message contains "Duplicate case ID".
• results contains an ID that is not in dataset: throw an Error whose message contains "Unknown result ID".

In Python, raise ValueError with the same messages.

Fixture

The tests score this fixture, the same fixture plus a case with no recorded output, and the edge cases above.

Example

summarize(
    [
        { id: 'A1', slice: 's', expected: 'answer', fact: 'f1' },
        { id: 'A2', slice: 's', expected: 'abstain', fact: null },
    ],
    new Map([['A1', { kind: 'answer', fact: 'f1' }]]),
);
// {
//     total: 2, success: 1, answered: 1, correctAnswers: 1, errors: 0, missing: 1,
//     taskSuccess: 0.5, answerCoverage: 0.5, answeredAccuracy: 1,
//     slices: { s: { total: 2, success: 1 } },
// }

A2 has no recorded output, so it counts as missing and does not pass, even though it expected an abstention.

In Python, implement summarize(dataset, results) with lists and dicts, and use the Python keys from the table.