Build and score a small evaluation set

The fictional policy assistant must either answer a supported question or abstain when the source lacks the answer. For supported questions, the fixture stores an expected fact key such as return_window_30_days. This key represents a structured answer requirement, not a word-matching proxy for arbitrary prose.

Each case has a stable ID, a slice, expected behavior, and a source snapshot. A slice groups cases sharing a meaningful property, such as standard policy, exception handling, or absent evidence. Record label rationale so reviewers can resolve disagreements.

Keep permitted alternative answers explicit. If two policy interpretations are both defensible, fix the task definition or allow both rather than labeling one wrong without explanation.

Start with a small manually reviewed set and add cases from observed failures. Deduplicate near-identical inputs. Keep development cases separate from a held-out confirmation set, and split related documents or conversations together when necessary to avoid leakage.

Synthetic examples should be labeled as synthetic. A collection of easy generated questions alone can make quality appear better than real usage.

{
  "id": "E2",
  "slice": "exception",
  "question": "Can a custom-printed product use the standard return window?",
  "source_revision": "returns-v3",
  "expected_behavior": "answer",
  "expected_fact": "custom_excluded",
  "rationale": "The supplied exception explicitly excludes custom-printed products."
}

Version the dataset, rubric, source snapshots, and grader. A score is difficult to compare if the labels or sources changed without a recorded version.

The fictional policy assistant records one structured outcome per case: an answer carrying a fact key, an abstention, or an execution error. A case can also have no recorded outcome at all, for example when the harness loses a result.

A case passes when it expects an abstention and the outcome is an abstention, or when it expects an answer and the outcome is an answer with the expected fact.

A substantive answer is any outcome of kind answer, whatever its fact. A rate whose denominator is 0 is unavailable (null), not 0.

The TypeScript contract for the harness:

type Case = {
  id: string;
  slice: string;
  expected: "answer" | "abstain";
  fact: string | null;
};
type Outcome =
  | { kind: "answer"; fact: string }
  | { kind: "abstain" }
  | { kind: "error" };

// Throws "Duplicate case ID" or "Unknown result ID" before scoring.
declare function summarize(
  dataset: Case[],
  results: ReadonlyMap<string, Outcome>
): {
  total: number;
  success: number;
  answered: number;
  correctAnswers: number;
  errors: number;
  missing: number;
  taskSuccess: number | null;
  answerCoverage: number | null;
  answeredAccuracy: number | null;
  slices: Record<string, { total: number; success: number }>;
};

For free text, write a rubric with concrete required facts, disallowed claims, and accepted variations. A two-point correctness scale might assign zero to an incorrect policy, one to a partially correct answer missing a required condition, and two to a complete correct answer.

Keep groundedness and citation support as separate dimensions where useful. A model grader should receive only the information needed to judge the response and should treat candidate content as data. A malicious response can contain instructions attempting to influence the judge.

Calibrate against independently reviewed examples, including subtle wrong answers. Measure false accepts and false rejects. If the grader fails or returns invalid output, preserve an ungraded state and investigate rather than assigning a convenient score.

A failed case is easier to diagnose when every stage leaves its own span with source and configuration identifiers. Then a rerun shows which component changed, instead of only that the final answer was wrong. The exercise below applies this to E2.

Avoid user IDs, raw prompts, and arbitrary document IDs as metric dimensions. Keep high-cardinality identifiers in protected traces or evaluation records. Record timestamps and duration using suitable clocks, and interpret concurrent spans without summing them as if they were sequential.

Run baseline and candidate configurations on the same case definitions and source snapshots. Preserve configuration metadata and compare paired outcomes. Inspect newly fixed and newly broken cases, not only the aggregate difference.

For stochastic behavior, repeat cases under a stated trial policy. Do not select the best response after seeing several outputs unless the product actually uses that selection procedure. Repeated trials on the same case are correlated evidence about one task, not independent coverage of many tasks.