Evaluation & Observability · 2 to 3 hours
Build and score a small evaluation set
Score a six-case evaluation set with a deterministic harness, report success, coverage, and answered accuracy with named denominators, and keep missing outputs visible in the totals.
3 h
1
Step 1: The question
Build the six-case evaluation and calculate total task success, answer coverage, answered accuracy, and slice results. Add a case with no recorded output and explain how it affects each denominator. Describe the trace needed to diagnose E2.
A deterministic scoring harness
The fictional policy assistant records one structured outcome per case: an answer carrying a fact key, an abstention, or an execution error. A case can also have no recorded outcome at all, for example when the harness loses a result.
| Case | Slice | Expected | Expected fact | Recorded outcome |
|---|---|---|---|---|
| E1 | standard | answer | return_window_30_days | answer return_window_30_days |
| E2 | exception | answer | custom_excluded | answer return_window_30_days |
| E3 | missing | abstain | none | abstain |
| E4 | missing | abstain | none | answer shipping_fee_10 |
| E5 | standard | answer | defect_window_7_days | error |
| E6 | exception | answer | custom_excluded | abstain |
A case passes when it expects an abstention and the outcome is an abstention, or when it expects an answer and the outcome is an answer with the expected fact.
| Metric | Definition |
|---|---|
| Task success | Passing cases / cases in the dataset |
| Answer coverage | Substantive answers / cases in the dataset |
| Answered accuracy | Passing answers / substantive answers |
| Slice result | Passing cases / cases in the slice |
A substantive answer is any outcome of kind answer, whatever its fact. A rate whose denominator is 0 is unavailable (null), not 0.
The TypeScript contract for the harness:
type Case = {
id: string;
slice: string;
expected: "answer" | "abstain";
fact: string | null;
};
type Outcome =
| { kind: "answer"; fact: string }
| { kind: "abstain" }
| { kind: "error" };
// Throws "Duplicate case ID" or "Unknown result ID" before scoring.
declare function summarize(
dataset: Case[],
results: ReadonlyMap<string, Outcome>
): {
total: number;
success: number;
answered: number;
correctAnswers: number;
errors: number;
missing: number;
taskSuccess: number | null;
answerCoverage: number | null;
answeredAccuracy: number | null;
slices: Record<string, { total: number; success: number }>;
};Constraints
- Score structured outcomes only, with no free-text or model grading
- Report a rate as unavailable (null) when its denominator is 0
- Keep the six cases and their recorded outcomes exactly as given
Deliverables
- Counts, the three rates as fractions, and per-slice results for the six cases
- The same report with one added case that has no output, and what changed
- The spans and fields needed to locate E2's failure, and what each possible trace outcome would implicate
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Code exercise
Scoring harness
Implement summarize for the fixture above: count each kind of outcome, compute the three rates with their own denominators, and report every slice. Reject duplicate case IDs and results for unknown cases before scoring.
Limitations
This is an offline harness for typed fixtures with one trial per case. Validate runtime data before converting external files into these types. Free-text grading, repeated trials, and ungraded judgments need fields of their own.
Diagnose
Trace E2's failure
E2 asks whether a custom-printed product can use the standard return window. The expected fact is custom_excluded, and the recorded answer is return_window_30_days. The policy has two relevant sections: the standard window and the custom-product exception. You can add spans to the request trace and rerun the case.
- List the spans you would record, and the fields on each, to locate the failing stage.
- For each trace outcome, name the component it implicates and what you would inspect or change next:
- Retrieval finds both sections, the context includes both, and the answer still returns the standard window.
- Retrieval finds both sections, but the context omits the exception.
- Retrieval never returns the exception.
- If the assistant were an agent that also called a tool, what would you add to the trace?
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10
Kept in this browser until you sign up, then it moves to your account.
Complete this level when the harness reproduces the expected values and you can explain how every failed or ungraded case appears in the report.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why include negative cases?
Why separate missing output from execution error?
Why use paired comparisons?
Can a model judge be the sole ground truth?
Why version source snapshots?
7