Evaluation & Observability · 90 minutes
Separate retrieval, packing, and generation failures
Use a request trace to tell retrieval, context-packing, and generation failures apart, and report answer quality with a named denominator next to operational signals.
1.5 h
1
Step 1: The question
A policy assistant gives an incorrect answer. Explain how to distinguish a retrieval failure, a context-packing failure, and a generation failure. Name one quality metric and one operational signal you would record.
Scenario. A fictional returns policy has two passages. S1 allows returns of standard products within 30 days. S2 excludes custom-printed products from that window. A user asks whether a custom-printed product can be returned after 20 days, and the assistant answers that the 30-day window applies. Every request records a trace with spans for retrieval, context packing, and generation.
Deliverables
- A stage-by-stage way to tell retrieval, context-packing, and generation failures apart, with the evidence for each
- One quality metric, with its denominator
- One operational signal, and why it cannot establish correctness on its own
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can define a case with observable success conditions, calculate success without excluding failed attempts, and read a trace to locate where evidence disappeared.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why separate retrieval and answer quality?
Is valid JSON a successful task?
What is a negative evaluation case?
Why include timeouts in task success?
What does a trace add?
7