Evaluation & Observability
Measure quality, investigate failures, and gate releases.
Separate retrieval, packing, and generation failures, build a small scored evaluation set, and design release gates that account for slices and uncertainty.
3 questions
Questions
Easy
1.5 h
Use a request trace to tell retrieval, context-packing, and generation failures apart, and report answer quality with a named denominator next to operational signals.
Medium
3 h
Score a six-case evaluation set with a deterministic harness, report success, coverage, and answered accuracy with named denominators, and keep missing outputs visible in the totals.
Hard
4 h
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
What you will be able to do
- Define an evaluation case with observable success conditions
- Distinguish retrieval quality from answer quality
- Choose between deterministic checks, model graders, and human review
- Calculate simple success rates with clear denominators
- Explain metrics, logs, and traces, and read a trace to locate lost evidence
- Design a small evaluation set with slices, expected behavior, and recorded label rationale
- Implement a deterministic scoring harness for structured outcomes
- Report task success, answer coverage, and answered accuracy with explicit denominators, overall and per slice
- Account for errors, missing outputs, and ungraded cases without hiding them
Hands-on exercises
Interview rounds that use this topic
Topic boundaries
This topic measures quality and investigates failures. Production & Reliability covers running the application.
Related on CodeSnatch
System Design
Covers metrics, logs, distributed traces, and SLOs, the operational signals you read next to quality metrics.
Practice problem
Implement task success, coverage, and answered accuracy with separate denominators and per-slice totals.
AI Engineering
Traces one missing policy passage stage by stage, the same evidence path this topic learns to evaluate.
System Design
Explains why offline wins can fail online, plus shadow and canary rollouts and drift monitoring for models.
AI Engineering
Takes evaluation online: an A/B test contract, guardrails, and the checks a lift must pass before shipping.
Behavioral Interview
Frames incident stories with mitigation, root cause, and prevention kept apart, for failure investigation rounds.
Community article
A practitioner's view of what each signal answers, why metric labels must stay low cardinality, and log sampling.
Community article
Lists RAG failure modes seen in production with the dashboard panels and weekly eval set that caught them.