Evaluation & Observability · 3 to 4 hours
Design a release gate with slices and uncertainty
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
Hard
System Design
4 h
Premium
Used in interview rounds:
Debugging & Evaluation (Hard)Behavioral & Project Deep Dive (Hard)Take-home & Review (Hard)1
Step 1: The question
Review the 400-case candidate in the release scenario. Design a release gate using overall performance, protected slices, human review, calibrated graders, traces, and uncertainty. Explain what zero failures in 100 tests establishes and how production sampling affects monitoring.
Constraints
- The evaluation has 400 independent task fixtures and one trial per configuration: 300 routine, 60 exception, and 40 missing-evidence cases
- Release criteria were declared before the run, including at least 85% observed success on the exception slice
- A model grader scores free-text answers, and its configuration can change between evaluation runs
- Production tracing keeps every error trace but only a sample of successful requests
Deliverables
- A ship or hold decision for the candidate, with slice rates and a reading of the paired outcomes
- A release gate: mandatory invariants, overall and protected-slice criteria, human review, grader calibration, and the uncertainty standard
- What zero failures in 100 independent tests establishes, with the bound and its assumptions
- A monitoring plan that accounts for trace sampling, missing telemetry, drift, and re-evaluation triggers
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.