Interview Practice
Debugging & Evaluation
Locate a failure stage by stage and prove the fix with evidence.
What interviewers evaluate
- Stage-by-stage hypotheses
- A confirming check per hypothesis
- A minimal fix and a regression test
- No weakened controls
What you will be asked
Decide whether a release ships, using evaluation and production evidence.
In the room
The interviewer gives offline results, slices, and canary data, some of them conflicting, and asks for a release decision you can defend.
Timebox
45 minutes
Mock interview
A timed answer to a random question from this level, then the reference answer and your score.
How to practice
- Work the linked questions: compute every rate and interval yourself before you read the reference answer.
- Write the decision rule before you look at the results, then apply it.
- With a peer: they argue for shipping while you hold the gate, then you swap roles.
What a strong answer shows
- Defines the gate before reading the results
- Reports slices and uncertainty, not only an aggregate
- Compares candidates on the same cases
- States what would trigger a rollback after release
Questions for this level
Each question opens on its Practice tab, and Previous and Next follow this list.
Evaluation & Observability
4 h
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
Production & Reliability
4 h
Design reliability controls for a multi-tenant AI assistant: SLOs and error budgets, quota-aware admission, fallback and cost reservations, staged rollout with a canary decision, rollback, and recovery.
RAG & Data Pipelines
4 h
Find why a partly failed re-embedding backfill left a help-center index missing, duplicating, and mixing vectors, then plan a repair and a cutover gate you could defend in a release review.
Production & Reliability
4 h
Find why an SMS reply webhook sent duplicate texts, left some messages unanswered, and multiplied model calls under rate limits, timeouts, and redeliveries. Fix each defect with a regression test, then declare the release check for the fix.
AI Product & Experimentation
4 h
Check the validity of a synthetic A/B test of an AI coach, compute its rates, intervals, and costs, and defend a release decision when offline ratings, immediate metrics, and matured outcomes disagree.
ML & Data Foundations
4 h
Design rolling-origin backtests, capacity-bound metrics, label-delay handling, and drift monitoring for a synthetic card-fraud model whose chargebacks arrive weeks late, then decide whether to promote, shadow, or hold a retrained candidate.
Follow-up questions
An interviewer may ask these next.
- The aggregate improved but a small slice regressed badly. Do you ship?
- Offline and online metrics disagree. Which do you trust?
- How do you keep the evaluation set from going stale?
Scoring rubric
Interview ready at 8 of 10 with no 0.
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Problem framing | Jumps to a solution or solves the wrong problem | Restates the problem but misses key constraints or success criteria | Clarifies goals, constraints, and success criteria before solving |
| Correctness | Key claims or code are wrong | Mostly right with gaps or unhandled cases | Correct and precise, including edge cases |
| Evidence or testing | No way to verify the answer | Generic testing ideas | Concrete fixtures, metrics, or traces that would confirm or refute the approach |
| Trade-offs | One option, no alternatives | Names alternatives superficially | Compares options against the stated constraints and says what would change the decision |
| Communication | Disorganized or unclear | Understandable but meandering | Structured and concise; states assumptions; checks in with the interviewer |