Interview Practice
Take-home & Review
Build a small feature with tests and a README, then defend it in review.
What interviewers evaluate
- Working code with tests
- A README with assumptions
- Defended trade-offs
- A credible next-steps list
What you will be asked
Defend an architecture, its evaluation, and its operations in review.
In the room
A review of a design or release plan you prepared offline. The panel challenges your guarantees, metrics, and operational trade-offs.
Timebox
2-4 hours plus a 45-minute review
Mock interview
A timed answer to a random question from this level, then the reference answer and your score.
How to practice
- Prepare a two-page design for one linked question, including the evaluation results you would expect and how you would operate it.
- List the three weakest points of your own design before the review.
- With a peer: they act as a skeptical reviewer and probe one weak point for ten minutes.
What a strong answer shows
- Defends each guarantee with the mechanism that enforces it
- Presents evaluation results with slices and uncertainty
- Explains operating cost, alerts, and rollback
- Acknowledges limits and gives a credible next-steps list
Questions for this level
Each question opens on its Practice tab, and Previous and Next follow this list.
Evaluation & Observability
4 h
Decide whether a candidate with better aggregate success but a regressed protected slice can ship, and design a release gate with calibrated graders, traces, and stated uncertainty.
Production & Reliability
4 h
Design reliability controls for a multi-tenant AI assistant: SLOs and error budgets, quota-aware admission, fallback and cost reservations, staged rollout with a canary decision, rollback, and recovery.
RAG & Data Pipelines
4 h
Design a private multi-tenant policy assistant whose answers and citations stay correct through asynchronous updates, retries, permission changes, and deletion.
AI Developer Tooling
4 h
Design one agent harness for a fictional payroll company's developer machines and its CI agent, covering permissions, sandboxing, extensions, hooks, evaluation, and budgets. Then decide from predeclared rules which configuration rolls out first, where, and with which permissions.
Take-home brief
Write a two-page design for one linked question (architecture, evaluation, and operations), then defend it in a 45-minute review.
A strong answer covers
- Guarantees tied to concrete mechanisms
- An evaluation plan with release gates
- Alerts, budgets, and a rollback path
- Known limits and next steps
Follow-up questions
An interviewer may ask these next.
- Which assumption, if wrong, breaks your design?
- What does it cost per month at ten times the traffic?
- How would you know the design is failing in production?
Scoring rubric
Interview ready at 8 of 10 with no 0.
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Problem framing | Jumps to a solution or solves the wrong problem | Restates the problem but misses key constraints or success criteria | Clarifies goals, constraints, and success criteria before solving |
| Correctness | Key claims or code are wrong | Mostly right with gaps or unhandled cases | Correct and precise, including edge cases |
| Evidence or testing | No way to verify the answer | Generic testing ideas | Concrete fixtures, metrics, or traces that would confirm or refute the approach |
| Trade-offs | One option, no alternatives | Names alternatives superficially | Compares options against the stated constraints and says what would change the decision |
| Communication | Disorganized or unclear | Understandable but meandering | Structured and concise; states assumptions; checks in with the interviewer |