Model Serving & Inference · 3 to 4 hours
Choose a serving strategy under tail-latency limits
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.
4 h
1
Step 1: The question
Using the serving scenario, choose how to serve live hints and offline drafts on the four accelerators. Compute each strategy's KV headroom with every device healthy and with one lost, check every gate, and state your confidence and the evidence still missing. Then explain the scheduling, parallelism, caching, overload, quality, and verification choices that make your strategy hold during the hint burst and a device restart.
Constraints
- Four 80 GiB accelerators and the builds A16 and A8 only: no extra hardware and no other model
- Every gate in the scenario is mandatory; use the stated service times and reservations, and state any other assumption
- The reliability team owns SLO definitions, error budgets, and canary rules: supply the runtime signals they need rather than redesigning them
Deliverables
- Workload classes, what counts as useful work, and which requirements are gates versus preferences
- Capacity table per strategy: KV per token, KV pool per device, reserved KV at peak and with one device lost
- Gate check for every strategy, a provisional choice with your confidence, and the result that would reverse it
- Scheduling, admission, and preemption policy at peak, during the burst, and during a device restart, plus your parallelism, speculative decoding, and prefix caching decisions
- Verification, readiness, and rollback plan with failure-injection tests
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.