Model Serving & Inference · 3 to 4 hours
Choose a serving strategy under tail-latency limits
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.
4 h
Overview
Choose and defend a serving strategy under memory, tail-latency, quality, and operational constraints. The goal is usable capacity: successful, sufficiently correct work delivered within each workload's objectives, not maximum GPU utilization or raw token throughput. You practice on a synthetic four-device scenario that you work by hand before defending a provisional choice.
Study time: 3 to 4 hours
Interview format: System design discussion with capacity calculations
By the end you can
- Define workload classes with their own latency, completion, and quality objectives, and a goodput definition
- Estimate KV capacity per device from model shape, weights, and runtime reserve, and convert arrival rates into reserved KV with Little's Law
- Check serving strategies against measured and computed gates, including a lost device, and state confidence and missing evidence
- Design scheduling, admission, and overload behavior that protects interactive traffic during bursts and device failures
- Gate builds and runtime optimizations on quality and on valid open-loop benchmarks, and plan failure-injection tests
Prerequisites
Helpful background: Comfort with percentiles, Little's Law, and GiB arithmetic helps.
This topic covers runtime scheduling, memory, and hardware. AI Application Integration covers streaming state in the client and server.
In this guide
- Define workload classes and objectives
- Model the bottlenecks
- KV memory and admission budgets
- Parallelism choices
- Scheduling and chunked prefill
- Speculative decoding and prefix caching
- Quality checks and release benchmarks
- Serving scenario
- Work the capacity inputs
- Check each strategy against the gates
- Admission, overload, and warm capacity
- Failure-injection matrix
- Design and defend the serving plan
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
System Design
N+1 headroom, readiness checks beyond liveness, and bulkheads that keep one failing part from draining shared capacity.
System Design
Little's Law fleet sizing with headroom, plus warm-up time and warm pools for capacity that cannot start quickly.
AI Engineering
Practice a ship or hold decision from slice rates and paired comparisons, with the uncertainty stated.
System Design
Compares round-robin, least-connections, and hash routing, plus health checks and connection draining for a server pool.
Behavioral Interview
Frames a decision with calibrated confidence and the result that would prove it wrong, as a provisional choice needs.
Practice the interview question
Answer it in your own words before you look at the reference answer.