Model Serving & Inference
Reason about prefill, decode, batching, memory, and tail latency.
Diagnose a slow first token, profile time to first token and throughput, and choose a serving strategy that holds tail latency under load.
4 questions
Questions
Easy
1.5 h
Split a slow first token into queue wait, prefill, and emit from a server trace, and choose a serving configuration per workload instead of by throughput alone.
Medium
3 h
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
3 h
Compare a hosted model API with a self-hosted open-weight model for a live tutor and a nightly scoring job: apply a quality gate, compute monthly cost at real utilization and the breakeven volume, check a data-location rule, and say what would reverse the choice.
Hard
4 h
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.
What you will be able to do
- Distinguish inference from training, and a serving system from a model checkpoint
- Explain prefill and decoding, and split a server TTFT into queue wait, prefill, and emit
- Compare latency with throughput and say where each clock starts and stops
- Explain how KV memory and batching affect queueing and concurrency
- Describe quantization and caching without treating them as guaranteed speedups
- Choose a serving configuration per workload from latency, failure, quality, and cost evidence
- Freeze a benchmark contract that names the model artifact, runtime, hardware, workload distributions, and success definition
- Validate per-request timing records and compute success latency, TTFT, request TPOT, and throughput over an observation window
- Keep timeouts, rejections, and other non-success outcomes visible in a latency report
Hands-on exercises
Interview rounds that use this topic
Topic boundaries
This topic covers runtime scheduling, memory, and hardware. AI Application Integration covers streaming state in the client and server.
Related on CodeSnatch
System Design
Peak-to-average ratios, headroom for losing a zone, and memory sizing: the arithmetic every serving plan starts from.
System Design
Sizes a fleet with Little's Law and headroom, and explains why slow warm-up means keeping capacity running.
System Design
Least-connections suits requests of very different lengths, and hash routing keeps cache locality on one node.
System Design
Reads latency as p50, p95, and p99 beside error rate and queue saturation, then turns them into SLOs.
System Design
Compares embedded, sidecar, and central model serving within a latency budget, and when a managed platform beats a custom build.
Behavioral Interview
Shows how infra loops probe capacity planning and on-call judgement, with a worked load-testing story to adapt.
AI Engineering
The product view of prefill: how prompt length moves the first token and what prefix caching saves per question.
AI Engineering
Covers the client side this topic leaves out: streamed partial output, and what pressing Stop really cancels.