Model Serving & Inference · 2 to 3 hours
Profile TTFT, throughput, and failures
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
3 h
Overview
Build a serving benchmark that measures the right things, estimate KV cache memory, and diagnose latency under load. The runnable exercise analyzes synthetic per-request timing records; it is not a network load generator, tokenizer, or GPU profiler.
Study time: 2 to 3 hours
Interview format: Hands-on timing analysis with deterministic fixtures, then a load-profile diagnosis
By the end you can
- Freeze a benchmark contract that names the model artifact, runtime, hardware, workload distributions, and success definition
- Validate per-request timing records and compute success latency, TTFT, request TPOT, and throughput over an observation window
- Keep timeouts, rejections, and other non-success outcomes visible in a latency report
- Estimate KV cache memory from a model's shape and turn it into a limit on concurrent sequences
- Tell queueing, prefill, decode, and memory limits apart using stage timings, active sequences, and an offered-load sweep
Prerequisites
Helpful background: Python or TypeScript for the runnable exercise; the linked practice problem runs in JavaScript or Python. The memory and load exercises need only arithmetic.
This topic covers runtime scheduling, memory, and hardware. AI Application Integration covers streaming state in the client and server.
Freeze the benchmark contract
Record the model weights and revision, tokenizer, chat template, precision, runtime version, hardware, parallelism, cache policy, sampling parameters, and output limits. Include the input and output length distributions and the arrival pattern.
A benchmark with tiny prompts and ten-token answers cannot predict a workload with long retrieval prompts and multi-page reviews. Use representative cases and keep important slices separate. Report cold-start and warmed-service behavior separately.
Define success as a valid terminal completion that meets the task contract. Track malformed outputs, timeouts, cancellations, and rejections independently. A fast incorrect answer is not useful capacity.
Instrument the request timeline
Capture arrival, admission, execution start, first output, and terminal time where they are observable. Client-observed latency includes transport effects. Server timestamps can separate queueing from computation, but never subtract clocks from different machines without synchronization or a tracing strategy.
Use monotonic time for elapsed durations within a process. Preserve request IDs so stages can be correlated. Do not infer token timings from arbitrary network chunks unless the protocol supplies token-level information.
For a request that produces N > 1 tokens, the average time per output token after the first (request TPOT) is (end - first_token) / (N - 1) under a defined completion boundary. It is an average, not a distribution of individual token gaps.
A runnable timing summary
A fictional hint service logs one record per request, in seconds on one monotonic clock. Each record has start, end, and status; a success also has firstToken and outputTokens (first_token and output_tokens in Python). The observation window must contain every record and all the work being counted.
summarize(records, windowStart, windowEnd) checks its input in this order and throws (Python raises ValueError) with the message of the first rule that fails:
- Both window bounds are finite numbers and
windowEnd > windowStart: "Invalid observation window". recordsis a non-empty list: "Expected nonempty records".- Records are then checked one at a time, in order. Each must be an object: "Record must be an object".
startandendare finite numbers withwindowStart <= start <= end <= windowEnd: "Invalid request times". Times are checked before status.statusis one ofsuccess,failed,timeout,cancelled, orrejected: "Invalid status".- A success also needs a finite
firstTokenwithstart <= firstToken <= endand an integeroutputTokensof at least 1: "Invalid successful output". Non-success records need onlystartandend.
Booleans never count as numbers.
| Field (JavaScript, then Python) | Value |
|---|---|
requests | All records |
successes | Records with status success |
nonSuccessFraction, non_success_fraction | 1 - successes / requests |
successfulRequestsPerSecond, successful_requests_per_second | Successes divided by the window length, windowEnd - windowStart |
successfulOutputTokensPerSecond, successful_output_tokens_per_second | Output tokens of successful requests divided by the window length |
successLatencyP95S, success_latency_p95_s | p95 of end - start over successes |
successTtftP95S, success_ttft_p95_s | p95 of firstToken - start over successes |
successRequestTpotP95S, success_request_tpot_p95_s | p95 of (end - firstToken) / (outputTokens - 1) over successes with more than one output token |
The percentile is nearest rank: sort the values and take sorted[max(0, ceil(0.95 * n) - 1)]. An empty list gives null (None in Python). The TPOT percentile summarizes per-request averages; it is not p95 inter-token latency. One-token requests have no interval after the first token and are left out of that metric.
Counting tokens only from successful requests measures useful output. When you diagnose resource efficiency, also report total generated work and wasted tokens. This summary groups non-success outcomes for brevity; a production report also breaks them down by status and cause, and checks that request IDs are unique and the record set is complete before summarizing.
The fixture uses the window 0 to 10 seconds:
| Record | Status | start | firstToken | end | outputTokens |
|---|---|---|---|---|---|
| R1 | success | 0 | 0.5 | 2.5 | 9 |
| R2 | success | 1 | 1.25 | 1.25 | 1 |
| R3 | timeout | 2 | none | 7 | none |
| R4 | rejected | 3 | none | 3 | none |
Code exercise
Timing summary
Implement summarize(records, windowStart, windowEnd) (summarize(records, window_start, window_end) in Python) for the contract above. Check the rules in the stated order and throw the first failing rule's message, count only successful requests toward latency, TTFT, TPOT, and throughput, and return null for a percentile with no values.
Expected
One object with the eight fields from the table, camelCase in JavaScript and TypeScript and snake_case in Python. The practice problem compares the whole summary for the R1 to R4 fixture with floats rounded to four decimal places, checks nearest-rank p95 on a longer list of values, checks null (None) for metrics with no values, checks that the input records are not reordered or changed, and checks the exact message for invalid input.
Limitations
The summary trusts that each record is one unique, complete request on a single monotonic clock: it does not detect duplicate request IDs, missing records, or clock skew between machines. Python rejects a float token count such as 9.0, but JavaScript cannot tell 9.0 from 9.
Predict the summary before you run it
Predict the outcome
Predict the summary
Using only the rules and the R1 to R4 fixture, predict every field of the summary and name the records each one counts. Then predict the nearest-rank p95 of the twenty values 1, 2, ..., 20. Write your answers before you reveal them.
| Field | Records counted | Value |
|---|---|---|
| requests | ? | ? |
| successes | ? | ? |
| nonSuccessFraction | ? | ? |
| successfulRequestsPerSecond | ? | ? |
| successfulOutputTokensPerSecond | ? | ? |
| successLatencyP95S | ? | ? |
| successTtftP95S | ? | ? |
| successRequestTpotP95S | ? | ? |
| p95 of the values 1 to 20 | ? | ? |
Load generation and saturation
A closed-loop test sends new work when earlier work finishes. An open-loop test schedules arrivals independently of completion. Closed-loop tests can reduce offered load when the system slows, masking queueing that occurs under real arrivals.
Test a range of arrival rates and concurrency levels. Report offered, accepted, completed, and rejected work. Include the load generator's own queueing and capacity limits. Keep timed-out and unfinished requests in the report; they must not disappear because no final token arrived.
Queue wait does not grow in proportion to load. The average number of requests in service is roughly the arrival rate times the time each one spends in service. As that number approaches how many requests the server can hold at once, waits rise sharply.
Find the load at which the defined latency and quality objectives stop being met. Maximum raw throughput beyond that point is not the same as usable serving capacity.
Estimate memory carefully
Worked example
A simple dense-weight estimate is parameter count × bytes per stored parameter. Quantization metadata, runtime buffers, and execution state add overhead.
For an ordinary decoder-only model with full-attention KV storage, a simplified cache estimate is:
KV bytes ≈ 2 × layers × KV_heads × head_dimension × cached_tokens × bytes_per_element
Here cached_tokens is the total stored across active sequences, ignoring sharing and allocation overhead. The factor 2 represents keys and values. Use KV heads, not necessarily query heads. Architecture-specific compression, sliding windows, sharding, or mixed cache layouts require different accounting.
Example: 32 layers, 8 KV heads, head dimension 128, and two-byte elements give 2 × 32 × 8 × 128 × 2 = 131,072 bytes per cached token. At 8,192 tokens that is 1,073,741,824 bytes, exactly 1 GiB per sequence under these assumptions. The estimate excludes weights and other memory and is not a hardware-sizing guarantee.
Dividing the memory left for the cache by the tokens one request needs gives the number of sequences that can be active at once.
Tune scheduling with evidence
Continuous batching improves utilization by letting requests enter and leave ongoing work. Chunked prefill processes large inputs in pieces so the scheduler can mix them with decoding work. These policies trade scheduling overhead and prompt progress against decode responsiveness.
Measure TTFT and post-first-token behavior together. A change that improves one can worsen the other. Include long-prompt and short-prompt slices and test fairness between tenants or workload classes.
Limit active sequences and scheduled token work using the runtime's supported controls. Configuration names and defaults change; pin and verify the installed version instead of copying unexplained tuning flags.
Cache experiments
Compare cold prefixes, repeated prefixes, and partially shared prefixes. Keep prompt formatting stable when studying prefix reuse; even small token differences can reduce matching. Include cache eviction and realistic request order.
Prefix caching mainly avoids recomputing matching input state; it does not remove the cost of generating a long new response. Measure both hit rate and useful latency reduction.
Response caches need a different contract: the model and configuration, input, source revision, permissions, and freshness may all affect validity. Never treat similarity alone as proof that a cached answer is interchangeable.
Quantization and export checks
Evaluate the exact served artifact on code correctness, structured-output validity, refusals, and long-context cases. Compare latency and memory at the same workload, not only file sizes.
A precision change can interact with kernels, batching, and KV representation. Check numerical failures, unsupported operations, and fallbacks that silently run slower code paths. Maintain a known-good artifact for rollback.
Admission, deadlines, and retries
Estimate work from tokenized input and a bounded output allowance, with architecture-specific handling for media. Request counts alone miss major differences in resource demand.
Use bounded queues and deadlines. Propagate cancellation and verify that resources are released. Retrying an overloaded server immediately can deepen the overload; backoff must fit the remaining request deadline and should be bounded.
Track whether partial output was delivered. A retry should not silently concatenate a new generation onto an earlier partial answer as if it were one coherent response.
Diagnose two load profiles
The two profiles in this exercise have the same request count and arrival rate but very different prompt lengths. Decide what you need to measure before you read their numbers.
Diagnose
Find the limit for two load profiles
Suppose a coding-practice platform self-hosts one open-weight model for two features: short hints (profile A) and long code reviews (profile B). The model has 32 layers, 8 KV heads, head dimension 128, and 2-byte cache elements. Estimate KV memory as 2 × layers × KV heads × head dimension × cached tokens × bytes per element. After weights and runtime buffers, 8 GiB of device memory is left for the KV cache. This fictional scheduler admits a request only when its prompt plus its full output cap fit in free KV memory, and requests share no cache entries.
Each profile ran for 300 seconds with open-loop arrivals averaging 1 request per second, 300 requests in total. Every request has a 200-token output cap, and every success generated 200 tokens. The request deadline is 20 seconds. TTFT here runs from arrival to first token, and the percentiles cover successful requests.
| Measurement | Profile A | Profile B |
|---|---|---|
| Prompt tokens (every request) | 1,000 | 7,800 |
| Requests, timeouts | 300, 0 | 300, 3 |
| Queue wait (arrival to admission) p50, p95 | 0.00 s, 0.02 s | 2.4 s, 8.7 s |
| Prefill (admission to first token) p50, p95 | 0.15 s, 0.25 s | 1.2 s, 1.9 s |
| TTFT p95 | 0.26 s | 10.3 s |
| Request TPOT p50 | 0.030 s | 0.031 s |
| Peak active sequences | 13 | 8 |
| Share of the run at that peak | under 1% | 72% |
- Before reading the numbers, list the measurements that tell a queueing limit, a prefill limit, a decode limit, and a KV memory limit apart, and the pattern that would point to each.
- Compute the KV bytes per cached token, the tokens the 8 GiB budget holds, each profile's reservation per request, and the maximum number of concurrent sequences for A and B.
- Estimate the average number of requests in service for each profile as arrival rate × time in service, where time in service is the median prefill plus 199 median TPOT intervals.
- Name what limits profile B, and whether anything limits A at this load, citing the measurements. Say what prefill contributes.
- Propose one experiment that would confirm your diagnosis and the result that would refute it.
Completion checklist
- I can reproduce the R1 to R4 summary and name the records behind each value.
- I can explain what request TPOT, success-only percentiles, and window throughput leave out.
- I can turn a KV estimate into a maximum number of concurrent sequences.
- I can name the measurements that separate queueing, prefill, decode, and memory limits.
Sources
- vLLM: Optimization and tuning Describes chunked prefill batching prefill chunks with decode work, and preemption when KV cache space runs out.
- vLLM: Automatic prefix caching States that prefix caching reduces prefill time for shared prefixes but not the time spent generating new tokens.
- Efficient Memory Management for Large Language Model Serving with PagedAttention Explains why per-request KV cache memory limits batch size, and block-based cache allocation.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Defines multi-query and grouped-query attention, which use fewer key-value heads than query heads.
- Orca: A Distributed Serving System for Transformer-Based Generative Models Introduces iteration-level scheduling, which lets requests join and leave a running batch (the basis of continuous batching).
The timing records, load profiles, memory budget, measurements, thresholds, and rubric are original synthetic teaching material about a fictional serving deployment. They are not benchmarks of any real model, runtime, or hardware. The timing analysis is executable; no model server or GPU workload is benchmarked here. Study times and readiness thresholds are editorial estimates. Product and library details are as of November 2026.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
System Design
Back-of-the-envelope memory and capacity sizing, the habit behind turning a KV estimate into a concurrency limit.
AI Engineering
Introduces Little's Law and bounded queues, the basis for estimating how many requests are in service at once.
Question bank
Finds the kth smallest value without a full sort, which is exactly what a nearest-rank p95 selects.
Practice problem
Count the most intervals open at once with a sweep, the step that turns start and end times into peak concurrency.
Practice the interview question
Answer it in your own words before you look at the reference answer.