Model Serving & Inference · 2 to 3 hours
Profile TTFT, throughput, and failures
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
3 h
1
Step 1: The question
Summarize records R1 to R4 over the 0 to 10 second window, name the records behind each value, and say what the summary cannot tell you about token-gap jitter, failure latency, and answer quality. Then work through load profiles A and B: compute the KV memory each request reserves, decide what limits profile B, and propose a benchmark that separates queueing, prefill, and decode bottlenecks with failures kept in the counts.
A runnable timing summary
A fictional hint service logs one record per request, in seconds on one monotonic clock. Each record has start, end, and status; a success also has firstToken and outputTokens (first_token and output_tokens in Python). The observation window must contain every record and all the work being counted.
summarize(records, windowStart, windowEnd) checks its input in this order and throws (Python raises ValueError) with the message of the first rule that fails:
- Both window bounds are finite numbers and
windowEnd > windowStart: "Invalid observation window". recordsis a non-empty list: "Expected nonempty records".- Records are then checked one at a time, in order. Each must be an object: "Record must be an object".
startandendare finite numbers withwindowStart <= start <= end <= windowEnd: "Invalid request times". Times are checked before status.statusis one ofsuccess,failed,timeout,cancelled, orrejected: "Invalid status".- A success also needs a finite
firstTokenwithstart <= firstToken <= endand an integeroutputTokensof at least 1: "Invalid successful output". Non-success records need onlystartandend.
Booleans never count as numbers.
| Field (JavaScript, then Python) | Value |
|---|---|
requests | All records |
successes | Records with status success |
nonSuccessFraction, non_success_fraction | 1 - successes / requests |
successfulRequestsPerSecond, successful_requests_per_second | Successes divided by the window length, windowEnd - windowStart |
successfulOutputTokensPerSecond, successful_output_tokens_per_second | Output tokens of successful requests divided by the window length |
successLatencyP95S, success_latency_p95_s | p95 of end - start over successes |
successTtftP95S, success_ttft_p95_s | p95 of firstToken - start over successes |
successRequestTpotP95S, success_request_tpot_p95_s | p95 of (end - firstToken) / (outputTokens - 1) over successes with more than one output token |
The percentile is nearest rank: sort the values and take sorted[max(0, ceil(0.95 * n) - 1)]. An empty list gives null (None in Python). The TPOT percentile summarizes per-request averages; it is not p95 inter-token latency. One-token requests have no interval after the first token and are left out of that metric.
Counting tokens only from successful requests measures useful output. When you diagnose resource efficiency, also report total generated work and wasted tokens. This summary groups non-success outcomes for brevity; a production report also breaks them down by status and cause, and checks that request IDs are unique and the record set is complete before summarizing.
The fixture uses the window 0 to 10 seconds:
| Record | Status | start | firstToken | end | outputTokens |
|---|---|---|---|---|---|
| R1 | success | 0 | 0.5 | 2.5 | 9 |
| R2 | success | 1 | 1.25 | 1.25 | 1 |
| R3 | timeout | 2 | none | 7 | none |
| R4 | rejected | 3 | none | 3 | none |
Constraints
- Use only the records, the load-profile measurements, and the model shape given
- Apply the summary rules and the nearest-rank percentile exactly as stated
- Keep every non-success request in the request count
Deliverables
- The eight summary values for R1 to R4, with the records each one counts
- What the summary cannot show about token-gap jitter, failure latency, and answer quality
- The KV reservation and maximum concurrent sequences for profiles A and B, and the limit for B with its evidence
- A benchmark plan: frozen contract, offered-load sweep, stage timings, cache states, and failure accounting
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Code exercise
Timing summary
Implement summarize(records, windowStart, windowEnd) (summarize(records, window_start, window_end) in Python) for the contract above. Check the rules in the stated order and throw the first failing rule's message, count only successful requests toward latency, TTFT, TPOT, and throughput, and return null for a percentile with no values.
Expected
One object with the eight fields from the table, camelCase in JavaScript and TypeScript and snake_case in Python. The practice problem compares the whole summary for the R1 to R4 fixture with floats rounded to four decimal places, checks nearest-rank p95 on a longer list of values, checks null (None) for metrics with no values, checks that the input records are not reordered or changed, and checks the exact message for invalid input.
Limitations
The summary trusts that each record is one unique, complete request on a single monotonic clock: it does not detect duplicate request IDs, missing records, or clock skew between machines. Python rejects a float token count such as 9.0, but JavaScript cannot tell 9.0 from 9.
Diagnose
Find the limit for two load profiles
Suppose a coding-practice platform self-hosts one open-weight model for two features: short hints (profile A) and long code reviews (profile B). The model has 32 layers, 8 KV heads, head dimension 128, and 2-byte cache elements. Estimate KV memory as 2 × layers × KV heads × head dimension × cached tokens × bytes per element. After weights and runtime buffers, 8 GiB of device memory is left for the KV cache. This fictional scheduler admits a request only when its prompt plus its full output cap fit in free KV memory, and requests share no cache entries.
Each profile ran for 300 seconds with open-loop arrivals averaging 1 request per second, 300 requests in total. Every request has a 200-token output cap, and every success generated 200 tokens. The request deadline is 20 seconds. TTFT here runs from arrival to first token, and the percentiles cover successful requests.
| Measurement | Profile A | Profile B |
|---|---|---|
| Prompt tokens (every request) | 1,000 | 7,800 |
| Requests, timeouts | 300, 0 | 300, 3 |
| Queue wait (arrival to admission) p50, p95 | 0.00 s, 0.02 s | 2.4 s, 8.7 s |
| Prefill (admission to first token) p50, p95 | 0.15 s, 0.25 s | 1.2 s, 1.9 s |
| TTFT p95 | 0.26 s | 10.3 s |
| Request TPOT p50 | 0.030 s | 0.031 s |
| Peak active sequences | 13 | 8 |
| Share of the run at that peak | under 1% | 72% |
- Before reading the numbers, list the measurements that tell a queueing limit, a prefill limit, a decode limit, and a KV memory limit apart, and the pattern that would point to each.
- Compute the KV bytes per cached token, the tokens the 8 GiB budget holds, each profile's reservation per request, and the maximum number of concurrent sequences for A and B.
- Estimate the average number of requests in service for each profile as arrival rate × time in service, where time in service is the median prefill plus 199 median TPOT intervals.
- Name what limits profile B, and whether anything limits A at this load, citing the measurements. Say what prefill contributes.
- Propose one experiment that would confirm your diagnosis and the result that would refute it.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Measurement correctness or Memory reasoning or Failure accounting
Kept in this browser until you sign up, then it moves to your account.
Complete this level when your summary reproduces the fixture values, you can explain what the summary leaves out, and you can name the measurements that separate queueing, prefill, decode, and memory limits for a load profile.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why record failed requests in a latency study?
Why can average TPOT hide a bad stream?
Why use KV heads in the memory estimate?
Why test cold and warm caches?
Why can a closed-loop load test hide queueing?
7