Model Serving & Inference · 90 minutes
Diagnose a slow first token
Split a slow first token into queue wait, prefill, and emit from a server trace, and choose a serving configuration per workload instead of by throughput alone.
1.5 h
1
Step 1: The question
Suppose a coding-practice platform's live hints stream smoothly once they start, but some start only after a long pause. Explain which measurements, and which clock start and stop points, separate a queueing problem from a prompt-processing (prefill) problem, and why the steady output rate does not explain the pause. Use the trace to name each request's dominant delay and one change you would test for each cause, with its trade-off. Finish with one memory reason the queue can grow while the model is loaded, and what the service should do when a hint cannot start before its deadline.
Scenario. A fictional coding-practice platform offers live hints during timed practice. It serves an open-weight model on its own serving runtime and streams each hint to the browser. Every hint prompt starts with the same instructions and problem statement, followed by the learner's code. H3 also carries the learner's full submission history, and H4 a long failing-test log. The product target is a server TTFT of at most 1,000 ms per hint.
Trace rule. All times are server-side milliseconds on one monotonic clock, starting when the request arrives at the serving runtime. For every row, TTFT = queue wait (arrival until the scheduler admits the request) + prefill (processing the prompt, which yields the first token) + emit (sampling that token and writing it to the stream). The dominant delay is the largest of the three parts. The browser's first visible text is not in this trace.
| Request | Prompt tokens | Requests queued at arrival | Queue wait (ms) | Prefill (ms) | Emit (ms) | TTFT (ms) | Output rate after the first token (tokens/s) |
|---|---|---|---|---|---|---|---|
| H1 | 700 | 0 | 20 | 70 | 10 | 100 | 40 |
| H2 | 650 | 14 | 2,325 | 65 | 10 | 2,400 | 33 |
| H3 | 18,000 | 0 | 30 | 1,860 | 10 | 1,900 | 38 |
| H4 | 6,000 | 6 | 970 | 620 | 10 | 1,600 | 35 |
All values are synthetic.
Constraints
- Use only the trace values; all times are server-side milliseconds on one clock
- Treat every change as a hypothesis to measure on representative traffic, not a guaranteed speedup
- Keep the answer to about three minutes spoken
Deliverables
- The measurements and clock boundaries that separate queue wait from prefill, and why the output rate does not explain the pause
- The dominant delay of H1 to H4, with its share of TTFT
- One change to test for the queueing case and one for the prefill case, each with a trade-off
- One memory reason the queue can grow while the model is loaded
- What the service does when a hint cannot start before its deadline
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Latency metrics from the trace
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can split a server TTFT into queue wait, prefill, and emit from a trace, say where each clock starts and stops, and explain a latency report without confusing user experience with aggregate hardware output.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why can a short answer take a long time to start?
Does streaming increase model speed?
Why can a loaded model still run out of memory?
Does quantization always improve latency?
Is throughput enough to choose a server?
7