Latency
latency
Practice Problems
Serving Timing Summary
Summarize an inference load test: validate the window and every timing record, count throughput from successful work over the whole window, and report nearest-rank p95 latency, time to first token, and per-request time per output token, with null when a metric has no values.
AI Engineering
Select a model under quality, cost, and latency limits
Recommend a model or routing policy from measured quality, cost, and latency, say what evidence is still missing, and revise the decision when the traffic mix changes.
Design hybrid retrieval within a latency budget
Design hybrid retrieval with reranking for a multi-tenant help center, choose a configuration from measured quality and latency gates, and keep access rules intact through timeouts and model migrations.
Explain latency, throughput, and rate limits
Explain why a slow model provider raises concurrency and retry load, using latency, throughput, rate limits, deadlines, and a bounded response strategy.
Diagnose a slow first token
Split a slow first token into queue wait, prefill, and emit from a server trace, and choose a serving configuration per workload instead of by throughput alone.
Profile TTFT, throughput, and failures
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
Choose a serving strategy under tail-latency limits
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.