Production & Reliability · 90 minutes
Explain latency, throughput, and rate limits
Explain why a slow model provider raises concurrency and retry load, using latency, throughput, rate limits, deadlines, and a bounded response strategy.
1.5 h
1
Step 1: The question
Using the incident, explain why a slow AI dependency can cause rising concurrency and retry load. Define latency, throughput, and rate limits, and describe a bounded response strategy.
Incident: a slow provider and stacked retries
All figures are synthetic. A support chatbot receives about 20 tasks per second. Normally each task spends about 3 seconds in the service. Then the model provider becomes slow, and tasks start to average about 9 seconds.
Two layers repeat failed or timed-out calls. The application makes up to three attempts per request, and the provider SDK it uses makes up to three attempts for each of those application attempts. The SDK setting is the library default, and nobody counted it in the application's retry plan.
Deliverables
- Average work in flight before and during the slowdown
- Worst-case downstream attempts for one logical task
- Definitions of latency, throughput, and rate limits
- A bounded response strategy, including what the user sees when a limit is hit
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can calculate the capacity and cost examples, distinguish attempts from retries, and explain why operational success needs a user-facing definition.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
How do latency and throughput differ?
Why use jitter?
Does a timeout mean an operation failed?
Can a queue solve sustained overload?
What makes a fallback acceptable?
7