Production & Reliability
Run AI features within latency, rate-limit, and cost budgets.
Reason about latency, throughput, and rate limits, implement bounded retries, and set SLOs, rollout plans, and cost controls for an AI feature.
4 questions
Questions
Easy
1.5 h
Explain why a slow model provider raises concurrency and retry load, using latency, throughput, rate limits, deadlines, and a bounded response strategy.
Medium
3 h
Implement a pure, deterministic retry policy with attempt limits, capped jittered backoff, retry-after, and a deadline check, then fit it into an executor that handles cancellation, partial streams, and uncertain writes.
Hard
4 h
Design reliability controls for a multi-tenant AI assistant: SLOs and error budgets, quota-aware admission, fallback and cost reservations, staged rollout with a canary decision, rollback, and recovery.
4 h
Find why an SMS reply webhook sent duplicate texts, left some messages unanswered, and multiplied model calls under rate limits, timeouts, and redeliveries. Fix each defect with a regression test, then declare the release check for the fix.
What you will be able to do
- Distinguish latency, throughput, and concurrency, and relate them with Little's Law
- Explain what rate limits and quotas restrict, and how to respond when capacity is exhausted
- Separate timeouts from deadlines, and explain why a timed-out write may still have succeeded
- Count attempts versus retries, and explain how stacked retries amplify load
- Judge whether a fallback or degraded response is acceptable and reported honestly
- Compute an error budget and the cost per successful task
- Classify failures by transience and replay safety before deciding to retry
- Implement a pure retry policy with an attempt limit, capped exponential backoff, full jitter, and retry-after
- Refuse a retry when the remaining deadline cannot fit the wait plus a useful attempt
Hands-on exercises
Interview rounds that use this topic
Topic boundaries
This topic runs the application. Evaluation & Observability measures its quality, and Model Serving & Inference covers the runtime underneath.
Related on CodeSnatch
System Design
Timeouts, backoff with jitter, circuit breakers, and bulkheads that stop one slow dependency from sinking a service.
System Design
Defines SLIs, SLOs, and error budgets, then shows burn-rate alerts that page on user impact rather than noise.
System Design
The server side of a rate limit: token bucket and window counters, 429 with Retry-After, and limits shared by replicas.
System Design
Sizes capacity from arrival rate and time per request, and shows why downstream limits cap what scaling out can add.
Code snippet
A runnable retry wrapper with capped exponential backoff, full jitter, and a predicate for which errors to retry.
AI Engineering
Goes one layer down to the serving runtime, where queue wait and prefill decide much of the latency you budget.
AI Engineering
Keys each side effect to a logical operation so retries, crashes, and duplicate jobs cannot repeat a write.
Community snippet
A small breaker with one half-open probe and an injectable clock, so its transitions can be tested without sleeps.
System Design
Tenant isolation patterns and recovery objectives for one shared service that many customers depend on.
Behavioral Interview
Shapes on-call, SLO trade-off, and capacity stories for the behavioral probes inside reliability rounds.