Model Serving
model-serving
Practice Problems
Serving Timing Summary
Summarize an inference load test: validate the window and every timing record, count throughput from successful work over the whole window, and report nearest-rank p95 latency, time to first token, and per-request time per output token, with null when a metric has no values.
System Design
ML System Design (Feature Store, Model Serving)
An ML system in production is mostly a data system with a model in the middle. The model is the smallest, most-discussed, and least-troublesome part. The hard parts are training data pipelines, feature freshness and parity between training and serving, the feature store that enforces that parity, model deployment and rollback, online and offline evaluation, and the operational concern that the model silently degrades as the world drifts. This lesson covers the canonical reference architecture: training pipeline, feature store with online and offline halves, model registry, serving infrastructure, monitoring, and the feedback loop. It is the senior-level mental model for designing 'add ML to product X' without falling into the standard traps.
AI Engineering
Diagnose a slow first token
Split a slow first token into queue wait, prefill, and emit from a server trace, and choose a serving configuration per workload instead of by throughput alone.
Profile TTFT, throughput, and failures
Summarize per-request timing records into latency, TTFT, request TPOT, and throughput with failures kept in the counts, then use a KV memory estimate and stage timings to find what limits a long-prompt load profile.
Choose a hosted API or a self-hosted model
Compare a hosted model API with a self-hosted open-weight model for a live tutor and a nightly scoring job: apply a quality gate, compute monthly cost at real utilization and the breakeven volume, check a data-location rule, and say what would reverse the choice.
Choose a serving strategy under tail-latency limits
Choose how to serve interactive hints and offline drafts on four memory-limited accelerators: compute KV headroom, check measured tail-latency and quality gates, and design scheduling, overload handling, and verification.