Diagnose a slow first token

Inference uses a trained model to produce outputs. Serving is the surrounding system that accepts requests, schedules computation, manages memory, streams results, and handles failures. A model checkpoint alone is not a reliable service.

Suppose a coding-practice platform runs an open-weight model on its own serving runtime. One learner asks for a short hint while another asks for a detailed code review. The same model may spend very different amounts of time on these requests because prompt length, output length, queueing, and concurrent demand differ.

Training updates learned parameters. Ordinary inference does not update them. Conversation history is supplied as context; the model does not learn it permanently.

The application validates and authorizes a request, prepares the prompt, and submits it to a model service. The service may hold the request in a queue until its scheduler admits it. It then processes the input and produces output tokens, often incrementally.

Record the actual model revision and inference configuration. A display name alone may not uniquely identify the deployed weights, tokenizer, template, or numeric precision.

Application latency includes more than model computation: authentication, retrieval, tools, network transfer, queueing, and client rendering may all contribute. Optimizing model kernels will not fix a slow retrieval query.

For a common autoregressive decoder-only language model, prefill processes the whole prompt in one pass (or in chunks, on runtimes that use chunked prefill), builds the attention state (the KV cache), and yields the first output token. Decoding then generates each further token one step at a time, using the cached state and the tokens generated so far.

Long inputs increase prefill work. Long outputs require more decode steps. The bottleneck varies with architecture, hardware, batch size, and implementation, so avoid assuming that all requests are limited by the same resource.

Streaming makes partial output available earlier. It does not necessarily reduce the time required to finish the response. A broken stream is not proof of successful completion.

State where clocks start and stop. A frontend's first visible text and a server's first generated token are different measurements.

Percentiles describe the distribution: p95 is a threshold at or below which approximately 95% of observations fall, subject to the chosen percentile convention. It is not the average of the slowest 5%.

Batching processes multiple requests together to use hardware efficiently. Static batching forms a group that is processed as a unit. Continuous batching (iteration-level scheduling) can admit and remove sequences as generation proceeds, depending on the runtime.

Batching can increase throughput but can also introduce waiting or resource contention. Admitting more sequences at once can raise total tokens per second while each decode step takes longer. A batch with long and short outputs presents a scheduling problem. There is no universally optimal batch size.

For interactive hints, protect responsiveness. For overnight content generation, higher throughput may matter more than each request's latency. Separate service objectives help choose a policy.

Memory is needed for model weights, attention state, temporary buffers, and runtime overhead. The key-value cache, usually called KV cache, stores attention keys and values so prior context need not be fully recomputed during each decode step.

KV usage generally grows with active sequence lengths and concurrency for ordinary full-attention decoder models. Architecture and cache policy affect the exact relationship. A model that loads successfully can still run out of memory when many long requests arrive.

Many runtimes admit a new sequence only when there is room for its KV cache. When room runs short, new requests wait in the queue, and some runtimes preempt a running sequence and recompute it later, which adds latency. Leave headroom and test representative traffic. Weight size alone is not a serving-capacity estimate.

Quantization represents some values with fewer bits, which can reduce storage and memory traffic. Weight quantization and KV-cache quantization are separate choices.

Lower precision can change quality, and acceleration depends on compatible hardware and kernels. A smaller model file does not guarantee faster end-to-end responses. Evaluate exact code correctness, structured output, and long-context behavior after changing precision.

Record the deployed representation as part of the release. A benchmark of the original high-precision model does not establish the quality of a quantized artifact.

Prefix caching does not mean the service simply returns an old answer. It reuses compatible internal state (the KV cache) for identical leading prompt tokens. New output generation still has all its decode work to do, and requests that share no prefix gain nothing.

Do not use cross-user caches without an explicit security design. Similar prompts can refer to different private resources or permissions.

Set input, output, concurrency, and queue limits. Bound waiting with deadlines. When demand exceeds safe capacity, explicit rejection can be better than allowing every request to time out after consuming resources.

Retries can amplify overload; Production & Reliability covers bounded retries. A client cancellation should propagate so the backend can stop unnecessary work, but stopping the browser alone does not prove computation has stopped.

For a hint feature, tell the learner that a hint failed, was rejected, or was interrupted. Do not display partial feedback as a complete assessment.

Apply the timeline method to four hint requests from the same platform. Every hint prompt starts with the same instructions and problem statement, followed by the learner's code. H3 also carries the learner's full submission history, and H4 a long failing-test log. The product target is a server TTFT of at most 1,000 ms per hint.

A hosted API delegates much infrastructure work to a provider. Self-managed serving gives more control over deployment and hardware but adds capacity planning, upgrades, monitoring, and incident response.

Compare total lifecycle cost and operational requirements, not only a quoted token rate or GPU price. Consider supported models, required quality, data handling, traffic variability, team skills, and reliability targets. A smaller fallback model should not silently violate the product's required behavior.

The platform tests two configurations of the same model revision on the same hardware and the same mixed workload. All figures are synthetic.