Explain latency, throughput, and rate limits

A working model endpoint is only one dependency in an application. Authentication, retrieval, storage, model generation, tool execution, and response delivery can each fail. A successful HTTP response does not establish that the user's task completed correctly.

Define what a usable outcome means. A support assistant may need a complete response within a deadline, valid citations, and appropriate handling of missing evidence. Operational availability and semantic correctness can be measured separately so each failure remains visible.

An intentional, accurate no-answer response can be valid product behavior. A generic fallback returned during every outage should not automatically count as successful task completion.

Latency measures elapsed time for an operation. For streaming AI interfaces, distinguish time to first useful output from total completion time. Fast initial tokens can make an interface feel responsive while the complete result still takes too long.

A percentile describes a distribution. If p95 latency is four seconds, approximately 95% of observations are at or below that value under the measurement's definition. It does not describe every user's experience or guarantee future performance.

Measure end-to-end task duration and important stage durations. Concurrent operations overlap, so summing their durations can overestimate total time. Averaging several services' p95 values does not produce the end-to-end p95.

Throughput is work completed per unit time, such as requests per second. Concurrency is work in progress at a moment. A system can have high concurrency because requests are slow, rather than because it completes work efficiently.

In a stable system, Little's Law relates average in-flight work, arrival rate, and average time in the system. If 20 tasks arrive per second and each spends an average of three seconds in the measured system, average in-flight work is about 60 tasks.

This is an average relationship under stability assumptions. It does not establish peak capacity or safe limits during bursts. Long requests, retries, and queueing can increase resource occupancy.

A provider may limit requests, tokens, concurrent jobs, or spend over a specified interval. Limits can differ by account, model, region, or endpoint. The application must understand the actual scope of the limit it encounters.

A token-heavy request can consume more capacity than a short request. Request count alone may therefore be a poor admission signal. Estimate expected work and observe actual usage.

When a service rejects a request because capacity is exhausted, respect any documented retry guidance. Immediate repeated retries can consume more resources and prolong an overload.

A timeout limits how long one operation waits. A deadline specifies when the overall request must stop trying to complete. Separate connection, response, idle-stream, and total-task limits when the client supports them.

If a user task has five seconds remaining, starting a dependency call with a 30-second timeout wastes work after the caller has stopped waiting. Propagate the remaining deadline through dependent operations.

Stopping a local wait does not guarantee that the remote service stopped processing. Cancellation depends on the transport and service contract. A timed-out mutation may already have succeeded, so it needs reconciliation or idempotency before retrying.

A retry repeats an attempt after a failure. It can help with temporary service problems, but it adds load, latency, and cost. Retry only failures classified as recoverable under the operation's contract.

Backoff increases the delay between attempts. Jitter varies the delay so many clients do not retry at the same instant. Limit attempts and total elapsed time, and account for retries already performed by SDKs.

Invalid credentials, malformed inputs, and denied access usually require correction rather than repeated identical requests. A provider refusal is a distinct product outcome and should not be treated as a generic transient error.

A fallback changes how the application responds when its preferred path fails. Examples include an evaluated alternate model, a cached authorized answer, search results without generation, or an explicit temporary-unavailability response.

Every fallback needs a defined contract. An alternate model may differ in output schema, context size, quality, privacy terms, or tool behavior. A cache may be stale or inappropriate for the current user's permissions.

Report degraded behavior accurately. If the application shows source links because generation is unavailable, the interface should make that behavior clear rather than claim it generated a verified answer.

Queues absorb short bursts by delaying work. They do not create processing capacity. If arrivals consistently exceed completion capacity, the queue grows and requests become stale.

Bound queue length and age. Expire work whose deadline has passed before starting an expensive model call. Apply fair scheduling or tenant quotas when one customer's workload could consume the entire service.

Load shedding rejects some work deliberately to preserve the service for accepted work. Define user-visible retry or resubmission behavior and measure these rejections honestly.

AI application costs can include input and output tokens, embeddings, retrieval, tools, retries, storage, and external services. Track cost per completed user task as well as cost per model call.

For a fictional service, 100 tasks each cost $0.01 for the initial call. If 20 tasks each require one equally priced retry, total call spend is $1.20. If 90 tasks complete successfully, call spend per successful task is about $0.0133.

The example excludes other costs and assumes equal call prices. Real billing can distinguish cached input, output, reasoning usage, and service-specific fees. Verify the provider's accounting.

A service-level indicator is a defined measurement. A service-level objective is its target over a specified window. An error budget expresses the tolerated amount of failure implied by the objective.

For a request-based 99% success objective across 10,000 eligible requests, the failure allowance is 100 requests. If 80 fail, the observed success rate is 99.2% and 20 failures remain within that fixed-volume illustration.

A time-based objective uses a different denominator. Do not convert request failures to downtime without assumptions about traffic and failure distribution. Define exclusions such as invalid input carefully so service-caused failures are not hidden.