Implement a bounded retry policy

HTTP status alone may be insufficient. A provider-specific error body can distinguish a temporary rate limit from exhausted billing quota. The adapter should normalize errors while retaining the original diagnostic category in protected telemetry.

The teaching policy permits at most three total attempts. It retries only explicitly classified transient failures when replay is safe. It uses capped exponential backoff with full jitter and respects a normalized retry-after delay as a minimum wait.

Before scheduling a retry, it requires enough remaining time for the delay plus a configured minimum useful attempt window. This check does not guarantee that the next request will finish. The runtime must still enforce the actual deadline.

The provider's raw Retry-After value may be a duration or date depending on the protocol. Parsing and validation happen before this pure policy function receives a nonnegative delay in milliseconds.

The policy is a pure function. The caller passes the failure classification, the number of attempts already made (including the one that just failed), the time left before the task deadline, and a random value for jitter. Nothing inside it sleeps, reads a clock, or generates randomness, so every decision is reproducible.

type Failure = {
  transient: boolean;
  safeToRetry: boolean;
  retryAfterMs?: number;
};
type Policy = {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  minimumAttemptMs: number;
};
type Decision =
  | { retry: true; delayMs: number }
  | { retry: false; reason: "unsafe" | "permanent" | "attempts" | "deadline" };

// Throws on a negative or non-finite duration, an attempt count that is not a
// positive integer, a random value outside [0, 1), or a zero minimumAttemptMs.
// Then stops, in order, for unsafe replay, permanent failure, and used-up attempts.
// Otherwise:
//   ceiling = min(maxDelayMs, baseDelayMs * 2 ** (completedAttempts - 1))
//   delayMs = max(floor(random * ceiling), retryAfterMs ?? 0)
// and stops for "deadline" when delayMs + minimumAttemptMs > remainingMs.
declare function decideRetry(
  failure: Failure,
  completedAttempts: number,
  remainingMs: number,
  random: number,
  policy: Policy
): Decision;

const policy: Policy = {
  maxAttempts: 3, baseDelayMs: 200, maxDelayMs: 1000, minimumAttemptMs: 300
};

Maintain one monotonic deadline for the logical task. Before every attempt, compute remaining time and cap dependency timeouts to it. After a failure, classify the error, ask the policy for a decision, and use a cancellation-aware wait.

Recheck the deadline after waiting because scheduling delays can consume more time than planned. Propagate cancellation to the client where supported, and stop dispatching new attempts once the user cancels.

A Promise.race timeout can stop local waiting without aborting remote processing. If a retry starts while the earlier call is still active, resource use can overlap. Track active work and avoid assuming timed-out attempts are free or cancelled.

Inspect SDK and proxy behavior. If the HTTP client performs retries internally, its attempts must fit inside the logical task's budget and deadline. Avoid three independent layers each interpreting maxAttempts as its own allowance.

Record both logical request count and physical attempt count. An apparent stable user-request rate can conceal a growing number of provider calls. Measure retry success as well as retry volume so ineffective recovery can be reduced.

For writes or tools with side effects, retain a stable operation identity and apply the idempotency or reconciliation design from Tool Calling & Agents. Do not infer safe replay from a function name.

Before any output is shown, a safe retry can replace the attempted response. After partial text is displayed, silently appending a newly generated answer may duplicate or contradict earlier content.

Define a UI contract for partial failure. It may preserve the partial response with an interruption label and offer a restart. True resumption requires protocol or application support and cannot be assumed from a new generation call.

Do not replay completed tool effects while restarting a response. Persist operation outcomes and feed their verified results into the resumed conversation.

A fallback can use another model, return source links, or defer work to an asynchronous job. Check output contract, data policy, tool support, quality, and available capacity before routing.

Fallback attempts consume the same logical deadline and cost budget unless the user explicitly begins a new task. Switching providers does not justify resetting the retry counter indefinitely.

When the primary provider is overloaded, sending all traffic immediately to a smaller secondary provider can overload it too. Use admission control and capacity limits for the fallback path. Report when degraded output is returned and evaluate it separately.

Limit concurrent expensive calls and bound waiting time. Reject or defer work before consuming substantial resources when the system cannot plausibly finish it within the task deadline.

A semaphore limits in-flight work but does not implement token-per-minute quotas by itself. A token budget or rate limiter addresses a different constraint. Multi-instance deployments need coordinated limits or deliberately partitioned capacity.

Use fair scheduling for tenants or workload classes where required. Large batch work should not consume all capacity needed for interactive requests.

Inject a sequence of fake outcomes and a controllable clock into the executor. Verify exact attempt counts, chosen delays, deadline exhaustion, cancellation, fallback eligibility, and partial-stream behavior.

For the pure policy, test invalid durations, random values outside [0,1), zero attempt counts, and a retry-after longer than the remaining deadline. Use fixed random values for deterministic assertions and separate tests for the distribution if needed.

Add integration tests that confirm the client actually receives cancellation and timeout settings. A policy function passing its unit checks does not prove the network stack honors them.