Choose long context or retrieval

Two designs answer questions about documents:

• Whole-document context sends the documents the user may see, in full, with every question. There is no index.
• Passage retrieval sends only the top k passages an index returns. Here it is a component with measured results; ranking belongs to Embeddings & Retrieval, and chunking and citations to RAG & Data Pipelines.

Decide per feature, from its goal and constraints: who asks, how much text one user may see, how often it changes, who may see which parts, the latency target, and the volume. Then rate both options in one table on the same dimensions: fit, effective use, cost, latency, freshness, access, traceability, and failure mode.

Use the simplified rule instructions + documents + question + output reserve <= window, and check the provider's own accounting. Count what one request would carry: the documents one user may see for that question, not the whole corpus, measured with the model's tokenizer, with headroom for growth. The largest documents decide, so check fit per document at request time, with a fallback for those that do not fit.

Fit is necessary, not sufficient: a request that fits can still be slow, costly, or poorly used.

Capacity is the number of tokens a request may hold; effective use is whether the model finds and combines the right ones. Models that find one planted fact almost perfectly can degrade on multi-fact, tracing, and aggregation tasks as input grows (RULER), and evidence in the middle of a long input is often used less well than evidence near its start or end (Lost in the Middle). A later amendment can override an earlier clause: whole-document context holds both and must apply the later one, while retrieval may return only one.

Test by slice on your own documents (single facts, evidence positions, synthesis, unanswerable questions), and re-ask with the evidence moved to the start, middle, and end.

The key measurement is whether every passage the answer needs is among the k returned, per slice: single facts can score high while synthesis scores low. A miss is silent, so "not in these passages" is weaker evidence than "not in the document". Retrieval also adds an index to build, update, and delete from, plus a search per question.

Long windows do not make it obsolete: in one study, a short-window model with simple retrieval performed comparably to a longer-window model on long-context tasks with much less computation (Retrieval meets Long Context).

Cost per question is (uncached input x uncached rate + cached input x cached rate + output x output rate) / 1,000,000, plus any per-question fee such as a search.

Prefix caching reuses the provider's processing of an identical leading part of a prompt. Reads bill at a fraction of the input rate; writes cost the same or more, depending on the provider. The cache lasts minutes by default (some providers sell longer lifetimes), is refreshed on each use, and breaks on any change at or before the cached point; very short prefixes are not cached. Put stable content first and the question last.

Reuse sets the discount. A shared document is reused across users only if all of them may see it, and a per-user document only by that user's follow-ups within the lifetime, so n questions pay one write and n - 1 reads. Caching changes input cost and time to first token, not output cost or the answer.

The model processes the whole prompt before its first token (prefill; see Model Serving & Inference), so longer input usually means a later first token. A cached prefix skips most of that work; retrieval adds a search but sends a far shorter prompt.

Measure p95 time to first token per request kind (cache miss, cache hit, retrieval) against the target, and never average p95 values across kinds. Writing the cache before the first request moves that work earlier, and is paid for even if no question follows.

Freshness. Whole-document context reads the current file, and a changed document stops matching its old cached prefix. Retrieval must re-index changes and retire old revisions, and a deletion must reach every derived copy, such as passages and embeddings.

Access. What reaches the prompt is the access boundary. Retrieval uses a permission-aware filter (OWASP LLM08), applied before ranking. Whole-document context must assemble only text the user may see, annexes and notes included. An instruction not to reveal text is not a control, and the identity comes from the session, never the question.

Traceability. Retrieval returns passage IDs to cite; a whole-document answer quotes the clause, and the application checks the quote.

A retrieval miss leaves a needed passage out: a wrong answer or a false "not covered". Diluted attention leaves the evidence in the prompt but missed or misread, such as a middle position, a near-duplicate clause, or an overridden clause. Hybrids sit between: retrieve, then send whole sections or short documents; send small documents whole and retrieve from large ones; or route each question to one or the other, which one study found kept long-context quality at much lower cost (Li et al.).

Settle the choice on the same labeled questions, sliced, comparing accuracy, cost per question, and p95 time to first token. Read each gap against its slice's size: a one-question gap is noise; a large gap with a measured mechanism is evidence. To price a quality gap, divide the daily cost difference by wrong answers avoided per day and compare with what a wrong answer costs.

Use the prices, prompt parts, and volumes in the scenario.

Compare each gap with the size of its slice before you call a winner.

Each row changes one fact; everything else stays as stated.