Retrieve evidence for a follow-up question

A turn that uses retrieval has two separate inputs.

• The generator request carries the instructions, the recent turns (usually resent by the application on every request), the latest message, and the retrieved passages.
• The retrieval call usually takes one query string plus filters. It does not read the conversation unless the application puts the needed parts into that string.

So the generator can follow "And for meals?" from the history while the retriever cannot, and a fluent answer built from whatever came back can hide that retrieval was asked the wrong question.

Suppose a travel assistant has just answered "What is the hotel limit per night for trips abroad?" Follow-ups lean on that turn in two ways:

• Coreference: "Does that include breakfast?" uses "that" for the hotel limit.
• Ellipsis: "And for meals?" leaves out "trips abroad" and "limit".

Sent as typed, such a message has few meaningful words. The retriever may match an incidental word ("meals" pulls up an office canteen page) or return weak closest matches, so the generator gets the wrong evidence or none.

Conversational search research, such as the QReCC dataset and the TREC Conversational Assistance Track (CAsT), studies rewriting a question that depends on earlier turns into one that stands alone, and evaluates whether the system then retrieves the right passages. In CAsT 2019, rewrites written by people still beat the best automatic system.

A standalone query is one question that a reader who has not seen the conversation could understand. It keeps every subject and condition the message refers to and leaves out filler such as thanks or "never mind".

Identity is not the query's job: "I" and "my region" are enforced by filters from the session (see "Scope comes from the session on every turn"). Writing a region into the query may change ranking, but it restricts nothing.

Skip the rewrite when there is no history, as on the first message. Frameworks package this step: LlamaIndex's condense-question chat mode, for example, builds a standalone question from the conversation and the last message, then queries the index with it (as of November 2026). A rewrite instruction can be short:

The standalone query is a retrieval tool, not a replacement for what the user wrote. The generator still answers the user's actual message, keeping its tone and any format request, together with recent history.

Logging both the original and the rewrite lets a reviewer tell a bad rewrite from a bad retrieval. Here is one turn of the travel assistant, for a contractor in the South region:

{
  "turn": 2,
  "original_message": "And for meals?",
  "standalone_query": "What is the limit for meals on trips abroad?",
  "rewrite": { "method": "model", "prompt_version": "rewrite-v3" },
  "decision": "retrieve",
  "filters": { "tenant": "tenant-a", "region": ["South", "all"], "audience": ["contractor", "all"] },
  "filters_source": "session",
  "retrieved_ids": ["TR4", "TR1"]
}

Access scope decides which documents a user may see: tenant, employee, groups, region, employment type. On every turn, follow-ups included, the server looks it up from the authenticated session and passes it to retrieval as filters. Azure AI Search calls this security trimming: at query time the caller's identity becomes the filter, so documents that do not match are excluded.

Scope never comes from the message text ("I'm on the Finance team"), the standalone query, earlier assistant replies, or retrieved passages. A model rewrite that writes "for the Finance team" into the query has no authority to add Finance to the filters. An employee may still ask about the Finance team's travel limit: the query can mention Finance, because that is the question, but the filter stays the same.

Passages reused from an earlier turn were retrieved under that turn's scope; if the session has changed since, recheck them (Security & Privacy covers revocation and caches). OWASP LLM08:2025 (Vector and Embedding Weaknesses) lists inadequate access control on embedding stores as a cause of unauthorized retrieval and recommends permission-aware stores.

Retrieving on every turn has costs: each call adds cost and latency, new passages can change an answer the user only asked to reformat, and questions about the conversation itself ("What did I ask you first?") are answered from history. LlamaIndex notes that a chat mode that always queries the index can struggle with such meta questions.

Apply these checks in order to each new message.

1. No new policy question. An acknowledgement, a greeting, an empty message, a question about the conversation itself, or a request to change the format, length, or tone of an earlier answer. Do not retrieve. Reply directly, or rework the earlier answer using only its facts and citations.
2. Two possible referents. The message points back with "it", "that", or "this", two or more earlier things fit, and the answer would differ between them. Ask one short clarifying question, and retrieve after the reply.
3. New topic. The message names its own subject and does not depend on earlier turns (this includes the first message). Retrieve with a query built from this message only, and drop entities from earlier turns.
4. Follow-up. Anything else that depends on earlier turns. Rewrite it into a standalone query, then retrieve.

On every retrieval, the filters come from the session.

Use the session facts, the passages, and the four checks in "When not to retrieve".

A rewrite is model or template output, so it can be wrong. It can drop a condition, add an entity or number the conversation never mentioned, keep a stale entity after a topic switch, answer instead of rewriting, or pick one meaning of an ambiguous "it". Even a correct rewrite does not guarantee that retrieval finds the passage or that the answer uses it.

Log the original message and the rewrite together, and keep a handful of multi-turn cases with the expected decision and query (follow-up, reformat, ambiguous reference, topic switch, out-of-scope request). Rerun them when the rewrite prompt or model changes. Measurement at scale belongs to Evaluation & Observability.

These rows come from the travel assistant, not the policy session.