Design a cited support assistant with measurable quality

Begin with the users, what a good outcome looks like for them, and the limits: volume, freshness, latency, and cost. Components come after.

Turn each requirement into a metric with a definition, a denominator, and a target. "90% accurate" means nothing until you say 90% of what: answerable questions, all questions, or answers shown.

Separate zero-tolerance guardrails, such as a citation to a source the user may not see, from targets that trade off against each other. Also separate offline metrics, measured on an evaluation set, from production metrics taken from traces.

The same metrics later become the evaluation plan and the rollout gate.

Ingestion copies each source's audience from the content system, the source of truth, onto every chunk. At query time, derive the eligible audiences from the authenticated session and a trusted account service, never from conversation text: a customer who types "I'm on Enterprise" has not changed plan.

Apply eligibility before ranking and generation. A prompt rule such as "never cite internal documents" is not a control, because content that reached the model can be paraphrased without a citation. Content no customer may ever see can stay out of the customer index entirely (OWASP LLM08).

Retrieval is one stage with its own latency budget; ranking belongs to Embeddings & Retrieval.

The application labels the selected chunks S1, S2, and so on, and maps each label to a trusted source record (article ID, revision, title, URL). The model writes labels, never links.

Labels not in the map are dropped. A statement left without a valid citation is removed, or the turn falls back to no-answer.

Check access when evidence is selected and again when a citation renders: a plan can change or an article can be deleted in between, even in a reopened conversation.

Citation precision is the share of citations that support their statement; citation recall is the share of statements fully supported by their citations (Liu et al. 2023; Gao et al. 2023, ALCE). A real article can still fail support.

The no-answer path has two triggers: no eligible evidence passes retrieval, or the supplied passages do not support an answer and the generator returns the no-answer form instead of guessing.

The customer sees a plain statement that the help center does not cover the question, with no guessed policy, and a handoff to a human agent that carries the conversation so the agent does not start over.

Keep the message generic. "There is an internal procedure for that" tells the customer that a restricted source exists.

Measure the path from both sides, as the evaluation set section below explains.

Drive reindexing from CMS events (publish, update, delete, audience change), backed by a periodic reconciliation against the index. Events can be late or lost, and some indexing services detect changes automatically but need an explicit strategy for deletions.

An update replaces the article's chunks with the new revision's chunks, a deletion removes them, and an audience change rewrites their audience metadata.

Derived copies count too: answer-cache entries built from the article, and citations in old conversations, which should resolve through current source records. Measure freshness lag from the event until the index and cache reflect it. This topic's Hard question covers races with in-flight answers and multi-tenant isolation.

Request tokens are instructions + capped history + question + k × chunk size, plus the output cap, counted with the model's own tokenizer.

Request cost is input tokens × input rate + output tokens × output rate, with rates per million tokens, plus any per-request retrieval charge. A cost per outcome divides all spend by the number of outcomes, so requests that end without that outcome still add to the spend.

Time to first token covers every stage before the first streamed token. Waiting for the whole answer before showing anything moves the first visible token to the full-answer time.

Stage percentiles do not add exactly. Use stage allocations for planning, and measure end-to-end p95 from traces.

Design the path from a customer's question to the rendered answer, then check it against the scenario's budgets.

Build cases of several types: answerable from one passage, multi-passage, unanswerable, restricted (answer only in a source the asker may not see), plan-boundary pairs (one question from two plans), stale after an update, and deleted source. Draw them from past tickets; label synthetic ones as synthetic.

Label each case with the expected behavior, the sources it must cite, and the sources it must not cite or use; the forbidden list catches leakage.

Name each metric with its denominator. Report correct and unnecessary no-answers together: an assistant that always declines is perfect on the first. Leakage is a count with target zero, not an average.

Write the gate before reading results. Scoring and uncertainty belong to Evaluation & Observability.

Each row is a candidate case for the scenario's evaluation set. Classify it with the case types above.

A prompt is versioned configuration. Record its version in every trace and in answer-cache keys.

Run the full offline set on the candidate version and apply the gate written before the run.

Then run a canary: send a small share of conversations to the new version and compare it with the current version over the same period, on restricted exposure, unknown citation labels, handoff rate, cost per answered question, and latency. State the stop conditions in advance.

Roll back by switching the version back. Cached answers from the new version stop being served, because their cache key carries the version.