RAG & Data Pipelines · 2 to 3 hours
Design a cited support assistant with measurable quality
Design a support assistant that answers from help articles with citations the customer may see, or hands off to a human. Turn its requirements into metrics, plan its evaluation set, keep it fresh through updates and deletion, and check its token, cost, and latency budgets.
3 h
Used in interview rounds:
AI System Design (Medium)1
Step 1: The question
Design the support assistant in the scenario: customers ask questions in the signed-in help panel and get a cited answer or a no-answer with a handoff to a human agent. Give the requirements and success metrics, the ingestion and query paths, how citations are built and access-checked, the evaluation plan, and how updates and deletions reach answers within 15 minutes. Compute the token, cost, and latency budgets, and explain how you would roll out a prompt change.
The scenario: sources and fixture documents
Suppose a fictional invoicing SaaS product adds a support assistant to its signed-in help panel. Customers on the Starter, Pro, and Enterprise plans ask how-to and limit questions. The assistant answers with citations to sources the customer may see, or says it cannot answer and offers a handoff to a human support agent. All names, counts, and documents below are fictional.
| Source | Count | Who may see it | Cited to customers | How it changes |
|---|---|---|---|---|
| Public help articles (HA) | 420 | Every signed-in customer | Yes | About 30 edits and 2 deletions a week |
| Plan articles (PA) | 60 | Customers on the named plan | Only to customers on that plan | About 5 edits a week |
| Internal runbooks (RB) | 150 | Support agents only | Never | Edited by the support team |
| Known-issues page (KI) | 1 page | Every signed-in customer | Yes | Entries opened and resolved daily |
All four live in one content management system (CMS). The CMS records each source's audience and sends an event for every publish, update, deletion, and audience change. Events can arrive late, and occasionally one is lost.
Fixture documents. Assume no other source covers the questions in this scenario's examples and evaluation cases.
| ID | Title | Audience | Current content |
|---|---|---|---|
| HA-12 | Export invoices to CSV | All customers | Revision 4, published yesterday: Invoices > Export > CSV, at most 10,000 rows per export. Revision 3 said 5,000 rows. |
| HA-31 | Plans and limits | All customers | Starter: 1 user. Pro: up to 5 users. Enterprise: unlimited users and SAML single sign-on (SSO). |
| HA-40 | Invite a team member | All customers | Settings > Team > Invite. The invite fails when the plan's user limit is reached. |
| PA-07 | Set up SAML SSO | Enterprise customers | Step-by-step SSO setup |
| RB-05 | Restore a deleted invoice | Support agents | Admin-console steps to restore an invoice a customer deleted |
| HA-55 | Connect the legacy payments plugin | Deleted yesterday | The plugin was retired |
Requirements and budget inputs
All requirements and figures are fictional. Retrieval ranking is out of scope: treat retrieval as one stage with its budget.
Requirements
- About 20,000 customer questions a day, each handled by one model request. A conversation has at most 4 turns, and follow-up questions use the conversation so far.
- The customer's account and plan come from the session and the billing service on every request.
- Every factual statement in an answer cites at least one source the customer may see.
- When the sources do not support an answer, the assistant says so and offers a handoff to a human agent. Planning assumption: 20% of questions end this way.
- Within 15 minutes of an update, deletion, or audience change in the CMS, answers stop using the old content.
- An answer cache for repeated first-turn questions is planned, with a 24-hour expiry.
- Latency, measured at the server: p95 time to first streamed token at most 2.0 s; p95 full answer at most 6.0 s.
- Cost: at most $0.02 per answered question. Cost per answered question is all model and retrieval spend for the day divided by the questions answered with citations that day.
Token caps per request. For budgeting, price every request at these caps.
| Part | Tokens |
|---|---|
| Instructions and output rules | 1,200 |
| Conversation history (cap) | 800 |
| Customer question (cap) | 100 |
| Retrieved chunks | 6 chunks of 350 |
| Answer output (cap) | 400 |
| No-answer output | 100 (the full input is still sent) |
Prices. Input $3.00 per million tokens, output $13.00 per million tokens, and $0.0006 per request for query embedding and search.
Latency budget per stage (p95)
| Stage | p95 |
|---|---|
| Session and plan lookup | 50 ms |
| Query embedding | 100 ms |
| Filtered retrieval | 250 ms |
| Context assembly and source labels | 50 ms |
| Model time to first token | 1,200 ms |
| Streaming the rest of a 400-token answer | 4,000 ms |
| Citation resolution and access recheck | 100 ms |
Constraints
- Plan articles are visible only to customers on that plan; internal runbooks are never shown, used, or cited for customers
- Updates, deletions, and audience changes reach answers, including cached ones, within 15 minutes
- p95 time to first token at most 2.0 s, p95 full answer at most 6.0 s, and at most $0.02 per answered question
- Short bullets and one budget table are enough
Deliverables
- Requirements and success metrics, each with a denominator and a target, including how no-answer behavior is measured
- Ingestion and query paths, showing where access is checked when evidence is selected and when a citation is rendered
- Citation design: how a source label in the model's output becomes a displayed link
- Evaluation plan: case types, the sources each case must and must not cite, named metrics, and one gate written before results
- Update and deletion handling across the index, the answer cache, and old citations
- Token, cost, and latency budgets with the arithmetic, and a prompt-change rollout with rollback
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your design
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: Turns the scenario's requirements into named metrics, each with a denominator and a target, and separates zero-tolerance guardrails from targets that trade off.
What earns a 2: Builds links from a server-side label map to trusted source records, drops unknown labels, and checks eligibility from the session before retrieval and again when a citation is rendered.
What earns a 2: Covers answerable, multi-passage, unanswerable, restricted, and stale cases with required and forbidden sources, measures no-answer behavior from both sides, and states one gate before results.
What earns a 2: Reindexes from CMS events with reconciliation for missed events, and removes updated or deleted content from the index, the answer cache, and rendered citations within 15 minutes.
What earns a 2: Computes request tokens and cost and the day's cost per answered question from the scenario's figures, checks each figure and both latency targets against their limits with the arithmetic for any change it proposes, and rolls a prompt change through a pre-stated offline gate, a canary, and rollback.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Citations and access or Evaluation plan
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can present the design in under an hour with every budget figure computed, and answer an interviewer's three likely follow-ups: a citation to a source the customer cannot see, how no-answers are measured, and how a prompt change rolls out.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
How do you prevent a citation to a document the user cannot see?
What is your no-answer behavior, and how do you measure it?
How do you roll out a prompt change safely?
Why does removing HA-55's chunks from the index not finish the deletion?
Which requests count toward cost per answered question, and which count in its denominator?
7