LLM Foundations · 2 to 3 hours
Choose long context or retrieval
Compare sending whole documents in a large context window with retrieving a few passages for two features of one product, using window fit, cost with prefix caching, latency, access, and a sliced pilot, and say what would change each choice.
Conceptual comparison
3 h
Used in interview rounds:
Technical Screen (Medium)1
Step 1: The question
A teammate says: "The model's window holds 128,000 tokens, so let's put the documents in the prompt and skip retrieval." For Feature A and Feature B, state the goal and constraints, then compare whole-document context with passage retrieval on the same dimensions, including window fit and each option's cost per question and per day. Recommend an approach for each feature, name the evidence that settles each choice, and say what change would reverse it.
Scenario: two assistants for a property manager
Suppose a property-management company adds two assistants to its apps, both on the same hosted model. The company, prices, and figures are fictional.
| Model and prices | Value |
|---|---|
| Context window | 128,000 tokens, input plus output |
| Input, uncached | $2.00 per million tokens |
| Input, cached prefix | $0.30 per million tokens |
| Output | $9.00 per million tokens; answers average 300 tokens, capped at 1,000 |
| Prefix cache | Reuses an identical prompt prefix for 5 minutes, refreshed on each use. In this scenario, a cache write costs the uncached input price. The application may also write the cache without a question; that write bills only the prefix. Prefixes under 2,048 tokens are not cached |
| Retrieval | Returns the top 5 passages of about 600 tokens each, for $0.0005 per question |
Every prompt starts with the same 1,000 tokens of instructions and ends with the question, about 200 tokens. Earlier turns are not resent.
Feature A: residents ask about their own lease.
- One lease per resident, about 40,000 tokens. It changes only at renewal or through a signed addendum, and the resident may read all of it.
- Residents open the lease chat about 1,200 times a day. 1,000 of those sessions include questions, 3 on average (3,000 a day), each within 5 minutes of the chat opening or of the previous question.
- Target: p95 time to first token of at most 3 seconds for every question.
- Option 1 sends the instructions, the whole lease, and then the question. Option 2 retrieves 5 passages from an index of all leases, filtered to the resident's lease.
Feature B: staff ask about internal policies.
- About 6,000 policy documents of about 2,500 tokens each; about 150 change each day, as new revisions or retirements.
- Each staff member may see the company-wide policies plus their own team's: 2,000 to 2,600 documents, depending on the team.
- 800 questions a day, 300 of them from the maintenance team.
- The maintenance lead proposes a team pack: send the team's 40 most-used policies (about 100,000 tokens) with every maintenance question and skip retrieval. Six of the 40 are visible only to maintenance supervisors.
- The alternative retrieves 5 passages from the policies the asker may see.
Scenario: the lease pilot and the policy sample
The team ran both Feature A options on 200 past resident questions, each in one slice, sampled so that each slice's share matches production traffic. Every answer was graded against the lease.
| Slice | Questions | Whole lease correct | Retrieval correct | Retrieval returned every needed passage |
|---|---|---|---|---|
| Single fact, such as the rent due date | 80 | 78 | 77 | 78 |
| Evidence in the middle third of the lease | 40 | 33 | 37 | 39 |
| Synthesis across clauses, such as an addendum that changes a clause | 50 | 42 | 31 | 34 |
| Unanswerable: the lease is silent, and a correct answer says so | 30 | 26 | 22 | Not applicable |
| Request kind (pilot traffic) | p95 time to first token |
|---|---|
| Whole lease, cache miss (first question in a session) | 4.6 s |
| Whole lease, cache hit (follow-up within 5 minutes) | 1.3 s |
| Retrieval, including the search | 1.6 s |
For Feature B, the team checked 50 past maintenance questions:
| Check | Result |
|---|---|
| The team pack holds every policy the answer needs | 35 of 50 |
| Retrieval over the asker's policies returned every needed passage | 44 of 50 |
Constraints
- Use only the figures in the scenario and the pilot; all of them are fictional
- Treat retrieval as a component with the measured results shown; ranking, chunking, and citation design are out of scope
- Use the scenario's pricing and simplifications: writing the cache costs the uncached input price, and earlier turns are not resent
- Keep the spoken answer to about five minutes, with one comparison table per feature
Deliverables
- Each feature's goal and constraints, then one comparison table per feature with both options on the same dimensions
- Window fit and cost per question and per day for each option, with the arithmetic
- How each option keeps text a user may not see out of the prompt
- The pilot and sample evidence behind each choice, and the checks still needed
- A decision per feature and the changes that would reverse it
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Your answer
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
What earns a 2: States each feature's goal and constraints (what one user may see, change rate, access rules, latency target, volume) before comparing, then rates both options on the same dimensions in one table per feature.
What earns a 2: Checks fit with the output reserve, prices a question with and without a reused prefix under a stated reuse assumption, and scales to a day with the arithmetic shown.
What earns a 2: Explains, for both options and both features, how text a user may not see stays out of the prompt, using the identity from the session and a filter or assembly step rather than an instruction to the model.
What earns a 2: Reads the pilot slice by slice against each slice's size, ties each gap to a mechanism, and names the checks or larger samples that would confirm it.
What earns a 2: Gives a decision per feature with the trade-off it accepts, and names concrete changes in size, reuse, price, freshness, or access that would reverse it or leave it standing.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10, with no 0 in Goal, constraints, and a shared comparison or Access boundary or Decision and what reverses it
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you can compare both options on the same dimensions for each feature with the arithmetic shown, keep restricted text out of every prompt, and say which evidence settles each choice and which change would reverse it.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
What is the difference between a context window's capacity and a model's effective use of it?
When does prefix caching lower the cost of sending a whole document?
Can an instruction to the model keep a restricted document private if the document is in the prompt?
What does a retrieval-based answer of "the document does not say" tell you?
What hybrid designs sit between whole-document context and passage retrieval?
7