RAG & Data Pipelines · 2 to 3 hours
Diagnose a missing document
Trace a missing policy exception through ingestion, eligibility, and context packing without weakening access rules.
Hands-on debugging
3 h
1
Step 1: The question
Diagnose the initial C2 failure and the later 180-token packing failure. Produce a stage-by-stage explanation, minimal correction, and regression checklist. Explain why removing access filters and increasing top-k are insufficient fixes.
Diagnose a missing policy
The fictional current source is returns revision 3. It contains the standard return window and a custom-product exception.
| Stage | Observed evidence | Interpretation |
|---|---|---|
| Source store | Revision 3 exists and contains both sections | Source acquisition succeeded |
| Parser | Both sections appear in extracted text | Parser preserved the needed evidence |
| Chunk builder | Section 2 created as C2 | Relevant chunk exists |
| Index lookup | C2 exists with department procurement | Index contains the chunk |
| Query filter | User is in support and only support chunks are eligible | C2 is excluded by metadata |
| Context | Only section 1 appears | Generator lacks the exception |
The first correction is to verify the intended access policy. If procurement is the correct permission, the support user must not receive the chunk. If the department metadata was mistakenly assigned and the source is authorized for support, correct the metadata through the normal publication process. Removing the filter would bypass the access rule.
Context budgets can cause a second failure
Suppose C2 is confirmed to belong to support and republished with correct metadata. Both C1 and C2 now pass eligibility. Their declared teaching token counts are 120 and 100. A context evidence budget of 180 tokens cannot include both whole passages.
A greedy packer that includes C1 first leaves 60 tokens and drops C2. Increasing retrieval top-k does not fix this context-selection failure. Options include selecting the exception passage for the specific question, including a compact verified representation of both rules, or increasing the evidence budget if the full request still fits.
When a chunk is omitted, record its ID and omission reason. Avoid silently truncating a sentence that carries a critical exception. A packing algorithm that skips oversized chunks can use later smaller chunks, but it still needs evaluation for whether all required evidence survives.
Constraints
- Access filters must remain enforced
- Use only the evidence in the trace and the fixtures
Deliverables
- Stage-by-stage diagnosis
- Minimal correction
- Regression checklist
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Code exercise
Chunk eligibility
Implement exclusionReason so each fixture returns its first exclusion reason, or null when the chunk is eligible. Check tenant, deletion, publication state, revision, and department, in that order.
Expected
C1 is eligible. C2 is excluded by department, C3 by tenant, C4 by revision, C5 by publication state, and C6 by deletion.
Limitations
This helper evaluates one in-memory snapshot and returns the first exclusion reason. It does not implement concurrency, authentication, retrieval ranking, or protection against permission changes after the snapshot. A production revision lookup must also be tenant-scoped, as the supplied map is for the already-selected tenant.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you reproduce the eligibility outcomes, explain both C2 failures, and propose checks that locate a defect without weakening authorization.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
How do you distinguish retrieval and generation failures?
Why retain revision IDs?
Why might more chunks make an answer worse?
Can a valid citation be unsupported?
What makes ingestion idempotent?
7