RAG & Data Pipelines · 3 to 4 hours
Debug a partially failed re-embedding backfill
Find why a partly failed re-embedding backfill left a help-center index missing, duplicating, and mixing vectors, then plan a repair and a cutover gate you could defend in a release review.
Hands-on debugging
4 h
Used in interview rounds:
Debugging & Evaluation (Hard)Overview
Debug a re-embedding backfill that failed partway and left a live index serving two embedding configurations. You will attribute each symptom to its cause from logs, code, and traces, reconcile the index against its manifest, and plan a repair and cutover gate under a budget and a deadline. Ingestion lifecycle design and retrieval tuning have their own questions.
Study time: 3 to 4 hours
Interview format: Debugging exercise with fixtures and a cutover decision
By the end you can
- Attribute each of a backfill's symptoms to its cause using job logs, code, and query traces
- Reconcile a derived index against its manifest by chunk identity instead of row counts or job progress
- Predict how queries behave when an index mixes embedding configurations, duplicates, and filtered-out records
- Plan a repair that contains first, fixes each cause at its source, and costs targeted re-embedding against a full rebuild
- Declare a cutover gate and rollback triggers before reading evaluation results
- Write fault-injection tests that would have caught each cause
Prerequisites
Helpful background: Familiarity with batch jobs, structured logs, and metadata filters in a vector store helps.
This topic covers ingestion, grounding, citations, and the document lifecycle. Embeddings & Retrieval covers how candidates are found and ranked.
In this guide
- Why a backfill fails quietly
- Checkpoints decide what a crash costs
- Deterministic record identity
- Versioned indexes and a paired switch
- Reconcile against the manifest
- Incident setup
- Query traces
- Diagnose the backfill
- Reconcile the live index
- Repair constraints
- Targeted repair or full rebuild
- Write the cutover gate first
- Test the failure windows
Hard questions are part of Premium: the study guide, reference answer, rubric, and exercises unlock with Premium.
Related on CodeSnatch
Lessons, problems, and articles that build the skills this question uses.
System Design
Walks through reindexing into a versioned index with dual writes and an atomic alias swap, the shape of a safe migration.
System Design
Covers idempotent reruns, backfills that swap in atomically at the end, and data tests that catch silent bad output.
Practice problem
Practice debugging with scripted fakes and a fake clock, the harness that fault-injection tests rely on.
System Design
Covers delivery guarantees, consumer offsets, retries, and idempotent consumers in queue-based systems.
AI Engineering
Practice turning paired evaluation results and their uncertainty into a ship decision written before the numbers.
Practice the interview question
Answer it in your own words before you look at the reference answer.