Embeddings & Retrieval · 2 to 3 hours
Compare lexical and dense retrieval on fixtures
Rank four help articles by shared words and by cosine similarity, score both rankings against labels, and explain a dense miss and an accidental lexical success.
3 h
1
Step 1: The question
Run or manually trace the three fixtures. Explain the E104 failure, calculate dense precision@2 and recall@2, and identify why the lexical result for “forgot secret” is misleading. Propose an evaluation that can test a hybrid improvement.
A reproducible ranking exercise
Four help articles, each one document with a stable ID. Every document is eligible.
| ID | Text | Vector |
|---|---|---|
| D1 | Reset your password using a recovery link | [1, 0] |
| D2 | Resolve E104 when a login token expires | [0.8, 0.6] |
| D3 | Download a billing invoice | [0, 1] |
| D4 | Change your profile display name | [-1, 0] |
Three queries, each with a supplied query vector and one relevant document.
| Query | Query vector | Relevant |
|---|---|---|
| forgot secret | [1, 0] | D1 |
| E104 | [1, 0] | D2 |
| billing invoice | [0, 1] | D3 |
Rules:
- Lexical score: lowercase both texts, split them into runs of letters and digits, and count the unique query words that also appear in the document. This measures shared words only; it is not BM25.
- Dense score: cosine similarity between the query vector and the document vector.
- Ranking: higher score first, equal scores broken by ascending ID, at most two results. Documents that score zero are still returned, so every query gets two results.
Metric conventions: precision@k is the number of relevant documents in the first k ranks divided by k, even when fewer than k results are returned. Recall@k is the number of relevant documents in the first k ranks divided by the number of relevant documents in the eligible corpus.
Constraints
- Return at most two results per query, and break equal scores by ascending ID
- The lexical score counts unique shared words; the dense score is cosine similarity
- Every document is eligible, and relevance labels are document-level
Deliverables
- Lexical and dense top-two rankings for each fixture
- Explanation of the E104 dense miss, with dense precision@2 and recall@2
- Why the lexical result for “forgot secret” is misleading
- An evaluation plan that can show whether a hybrid method helps
2
Step 2: Hints
Optional. Each hint gives a little more away.
3
Step 3: Work through it
Code exercise
Lexical and dense ranking
Implement cosine, rankLexical, rankDense, precisionAtK, and recallAtK (snake_case names in Python) for the fixture above. Rankings return at most k IDs, highest score first, with equal scores broken by ascending ID. Cosine throws for empty, mismatched, non-finite, or zero vectors, and recall throws when there are no relevant documents.
Limitations
The vectors are hand-made fixtures, not model output, and the cosine helper is meant for small finite vectors of ordinary magnitude; production numerical handling should follow the chosen library. The lexical scorer counts shared unique words rather than BM25 and returns zero-score items to keep a fixed ranking. There is no approximate index, filtering, or fusion.
Kept in this browser. Saving your score below keeps a copy with it.
0 / 5000
4
Step 4: Reference answer
5
Step 5: Score yourself
Score each dimension 0, 1, or 2, then save the score to your history. Readiness thresholds are editorial estimates.
Grades your written answer against this rubric with your own AI provider key from Settings. It is a second opinion: the score you save is still yours.
0 of 10 · Scored 0 of 5 dimensions
Ready at 8 of 10
Kept in this browser until you sign up, then it moves to your account.
Complete this level when you reproduce the expected rankings, explain accidental lexical success, and locate retrieval failures before proposing changes.
Your saved scores
Practice saves and mock interview answers on this question, newest first.
No saved scores yet. Score every dimension, then save.
6
Step 6: Follow-up questions
An interviewer may ask these next.
Why compare approximate search to exact search?
Why can recall improve while precision worsens?
Can fusion recover a document absent from both lists?
Why inspect ties?
How do filters change evaluation?
7