Compare lexical and dense retrieval on fixtures

Define the searchable unit, eligible corpus, query preprocessing, ranking direction, tie behavior, and result count. Decide whether evaluation judgments apply to documents or passages. Multiple passages from one document should not accidentally count as several relevant documents when evaluation is document-level.

For this exercise, each item is one document with a stable ID. All items are eligible. Return at most two results, break equal scores by ID, and evaluate against explicit document-level labels. Production access control is a separate mandatory eligibility check.

Start with a lexical baseline and an exact dense search baseline. Exact dense search gives a reference for later approximate-index tuning. Record model configuration, metric, normalization, and document revision along with results.

A simple term-overlap scorer can expose behavior on tiny fixtures. Use a production full-text ranking method such as BM25 when evaluating a real search stack. The exercise scorer below deliberately measures shared unique words only.

Cosine similarity is dot(query, document) / (norm(query) × norm(document)). Vectors must have matching dimensions, finite values, and nonzero length. A zero vector has no defined cosine similarity.

The fixture uses two-dimensional vectors to make calculations inspectable. Real embedding generation is intentionally replaced by supplied numbers so the exercise runs offline. These numbers illustrate ranking mechanics rather than model quality.

Four help articles, each one document with a stable ID. Every document is eligible.

Three queries, each with a supplied query vector and one relevant document.

Rules:

• Lexical score: lowercase both texts, split them into runs of letters and digits, and count the unique query words that also appear in the document. This measures shared words only; it is not BM25.
• Dense score: cosine similarity between the query vector and the document vector.
• Ranking: higher score first, equal scores broken by ascending ID, at most two results. Documents that score zero are still returned, so every query gets two results.

Precision@k is the number of relevant items in the first k ranks divided by k under this exercise's fixed-k convention. Recall@k is the number of relevant items retrieved divided by the total number of relevant items in the eligible corpus. If fewer than k results are returned, document whether precision uses k or the actual returned count.

The reciprocal rank of a query is 1 divided by the rank of its first relevant result. Mean reciprocal rank (MRR) averages reciprocal ranks across queries, with zero assigned when no relevant item appears within the evaluated result window. It emphasizes the first relevant hit. Recall is more useful when several pieces of evidence must be recovered.

When a query has no relevant items, recall has a zero denominator. Define a separate no-answer metric or exclusion rule instead of inventing a recall value. Report how many queries fall into that category.

Increasing top-k can help expose a relevant candidate at a lower rank, but it cannot recover an item that was never indexed or was excluded by policy. More results can also increase downstream cost and distract a generator.

Lexical and dense scores may have unrelated scales and distributions. Adding a BM25-like score of 12 to a cosine score of 0.8 has no automatic interpretation. Score normalization requires evaluation and can be unstable across queries.

Rank-based fusion is an alternative. Reciprocal rank fusion (RRF) adds contributions of 1 / (c + rank) for each list in which a document appears, using one-based ranks and a positive rank constant c. An absent document contributes zero. Candidate list size and c affect results.

When two lists hold the same two documents in opposite orders, such as lexical [A, B] and dense [B, A], equal-weight RRF gives both documents the same score, 1 / (c + 1) + 1 / (c + 2). A deterministic tie rule ensures reproducibility, but it does not resolve which document is relevant. Query-aware exact-code handling or a reranker may be needed.

Use the same eligible set when comparing methods. Otherwise apparent quality differences may actually be permission or metadata differences. Apply authorization before any content reaches an external reranker or generator, and verify authorization again when fetching mutable records if required by the consistency model.

Check query and document model versions, dimensions, and normalization. Equal dimensions can still hide incompatible embeddings. If an engine reports distance, inspect its direction before sorting. A reversed sort can produce convincing-looking yet irrelevant results.