Skip to content

Knowledge · Pattern

Hybrid search with rank fusion

Run lexical (BM25) and vector retrieval on the same query and merge the two ranked lists with reciprocal rank fusion. Covers what the merge gains and costs, where it fails, and how to test it on your own corpus and queries.

7 sections2 tables1 diagram

Fast understanding

Definition

Hybrid search sends each query to two retrievers over the same corpus, a lexical one and a vector one, and merges their results into a single ranked list. The lexical retriever matches the words in the query, most often scoring with BM25. The vector retriever embeds the query and returns the chunks whose embeddings are nearest. The two have different blind spots, and the merge step is what makes the pattern work, because their scores are not on the same scale.

BM25 scores a document by summing a contribution from each query term. A term counts for more when it is rare in the corpus, repeated occurrences help with diminishing returns, and documents longer than average are scored down (Elastic, BM25 explainer). It rewards the literal presence of the query's words. An embedding model rewards closeness in meaning, so a paraphrase with no shared words can still land near the query. Anthropic describes the gap in retrieval terms: embeddings "excel at capturing semantic relationships" but can miss exact matches, and for a query containing an error code the exact string may be what identifies the right chunk (Anthropic, 2024).

Explanation

When to use

Use it when queries mix natural language with strings that must match literally: error codes, policy numbers, product names, function names, people's names. Use it too when you cannot predict which kind of query will dominate.

The published evidence is easy to over-read, so keep the claims apart. BEIR is a benchmark of 18 datasets for zero-shot retrieval. It reports that BM25 was a robust baseline, that re-ranking and late-interaction models achieved the best average zero-shot performance at high computational cost, and that dense and sparse retrievers were cheaper to run but generally trailed those approaches (Thakur and others, 2021). That is a reason to keep a lexical retriever in the comparison and to consider a reranker. It does not test whether fusing lexical and vector lists beats either alone.

Anthropic's contextual-retrieval evaluation tested one particular configuration. It reports the top-20 retrieval failure rate, the share of relevant chunks missing from the top 20 results (1 minus recall@20), using Gemini Text 004 embeddings, with its charts showing averages across its test domains: 5.7 percent for the embedding baseline, 3.7 percent with contextual embeddings, and 2.9 percent with contextual embeddings plus Contextual BM25 (Anthropic, 2024). The 2.9 percent belongs to that combined setup, in which a model also wrote added context for every chunk, so it cannot be credited to BM25 fusion alone. Anthropic's results support its contextualized retrieval configuration, but they do not isolate the benefit of a conventional BM25-plus-vector merge such as RRF, and the source advises running your own evaluations. Neither source shows that a hybrid configuration wins on every corpus. Evaluate the merge separately on your own documents and labelled queries.

The measurement that justifies the extra machinery is simple: on a labelled question set, compare the recall of each retriever alone with the recall of the fused list. If one retriever already finds everything the other finds, hybrid search adds cost and no benefit.

Do not reach for it when something simpler fits. If the whole corpus fits in the prompt, Grounding answers in your own data explains why retrieval can be skipped. If the question needs an exact stored fact, such as an order status, look it up directly instead of searching text. If your queries are all natural-language questions over prose and the lexical list never contributes a relevant result, a second index is overhead.

How it works

Each retriever returns its top N candidates, called the rank window. The fusion step merges the two lists, duplicates are removed, and the first few results of the merged list go on to a reranker or into the prompt. (In this entry, k always means the fusion constant introduced below, never the number of results kept.)

Both retrievers apply the same filters and return a rank window each, and fusion merges the two lists by rank

The diagram places the two decisions that the failure modes below turn on: the filters belong on both paths, and the rank window N decides what can reach the merge at all.

The difficulty is merging. A BM25 score has no upper bound, while a cosine similarity sits in a narrow range, so adding the raw scores lets whichever scale is larger decide the order. Azure AI Search's documentation lists BM25 as having no upper limit and cosine-based vector scores as bounded (Microsoft, 2026). There are two common ways around it. One normalises each score list and then takes a weighted sum, which needs a normalisation rule and a weight to tune, and the result depends on which candidates happened to be retrieved. The other ignores scores and uses only ranks.

Reciprocal rank fusion, or RRF, is the rank-only option, introduced by Cormack, Clarke and Buettcher at SIGIR 2009, who reported that it generally beat the individual systems and the Condorcet fusion method in their experiments (Cormack and others, 2009). Each document receives, from every list that contains it, a contribution of 1 / (k + rank), counting rank from 1, and the contributions are summed. Elasticsearch documents the same formula with a default k of 60, and notes that a higher k gives lower-ranked documents more influence (Elastic documentation). A document that appears in both lists collects two contributions, so agreement between retrievers is rewarded without any comparison of their scores.

Some engines let you weight one retriever more than another when fusing; Azure AI Search documents a weighting multiplier for vector queries, for example. Weights are one more thing to validate against labelled queries.

Trade-offs

ChoiceWhat it buysWhat it costs
Hybrid instead of vector onlyExact strings and rare terms can be retrieved reliablyA second index to build, keep in sync and query
Hybrid instead of lexical onlyParaphrased questions find matching chunksAn embedding model, an embedding call on every query and a vector index
RRF fusionNo score normalisation and little to tuneIgnores score magnitude, so a decisive top result and a marginal one count the same
Weighted score fusionCan express that one retriever is more trustedNeeds normalisation, weights and re-tuning when the corpus or embedding model changes
Larger rank windowMore candidates to agree or disagree overElastic notes that a larger window improves relevance at the cost of performance
Reranker after fusionA more expensive model rescoring a short list for precisionAdded latency and cost on every request (see the grounding lesson)

The two retrievers can run in parallel, so added latency is roughly the slower of the two plus the merge, but each query now does the work of both.

Fusion also does not repair a bad split. If a rule and its exception sit in different chunks, both retrievers see the same fragmentation, which is the problem Chunking long inputs for models addresses. The chunking entry's Contextual BM25 is a related technique: a lexical index built over chunks that have been given added context. Anthropic evaluated it only together with contextual embeddings, so treat it as a variant to test and not as evidence for a plain BM25-plus-vector merge.

Deeper reference

Example

A constructed example with invented ranks. The help assistant receives "ERR_SYNC_4021 after badge reset". Five documents come back from the lexical retriever and four from the vector retriever (rank window of 5 and 4 for illustration):

DocumentLexical rankVector rankRRF score, k = 60Fused rank
A: badge reset, what to expect221/62 + 1/62 = 0.032261
E: sync error code reference141/61 + 1/64 = 0.032022
B: device sync troubleshooting331/63 + 1/63 = 0.031753
F: cannot sign in after a hardware changenone11/61 = 0.016394
C: unrelated runbook4none1/64 = 0.015635
D: unrelated runbook5none1/65 = 0.015386

Documents A, E and B appear in both lists, so they take the top three places. The vector retriever's first choice, F, shows up fourth, and the lexical retriever's first choice, E, shows up second because it is weak in the vector list.

Now suppose the raw scores were 18.4, 15.1, 9.7, 8.2 and 6.0 for the lexical list and 0.83, 0.81, 0.79 and 0.74 for the vector list (all invented). Adding raw scores gives E 19.14, A 15.91, B 10.49, C 8.2, D 6.0 and F 0.83. F, the vector retriever's top result, falls to last place because its score range is tiny next to BM25's, and the fused ranking is essentially the lexical one. RRF avoids that, at the price of discarding how big the gaps were.

The constant matters at the top. With k = 1 the scores become E 0.700, A 0.667, B 0.500 and F 0.500: the exact-match document E, rank 1 in one list, overtakes A, which is rank 2 in both. A small k favours the top of each list, and a large k favours agreement across lists. Which is right is an empirical question for your queries.

Failure modes

  • A filter applied to one retriever only. If the permission or version filter reaches the vector index but not the lexical one, restricted or superseded text enters through the other path. Apply the same filters to both, before fusion, as the grounding lesson requires of retrieval in general.
  • The same passage under two identifiers. RRF adds contributions only for the same document identifier. If the two indexes label chunks differently, the same passage is counted twice as two candidates. Use one chunk identifier across both indexes.
  • The identifier is not matchable. A lexical index applies its own tokenisation, and a string like ERR_SYNC_4021 may be split in ways that stop an exact query from matching. Test that the real identifiers in your data can be retrieved by themselves.
  • Reading the fused score as relevance. An RRF score reflects ranks only. Azure's documentation notes that its upper bound depends on the number of lists being merged. A query with no answer in the corpus still produces ranks and therefore a fused list, so the score cannot say that nothing relevant was found. That needs a separate check, as the grounding lesson describes for missing data.
  • A rank window too small. If the relevant chunk is rank 40 in one list and the window is 20, it never reaches the merge.

Interview considerations

Say what each retriever is for: lexical for exact strings and rare terms, vector for paraphrase. Name the fusion method and why. Rank-based fusion is the lower-maintenance start because scores from different retrievers cannot be added directly. Then say how you would verify the extra index is worth it: a labelled question set that includes exact-string questions, recall for each retriever alone and for the fused list, and the same filters on both paths.

Tip

Sample wording for a candidate: "Policy questions are mostly natural language, but people also paste error codes and policy numbers. So I'd run BM25 and vector search in parallel, apply the same permission filters to both, and fuse by rank with reciprocal rank fusion so I don't have to normalise two different score scales. I'd keep it only if recall on a labelled set, including exact-code questions, beats either retriever alone."

  • Decide when a model needs retrieved context, how that context is chosen and fitted into a prompt, and how stale or missing data changes the answer.

  • How to split long documents or event streams into pieces a model can use, the main ways to choose boundaries, and what each choice costs in recall, context use and complexity.