Most RAG systems that "feel dumb" are not reasoning badly. They are answering from the wrong passages. If the evidence never enters the context window, the model's only options are to refuse or to improvise, and it usually improvises.
We have written before that chunking is a retrieval decision. This is the next layer: once your boundaries are sane, recall comes from how you search and how you re-order what you find. Two mechanisms do most of the work — hybrid search and reranking — and both are easy to add badly.
The failure modes of a single retriever
A vector-only retriever is trained to match meaning. That is exactly what you want for "how do I cancel an order after it ships" against a page titled "Post-dispatch returns." It is exactly what you do not want for:
- Identifiers:
ERR_4471,SKU-99183,CVE-2025-1234. Rare tokens get averaged away in a dense embedding. The retriever returns documents that are about errors, not the document about that error. - Domain jargon and product names the embedding model never saw. If your company renamed a feature last year, the embedding has no idea the new name and the old name are the same thing.
- Negation and near-duplicates: "eligible" and "not eligible" sit close together in vector space and far apart in meaning.
A keyword-only retriever — BM25 — has the mirror problem. It nails the error code and misses every paraphrase. Users do not phrase queries in corpus vocabulary, so lexical-only systems fail quietly on the most natural questions.
In a labeled retrieval eval, these show up as distinct buckets. It is worth tagging them: lexical-miss, paraphrase-miss, wrong-doc-right-topic. The distribution tells you which mechanism to add first.
Hybrid search: run both, then fuse
Hybrid retrieval runs the lexical and dense queries in parallel and merges the two ranked lists. The merge is where people go wrong.
Do not add raw scores. BM25 scores are unbounded and corpus-dependent; cosine similarities live in a narrow band near the top. Summing them means one signal silently dominates, and the balance drifts as your corpus grows.
Two defensible options:
- Reciprocal rank fusion (RRF). Score each document by the sum of
1 / (k + rank)across lists, withkaround 60. It uses only ranks, so it needs no calibration and no tuning per corpus. It is the right default and it is hard to break. - Normalized weighted fusion. Min-max normalize each score list over the returned window, then take a weighted sum. More expressive — you can bias toward lexical for a code-heavy corpus — but the weight is now a parameter you own, and you must re-check it when the corpus or embedding model changes.
Start with RRF. Move to weighted fusion only if your evals show a consistent, directional gain that justifies maintaining the weight.
One practical note: retrieve deeper from each leg than you plan to keep. Pulling top 25 from each and fusing down to 8 gives fusion something to work with. Pulling top 5 from each mostly gives you the union of two shallow lists.
Reranking: precision, bought with latency
Fusion improves the candidate pool. It does not read the query and the passage together. A cross-encoder reranker does: it scores each (query, passage) pair jointly, which catches the relevance nuances a bi-encoder's independent embeddings cannot.
The pattern is a funnel:
- retrieve ~50 candidates with hybrid search (cheap, high recall),
- rerank them with a cross-encoder (expensive per pair, but only 50 pairs),
- keep the top 5–8 for generation.
What this buys you is precision at small k. That matters more than it sounds. Models attend unevenly to long contexts, and an irrelevant-but-plausible passage in the prompt is an active hazard — it gets cited. Cutting from 20 mediocre chunks to 6 good ones often improves faithfulness and cuts token spend at the same time.
What it costs is latency: typically 100–400ms for a hosted reranker over 50 candidates, more if you self-host a large model without batching. Budget for it explicitly. If you stream, the reranker sits before first token, so it is fully visible to the user.
When a reranker is not worth it: corpora small enough that recall@5 is already near 1.0, and hard-latency paths (autocomplete, inline suggestions). Late-interaction models like ColBERT-style retrievers sit in between — more storage, less per-query compute — and are worth evaluating if reranker latency is your binding constraint.
Fix the query before you blame the index
Two cheap query-side moves, both measurable:
Rewriting for context. In a chat app, turn "what about the second one?" into a standalone query using the conversation history. Follow-up turns are where retrieval quality falls off a cliff, and the fix is a small model call on the query, not a bigger index.
Multi-query expansion. Generate two or three phrasings of the query, retrieve for each, fuse the results. This helps most on sparse corpora and vocabulary mismatch. It also multiplies your retrieval calls and adds a model call to the critical path — so treat it as a measured trade, not a default.
Both belong behind an eval. We have seen rewriting add four points of recall on multi-turn traffic and, on another engagement, add latency for nothing because the app was single-turn.
Measure each stage separately
The reason retrieval work stalls is that teams evaluate only the final answer, so every change produces a vague "seems better." Instrument the stages:
| Stage | Metric | Question it answers |
|---|---|---|
| Candidate generation | recall@50 | Is the answering passage in the pool at all? |
| Fusion | recall@8, MRR | Does the merge surface it? |
| Reranking | precision@5, NDCG@5 | Is the top of the list actually relevant? |
| Generation | faithfulness, required-fact presence | Given good context, is the answer right? |
The rule follows from the table: if recall@50 is your problem, a reranker cannot help you. Reranking only re-orders what retrieval already found. Teams routinely buy a reranker to fix a candidate-generation problem and get a small, confusing delta for their latency.
Building the labeled set is the same exercise as bootstrapping an eval suite: 50–100 real queries, each annotated with the passage that should answer it. Retrieval metrics need no judge model and run in seconds, which means you can sweep fusion strategies, candidate depth, and reranker choice in an afternoon and keep the receipts.
A working order of operations
- Label 50–100 real queries with their answering passages. Tag the misses by type.
- Baseline recall@k for your current retriever. Write the number down.
- Add BM25 alongside dense and fuse with RRF. Re-measure. This is usually the largest single jump.
- Deepen the candidate pool until recall@50 plateaus.
- Add a cross-encoder reranker over that pool. Measure precision@5 and the latency it adds.
- Only then consider query rewriting or multi-query, and only if your traffic looks like it needs them.
- Re-run the generation-side evals to confirm the better context produced better answers — occasionally it does not, and that points at the prompt.
None of this is exotic. It is roughly a week of work on most stacks, it is cheap to keep in CI, and it converts retrieval from the part of the system nobody can reason about into a set of numbers you can defend in a design review.