Embedding models turn over every few months. A new one posts better MTEB numbers, someone on the team reads the announcement, and the question lands in standup: should we re-index?
It is a fair question and an expensive one to answer badly. Re-embedding is the one change in a RAG system that touches every stored vector, cannot be rolled back with a deploy, and degrades quietly when it goes wrong. Retrieval does not throw an error when recall drops four points. It just returns slightly worse passages, and the answers get slightly worse, and nobody can point at the commit.
This is the procedure we use on client systems. It has two halves: decide with your own data, then migrate without a window where the index and the query encoder disagree.
Leaderboard rank is not your recall
Public benchmarks average over corpora that are not yours. They tell you a model is broadly competent. They do not tell you how it handles your domain vocabulary, your chunk lengths, your query phrasing, or the language mix in your documents. We have measured new-and-higher-ranked models coming in flat on a client corpus, and we have measured a 9-point recall-at-10 gain from a model two rows lower on the public table. Neither result was predictable from the leaderboard.
Before any migration work, you need a retrieval eval on your own data. If you already built one — query, expected passage, rank scoring — you are ready. If not, it is a small job: take 80–150 real queries from logs, and for each one record the chunk that should have been retrieved. You can label that fast by reading the answer the system gave and the context it used. Half from known failures, half from median traffic.
That set gives you mechanical metrics, no judge model involved:
- recall@k at your production k, and at 2k to see whether the evidence is nearby or absent,
- MRR or mean rank of the target passage, which catches improvements that recall@k rounds away,
- recall on the failure slice specifically — if the new model only helps queries that already worked, it will not move your product.
Run the candidate offline on a corpus sample
You do not need to embed 12 million chunks to get a signal. Embed a stratified sample — enough documents to cover every content type and every high-traffic tenant, typically 5–10% — plus every chunk referenced as a target in your eval set. Build a throwaway index, run the query set, and compare against the same queries on production.
Three cost details that decide the answer as often as quality does:
- Dimensionality. Going from 768 to 3072 dimensions quadruples vector storage and raises query latency and index memory. Some models support truncation (Matryoshka-style); measure recall at the truncated size you would actually deploy, not the full one.
- Full re-embedding cost. Chunk count times average tokens times provider rate, plus the ingest wall-clock at your rate limit. Get the number before the meeting, not after.
- Self-hosted vs API. A self-hosted encoder removes per-token cost and adds a GPU service you now have to keep up in the request path. That is a real operational line item.
Set the bar in advance. Ours is usually: a meaningful recall gain on the failure slice, no regression on the median slice, and end-to-end answer quality no worse on the generation eval. A model that trades 3 points of recall for half the storage can also clear a bar — just say which bar you are aiming at before you see the numbers.
Never let the query encoder and the index disagree
The actual migration risk is mechanical. Vectors from two different models are not comparable. A single request that embeds the query with model B and searches an index built with model A returns near-noise, and it returns it with a normal 200 and plausible-looking text. Any migration plan that has both models live in the same namespace has a window where that happens.
So pin the encoder to the index, explicitly. Stamp every index — collection, namespace, or table — with the embedding model name, version, dimension, and the normalization and prompt-prefix convention it was built with. Instruction-tuned encoders expect a prefix like query: on queries and passage: on documents; forgetting it on one side is a silent recall killer. Read that stamp at query time and refuse to search when the running encoder does not match. Loud failure beats quiet noise.
Dual-write, backfill, shadow, cut over
With the encoder pinned, the migration is boring in the good way:
1. Dual-write. Ingest writes new and updated chunks to both the old and the new index. From this moment, fresh content exists in both places, and the new index will not be stale when you need it.
2. Backfill. Batch-embed the historical corpus into the new index, oldest-first or lowest-traffic-first. Checkpoint by document id so a rate-limit failure resumes instead of restarting. Expect to run at a fraction of the provider's ceiling and to be throttled; plan the wall-clock accordingly.
3. Reconcile. Before trusting the new index, verify it: chunk counts per document match, no document has zero chunks, spot-check that a sample of chunk texts and metadata round-trip identically. Backfills lose documents. Find yours here rather than in production.
4. Shadow reads. Route a copy of live queries to the new index without using the results. Log, for each query, the two result sets and their overlap. You are looking at rank-level diffs, not just aggregate scores: which queries changed top-1, which lost their previously-cited passage. Latency and error rate come free from the same traffic.
5. Gated cutover. Flip a percentage of traffic — per tenant, not randomly, so sessions stay coherent — and watch the product signals that actually move: answer-level eval pass rate on the live slice, citation rate, user retries, thumbs-down, escalations to support. Hold at 10% long enough to see a full traffic cycle, then ramp.
6. Keep the old index hot. Retain it, queryable, for at least one full eval cycle after 100%. Rollback should be a flag, not a re-index. Delete it when the storage bill annoys you more than the risk does.
Re-tune the things that were fit to the old model
A new encoder changes the numbers everything downstream was tuned on. Check these after cutover, not before:
- Similarity thresholds. Any hard cutoff — "drop matches below 0.75" — was calibrated to the old score distribution and is now arbitrary. Re-derive it from the new one.
- Hybrid search weights. The dense/sparse blend or RRF weighting shifts when dense recall shifts. Re-sweep it on the eval set; it is a cheap sweep.
- Reranker interaction. A stronger first stage sometimes means fewer candidates are needed, which buys back latency. Sometimes it means the reranker was covering for the old encoder and the gain is smaller than the retrieval numbers suggested. Measure the stack end to end, not the stage in isolation.
- Chunk sizing. Models differ in effective context per chunk. If the candidate handles longer passages well, a re-chunk may be worth more than the encoder swap itself — but test one variable at a time, or you will not know which change paid.
When the answer is no
Plenty of times the honest recommendation is to skip the migration. If your eval shows recall@10 at 0.94 and the failure slice is dominated by wrong-synthesis rather than retrieval-miss, a better encoder has almost nothing to fix. Re-indexing is then a week of engineering, a storage increase, and a real regression risk in exchange for a rounding error.
The work that pays in that case is usually elsewhere: parsing and metadata quality, query rewriting, hybrid search, reranking, or the generation prompt. "Which model is best" is the easy question to ask and rarely the one holding the system back.
The discipline is the same either way. Measure on your corpus, gate on your own bar, pin the encoder to the index, and keep a rollback that takes seconds. Done that way, an embedding upgrade is a scheduled maintenance task instead of an outage with good intentions.