Retrieval work gets the attention: chunk boundaries, hybrid search, rerankers, embedding choice. The pipeline that puts documents into the index usually gets a script. Someone wrote load_all_docs.py during the prototype, it ran once, and it has been run manually four times since, each time by a different person, each time with a slightly different flag.
That script is now a production dependency. Every retrieval metric you trust is measured against whatever it last wrote. When a RAG system starts answering with policies that were replaced in March, no amount of reranking helps — the wrong text is the best matching text, because the right text is not in the index.
This is a data-engineering problem wearing an LLM hat. It has boring, well-understood answers.
The three failure modes, named
Staleness. The source changed; the index did not. Users get confidently wrong answers sourced from a real document, which is the hardest failure to catch by reading outputs — the answer is well-formed, cited, and obsolete.
Orphans. The source document was deleted or unpublished; its chunks are still in the vector store. Deletes are the failure teams forget entirely, and they are the one with legal weight. "Retract that memo" and "honor this deletion request" both fail silently if your pipeline only ever inserts.
Duplicates. The loader ran twice, or a document moved path, or an ID scheme changed between runs. Now near-identical chunks occupy several of your top-k slots, crowding out the other evidence the answer needed. Recall looks fine on a per-document basis and the answers get worse anyway.
All three come from the same root cause: the pipeline has no stable identity for a document and no record of what it wrote last time.
Give every chunk a stable, derivable ID
Start here, because everything else depends on it. Each chunk needs an ID you can recompute from the source without consulting the index, typically a composite:
- a source ID — the canonical identifier in the system of record (Confluence page ID, S3 key, database primary key, not the file name and not the URL if URLs change),
- a chunk index or section path within that document,
- a content hash of the chunk text.
Store the source ID and a document-level content hash in chunk metadata. Now an ingestion run can ask cheap questions: does this document exist in the index, and is the hash the same as what I have on disk? If yes, skip it — no parsing, no embedding, no write. On a corpus where a few dozen of fifty thousand documents change per day, hash-gating turns a six-hour nightly rebuild into a few minutes of work and a much smaller embedding bill.
LangChain's indexing API implements roughly this pattern with a record manager, and it is a reasonable starting point. The concept matters more than the library: a durable record of what you wrote, for which source version.
Make writes idempotent, per document
The safe unit of update is the whole document, not the individual chunk. When a document's hash changes:
- Re-parse and re-chunk it.
- Upsert the new chunks.
- Delete every chunk in the store whose source ID matches and whose chunk IDs are not in the new set.
Step 3 is the one people skip. Without it, an edit that shortens a document leaves the removed sections retrievable forever. Do the delete after the upsert and the document is never absent from the index mid-run; do it before and you have a window where queries return nothing. Choose the order deliberately, and make sure your vector store's metadata filtering can express "all chunks where source_id = X" — if it cannot, that is a real selection criterion for the store.
Propagate deletes as events, not as a diff you hope to notice
Full-corpus diffing catches deletes only if you enumerate the entire source every run, which stops being practical somewhere around a few hundred thousand documents or the first API rate limit. Two workable approaches:
- Subscribe to source events where the system offers them: webhooks, change feeds, CDC streams, S3 event notifications. Deletes and edits arrive as first-class messages and your pipeline becomes a consumer rather than a crawler.
- Soft-delete with a sweep where it does not. Every run stamps
last_seen_aton the documents it observed; a periodic job tombstones anything not seen in the last N successful full enumerations. Filter tombstoned chunks out at query time immediately, and hard-delete later.
Whichever you pick, exclude deleted documents at the retrieval filter as well as removing them from the store. Belt and braces, because eventual consistency in vector stores is real and a deletion request is not the place to discover the replication lag.
Treat parsing failures as data, not as log lines
PDF extraction returns empty strings. A scanned page yields a column of ligature noise. A table becomes one long run-on line. The ingestion job reports success, because nothing threw.
Add assertions at the parse step and record their results per document: character count against file size, ratio of alphanumeric characters, heading count for formats that should have headings, and — for the corpora where it matters — whether the page had extractable text at all before you fall back to OCR or a vision model. Write the outcomes to an ingestion table. What you want on a dashboard is not "job succeeded" but "1,412 documents processed, 37 below the text-density threshold, 4 zero-length." Those 41 documents are a retrieval hole you can now go fix, instead of a mystery a user reports in six weeks.
Vision-model parsing for complex PDFs is legitimately better than the old text extractors on tables and multi-column layouts, and it is slower and meaningfully more expensive per page. Route to it by detection — low text density, tables present — rather than sending the whole corpus through it.
Reindex behind a shadow index
Some changes invalidate everything: a new embedding model, a changed chunking strategy, added metadata fields. Do not mutate the live index in place.
Build a second index, write to both while backfilling, and keep the live one serving. Then run your retrieval eval set against both — recall-at-k on the labeled queries you already collected — and compare. Cut over when the new index wins on the numbers, keep the old one until you have a week of clean production traces, and keep the cutover behind a config flag so reverting is a deploy, not a rebuild. The same discipline applies whether the trigger is an embedding swap or a chunking change; the shadow index is what makes the decision measurable instead of a leap.
Monitor freshness like you monitor latency
Four numbers, on the same dashboard as your p95:
- Ingestion lag: median and max age between a source document's last modification and its index write.
- Coverage: documents in source versus documents in index, by collection. Any persistent gap is a parsing or permissions failure.
- Orphan count: indexed source IDs that no longer exist upstream.
- Failed documents: the parse-assertion failures from the last run, listed by ID, not summarized to a percentage.
Add one eval that most teams miss: a small set of queries whose correct answer changed when a document was updated. Run it after ingestion. It is the only check that actually proves the pipeline propagated a change end to end, rather than proving the index is non-empty.
The short version
Stable IDs, content hashes, document-scoped idempotent writes, explicit delete propagation, parse assertions recorded as data, and reindexing behind a shadow index. None of it is novel and all of it is skipped, because ingestion looks like plumbing until the day a customer quotes your assistant repeating a policy you retired last quarter.
If your retrieval metrics are decent and your users still say the answers are wrong, check the index's age before you touch the retriever.