Ask a team with a struggling RAG system how they chunk their documents and the answer is usually the same: a recursive character splitter, 1,000 tokens, 200 overlap — whatever the tutorial they started from used. Chunking gets treated as preprocessing plumbing, decided once and forgotten. It deserves more respect: chunk boundaries determine what your retriever can return. No amount of embedding model tuning or prompt work recovers a fact that got split across two chunks or buried in an off-topic one.
What fixed-size chunking actually does to your corpus
A fixed-size splitter draws boundaries wherever the token counter says to, and real documents do not arrange themselves around token counts. In practice this produces three recurring failures:
Split answers. A clause in a contract, a step list in a runbook, or a definition and its exceptions get cut mid-thought. Each fragment embeds to a vector that only half-represents the idea, so neither fragment ranks well for a query that needs the whole thing.
Diluted chunks. The opposite case: a chunk spans the end of one section and the start of another. Its embedding is an average of two topics, which means it ranks mediocrely for both. A corpus full of diluted chunks shows up in evals as "the right document is in the index but never in the top k."
Orphaned context. Tables lose their headers, list items lose the sentence that introduces them, and a chunk that says "in that case, the fee is waived" retrieves confidently while meaning nothing.
None of these are model problems. All of them are invisible if you only ever look at final answers.
Chunk along the structure the author already gave you
Most corpora arrive with boundaries built in: headings, sections, clauses, ticket fields, docstrings. Structure-aware chunking uses them.
- Markdown and HTML: split on heading levels first, and only fall back to size-based splitting inside sections that exceed your ceiling. Keep the heading path ("Returns > International orders") in the chunk text or metadata — it is cheap disambiguation for both the embedding and the reader.
- Contracts and policies: split on numbered clauses. A clause is the unit lawyers cite for a reason: it is the unit of meaning.
- Tables: keep them intact with their headers, or serialize rows with the header repeated. A table shredded into token windows is unrecoverable.
- Code: split on function and class boundaries, not lines.
The principle: the author already segmented the document into ideas. Reuse their work.
Decouple the retrieval unit from the generation context
The most useful realization in chunking is that the thing you match on and the thing you hand the model do not have to be the same text.
Small chunks match precisely — a two-sentence chunk about fee waivers embeds crisply and ranks well. But two sentences may be too little context for the model to answer safely. The parent-document pattern (sometimes called small-to-big) resolves the tension: index small chunks for matching, then expand each hit to its parent section before generation. You get precise recall and sufficient context, at the cost of a slightly more complex pipeline and larger prompts.
This pattern alone has fixed more "RAG feels dumb" complaints in our work than any embedding model swap.
Overlap is a patch, not a strategy
Token overlap between adjacent chunks exists to soften boundary damage — if the split lands mid-idea, the neighboring chunk carries a copy of the severed text. With structure-aware boundaries, most of that damage never happens, and large overlaps mostly buy you index bloat and near-duplicate retrieval results that crowd out diverse evidence. A modest overlap is fine insurance. If removing overlap tanks your retrieval quality, that is a signal your boundaries are wrong, not that you need more overlap.
Measure it or you are guessing
Chunking changes are cheap to evaluate, because you can evaluate retrieval without touching generation:
- Collect 50–100 real queries (production traces beat invented questions).
- For each, record which document — and ideally which passage — contains the answer.
- Measure whether the answering passage appears in the top k, and where it ranks, across chunking variants.
Recall-at-k over a labeled set like this turns chunking from a matter of taste into an engineering decision. In our experience the delta between naive and corpus-aware chunking on this metric is regularly larger than the delta between embedding models — which is worth knowing before you pay to re-embed everything with something newer.
Where to start
Take ten real failed queries. For each, find the passage that should have answered it, and look at what your splitter did to that passage. That single exercise usually tells you more about your RAG system than a week of prompt tuning — and it points directly at which of the patterns above your corpus needs.