Chunking Strategies for RAG: Getting Retrieval Right

The retriever never reads a whole document. It reads chunks — embedded slices that must each carry enough context to be useful on their own. If a chunk is too small, it is a fragment with no meaning; if too large, it buries the relevant sentence inside a wall of noise.

Everything downstream inherits that decision: chunk boundaries determine what embeddings represent, what the reranker scores, and what the model sees in the prompt. Tuning the prompt or the index while ignoring chunking usually plateaus fast.

That is why chunking deserves its own design pass rather than a default character count.

The simplest strategy splits documents into fixed-size pieces measured by words, characters, or tokens, often with an overlap between neighbors. Overlap exists to catch content that spans a boundary — a concept introduced near the end of one chunk and explained at the start of the next.

Fixed chunking is cheap, predictable, and easy to reason about, which makes it a fine baseline. Its weakness is that it ignores meaning: a sentence can literally be cut in half, and paragraphs with strong internal logic can be scattered across chunks.

Semantic chunking splits where meaning shifts rather than where an arbitrary counter runs out. Simpler versions detect natural breakpoints using embeddings or heuristics such as paragraph boundaries; more elaborate approaches use a model to identify topical sections.

The result is chunks that correspond to ideas rather than line counts. That tends to improve retrieval relevance and answer coherence, at the cost of more computation during ingestion and less predictable chunk sizes.

For structured documents, hierarchy is the strongest signal available, and respecting it usually beats both fixed and purely semantic approaches.

The recurring principle is to keep the chunk aligned with the document's own unit of meaning. A manual with sections, subsections, and numbered lists chunks cleanly; a wall of unstructured text does not offer that gift.

Chunk size interacts with embedding, retrieval, and generation. Smaller chunks embed more precisely and match tightly, but they send less context to the model and often require stitching several chunks together. Larger chunks pack more context but blunt the similarity signal and consume more of the model's window.

There is no universal number. Teams iterate between a few hundred and a few thousand tokens per chunk depending on document type, question style, and model context budget, measuring retrieval quality at each step rather than guessing.

Even good chunking misses questions that span multiple chunks or sit at boundaries. Production systems layer mitigations: retrieve several chunks and let the model synthesize, link child chunks back to their parent sections, and return the full headed unit rather than a slice when a query needs it.

None of these rescues a chunking scheme that cut every idea in half to begin with, which is why the upstream decision still deserves the first pass.