RAG Chunking Strategies
A support document explaining refund policy, shipping policy, and warranty terms gets embedded as one 2,000-token chunk — and a query about refunds retrieves it correctly, but its embedding is really an average of three unrelated topics, so a more specific refund question loses to a different document that talks about nothing but refunds. Chunk it too small instead, and a query about a multi-step process retrieves only step 3 with no surrounding context to make sense of it. Chunking is the first decision in any RAG pipeline, and getting it wrong doesn't throw an error — it just quietly makes retrieval worse.
This guide covers why chunk size is a real precision-versus-context trade-off (not an arbitrary tuning knob), the mechanical difference between fixed-size, recursive, and semantic chunking, and a real failure where a naive chunk boundary split a table in half — traced to the structure-aware fix. Chunking feeds directly into Embeddings Explained; for how the resulting chunks get used in a full retrieval pipeline, see Classic vs Graph vs Agentic RAG.
Why Chunk at All: The Precision-vs-Context Trade-off
An embedding model produces one vector per chunk of text, and that vector is effectively a blended representation of everything in the chunk. A chunk covering three unrelated topics produces an embedding that partially matches queries about any of the three — and fully matches none of them, losing a ranking comparison to a more focused chunk from a different document that discusses only the query's actual topic. This is the precision cost of chunks that are too large.
The opposite failure is just as real: a chunk that's too small might contain step 3 of a five-step process with no surrounding steps, so even a perfect retrieval match hands the LLM a fragment it can't properly use to answer "how do I do this whole process." Chunk size is a genuine trade-off between focused, precise retrieval and enough surrounding context for an answer to actually make sense — not a parameter to maximize or minimize, but one to tune against your specific documents and query patterns.
Quick reference
- A multi-topic chunk's embedding is a blend — it partially matches several queries and fully matches none as well as a focused chunk would.
- A too-small chunk can win the retrieval match and still fail to provide enough context for a usable answer.
- There is no universally correct chunk size — it depends on your documents' natural structure and how self-contained a single idea tends to be.
- Measure retrieval quality against a representative query set when tuning chunk size — don't guess a number and assume it generalizes.
- Chunking happens before embedding — every downstream retrieval decision inherits whatever chunking got wrong upstream.
Remember this
Chunk size trades retrieval precision against usable context — a chunk covering multiple topics dilutes its own embedding, while a chunk that's too small can win the match and still not contain enough to answer with.
Fixed-Size and Recursive Splitting
Fixed-size chunking splits text at a target token or character count with some overlap between consecutive chunks — commonly a few hundred tokens per chunk with an overlap of 10-20% of that size. The overlap exists because an idea spanning exactly the boundary between two chunks would otherwise be under-represented in both chunks' embeddings; repeating the tail of chunk N at the start of chunk N+1 gives that boundary idea a fighting chance of being retrieved. The failure mode is cutting mid-sentence or mid-paragraph purely because the character count hit its limit, regardless of what was actually being said.
Recursive splitting fixes this by trying natural boundaries first: split on paragraph breaks, and only if a resulting piece still exceeds the target size, split that piece on sentence breaks, and only if still too large, fall back to a hard character cut. This respects document structure far better than naive fixed-size splitting while still guaranteeing every chunk stays under the size limit — it's the default behavior of popular splitters like LangChain's RecursiveCharacterTextSplitter for exactly this reason.
Quick reference
- Overlap gives a boundary-spanning idea a chance to be fully represented in at least one chunk's embedding.
- Too much overlap wastes storage and can crowd a top-k retrieval result with near-duplicate chunks.
- Recursive splitting tries paragraph breaks, then sentence breaks, before ever falling back to a hard character cut.
- Recursive splitting still guarantees a maximum chunk size — it's a better default, not an unbounded one.
- Neither fixed-size nor recursive splitting understands document meaning — both split on structure or size, not topic.
Remember this
Recursive splitting respects paragraph and sentence boundaries before falling back to a hard character cut — a meaningfully better default than naive fixed-size chunking, though neither actually understands what the text means.
Semantic Chunking: Splitting by Meaning, Not Size
Semantic chunking decides split points based on meaning rather than size: embed consecutive sentences, measure the similarity between each adjacent pair, and cut where similarity drops sharply — the signature of a genuine topic shift rather than just "500 characters happened." The result is variable-length chunks that each tend to cover one coherent idea, regardless of whether that idea took 100 or 800 tokens to express.
The cost is real: semantic chunking requires embedding calls just to decide where to split, before the actual chunk embeddings for retrieval are even computed — meaningfully more compute and latency at ingestion time than a free character count. It's worth that cost for document collections where topics genuinely vary in length and a fixed size would either constantly split short ideas unnecessarily or blend several short ideas into one chunk — less clearly worth it for uniformly structured content where fixed-size or recursive splitting already aligns well with natural boundaries.
Quick reference
- Cut points are chosen where adjacent-sentence similarity drops — a proxy for a genuine topic shift.
- Produces variable-length chunks aligned with actual ideas, not a fixed size.
- Costs extra embedding calls at ingestion time purely to decide where to split — a real latency and compute trade-off.
- Most valuable for content where topic length genuinely varies — less clearly worth it for uniform, already-well-structured documents.
- Semantic chunking still benefits from a maximum size fallback, so one unusually long coherent topic doesn't produce an oversized chunk.
Remember this
Semantic chunking cuts at genuine topic shifts instead of a fixed size, at the real cost of extra embedding calls during ingestion — worth it when topic length varies a lot, less clearly worth it on uniformly structured content.
Recover One Chunk Boundary That Cut Through a Table
Trigger. A pricing document containing a comparison table gets processed by a naive fixed-size chunker with no awareness of table structure — the 500-character boundary lands in the middle of the table's rows. Symptom. A query about a specific plan's price retrieves the chunk containing that row, but the row arrives with no header row for context — the LLM receives numbers with no labels for what they represent, and either guesses wrong or (correctly) says it can't determine the answer from the given context.
Root mechanism. Fixed-size and even recursive splitting operate on paragraph and sentence boundaries — neither treats a table, code block, or other structural unit as something that must stay whole. Recovery: re-chunk with a structure-aware splitter that detects markdown tables (or code fences, or list structures) and treats each as one atomic unit, never split internally, repeating the header row in the chunk with its data rows. Verification: re-run the same pricing query and confirm the retrieved chunk now includes the table's header alongside the relevant row. Prevention: for any document corpus containing tables, code blocks, or other structured elements, use a chunker that specifically recognizes and preserves those structures — a generic recursive text splitter alone is not enough for structured content.
Quick reference
- Trigger: a fixed-size or recursive splitter treats a table's raw text like any other paragraph.
- Symptom: a retrieved chunk contains data rows with no header — numbers with no labels.
- Root mechanism: neither fixed-size nor plain recursive splitting recognizes structural elements as atomic units.
- Recovery: a structure-aware chunker detects tables/code blocks and preserves them whole, including headers.
- Prevention: any corpus containing structured content needs a chunker built for that structure, not a generic text splitter alone.
Remember this
A chunk boundary that splits a table mid-row hands the LLM numbers with no labels to interpret them — the fix is a chunker that recognizes structural elements like tables as atomic units, never splitting them internally.
Key takeaway
Chunk size trades retrieval precision (a focused chunk matches its topic well) against usable context (a chunk needs enough surrounding material to actually answer with). Fixed-size chunking is simplest but can cut mid-thought; recursive splitting respects paragraph and sentence boundaries first; semantic chunking cuts at genuine topic shifts at the cost of extra embedding calls during ingestion. Structural elements like tables and code blocks need their own atomic-unit handling, or a chunk boundary can hand the LLM data with no labels to interpret it.
Practice (25 min): chunk the same multi-topic document with fixed-size (no structure awareness), recursive, and semantic chunking, and compare the resulting chunk boundaries directly against the document's actual paragraph and topic structure. Include a table in your test document and confirm which chunking approach preserves it intact. Run a query whose answer depends on a specific table row and confirm whether the retrieved chunk includes the header row needed to interpret it. Pass when you can explain, for your own document corpus, which chunking approach's boundaries actually align with where your content's real ideas begin and end.
Related Articles
Explore this topic