Chunking looks like a preprocessing detail and is usually the single largest determinant of whether a retrieval system works. Get it wrong and no amount of prompt engineering or model upgrading compensates — the model cannot reason about a passage it was never shown. This page covers the decisions that matter and how to make them.
Sensible defaults
400–800 tokens per chunk, 10–20% overlap, split on document structure rather than character count, and prepend a short contextual summary to each chunk before embedding. Measure and adjust from there — those defaults are a starting point, not an answer.
Why chunk at all
Three separate reasons, and they pull in different directions:
- Embedding models have input limits, typically 512 to 8,192 tokens. A long document simply will not fit.
- Precision. An embedding is a single vector representing everything in its input. A whole chapter averages into a vague vector that matches many queries weakly and none strongly.
- Context budget. You can only pass a limited amount to the model, so retrieving whole documents wastes the window on irrelevant material.
💡 Why big chunks embed badly
An embedding is roughly a summary of meaning in a fixed number of dimensions. A chunk covering five topics produces a vector that sits somewhere between all five — close to none of them.
This is the core tension. Smaller chunks give sharper vectors that match precisely. Larger chunks give more complete passages that read better. You cannot maximise both, and the whole craft of chunking is managing that trade.
Choosing a size
| Size | Retrieval precision | Context preserved | Suits |
|---|---|---|---|
| 100–200 | Very high | Poor | FAQs, definitions, product specs |
| 300–500 | High | Moderate | Documentation, knowledge bases |
| 500–800 | Good | Good | General prose — the default |
| 800–1,500 | Moderate | Strong | Narrative, reports, legal text |
| 1,500+ | Weak | Very strong | Rarely the right choice |
The right size depends as much on your questions as your documents. Narrow factual questions — "what is the refund window?" — favour small chunks. Questions requiring synthesis — "how does the escalation process work?" — favour larger ones that hold a complete explanation.
If your queries are mixed, index at two sizes and search both. Storage is cheap relative to the cost of failed retrieval.
Splitting strategies
Fixed size — the fallback
Recursive — the sensible default
Try to split on the largest natural boundary available, and fall back to smaller ones only when a piece is still too big:
The mergeToLimit step matters and is often omitted. After splitting on paragraphs you have many small pieces; combining adjacent ones up to the size limit produces full chunks rather than a scatter of one-sentence fragments.
Structural — the best where available
Markdown, HTML and well-formed documents carry explicit structure. Use it.
Keeping the heading path is valuable twice over: it can be prepended to the chunk text so the embedding knows what the passage is about, and it gives the model a citation when it answers.
Semantic — sometimes worth it
Embed each sentence, measure similarity between consecutive sentences, and split where similarity drops — the point where the topic changes.
It produces genuinely coherent chunks and costs an embedding call per sentence at index time. Worth it for high-value corpora with poor structural markup; unnecessary when headings already tell you where the boundaries are.
Overlap
Overlap exists to stop a passage that straddles a boundary being lost from both chunks.
| Overlap | Effect |
|---|---|
| 0% | Boundary passages lost |
| 10–20% | Sensible default |
| 30–50% | Near-duplicates crowd the results; storage inflates |
The failure mode at high overlap is subtle: your top five results become five overlapping views of the same passage, and genuinely different relevant material never surfaces. If your retrieval returns repetitive results, check overlap before blaming the embedding model.
Note that overlap is largely unnecessary with structural splitting — a chunk that ends at a section boundary is not cutting anything mid-thought.
Contextual retrieval
This technique fixes more retrieval failures than any amount of size tuning, and it is worth implementing before anything else.
The problem: a chunk extracted from the middle of a document loses everything the surrounding text established.
The fix is to generate a short contextual preamble for each chunk — using a cheap model, at index time — and prepend it before embedding:
The chunk now contains the entities the question uses. Published results on this approach report large reductions in retrieval failure rate, and it is straightforward to add to an existing pipeline.
✅ It is cheaper than it sounds
The cost is one small-model call per chunk, paid once at indexing. With prompt caching on the document — which is resent for every chunk from the same document — the marginal cost per chunk drops substantially. Against the cost of a retrieval system that returns the wrong passages, it is not a close call.
Tables, code and lists
| Content | Rule |
|---|---|
| Tables | Never split from the header. If it must split, repeat the header row in each part. |
| Code | Split on function or class boundaries. Include the signature and any imports it needs. |
| Lists | Keep whole where possible; carry the introducing sentence into each part. |
| Definitions | Term and definition must stay together. |
| Q&A pairs | One chunk per pair — they are naturally sized. |
A table fragment without its header is genuinely worse than nothing: it contains numbers with no indication of what they measure, and the model will confidently interpret them wrongly.
Retrieve small, return large
A pattern that gets much of the benefit of both sizes.
Index small chunks for precise matching, but store a pointer from each to a larger surrounding passage. Search the small ones; return the large ones to the model.
You get the retrieval precision of a 150-token chunk and the contextual completeness of an 800-token one. The main cost is a second store and slightly more complex retrieval code.
Evaluating it
Chunking cannot be tuned by intuition. Build a small evaluation set — thirty to fifty questions with the passages that should answer them — and measure.
| Metric | Question it answers |
|---|---|
| Recall@k | Is the correct passage in the top k results? |
| MRR | How highly is it ranked when it is found? |
| Context precision | What fraction of retrieved tokens are actually relevant? |
| Answer accuracy | The one that matters — end to end |
Recall@k is the diagnostic to start with. If the right passage is not being retrieved, nothing downstream can fix it — a better model will simply be confidently wrong using the wrong source. Fix retrieval first, then tune generation.
Preparing documents for indexing?
Convert HTML to Markdown for cleaner chunking, or extract and split PDF pages — all in your browser.
Open the HTML to Markdown Converter →Summary
- Chunk size is a precision-versus-context trade-off. 400–800 tokens suits most prose.
- Split on structure — headings and paragraphs — before falling back to character counts.
- 10–20% overlap. More returns near-duplicates that crowd out real results.
- Contextual retrieval fixes the most failures. Prepend a situating summary before embedding.
- Never split a table from its header or a function from its signature.
- Retrieve small, return large to get precision and completeness together.
- Keep heading paths as metadata — they improve embeddings and provide citations.
- Measure recall@k first. Retrieval failures cannot be fixed downstream.
Frequently Asked Questions
What is the best chunk size for RAG?
There is no universal answer, but 400 to 800 tokens suits most prose. Smaller chunks give sharper embeddings and more precise retrieval while losing surrounding context; larger chunks preserve context but produce diluted embeddings that match many queries weakly. The right size depends on your documents and questions, and should be measured rather than assumed.
How much overlap should chunks have?
Typically 10 to 20% of the chunk size — about 50 to 100 tokens for a 500-token chunk. Overlap exists so that a passage spanning a boundary is not lost entirely from both chunks. Too much overlap wastes storage and returns near-duplicate results that crowd out genuinely different passages.
Should I split on characters or on structure?
Structure, wherever the document has any. Splitting on headings, paragraphs and sections produces chunks that are individually coherent, which both embeds better and reads better when passed to the model. Fixed-size character splitting is a fallback for unstructured text, not a default.
Why does my RAG system retrieve the wrong chunks?
The most common cause is chunks that lost their context. A passage saying 'the rate increased to 4.5%' embeds without any indication of which rate, which product or which year, so it matches poorly against a specific question. Prepending a short contextual summary to each chunk before embedding fixes a large share of these failures.
How do I chunk tables and code?
Do not split them mid-structure. A table fragment without its header row is meaningless, and a function split across two chunks is worse than useless. Keep tables and code blocks whole where they fit, and where they do not, repeat the header or signature in each part so every chunk stands alone.