Knowledge · Pattern
Chunking long inputs for models
How to split long documents or event streams into pieces a model can use, the main ways to choose boundaries, and what each choice costs in recall, context use and complexity.
Fast understanding
Definition
Chunking means splitting an input that is too long, or too coarse, for a model-facing step into pieces that the step can handle. Each piece is a chunk. The word covers two jobs that are judged differently.
In retrieval, chunks are the units that get indexed and later returned. A chunk is good when a question about its content finds it, and when the model can use it without the rest of the document. In multi-call processing, such as reading a long transcript across several model calls, chunks are windows over the input. A window is good when a call over it can produce a correct partial result, and when the partial results can be combined afterwards.
Three constraints apply to every chunk at once. It must fit the step that consumes it: an embedding model's input limit, or the context window of a request after instructions and the reserved answer are subtracted. It must be understandable without text outside it. And it should be specific enough that one embedding vector, or one model call, is not spread over unrelated topics. The first is a hard limit. The other two pull in opposite directions, and most of the choices below are ways of trading between them.
Explanation
When to use
Chunk when the input cannot or should not go to the model whole: a corpus too large for the prompt, a single document or event stream longer than the context window, an embedding model whose input limit is shorter than the document, or a cost and latency budget that rules out sending everything on every request. Embedding services can differ in what they do with text past the limit, truncating it or rejecting the request, so check the documentation of the model in use instead of assuming.
Do not chunk when something simpler works. If the whole source fits comfortably in the prompt and changes rarely, Grounding answers in your own data explains why putting it in the prompt removes a whole class of retrieval errors. If the data already comes in natural units, such as FAQ entries, tickets or product records, the record is the chunk, and splitting inside it only destroys structure. If a question needs an exact stored fact, look the fact up directly instead of retrieving text that happens to sit near it.
How it works
A chunking design is a boundary rule plus up to three additions.
| Boundary rule | Where it cuts | Strength | Weakness |
|---|---|---|---|
| Fixed length | Every N tokens or characters | Predictable size, cost and storage; trivial to build | Cuts through sentences, tables and code |
| Structure-aware | At headings, paragraphs, list items or speaker turns | Keeps the author's units together | Needs input with usable structure; chunk sizes vary |
| Semantic | Where the embedding similarity between adjacent sentences drops | Can follow topic shifts that have no markup | Extra embedding pass; sensitive to its threshold |
| Window over a stream | Every N events or every T minutes, or at pauses and topic changes | Matches how logs and chats arrive; works incrementally | A window boundary can fall in the middle of an exchange |
The structure-aware family often uses a fallback. LangChain's recursive character splitter takes an ordered list of separators, by default paragraph break, line break, space and finally the empty string, and moves to the next separator only when a piece is still too large (documentation). Large units stay whole whenever they fit, and the cut falls on the coarsest boundary available. In that documentation's example, size is measured by a configurable length function that counts characters. The general point is to measure size in the units of the thing that consumes the chunk, which is tokens of the target model, not characters.
Three common additions, which differ in where they apply:
- Overlap. The last few units of one chunk are repeated at the start of the next. This can keep material near a boundary together in at least one chunk, provided the overlap is large enough to hold the context that material depends on. It applies to retrieval chunks and to processing windows alike.
- Attached context. Each chunk carries where it came from: document title, heading path, speaker or date. In a retrieval system this can be stored as metadata for filtering, provenance and citation, or prepended to the chunk text before embedding. A heavier variant asks a model to write a short description that places each chunk within its document, and prepends that. Anthropic describes this as contextual retrieval and gives its motivation: a chunk saying revenue grew 3 percent names neither the company nor the quarter (Anthropic, 2024).
- Small-to-big retrieval. This is a retrieval-system technique. The index holds small chunks, and each one points to a larger parent section. Indexing smaller chunks can improve retrieval precision, and returning the parent or the neighbouring chunks when a small chunk matches can give the model more context. The cost is more tokens per hit and a parent link to maintain.
For multi-call processing the chunks are consumed by model calls instead of an index. The windows can be processed independently and then combined, or one after another with some state carried forward. A published example of the first shape is Wu and others, 2021, who summarised books by summarising sections and then summarising those summaries recursively. How partial results are combined is a design problem of its own and outside this entry. Chunking decides where each window starts and ends, and what neighbouring windows share. Of the three additions, only overlap carries over directly to windows. Small-to-big retrieval has no counterpart in a window-based workflow, which shares overlap and any state carried forward instead.
Trade-offs
No chunk size is correct in general. The sizes in this entry are examples to make the arithmetic visible, and defaults in a library or hosted service are starting points to test.
| Lever | What it buys | What it costs |
|---|---|---|
| Smaller chunks | More precise matches, fewer irrelevant tokens per hit | Less context per chunk, more chunks to store and search, more calls for processing |
| Larger chunks | More context in each piece, fewer pieces | Blurred topics per vector, more of the token budget spent per hit, risk of exceeding an embedding limit |
| Overlap | Boundary material can stay together in one chunk when the overlap is large enough to hold its context | About w / (w - o) times the storage or processing, and near-duplicate hits |
| Semantic boundaries | Splits that follow topic shifts | An extra embedding pass and a threshold to tune |
| Attached context | Chunks that can be found and cited alone | Extra tokens per chunk, and for model-written context an extra model call per chunk at indexing time |
| Small-to-big (retrieval only) | Potentially more precise matching, with fuller context returned | More tokens per hit, and a parent link to keep correct |
Evidence for the fancier options is mixed, so test them before paying for them. One study compared semantic and fixed-size chunking on document retrieval, evidence retrieval and answer generation, and concluded that the extra computation of semantic chunking was not justified by consistent gains (Qu and others, 2024). That is one study on its own datasets, not a verdict on every corpus. Anthropic's own tests of model-written context report the top-20 retrieval failure rate, the share of relevant chunks missing from the top 20 retrieved results (1 minus recall@20). It was 5.7 percent for the baseline, 3.7 percent with contextual embeddings, and 2.9 percent with contextual embeddings plus Contextual BM25, a keyword index built over the same contextualised chunks. The headline results are averages across several domains, using Gemini Text 004 embeddings. They are reported experimental results, not guaranteed improvements for every corpus or retrieval system, and the source recommends running your own evaluation (Anthropic, 2024).
Choose by measurement. Build a set of questions whose answers you have located in the source, run each candidate configuration, and check whether the right chunk appears in the top results before judging the generated answers. Retrieval and generation fail separately, as Evaluating output you can't unit-test explains. Changing the chunking rule changes every chunk, so it also means re-embedding the corpus. Record the rule and its parameters alongside the index so that a result can be traced to the configuration that produced it.
Deeper reference
Example
A constructed example. A runbook has a step that reads "Drain the node before restarting it", followed by a twelve-row table of exceptions for nodes that hold state. A fixed 300-token split happens to end after row 5. A question about restarting a stateful node retrieves the chunk holding the step and rows 1 to 5, and the answer is built without rows 6 to 12, where the exception that applies may be listed.
Splitting at headings and keeping the table whole fixes this. If the table is longer than the size limit, split between rows and repeat the header row in every piece, so each piece can still be read alone. The cost is that chunk sizes now vary, and the largest must still fit the limit of the embedding model.
Example
Assumed workload for illustration: an incident channel export of 900 events, processed in windows of 150 events. With no overlap the windows start at events 0, 150, 300 and so on. A reply at event 152 says "that deploy is the cause", and the deploy was announced at event 147. The second window starts at event 150, so it contains the reply but not the deploy.
With an overlap of 20 events, the second window starts at event 130 and contains both. Each window advances by 130 events, so the 900 events need 7 windows, starting at 0, 130, 260, 390, 520, 650 and 780.
window 1 events 0 to 149
window 2 events 130 to 279 (events 130 to 149 appear in both)
window 3 events 260 to 409 (events 260 to 279 appear in both)
The cost of overlap follows from one ratio. If a window holds w units and neighbours share o units, each new window adds w minus o new units, so the input is processed about w / (w - o) times. Here that is 150 / 130, about 1.15 times. For retrieval with 400-token chunks and 80 tokens of overlap, it is 400 / 320 = 1.25, so a 1,000,000-token corpus becomes roughly 1,250,000 stored tokens across about 3,125 chunks. Overlap helps only when it is large enough to hold the context a reference needs, and on its own it cannot guarantee that every dependency appears whole in at least one window. A reply at event 160 that points at event 100, 60 positions earlier, is still separated from it by a 20-event overlap: window 1 ends at event 149 and window 2 starts at event 130, so neither window contains both. An overlap of at least 60 events would keep every pair of events up to 60 positions apart together in some window, at a processing cost of 150 / 90, about 1.67 times the input. A larger window can also preserve these references if its overlap is at least 60 events; the resulting processing cost depends on the window size and stride. When long-range dependencies must be preserved and that cost is too high, carrying state from one window to the next is the other option. The live incident summary design problem works through that choice for a full system.
Failure modes
- A boundary cuts through a unit. A table, a code block, a numbered procedure or a sentence is split, and each half is misleading alone.
- Chunks that cannot stand alone. Text such as "as above" or "it then restarts" loses its referent. Attached context and small-to-big retrieval address this.
- Overlap fills the results with near-duplicates. Two adjacent chunks share text, both match, and together they use up the result slots. Deduplicate or merge adjacent hits before building the prompt.
- Stale chunks after an edit. If a source changes, its old chunks keep matching unless each chunk records its source and position and the update replaces them all.
- Lost provenance and permissions. A chunk with no source identifier cannot be cited, and one with no access scope cannot be filtered. As the grounding lesson notes, permission checks have to happen at retrieval time.
- Size counted in the wrong unit. Prose, code and identifiers tokenize differently, as the first lesson in the Track shows, so a character-based size gives uneven token counts. Measure with the tokenizer of the model that consumes the chunk.
Interview considerations
State the unit and why it fits the data, then the size in tokens, the overlap, and what metadata each chunk carries. Say how the choice will be checked: a labelled question set and a retrieval measurement before any answer-quality judgement. Name the first change you would try if retrieval misses, such as splitting on structure or attaching headings, before reaching for a more expensive technique.
Sample wording for a candidate: "I'd split on the document's own headings and keep tables whole, cap chunks at the embedding limit in tokens, and carry the heading path with each chunk for filtering and citation. I'd start with a small overlap and measure whether the right chunk is in the top results on questions with known answers, then change one setting at a time."
Related Lessons
- Grounding answers in your own dataApplied in
Decide when a model needs retrieved context, how that context is chosen and fitted into a prompt, and how stale or missing data changes the answer.
Related Problems
- Design a live incident summary that an on-call engineer can trustMedium · Applied in
Related Knowledge
Run lexical (BM25) and vector retrieval on the same query and merge the two ranked lists with reciprocal rank fusion. Covers what the merge gains and costs, where it fails, and how to test it on your own corpus and queries.