What Is Chunking in RAG? Splitting Data So Retrieval Works

You need a chunking strategy because models operate with fixed token limits. Without splitting data, you will truncate critical information or retrieve irrelevant noise that kills accuracy. Treat this as an optimization problem where you balance vector density against token limits.

In production, your choice of chunking strategy can account for a 9% difference in recall performance. If you split chunks too aggressively, the LLM loses the semantic context required to answer complex queries. Conversely, if your chunks are too large, the vector representation dilutes the "signal" of the relevant information and your search results suffer.

Effective chunking transforms raw text into a format suitable for vector-based search while balancing the computational costs of embedding and inference. To build a reliable RAG (retrieval-augmented generation) pipeline, you must analyze how data is retrieved and processed. Metrics like Intersection over Union (IoU) and Recall@K are the only way to measure success and avoid the "vibe checks" that plague amateur implementations.

Why does RAG have to split documents at all?

You are forced to split documents primarily because of the hard technical constraints of embedding models. Every embedding model has a maximum context window, typically ranging from 512 to 8192 tokens. When you try to process text that exceeds this limit, the model truncates the excess tokens. This results in permanent data loss; any information buried in the discarded text will never be represented in the vector database and remains invisible to your retrieval system.

Even if a model could ingest an entire document, your search precision would suffer. Embedding models compress text into a fixed-size vector (often 1536 dimensions). When you embed a massive document as a single unit, that vector must represent the "average" meaning of all its parts. This dilution makes it nearly impossible for the retriever to find specific facts. By splitting the document into smaller chunks, you ensure that each vector represents a focused semantic unit, which makes a match far more likely during a similarity search.

You must also account for the "lost-in-the-middle" problem observed in LLMs. Even models with large context windows struggle to retrieve and use information buried in the center of a long prompt. Performance tends to peak at the very beginning and the end of the provided context. Chunking lets you isolate the most relevant pieces of information and place them directly in the prompt, bypassing the degradation that comes with massive, unorganized context blocks.

The way embedding models process fixed-size vectors means semantic density per token is higher in smaller segments. If 1,000 tokens are compressed into a 1536-dimension vector, the mathematical representation is noisier than if only 100 tokens were compressed. Smaller chunks allow the vector to more accurately "point" to a specific concept in the embedding space. Without this granularity, your retrieval system is searching through a library where every book has the same generic summary on its cover.

The core trade-off: large chunks or small chunks

Your primary engineering challenge is balancing retrieval precision against context preservation. Small chunks, such as single sentences, offer high precision. They allow your retriever to pinpoint the exact location of a specific fact. However, these "micro-chunks" often lack the surrounding context required for a model to make sense of the data. A chunk that reads "Its population is 3.85 million" is useless if the preceding sentence identifying the city sits in a different, unretrieved chunk.

Large chunks, such as entire sections or thematic blocks, do a better job of preserving relationships between ideas and the author's "train of thought." This contextual padding helps the model generate more coherent and accurate responses. The risk here is the "averaging chapters" problem: as the chunk size increases, the embedding vector becomes noisier. If a single chunk covers three different sub-topics, its position in the vector space will be a mathematical average of those topics, matching none of them well during retrieval.

The mechanics of this averaging problem come down to vector dimensionality. In a standard 1536-dimension vector, the semantic density per token decreases as the number of tokens in the chunk increases. When you try to represent 1,000 tokens in the same space as 100 tokens, the individual nuances of each sentence are flattened into a global average. In production, this often shows up as a "missed" retrieval where the document is clearly relevant but the vector was pulled toward an irrelevant sub-topic within the same large chunk.

Treat this as a spectrum between sentence-level and thematic focus. If your users typically ask factoid-based questions — specific dates or names — lean toward smaller chunks for better precision. If your system is designed for analytical queries that require synthesizing multiple paragraphs, larger chunks are necessary. In most production environments, you will find a middle ground where the chunk is large enough to be self-contained but small enough to remain semantically distinct, typically around the 400-512 token mark.

The common chunking strategies

Recursive character splitting is the industry standard for general text. Unlike basic fixed-size splitting, which often cuts a word or sentence in half, recursive splitting uses a hierarchy of separators to break text at the most logical points. It first looks for double newlines to split paragraphs, then single newlines, and finally spaces or characters. This maintains the natural structure of the document while staying under your target token limit. Recursive splitting reaches 88.1% recall at 200-token chunks and 89.5% at 400-token chunks.

For specialized technical data, use code-aware or structure-aware separators. When chunking source code, prioritize splitting at class and function definitions to prevent logic from being fragmented across chunks. You can implement this with the following hierarchy:

separators = [
    "\n\nclass ",  # Class definitions
    "\n\ndef ",    # Function definitions
    "\n\n",        # Paragraph breaks
    "\n",          # Line breaks
    " ",           # Spaces
    ""
]

Beyond recursive splitting, implement page-level chunking for structured documents like PDFs. In NVIDIA's 2024 benchmarks, page-level chunking achieved the highest accuracy (0.648) and the lowest variance across datasets. Because many documents — financial reports and legal filings — organize information visually by page, respecting these boundaries preserves the relationship between text, tables, and figures. Generic splitters often destroy these visual-semantic links, leaving fragmented and unusable context.

Do not assume "advanced" methods are always better. A Vecta February 2026 benchmark of seven strategies across academic papers found that recursive 512-token splitting achieved 69% accuracy, while semantic chunking fell to 54%. The failure was fragmentation; semantic chunking often produced fragments averaging only 43 tokens. These micro-fragments lacked enough semantic signal for the retriever to latch onto, making them invisible. Start with structure-aware defaults and only move to semantic splitting if your data demands it.

How do you pick chunk size and overlap?

For most text-based RAG applications, start with engineering defaults of 400 to 512 tokens. This range is large enough to capture several sentences of context but small enough to maintain a clear semantic signal. You should also implement a "sliding window" by including a 10% to 20% overlap between chunks (roughly 50 to 100 tokens). This overlap acts as a safety net, ensuring that if a key piece of information sits at a split point, it appears in full in at least one chunk.

The query types you expect should drive this decision. If your system handles mixed workloads, stick to the 400-512 token range. Use the table below to match your chunk size to your system goals:

Evaluate these sizes against your storage and compute budget. While increasing overlap can improve retrieval, it also drives higher indexing costs and storage requirements. A January 2026 systematic analysis using SPLADE retrieval and Mistral-8B on Natural Questions found that overlap provided no measurable benefit while significantly increasing indexing cost. Test your own corpus; don't assume overlap is a free accuracy boost.

The storage-versus-performance trade-off is real. Larger overlaps mean your vector database will grow significantly, increasing index build times and the cost of memory-resident indexes. In production, you must decide whether the marginal gain in recall justifies the linear increase in storage costs. For multi-billion token repositories, the cost of a 20% overlap can be the difference between a profitable and an unprofitable product.

Late chunking and contextual retrieval: two ways to restore lost context

Advanced RAG pipelines use late chunking to solve the isolated chunk problem. In standard pipelines, you split the document and then embed the chunks, meaning the vector for a chunk has no knowledge of the rest of the document. Late chunking, pioneered by Jina AI, embeds the full document first so that every token carries bidirectional attention context. This preserves the hidden states of preceding and succeeding tokens across what would otherwise be a hard boundary.

The "Berlin" versus "its population" example illustrates this. If a chunk says "Its population is 3.85 million," the embedding for the word "Its" in a late chunking system already encodes the fact that it refers to Berlin from three paragraphs earlier. Because the transformer processes the full sequence before the pooling step, the attention weights are shared across the entire document. This maintains global semantic context without forcing you into massive, inefficient chunks.

Contextual retrieval, introduced by Anthropic, takes a lexical approach. It uses a cheap model to generate a 50-to-100-token explanatory summary for each chunk, which requires a two-pass approach: one pass to generate the context and one to embed the final string. Contextual embeddings alone cut the top-20 retrieval failure rate by 35%, adding contextual BM25 takes it to 49%, and layering reranking on top reaches 67%. If your original chunk is a single line of revenue data, the prepended context would specify the company name and fiscal quarter.

The primary trade-off with contextual retrieval is production complexity and cost. You are adding an LLM inference step into your ingestion pipeline, which runs about $1.02 per million document tokens as a one-time preprocessing expense. For millions of chunks, this demands robust error handling and prompt caching to stay economical. For technical documentation or legal contracts where chunks frequently reference definitions introduced elsewhere, that enrichment is often the only way to avoid retrieval failure.

How do you measure chunking quality?

You cannot rely on "vibe checks" to evaluate your chunking strategy; you need technical metrics. The most effective metric for chunking efficiency is Intersection over Union (IoU) at the token level. By treating your retrieved chunks as bounding boxes and comparing them to the ground-truth relevant tokens, IoU measures how accurately your retriever covers the necessary information without pulling in too much noise. This is the only way to measure how much distractor text you are forcing the model to process.

Track Recall@K and precision too. Recall@K tells you what percentage of the relevant information was captured within the top K results. In production, engineers often use a high K (like 20) to mask poor chunking performance. This is a mistake. While a high K increases recall, it also induces context rot, where reasoning performance degrades as the context window fills with noise. If you need K=20 to find a basic fact, your chunking strategy is likely the culprit.

Precision measures how much of the retrieved content is actually useful. If you have high recall but low precision, your chunks are likely too large, wasting tokens and potentially leading to hallucinations. You should optimize for a tight cluster of relevant tokens. If your precision is consistently low, irrelevant context inside the chunk is distracting the model and degrading its ability to follow complex reasoning chains.

Finally, respect the "context cliff" identified in 2026 research. Response quality drops sharply once the context you give a model exceeds roughly 2,500 tokens. Optimize your chunking strategy to keep the total retrieved context well below this cliff. If your retrieval strategy requires pushing 5,000 tokens to get a reliable answer, you haven't solved the problem — you've moved the failure point from retrieval to inference.

Do million-token context windows kill chunking?

There is a common misconception that the massive context windows of today's frontier models make chunking obsolete. This is false. Latency increases significantly as you feed more tokens into a prompt. Retrieving only the 500 relevant tokens is always faster than sending 500,000 irrelevant ones. Even with prompt caching, the cost of processing millions of tokens for every user query is economically unsustainable for most production applications.

Context rot quietly erodes long-context performance. July 2025 research across 18 models showed that retrieval performance degrades as context length increases, even on simple tasks. Models become less accurate at pinpointing details as the haystack grows. Efficient chunking remains the most effective way to prune the search space, so the model only receives high-density, relevant information.

Cost decides it in the end. Even if latency were solved, the token cost of a no-chunking strategy would kill most business cases. While prompt caching reduces the cost of repeated calls, it does not eliminate the base cost of high-volume token ingestion. Moving from a system that retrieves exactly what it needs to one that ingests everything is an engineering regression that trades architectural rigor for brute-force computation.

Chunking also lets you scale your knowledge base far beyond the limits of even a million-token window. Enterprise data repositories often contain billions of tokens. RAG pipelines using efficient chunking and vector search remain the only viable way to search across these datasets while keeping the low latency and cost-effectiveness that production systems require. Chunking is a fundamental requirement for efficient data architecture, not a workaround for small context windows.

Which strategy should you start with?

Do not over-engineer your initial solution. Start with recursive character splitting using a target size of 400–512 tokens and a 10% overlap. This is the most robust default and typically yields 85–90% recall with minimal computational overhead.

Only move to complex strategies like semantic chunking or LLM-based splitting if your metrics — specifically Recall@K and IoU — show that recursive splitting is failing to capture the semantic boundaries you need. The simplest reliable path is the correct one until your own data proves otherwise.

References

• Chunking Strategies for LLM Applications — Pinecone
• Chunking Strategies to Improve LLM RAG Pipeline Performance — Weaviate
• Evaluating Chunking Strategies for Retrieval — Chroma
• Late Chunking in Long-Context Embedding Models — Jina AI
• Contextual Retrieval in AI Systems — Anthropic
• Reconstructing Context: Evaluating Advanced Chunking Strategies for RAG — Merola & Singh
• Best Chunking Strategies for RAG (and LLMs) in 2026 — Firecrawl
• Adaptive Chunking: Optimizing Chunking-Method Selection for RAG — de Moura Júnior et al.