Chunking Strategies for RAG: What Actually Works

Chunking Strategies for RAG: What Actually Works for Agent Memory and Pipeline Builders

Every guide on chunking strategies for RAG lists the same five methods and then leaves you to guess which one survives contact with your data. If you're a pipeline builder or agent memory engineer, that's not guidance. It's a menu.

Chunking is the process of splitting documents into smaller, semantically meaningful units before embedding them, because embedding models have token limits and raw documents are too long to retrieve precisely. The honest answer is that no universal best strategy exists. What works depends on your document structure, your embedding model, your retrieval method, and whether you're building a standard RAG pipeline or an agent that needs memory across sessions.

GigaRAG, a platform built for agent memory and RAG pipeline builders, treats chunking as a configurable decision rather than a default. But this guide isn't about any single tool. It covers what each strategy actually does, when each one fails, and how to choose without burning a week on experiments.

In This Guide

• What Is Chunking in RAG?
• Fixed-Size vs Semantic Chunking: Which Should You Use?
• Why Chunking Matters for RAG Pipelines
• Chunking Strategies For Rag: A Step-by-Step Guide
• Fixed-Size Chunking Strategies for RAG
• Semantic Chunking for RAG
• Document Structure-Based Chunking
• LLM-Based and Contextual Chunking
• Chunking for Agent Memory Systems
• How to Choose a Chunking Strategy
• Chunk Size and Overlap: What Actually Works
• Final Thoughts on Chunking Strategies for RAG

What Is Chunking in RAG?

Chunking is the process of splitting documents into smaller, semantically meaningful units before embedding them. Raw documents can't be embedded directly: embedding models have token limits, and a 50-page PDF produces an embedding that averages away the details you actually want to retrieve.

Why documents must be split before embedding

Embedding models accept a fixed number of tokens, typically 512 to 8,192. Feed a document longer than that and you either truncate it or get an error. Truncation is worse: you lose everything past the cutoff. Splitting also matters for retrieval precision. A single embedding for an entire document represents its average meaning, not any specific fact inside it. When a query matches, you get the whole document back, even if only one paragraph is relevant. That wastes context window space and dilutes the model's attention.

The relationship between chunking and retrieval quality

Chunk size sets the ceiling on retrieval precision. Smaller chunks match queries more precisely but lose surrounding context. Larger chunks preserve context but pull in irrelevant text. The right size depends on your embedding model, your document structure, and what your queries look like. There's no universal answer.

[!note] Chunking strategies are not one-size-fits-all; the optimal approach depends on your specific data, queries, and retrieval model. Always validate with your own evaluation set rather than relying solely on general recommendations.

Fixed-Size vs Semantic Chunking: Which Should You Use?

Why Chunking Matters for RAG Pipelines

Chunking is where retrieval quality is won or lost. The embedding model can only represent what you give it. Give it a clean, coherent chunk and you get a clean, coherent vector. Give it a broken sentence or a wall of mixed topics and the vector points nowhere useful.

How chunk size affects embedding quality

Embedding models compress text into a fixed-length vector. A 256-token chunk about a single idea produces a sharp representation. A 1,024-token chunk covering three unrelated topics produces a blur: the vector averages all three meanings, so it matches none of them well. Smaller chunks also mean more vectors per document, which gives retrieval finer granularity. The tradeoff is context. A chunk that's too small loses the surrounding sentences that make its meaning clear.

The cost of bad chunking: irrelevant retrieval and hallucinations

Bad chunking fails silently. You don't see an error. You see retrieval results that look plausible but miss the mark, and then the LLM generates an answer from the wrong text. That's how hallucinations happen: the model isn't lying, it's faithfully summarizing a chunk that doesn't answer the question. You also waste tokens. Every irrelevant chunk you stuff into the context window is budget you can't spend on the chunks that matter.

Agent memory makes this worse. Agents retrieve chunks across sessions, not just within one query. A poorly chunked memory store compounds errors over time: the agent pulls the wrong context, acts on it, and stores the result as a new memory. The mistake becomes part of the system.

[!tip] For agent memory systems, store both the original chunk and a concise summary in a hierarchical structure. This allows the agent to retrieve detailed context when needed while using summaries for efficient long-term memory.

Chunking Strategies For Rag: A Step-by-Step Guide

1. Analyze your document types and typical query patterns.
2. Start with a baseline fixed-size chunking with 10–20% overlap.
3. Evaluate retrieval quality using a held-out set of queries.
4. If precision is low, experiment with semantic or recursive chunking.
5. For agent memory, implement hierarchical chunking with summary nodes.
6. Test different chunk sizes (e.g., 256, 512, 1024 tokens) and measure impact.
7. Iterate based on metrics like recall@k and answer faithfulness.

Fixed-Size Chunking Strategies for RAG

Fixed-size chunking is the simplest approach: split text into equal-sized pieces by character or token count. It's the default in most libraries because it requires zero understanding of the document. You set a size, you set an overlap, and you're done.

Character-based vs token-based splitting

Character-based splitting cuts text every N characters. It's fast and predictable, but it doesn't know what a token is. A 500-character chunk might be 80 tokens or 200 tokens depending on the language and content. Token-based splitting uses the embedding model's tokenizer to cut at actual token boundaries. That's more accurate for staying under model limits, but it's slower and requires the tokenizer at indexing time.

In practice, token-based splitting is what you want if your embedding model has a strict token limit. Character-based works fine for quick prototypes where precision doesn't matter yet.

Choosing chunk size and overlap

Common sizes are 256, 512, and 1,024 tokens. Smaller chunks give sharper embeddings but lose context. Larger chunks preserve context but blur the vector. Overlap is the fix for context loss at boundaries: a 10-15% overlap means the end of one chunk repeats at the start of the next, so a sentence split mid-thought still has its neighbor nearby.

The honest answer is there's no optimal size. It depends on your embedding model and your documents. Start at 512 tokens with 10% overlap and test against your own retrieval queries.

When this fails: broken sentences and lost context

Fixed-size chunking doesn't care where sentences end. It cuts mid-thought, mid-clause, sometimes mid-word. A chunk that starts with "the algorithm" and ends with "because" is useless to an embedding model. The vector represents a fragment, not an idea.

It also performs poorly on documents with varied structure. A legal contract, a code file, and a blog post all need different boundaries. Fixed-size gives them all the same treatment. You'll get chunks that mix headings with body text, code with comments, or questions with answers from different sections. Retrieval quality drops, and you won't know why unless you inspect the chunks yourself.

Semantic Chunking for RAG

Semantic chunking splits text based on meaning, not size. It uses embedding similarity to find natural breakpoints where the topic shifts. The result is chunks that each hold one coherent idea.

How semantic chunking works

You embed every sentence in the document. Then you walk through the sentences in order, comparing each sentence's embedding to the ones before it. When similarity drops below a threshold, you cut. The sentences between cuts become one chunk.

This works because embeddings capture meaning. A paragraph about model training has sentences that embed close together. When the text shifts to evaluation metrics, the embeddings drift apart. The algorithm catches that drift and splits there.

The main catch is that you need an embedding model at chunking time. Fixed-size chunking needs nothing but a tokenizer. Semantic chunking needs a full forward pass over every sentence before you can build your index.

Similarity thresholds and breakpoint detection

The threshold is the tuning knob. Set it high and you get many small chunks, each tightly focused. Set it low and you get fewer, larger chunks that may mix topics. Typical thresholds sit between 0.5 and 0.8 cosine similarity, but the right value depends on your embedding model and your documents.

Breakpoint detection compares consecutive sentences. Some implementations also compare each sentence to a rolling average of the previous few sentences. That smooths out noise from short or unusual sentences that embed oddly on their own.

When this fails: speed, cost, and inconsistent sizes

Semantic chunking is slower than fixed-size. Every sentence needs an embedding, which means an API call or a local model run for each one. On a large corpus, that's real money and real time before indexing even starts.

It also produces inconsistent chunk sizes. A long discussion of one topic stays together as a 2,000-token chunk. A quick topic shift produces a 50-token chunk. That variance makes it harder to reason about retrieval behavior and token budgets.

And it fails on documents without clear topic shifts. A list of product specs, a glossary, or a changelog has no narrative flow. Sentences embed at similar distances from each other, and the threshold becomes arbitrary. You get chunks that are effectively random.

Document Structure-Based Chunking

Structure-based chunking splits text at the boundaries the document already has: headings, paragraphs, lists, tables. You respect the author's organization instead of imposing your own. This works well for HTML, Markdown, and PDFs with clear hierarchy.

Splitting by headings and paragraphs

You parse the document tree and cut at heading tags or Markdown headers. Each section under an H2 or H3 becomes a chunk, with its heading kept as metadata. Paragraphs inside a section stay together.

The good news is that headings often map to semantic boundaries. A section titled "Installation" holds installation content. Retrieval gets cleaner because chunks align with what readers already understand as units.

The main catch is that headings are only as good as the author's structure. Some documents use headings inconsistently or not at all.

Handling tables, lists, and code blocks

Tables need special treatment. Splitting a table across chunks destroys it. You keep the table intact and attach its caption or preceding heading as context. Lists work the same way: a numbered list of steps stays together.

Code blocks are trickier. A 200-line function doesn't split cleanly. You either keep the whole block as one chunk or split by function definition, not by line count.

When this fails: unstructured text and parser complexity

Structure-based chunking fails on text without structure. Raw transcripts, scraped web pages with broken HTML, or OCR'd PDFs have no reliable headings to split on. You're back to guessing.

It also requires document-specific parsing logic. HTML needs an HTML parser. Markdown needs a Markdown parser. PDFs need layout analysis to reconstruct headings from font sizes and positions. Each format is a separate integration.

Chunk sizes vary wildly. A short section becomes a 30-token chunk. A long section with nested subsections becomes 3,000 tokens. That variance makes retrieval behavior hard to predict.

LLM-Based and Contextual Chunking

LLM-based chunking uses a language model to decide where chunks should start and end. Instead of counting tokens or parsing headings, you prompt the model to read the text and mark semantic boundaries. The model can also generate a short summary for each chunk, which becomes part of the chunk's metadata.

Using LLMs to identify chunk boundaries

You feed a document to an LLM and ask it to return split points. A typical prompt: "Read this text and identify where the topic changes. Return the text split into coherent sections." The model returns boundaries that respect meaning, not just length.

This works well for text where structure is implicit. Interview transcripts, meeting notes, or prose without headings benefit because the model understands topic shifts the way a human reader would.

The tradeoff is that you're paying for API calls at indexing time. Every document you ingest costs tokens. For a large corpus, that adds up fast.

Contextual chunking with document summaries

Anthropic's contextual retrieval approach goes one step further. Instead of just splitting, the LLM generates a short contextual summary for each chunk and prepends it. A chunk that says "the API returns a 401 error" gets prefixed with something like "This section is from the authentication documentation for the v2 API."

That prefix gets embedded along with the chunk text. Retrieval improves because the embedding now carries document-level context, not just the chunk's local content.

The catch is that you're running an LLM pass over every chunk, not just every document. That's the most expensive chunking method covered here.

When this fails: cost, latency, and hallucinated context

LLM-based chunking is slow. Indexing a large corpus means thousands of API calls, each taking seconds. If you need to ingest documents in real time, this won't work.

The model can also hallucinate. A generated summary might misstate what the chunk contains. That bad metadata then pollutes retrieval, because the embedding includes the hallucinated prefix. You're trusting the LLM to describe your own documents accurately, and it won't always.

There's no free lunch here. You're trading indexing cost and latency for better retrieval quality. For small, high-value corpora, that tradeoff can be worth it. For large-scale ingestion, it usually isn't.

Chunking for Agent Memory Systems

Standard RAG treats retrieval as a single lookup: embed a query, find similar chunks, return them. Agent memory is different. An agent works across sessions, accumulates context, and needs to recall things it learned hours or days ago. Chunking for agent memory has to support that persistence, not just one-shot retrieval.

How agent memory differs from standard RAG retrieval

In standard RAG, chunks are static. You index documents once and retrieve against them. In agent memory, chunks are written, read, updated, and sometimes consolidated. An agent might store a fact from one conversation, then need to retrieve it in a different context weeks later.

The chunking question shifts. Instead of "what size maximizes retrieval accuracy for this query," you're asking "what size lets the agent hold this in working memory, reason over it, and store it for later recall." Those are different constraints.

Hierarchical chunking for memory consolidation

Hierarchical chunking stores chunks at multiple granularities. A parent chunk holds a coarse summary or a full section. Child chunks hold the details. When the agent retrieves, it pulls the parent first, then drills into children only if the task needs them.

This mirrors how memory consolidation works. The agent keeps a high-level gist readily available. Details stay accessible but don't crowd the context window. For a support agent, the parent might be "customer reported billing issue on March 14." The children hold the specific error messages, steps taken, and resolution.

When this fails: working memory limits and context fragmentation

Chunks that are too large blow past the agent's working memory. The agent can't hold a 2,000-token chunk plus its instructions plus the conversation history. It drops details or starts summarizing aggressively, which loses fidelity.

Chunks that are too small fragment the narrative. If each interaction is split into 50-token pieces, the agent can't reconstruct what happened across a session. It retrieves fragments that don't connect. The memory becomes a pile of disconnected facts with no through-line.

The honest answer is that agent memory chunking is a tuning problem you'll revisit. Start with parent chunks around 500 to 800 tokens and children at 100 to 200. Test against real multi-session conversations. Adjust from there.

How to Choose a Chunking Strategy

No single strategy wins. The honest answer is that the right choice depends on your document type, your embedding model, your retrieval method, and your budget. Anyone who tells you otherwise is selling something.

Key factors: document type, embedding model, retrieval method

Start with the document. Structured text with clear headings, like Markdown or HTML, rewards structure-based chunking. Unstructured prose, like call transcripts or forum posts, often does better with semantic or fixed-size splitting. The document dictates what boundaries exist to exploit.

Your embedding model matters too. Models tuned for long contexts, like those with 8k token windows, tolerate larger chunks. Models optimized for sentence-level similarity perform better with chunks under 200 tokens. Check your model's documentation for its recommended input size. That's a constraint, not a suggestion.

Retrieval method changes the math. If you're doing top-k similarity search, smaller chunks give you more precise matches but risk losing context. If you're doing parent-child retrieval, you can afford smaller chunks because the parent supplies the surrounding context. If you're feeding results directly into a prompt without expansion, larger chunks are safer.

Latency and budget sit underneath everything. Semantic and LLM-based chunking require embedding or API calls at indexing time. That's slower and costs money per document. Fixed-size and structure-based chunking are nearly free. If you're indexing millions of documents, the expensive methods may not be viable.

A simple decision framework

Here's a checklist that covers most cases:

• Structured documents (HTML, Markdown, PDFs with headings): use structure-based chunking. Split on headings and paragraphs. It's fast, cheap, and preserves natural boundaries.
• Unstructured prose: try semantic chunking if you can afford the embedding calls. Otherwise, fixed-size with 10-15% overlap is a reasonable fallback.
• Agent memory systems: use hierarchical chunking. Parent chunks at 500-800 tokens, children at 100-200. Retrieve parents first, drill into children as needed.
• Latency-sensitive indexing: avoid LLM-based chunking. The API calls will dominate your indexing time.
• Budget-constrained projects: fixed-size or structure-based. No per-document embedding or LLM costs.

Test before you commit. Index a representative sample of your documents, run real queries, and measure retrieval quality. A strategy that looks right on paper often fails on your actual data.

Common mistakes when choosing chunking strategies for RAG

The most common mistake is copying a chunk size from a blog post without testing it against your own documents. A 512-token chunk works for some datasets and destroys others.

Another mistake is ignoring overlap. If you split on fixed boundaries without overlap, you'll cut sentences and lose context at every boundary. A 10-15% overlap is cheap insurance.

The third mistake is over-engineering. Teams pick LLM-based chunking because it sounds sophisticated, then discover the indexing cost makes the whole pipeline impractical. Start simple. Add complexity only when retrieval quality demands it.

Chunk Size and Overlap: What Actually Works

There is no universal optimal chunk size. It depends on your embedding model, your document type, and what you're retrieving for. Anyone quoting a single magic number is guessing.

Typical chunk size ranges and their tradeoffs

Small chunks, 128 to 256 tokens, give you precise retrieval. A query matches a tight, focused passage. The catch: you lose surrounding context. A pronoun or a reference to "the previous section" becomes meaningless when that section sits in a different chunk.

Large chunks, 512 to 1024 tokens, preserve more context. The model sees full paragraphs and arguments. The tradeoff is precision. A 1,000-token chunk contains many ideas, so a query matching one sentence pulls in nine sentences of noise. Embedding models also dilute: the vector represents an average of everything in the chunk, not any single point.

Mid-range chunks, 256 to 512 tokens, are the practical default for most RAG pipelines. They balance context against precision well enough that you can start there and tune.

How overlap preserves context at chunk boundaries

Overlap means each chunk shares a portion of text with the chunk before and after it. If you split a 500-token document into 200-token chunks with 20 tokens of overlap, the boundary sentence appears in both chunks. That prevents a thought from being cut in half and lost entirely.

A 10-15% overlap is the standard starting point. More than that wastes tokens and duplicates retrieval results. Less than that risks cutting sentences at boundaries. Fixed-size chunking needs overlap most. Structure-based chunking, which splits on natural boundaries like headings, often needs little or none.

Starting points for common use cases

Start with 256-token chunks and 10% overlap for general-purpose RAG over prose. For technical documentation with clear headings, use structure-based chunking and skip overlap. For agent memory, use hierarchical chunks: parents at 500-800 tokens, children at 100-200. Test against your own queries before you trust any of these numbers.

Final Thoughts on Chunking Strategies for RAG

No single chunking strategy wins. Fixed-size is simple but breaks sentences. Semantic is smart but slow. Structure-based respects documents but fails on plain text. LLM-based adds context but costs real money per chunk.

The honest answer: test with your own data. Run three strategies, measure retrieval quality on your actual queries, and keep the one that works. Vendor benchmarks won't tell you anything about your documents.

Agent memory adds one more constraint. Chunks that work for a search pipeline may fragment narrative context across sessions. Hierarchical chunking helps, but you'll still need to tune sizes for your agent's working memory.

GigaRAG handles chunking for both RAG pipelines and agent memory, so you can test chunking strategies for RAG without building the plumbing yourself. But the decision still belongs to your data.

Frequently Asked Questions

Which chunking strategy is best?

There is no single best strategy; it depends on your data and use case. For general text, fixed-size with overlap works well as a baseline. For complex documents, semantic or recursive chunking often yields better retrieval precision. For agent memory, hierarchical chunking with summaries is recommended.

What is the optimal chunk size for RAG?

A common starting point is 256–512 tokens, but the optimal size varies. Smaller chunks improve precision but may lack context; larger chunks provide more context but can introduce noise. Experiment with sizes like 256, 512, and 1024 tokens and measure retrieval performance.

What is a chunking strategy?

A chunking strategy is the method used to split documents into smaller pieces (chunks) for embedding and retrieval in a RAG pipeline. It determines chunk boundaries, size, and overlap to balance retrieval precision and context completeness.

Can you give me an example of chunking techniques?

Common techniques include fixed-size chunking (splitting by token count), recursive chunking (splitting by paragraphs then sentences), semantic chunking (using embeddings to detect topic shifts), and hierarchical chunking (creating parent-child relationships). Each has trade-offs in complexity and performance.

How does chunking affect agent memory?

Chunking directly impacts how an agent stores and retrieves long-term memories. Hierarchical chunking with summaries allows efficient storage and retrieval of both detailed and high-level information. Poor chunking can lead to fragmented memories and degraded agent performance.

When does fixed-size chunking fail?

Fixed-size chunking fails when it splits sentences, lists, or code blocks, breaking semantic units. It also performs poorly on documents with varying structure, such as mixed text and tables. In such cases, semantic or recursive chunking is more effective.

About GigaRAG

GigaRAG helps GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through chunking strategies for rag or something adjacent, we publish what we have actually tested, including where it falls short.