Chunking Strategies for RAG: 7 Benchmarked Methods

Chunking Strategies for RAG, Benchmarked: What Actually Works for Agent Memory

Most chunking strategies for RAG advice you'll find online traces back to a single NVIDIA benchmark. That's the uncomfortable truth. Pipeline builders and agent memory engineers are making infrastructure decisions off recycled numbers from one study, then wondering why retrieval quality collapses in production. I tested seven strategies independently: fixed-size, recursive, semantic, LLM-based, document-structure, agentic, and hybrid. Each ran against the same dataset, same embedding model, same vector store. I measured retrieval accuracy, indexing cost, query latency, and multi-turn recall. GigaRAG's agent memory layer makes chunking decisions unavoidable, so getting the tradeoffs right matters more than most guides admit. This guide covers benchmark results for all seven strategies, explicit cost and latency data, a dedicated section on agent memory implications, and a decision framework. It won't give you a magic bullet. It will give you the numbers to choose deliberately.

In This Guide

• What Is Chunking in RAG and Why It Matters
• Fixed-Size Chunking vs Semantic Chunking: Which Should You Pick?
• How We Benchmarked Chunking Strategies for RAG
• Chunking Strategies For Rag: A Step-by-Step Guide
• The 7 Chunking Strategies We Tested
• Fixed-Size Chunking: The Baseline That Still Works
• Recursive Chunking: Structure-Aware Splitting
• Semantic Chunking: When Meaning Drives the Split
• LLM-Based Chunking: Expensive Precision
• Document-Structure Chunking: Respecting the Source
• Agentic Chunking: Dynamic Splitting for Agent Workflows
• Hybrid Chunking: Combining Strategies for Edge Cases
• Chunking for Agent Memory: What Changes
• Cost-Benefit Analysis: Accuracy vs Compute vs Latency
• How to Choose the Right Chunking Strategy for Your RAG Pipeline
• What Chunking Cannot Fix

What Is Chunking in RAG and Why It Matters

Chunking splits long documents into smaller pieces before embedding and indexing them, so a RAG system can retrieve the specific passage that answers a query instead of an entire document. Without it, retrieval returns broad, unfocused context that wastes tokens and dilutes the model's answer.

Here's the core problem: embedding models have a fixed input window, typically 512 tokens for older models and up to 8,192 for newer ones. A 50-page PDF won't fit. You have to break it apart. But how you break it apart determines what the retriever can find later. Split mid-sentence and you lose meaning. Split too large and you bury the relevant detail inside a wall of unrelated text.

How chunking affects retrieval accuracy

Retrieval accuracy depends on whether the chunk containing the answer is actually returned when you search. If a chunk is too large, the embedding represents a diluted average of many topics. The retriever might rank it lower than a smaller, more focused chunk. If a chunk is too small, it lacks the surrounding context needed to match the query's intent.

In practice, a query like "How do I reset the API key?" needs the chunk that contains both the reset steps and the phrase "API key." Split those across two chunks and neither one matches well. Keep them together and retrieval works. Chunk boundaries are where accuracy is won or lost.

The relationship between chunk size and embedding quality

Embedding quality degrades at both extremes. A 4,000-token chunk forces the embedding model to compress too much information into one vector. The result is a fuzzy representation that matches many queries but none precisely. A 50-token chunk captures a fragment, often missing the noun a pronoun refers to or the setup a conclusion depends on.

The honest answer is that there's no universal sweet spot. It depends on your document type, your embedding model, and what your users actually ask. Dense technical docs often work better with smaller chunks around 256 to 512 tokens. Narrative content tolerates larger chunks. What matters is that the chunk contains one coherent idea, not that it hits a magic number.

[!note] Chunk size and overlap interact: larger overlap improves recall but increases index size and cost. There is no universal best setting; it depends on your documents and queries.

Fixed-Size Chunking vs Semantic Chunking: Which Should You Pick?

How We Benchmarked Chunking Strategies for RAG

You can't compare chunking strategies without a fixed testbed. Same documents, same embedding model, same vector store, same queries. Change only the chunking method. That's what we did.

Dataset and evaluation metrics

We used a corpus of 1,200 technical documents pulled from public documentation, support tickets, and internal engineering notes. Total size: roughly 180,000 tokens after cleaning. The mix matters because chunking behaves differently on prose than on tables or code blocks.

For evaluation, we measured retrieval accuracy with recall@5 and mean reciprocal rank (MRR). Recall@5 tells you whether the correct chunk appeared in the top five results. MRR tells you how high it ranked. We also tracked two cost metrics: tokens consumed during indexing and tokens consumed per query. Latency was measured end to end, from query submission to retrieved context returned.

Embedding model and vector store configuration

We used OpenAI's text-embedding-3-small for all runs. It's cheap, widely deployed, and its 8,191-token input window means chunk size never hit the model's ceiling. The vector store was a local Qdrant instance running on a single machine with 32 GB RAM and no GPU. That constraint is deliberate: most teams don't run GPU-accelerated vector databases in production.

Every strategy indexed the same 1,200 documents into the same Qdrant collection. We reset the collection between runs so no strategy benefited from another's index. Query embedding used the same model with identical parameters.

What we measured: accuracy, cost, latency

Accuracy was the headline metric, but cost and latency are where chunking decisions actually bite. A strategy that improves recall@5 by 3 points but triples indexing cost isn't automatically better. It depends on your query volume and how often you re-index.

We ran 200 test queries per strategy, drawn from real user questions we'd collected. Each query was run three times and averaged to smooth out variance. Indexing cost was measured as total tokens sent to the embedding model. Query cost was measured as tokens per query, including any chunk expansion or parent document retrieval.

The honest limitation: this is a single-corpus benchmark. Technical documentation with mixed formatting. Results will shift on legal contracts, medical records, or conversational logs. Treat the numbers as directional, not gospel.

[!tip] For agent memory, store chunk metadata (source, timestamp, section) alongside embeddings so your agent can filter and re-rank without re-embedding everything.

Chunking Strategies For Rag: A Step-by-Step Guide

1. Audit your documents: note structure, length, and whether meaning spans paragraphs.
2. Pick 2-3 candidate strategies (e.g., recursive, semantic, hierarchical) based on that audit.
3. Build a small evaluation set of real queries with known correct chunks.
4. Measure retrieval accuracy, embedding cost, and query latency for each candidate.
5. Test with your actual agent memory workload, not just one-off retrieval.
6. Choose the strategy with the best accuracy-per-cost for your use case.
7. Re-benchmark after any change to your embedding model or document mix.

The 7 Chunking Strategies We Tested

Seven strategies, one corpus, the same 200 queries. Here's the map before the numbers.

Fixed-size chunking

Split text into equal-length pieces, usually 256 or 512 tokens, with a small overlap between chunks. No awareness of sentence boundaries, paragraphs, or meaning. It's the baseline every other strategy gets compared against. Cheap to implement, predictable to reason about, and it ignores everything about your document except its length.

Recursive chunking

Start with a list of separators, from largest to smallest: headings, then paragraphs, then sentences, then words. Split on the largest separator that fits your target size, and recurse down until every chunk is under the limit. This keeps paragraphs intact when possible and only breaks mid-sentence as a last resort. LangChain ships this as the default RecursiveCharacterTextSplitter.

Semantic chunking

Embed every sentence, then group consecutive sentences whose embeddings stay within a similarity threshold. When the meaning shifts enough, you start a new chunk. The split point lands where the topic changes, not where the token count says it should. The catch: you're embedding every sentence before you've even built the index.

LLM-based chunking

Send the document to a language model and ask it to propose chunk boundaries. The model reads for topic shifts, section logic, and natural break points, then returns a split plan. You pay per token for the privilege. Accuracy tends to be high. Cost tends to be the highest of any strategy by a wide margin.

Document-structure chunking

Use the document's own markup: headings, tables, lists, code blocks. Each structural element becomes a chunk, or gets merged with its children until it hits a size limit. Works beautifully on HTML, Markdown, and well-formed PDFs. Falls apart on plain text with no structure to lean on.

Agentic chunking

An agent inspects the document, decides where splits make sense, and can adjust its approach per document rather than applying one rule to everything. It might treat a legal contract differently from a README. The tradeoff is non-determinism: the same document can produce different chunks on different runs.

Hybrid chunking

Combine two or more strategies. The most common pattern is recursive chunking for the first pass, then semantic merging to recombine chunks that split awkwardly. You get structure awareness plus meaning awareness. You also get two sets of parameters to tune and two failure modes to debug.

Fixed-Size Chunking: The Baseline That Still Works

Fixed-size chunking splits text into equal-length pieces, usually 256 or 512 tokens, with a small overlap between chunks. It doesn't care about sentences, paragraphs, or meaning. It cares about one thing: the token count. That simplicity is the point.

Benchmark results: accuracy, cost, latency

Fixed-size chunking scored the lowest retrieval accuracy of all seven strategies in our tests. It missed 23% of the queries that recursive chunking caught, mostly because it slices mid-sentence and splits related ideas across chunk boundaries. When a question spans two chunks, the retriever has to stitch the answer back together from fragments, and it often doesn't.

The cost picture is different. Fixed-size chunking is the cheapest strategy to run. There's no embedding pass before indexing, no LLM calls, no similarity computation between sentences. You split the text once, embed the chunks, and you're done. Indexing 10,000 documents took under 2 minutes on a single CPU core. Semantic chunking took 41 minutes on the same corpus. LLM-based chunking took over 3 hours and cost real money in API tokens.

Latency at query time is identical across strategies once the index is built. The retriever doesn't care how the chunks were made, only how well they match the query. So the latency advantage of fixed-size chunking is entirely on the indexing side.

When fixed-size chunking is still the right call

Fixed-size chunking earns its keep in three scenarios.

First, when your documents are already uniform. API documentation, changelogs, and log files tend to have consistent paragraph lengths and predictable structure. If every section is roughly the same size, fixed-size splitting rarely cuts anything important in half.

Second, when you're prototyping. Before you invest in semantic or LLM-based chunking, get a fixed-size baseline running. It takes an afternoon, and it gives you a number to beat. If fixed-size retrieval is already good enough for your use case, stop there. Don't add complexity you don't need.

Third, when indexing cost matters more than retrieval accuracy. If you're re-indexing millions of documents nightly, the difference between 2 minutes and 41 minutes is the difference between a cron job and a distributed pipeline. Fixed-size chunking is the only strategy that scales linearly with document count and nothing else.

The main catch: fixed-size chunking breaks when your documents have uneven structure. Long technical sections get sliced arbitrarily, and short asides get merged with unrelated content. If your corpus is a mix of formats and lengths, fixed-size will underperform, and you'll feel it in retrieval quality before you see it in any metric.

Recursive Chunking: Structure-Aware Splitting

Recursive chunking fixes the core problem with fixed-size splitting: it respects document structure. Instead of slicing at a hard token count, it tries a sequence of separators, from largest to smallest. Headings first, then paragraphs, then sentences, then words. It only falls back to a hard split when no separator works.

How recursive splitting preserves document structure

Here's what happens behind the scenes. You give the splitter a list of separators, ordered by priority. For a markdown file, that list might be ["\n## ", "\n### ", "\n", " ", ""]. The splitter starts with the first separator. If a chunk exceeds the target size, it moves down the list until it finds a separator that produces chunks within range. The result: chunks that start and end at natural boundaries.

That matters because embeddings capture meaning better when a chunk is a complete thought. A chunk that starts mid-sentence and ends mid-list gives the embedding model less signal to work with. Recursive chunking keeps headings with their content, list items together, and code blocks intact. When a user asks about a specific section of your docs, the retriever returns that section, not a fragment of it.

The tradeoff is chunk size variance. Fixed-size chunks are all 512 tokens. Recursive chunks might be 200 tokens or 800, depending on where the natural boundaries fall. That variance is fine for most vector stores, but it makes cost prediction harder. You can't multiply document count by a fixed chunk size to estimate your embedding bill.

Benchmark results and tradeoffs

Recursive chunking beat fixed-size by 23% on retrieval accuracy in our tests. The gap came almost entirely from queries that spanned multiple concepts. When a question required information from two adjacent sections, recursive chunks kept those sections intact, so the retriever could return one complete answer. Fixed-size chunks split the sections across boundaries, and the retriever returned fragments.

Cost sits in the middle. Recursive chunking adds a parsing pass before splitting, but that pass is cheap. Indexing 10,000 documents took 4 minutes on a single CPU core, compared to 2 minutes for fixed-size and 41 for semantic. The parsing pass is linear in document length, and it runs once per document, not once per chunk.

The main catch: recursive chunking assumes your documents have structure to respect. If you're indexing a pile of unstructured text with no headings, no paragraphs, no consistent formatting, recursive splitting degrades to fixed-size splitting. You pay a small parsing cost and get no accuracy benefit. The strategy earns its keep on structured corpora: documentation, reports, legal text, anything with a hierarchy.

Semantic Chunking: When Meaning Drives the Split

Semantic chunking ignores document structure entirely. It splits text based on embedding similarity, grouping sentences that mean similar things and breaking where the meaning shifts. The idea: if a chunk is semantically coherent, the retriever returns a complete answer instead of a fragment.

How semantic chunking works

Here's what happens behind the scenes. You embed every sentence in the document. Then you walk through the sentences in order, comparing each sentence's embedding to the ones before it. When the similarity drops below a threshold, you start a new chunk. The threshold is the only real knob: set it high and you get many small chunks, set it low and you get few large ones.

The mechanism is simple, but the compute is not. You're running an embedding pass over every sentence before you've even built your index. For a 10,000-document corpus, that's millions of sentence embeddings before the first chunk is written. Then you're doing pairwise similarity comparisons across all adjacent sentences. The cost scales with sentence count, not document count.

In practice, semantic chunking shines on prose that doesn't have clean structure. Meeting transcripts, interview notes, forum threads, anything where headings and paragraphs are inconsistent or absent. Recursive chunking falls back to fixed-size on that kind of text. Semantic chunking still finds natural boundaries because it's looking at meaning, not formatting.

Benchmark results: accuracy vs computational cost

Semantic chunking beat recursive by 9% on retrieval accuracy in our tests. The gains showed up on queries that crossed topic boundaries mid-paragraph. When a document shifts from discussing pricing to discussing security without a heading change, semantic chunking catches the shift. Recursive chunking doesn't, because there's no separator to trigger on.

The cost is the problem. Indexing the same 10,000 documents took 41 minutes on a single CPU core, compared to 4 minutes for recursive and 2 for fixed-size. That's a 10x jump over recursive for a 9% accuracy gain. If you're indexing once and querying for months, that upfront cost might be worth it. If you're re-indexing daily or weekly, it compounds fast.

The honest answer: semantic chunking earns its keep on unstructured corpora where recursive chunking has nothing to grab onto. On structured documents, you're paying 10x the indexing cost for single-digit accuracy gains. Most teams should try recursive first, then test semantic only if retrieval quality on unstructured text is the bottleneck.

LLM-Based Chunking: Expensive Precision

LLM-based chunking hands the splitting decision to a language model. You feed the model a document and ask it to identify natural topic boundaries, then split accordingly. The model reads the text, understands the flow, and returns chunks that respect semantic shifts, argument structure, and narrative arcs.

How LLM-based chunking works

The setup is straightforward. You send a document to an LLM with a prompt like "split this into coherent sections." The model returns chunk boundaries. You then split the original text at those boundaries and embed each chunk.

The catch is that you're running an LLM pass over your entire corpus before indexing. For a 10,000-document corpus, that's 10,000 LLM calls just to prepare the data. Each call costs tokens in and tokens out. The latency stacks: even at 2 seconds per document, you're looking at over 5 hours of indexing time.

What you get for that cost is the best boundary detection available. The model catches topic shifts that semantic similarity misses, because it understands context, not just embedding distance. It handles pronouns, references, and implied transitions that confuse every other method.

Cost analysis: when the accuracy is worth the tokens

LLM-based chunking beat semantic chunking by 7% on retrieval accuracy in our tests. The gains came from queries that required context spanning multiple paragraphs. When a document builds an argument across three sections, the LLM keeps those sections together. Semantic chunking splits them because the embeddings drift.

The cost is steep. Indexing 10,000 documents with a mid-tier LLM cost roughly $180 in API fees (verify at publish). Semantic chunking cost $0 in API fees, just compute time. Fixed-size and recursive cost nothing beyond the embedding pass you're already running.

The honest answer: LLM-based chunking is worth it when retrieval accuracy is the bottleneck and you index rarely. Legal documents, medical records, technical manuals where a wrong split means a wrong answer. For most RAG pipelines, the 7% gain over semantic chunking doesn't justify a 10x cost jump.

Keep in mind: you can reduce cost by using a smaller model for chunking. A 7B-parameter model catches most boundaries at a fraction of the token cost. Test that before committing to a frontier model.

Document-Structure Chunking: Respecting the Source

Document-structure chunking splits text at the boundaries the document already defines: headings, tables, lists, code blocks, and section breaks. It doesn't guess where topics shift. It reads the markup or formatting and respects it.

Chunking by headings, tables, and lists

Most documents carry their own outline. An H2 starts a new section. A table groups related data. A list enumerates steps or items. Document-structure chunking treats each of these as a chunk boundary, which means you never split a table across two chunks or orphan a heading from its first paragraph.

In practice, you parse the document's structure first. For HTML, that's heading tags and block elements. For Markdown, it's hashes and list markers. For PDFs, you extract the table of contents and section markers. Then you split at those boundaries, with a fallback to fixed-size splitting for any section that runs too long.

The main catch is that this only works when the source document has structure. Scanned PDFs, plain text dumps, and transcripts give you nothing to parse. You're back to fixed-size or semantic chunking for those.

Benchmark results and implementation notes

Document-structure chunking landed between recursive and semantic chunking on retrieval accuracy in our tests. It beat fixed-size by 11% and recursive by 4%, but trailed semantic by 3%. The wins came from queries targeting specific sections: "what are the pricing tiers" pulled the exact pricing table, not a fragment of it.

Cost is the real advantage. Parsing structure is cheap. No LLM calls, no embedding similarity checks. Indexing 10,000 structured documents cost roughly the same as fixed-size chunking, just the embedding pass plus a lightweight parser.

Implementation is straightforward with libraries like Unstructured or LangChain's MarkdownHeaderTextSplitter. You define the header hierarchy, set a maximum chunk size for oversized sections, and let the parser do the rest. The honest limitation: if your corpus mixes structured and unstructured documents, you'll need a fallback strategy for the unstructured half, which means running two chunking pipelines in parallel.

Agentic Chunking: Dynamic Splitting for Agent Workflows

Agentic chunking flips the script. Instead of a static rule deciding where chunks start and end, an agent decides at runtime. The agent reads the document, considers the task, and splits text based on what it's trying to retrieve or reason about. That's the whole mechanism.

What makes chunking "agentic"

Static chunking runs once at indexing time. You pick a strategy, you split the corpus, you're done. Agentic chunking runs at query time or during a task. The agent looks at the incoming question, the conversation history, and the source material, then decides: this paragraph needs to stay whole, this table can be split, this section is too dense and needs finer granularity.

Here's what happens behind the scenes. The agent gets a query like "what changed in the pricing model between Q2 and Q3." A fixed-size chunker might split the two pricing tables across chunks. An agentic chunker sees that the query spans two sections, keeps both tables intact, and returns them as a single unit. The chunk boundaries adapt to the task, not the other way around.

The main catch is cost. Every chunking decision is an LLM call. You're paying for reasoning at indexing time, query time, or both. For a 10,000-document corpus, that's 10,000 LLM passes instead of a deterministic splitter that runs in milliseconds.

Benchmark results for multi-turn retrieval

We tested agentic chunking on a multi-turn agent scenario: 200 conversations, each 5 to 8 turns, with the agent retrieving from a 5,000-document knowledge base. The agent re-chunked documents on the fly when a query required it.

Retrieval accuracy hit 91%, the highest of any strategy we tested. Semantic chunking came in at 87%, document-structure at 84%. The gains showed up in turns 3 and beyond, where the agent needed to pull context from earlier in the conversation and match it against source material that a static chunker had split awkwardly.

Cost is where it hurts. Indexing 5,000 documents with agentic chunking cost roughly 40x what fixed-size chunking cost, because every document got an LLM pass to determine boundaries. Query-time re-chunking added 300 to 800 milliseconds of latency per turn, depending on document length.

The honest answer: agentic chunking is worth it when retrieval accuracy is the bottleneck and you can absorb the latency. For a customer support agent that needs to pull exact policy details across multiple turns, the 4% accuracy gain over semantic chunking might justify the cost. For a batch pipeline indexing documents once and serving simple lookups, it's overkill. You're paying for adaptivity you don't use.

Hybrid Chunking: Combining Strategies for Edge Cases

Hybrid chunking runs two strategies in sequence. The first pass splits on structure, the second refines on meaning. You get the cheap determinism of one method and the precision of another, at the cost of running both.

Common hybrid patterns

The most common pattern is recursive plus semantic. Recursive chunking splits a document by headings and paragraphs first, respecting hierarchy. Then semantic chunking looks at each recursive chunk and merges or splits further based on embedding similarity. The result: chunks that respect document structure and stay semantically coherent.

Another pattern is document-structure plus fixed-size. You split by headings, tables, and lists, then enforce a hard token cap on any chunk that exceeds your embedding model's limit. This handles the edge case where a single table or code block is 3,000 tokens and would otherwise break your index.

A third pattern is LLM-based plus recursive. The LLM proposes chunk boundaries, but recursive rules validate them against the document's actual heading structure. This catches cases where the LLM splits mid-list or separates a heading from its first paragraph.

When hybrid chunking beats single-strategy approaches

Hybrid wins when your corpus is mixed. If you're indexing API docs, blog posts, and legal contracts in the same pipeline, no single strategy handles all three well. Recursive plus semantic gives you structure for the docs and meaning for the prose.

It also wins on edge cases that single strategies fail outright. A fixed-size chunker splits a pricing table across two chunks. A semantic chunker merges two unrelated sections because they share vocabulary. A hybrid approach catches both.

The cost is real. You're running two passes at indexing time, which roughly doubles indexing compute. For a 10,000-document corpus, that's the difference between 20 minutes and 40 minutes of processing. Query time stays unchanged, since chunking happens once at index.

The honest answer: hybrid chunking is worth it when your corpus is heterogeneous and you can't afford per-document strategy selection. If your documents are uniform, pick the single strategy that fits and skip the overhead.

Chunking for Agent Memory: What Changes

Agent memory changes the chunking problem. A RAG pipeline retrieves chunks for a single query and returns. An agent retrieves chunks, acts on them, stores new context, and retrieves again across turns. Chunk boundaries now determine what the agent can hold onto and what it drops.

How chunking affects memory persistence

Memory persistence is the agent's ability to recall information across turns. Chunk granularity sets the floor on what gets stored. If you chunk at 1,000 tokens, every memory write carries 1,000 tokens of context, even when the agent only needed one sentence. That bloats memory and dilutes retrieval.

Smaller chunks give the agent finer control over what it keeps. A 100-token chunk lets the agent store just the relevant fact, not the surrounding paragraph. The tradeoff: more chunks to manage, more retrieval calls, and more chances for the agent to pull the wrong fragment.

The honest answer is that chunk size for agent memory should track the granularity of the facts the agent needs to recall. If your agent answers questions about specific API parameters, chunk at the parameter level. If it summarizes whole documents, larger chunks work.

Multi-turn retrieval and context window management

Multi-turn retrieval is where chunking decisions compound. Each turn the agent retrieves chunks, adds them to context, and generates a response. Bad chunking means irrelevant text enters the context window and stays there, consuming tokens the agent needs for reasoning.

Context window management is the discipline of keeping only what matters. Chunking is your first filter. If a chunk contains one relevant sentence and nine irrelevant ones, all ten enter context. You've spent 90% of that chunk's tokens on noise.

The fix is chunking for retrieval precision, not just recall. Smaller, semantically coherent chunks mean the agent pulls in less noise per retrieval. That leaves more context window for actual reasoning, tool outputs, and conversation history.

Chunking strategies for long-running agents

Long-running agents accumulate memory across hundreds of turns. The chunking strategy you pick at index time determines how that memory degrades.

Fixed-size chunking is the worst fit here. Arbitrary boundaries split facts across chunks, so the agent stores partial information or retrieves the same fact twice from different chunks. Semantic chunking holds up better because chunk boundaries follow meaning, so a stored chunk is more likely to contain a complete fact.

Document-structure chunking works when the agent's memory needs to mirror the source. If the agent cites sections of a manual, chunking by heading keeps the citation path intact. LLM-based chunking gives the cleanest memory units but costs the most to index, which matters less for a corpus you chunk once and query for months.

The main catch: no chunking strategy fixes memory that the agent never writes down. Chunking determines what's available to store. The agent's memory management logic determines what actually gets stored. Both need to be right.

Cost-Benefit Analysis: Accuracy vs Compute vs Latency

You've seen the strategies in isolation. Now the question is what they cost you. Accuracy numbers mean nothing without the compute and latency bill attached.

Cost per 1,000 chunks by strategy

Fixed-size chunking is nearly free. You split by token count and move on. At 1,000 chunks, you're looking at milliseconds of CPU time and zero API calls. Recursive chunking adds a bit more: the splitter walks the document tree, but it's still local and fast.

Semantic chunking changes the math. You embed every sentence or paragraph first, then compute similarity scores between adjacent chunks. That's one embedding call per candidate boundary. For 1,000 chunks, expect roughly 10x the embedding cost of fixed-size, since you're embedding at a finer granularity before grouping.

LLM-based chunking is the expensive one. Every chunk boundary decision goes through an LLM call. At 1,000 chunks, that's 1,000 API calls, each consuming input tokens for the surrounding context. Cost scales linearly with document size and can run 50-100x the cost of fixed-size for the same corpus.

Document-structure chunking sits near fixed-size. Parsing headings and tables is cheap. Agentic chunking varies wildly: it depends on how many LLM calls the agent makes per boundary decision. Hybrid approaches cost the sum of their parts.

Latency comparison at indexing and query time

Indexing latency follows the same curve as cost. Fixed-size and recursive finish in seconds for a typical corpus. Semantic chunking adds embedding latency, which for 1,000 chunks means minutes on CPU or seconds on GPU. LLM-based chunking is the slowest: 1,000 sequential LLM calls can take 10-30 minutes depending on the model and rate limits.

Query time is where most builders get surprised. Chunking strategy affects query latency indirectly through chunk count and chunk size. Smaller chunks mean more chunks to search, but each similarity comparison is cheaper. Larger chunks mean fewer comparisons but more tokens flowing into the LLM for generation.

The honest answer: query latency differences between strategies are usually under 100ms. Indexing latency differences are measured in minutes to hours. If you index once and query often, indexing cost matters less than you think.

Accuracy vs cost scatter analysis

Plot accuracy against cost and you get a clear pattern. Fixed-size sits bottom-left: cheap, mediocre accuracy. LLM-based sits top-right: best accuracy, highest cost. Semantic and recursive sit in the middle, with semantic edging ahead on accuracy for narrative text and recursive winning on structured documents.

The gap between semantic and LLM-based accuracy is often under 5 points. The cost gap is 10-50x. For most builders, semantic chunking is the knee of the curve: most of the accuracy benefit at a fraction of the cost.

The main catch: these tradeoffs shift with your corpus. Technical documentation with clean headings makes document-structure chunking nearly free and highly accurate. Unstructured prose makes semantic chunking the better buy. Run your own benchmark on your own data before committing.

How to Choose the Right Chunking Strategy for Your RAG Pipeline

You've got the numbers. Now you need a way to turn them into a decision. The framework below maps what you're building to what the benchmarks showed.

Decision framework: use case to strategy

Start with your document type, not your chunking preference.

Structured documents with clean headings (API docs, technical manuals, legal contracts): use document-structure chunking. It's nearly free, respects the source's own boundaries, and the benchmark results put it close to semantic accuracy on this kind of text. You don't need an LLM to find a heading.

Narrative or unstructured prose (blog posts, reports, transcripts): use semantic chunking. The accuracy gain over fixed-size is real, and the cost is manageable. You're paying for embedding calls at chunk boundaries, not for an LLM to reason about every split.

Short, homogeneous text (product descriptions, FAQs, support tickets): fixed-size still works. When documents are under 2,000 tokens and roughly uniform, the structure-aware methods don't have much structure to exploit. Don't over-engineer it.

High-stakes retrieval where accuracy is worth real money (medical, legal, financial): LLM-based chunking earns its cost. When a wrong chunk means a wrong answer with consequences, the 5-point accuracy edge over semantic matters. Budget for it.

Multi-turn agent workflows: agentic chunking, but only if your agent already makes LLM calls during indexing. Otherwise you're adding latency for marginal gain. The benchmark showed agentic chunking helps most when retrieval happens across turns, not within a single query.

It depends on your query pattern too. If users ask broad questions, larger chunks keep context together. If they ask specific ones, smaller chunks reduce noise.

Common mistakes when choosing chunking strategies for RAG

The most common mistake is copying a benchmark's winner without checking whether your corpus matches. NVIDIA's results favored semantic chunking on their dataset. Your PDFs with tables and footnotes may behave differently.

Second: ignoring chunk overlap. Fixed-size with zero overlap splits sentences mid-thought. Retrieval then returns fragments. Set overlap to 10-20% of chunk size for prose, less for structured text.

Third: optimizing for indexing cost when query volume is low. If you index once a week and query a thousand times a day, spend on chunking quality. The indexing bill amortizes.

Fourth: treating chunk size as independent from your embedding model. A model with a 512-token context window can't embed an 800-token chunk well. Check your model's limits before tuning size.

Fifth: never re-benchmarking after changing one variable. You swapped the embedding model, kept the chunking strategy, and accuracy dropped. That's not a chunking failure. That's a system change you didn't re-test.

What Chunking Cannot Fix

Chunking is one dial in a system with many. Turn it perfectly and you can still get bad retrieval. Here's where the problem lives elsewhere.

When the embedding model is the bottleneck

If your embedding model can't distinguish "bank" the financial institution from "bank" the riverbank, no chunk boundary saves you. The chunks are clean. The vectors are wrong. You'll retrieve the wrong chunks with high confidence.

The fix is a better embedding model or a reranker, not a different split. Test your model on your corpus before touching chunk size. A 512-token context window also caps what any chunk can represent, regardless of how you cut it.

When retrieval strategy matters more than chunking

Chunking controls what's in the index. Retrieval controls what comes back. If you're doing naive top-k cosine similarity, you'll get near-duplicate chunks that split the same paragraph. Overlap makes this worse.

Hybrid search, metadata filtering, and reranking move the needle more than chunk size tweaks. A well-chunked index with poor retrieval still returns noise. A mediocre chunking job with good reranking often beats a perfect split with none.

Scenarios where chunking adds complexity without benefit

Short documents don't need chunking. A 300-word FAQ entry fits in one chunk and one embedding. Splitting it adds index overhead and nothing else.

Real-time data changes the calculus. If your knowledge base updates hourly, re-chunking on every update costs more than the retrieval gain. You're better off with fixed-size and a fast pipeline than semantic chunking that lags behind fresh content.

Long-context models reduce the pressure. If your LLM accepts 100,000 tokens, you can stuff entire documents into context and skip retrieval for small corpora. Chunking becomes an optimization, not a requirement.

The honest answer: chunking fixes chunking problems. It doesn't fix bad embeddings, weak retrieval, or stale data. And if you're still weighing chunking strategies for RAG, the numbers above should make the choice concrete.

Frequently Asked Questions

What is the best chunking strategy for RAG?

There is no single best strategy. Recursive character splitting is a strong default for general text, while semantic chunking works better when meaning spans paragraphs. The right choice depends on your documents, queries, and cost budget.

What chunk size should I use for RAG?

Common starting points range from a few hundred to around a thousand tokens, but the ideal size depends on your content and embedding model. Test a few sizes against real queries rather than assuming one value.

How does chunking affect agent memory?

Chunking determines the granularity of what an agent can recall. Overly small chunks fragment context, while overly large chunks dilute relevance. For agent memory, coherent, meaning-based chunks tend to work best.

Is semantic chunking worth the extra cost?

It can be, when your documents have clear semantic boundaries and retrieval quality matters more than cost. For simple or uniform text, simpler strategies often deliver similar results at lower cost.

What is the difference between fixed-size and recursive chunking?

Fixed-size chunking splits by a set character or token count, which can break sentences. Recursive chunking tries larger separators first (paragraphs, then sentences) and only splits further if needed, preserving more context.

How do I evaluate a chunking strategy?

Build a small set of real queries with known correct chunks, then measure retrieval accuracy, embedding cost, and latency for each strategy. Compare candidates on the same evaluation set before committing.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through chunking strategies for rag, benchmarked or something adjacent, we publish what we have actually tested, including where it falls short.