Chunking Config Cheat Sheet: RAG Settings for Developers
Chunking breaks a corpus into smaller pieces for retrieval-oriented RAG workflows.
Choose chunking settings as a testable configuration, not a fixed recipe. Start with the source formats, the limits of the models you use, and the project questions the system must answer.
Quick answer
Choose a chunking method that matches the source material, establish a simple baseline, and compare retrieved results on representative documents and queries.
- Use fixed-size splitting as a straightforward baseline; test structure-aware splitting where source boundaries matter.
- Use overlap to share text at adjacent chunk boundaries when that boundary context is useful.
- Compare retrieval results and processing cost before adding semantic chunking.
What a chunking config cheat sheet should help you decide
A chunking config cheat sheet organizes the settings you need to test: splitting method, character or token measure, size, overlap, structural boundaries, metadata, and evaluation queries. Use it to document a baseline and compare changes against representative retrieval tasks.
Chunking is not the same as embedding or retrieval
Chunking creates smaller pieces from a document. Treat chunk boundaries, embedding settings, and retrieval settings as separate configuration choices.
The six configuration decisions behind a useful chunk
Choose a method, measurement unit, maximum size, overlap, structural boundaries, and metadata. Then define evaluation criteria for representative queries, such as whether the retrieved material contains the needed information within the context budget. Token-based splitting divides text into chunks containing a selected number of tokens, although a token boundary can still divide a sentence or idea.
What chunking cannot fix
Overlap can carry text across a boundary, but it can't repair a split that separates a function from its needed context. No configuration can replace retrieval evaluation on your own corpus.
Fixed-size vs recursive vs semantic chunking
| Factor | Fixed-size | Recursive or semantic |
|---|---|---|
| How it splits | Divides text by a chosen character or token count. | Semantic chunking divides documents into semantically coherent segments; structure-aware methods can follow source units. |
| Useful starting role | A straightforward baseline for comparison. | A structure-aware option when arbitrary boundaries break useful context. |
| Main limitation | Can cut sentences or ideas at arbitrary boundaries. | Semantic chunking can add computational cost, so compare it with a simpler baseline for the same task. |
| When to tune | When retrieved chunks omit needed context or exceed model limits. | When meaningful sections still split poorly, or added cost is not justified. |
How fixed, recursive, semantic, and document-based chunking work
Compare the chunk boundaries produced by each approach against the documents and retrieval tasks that matter to your application.
[!tip] For engineering knowledge bases, include nested Markdown, API references, tickets, runbooks, and code in your evaluation set so the configuration is tested against the document structures your developers query.
Fixed-size chunking: character versus token boundaries
Fixed-size chunking divides text at a set length without using its content structure. Character splitting is simple and language-agnostic, but it can divide words or sentences. Token-based splitting uses the units language models process, helping you measure chunk length against model context constraints.
Fixed-size chunks can still divide sentences or logical ideas at an arbitrary token boundary. Use the resulting chunks as a baseline when comparing other methods.
Recursive chunking: preferred separators and hard limits
Inspect the boundaries produced by a recursive splitter on representative documents. Keep its configuration only if the resulting chunks serve the structures you need to retrieve.
For mixed prose, retain useful structural information such as document titles and heading paths as metadata when your retrieval workflow needs it.
Semantic chunking: coherent segments and extra processing
Semantic chunking aims to divide documents into semantically coherent segments. Its implementation and processing cost should be evaluated against a simpler baseline for the same corpus and retrieval task.
In Is Semantic Chunking Worth the Computational Cost? by Renyi Qu, Ruixuan Tu, and Forrest Bao, semantic chunking did not show consistent gains over fixed-size chunking across document retrieval, evidence retrieval, and retrieval-based answer generation. Compare the methods on the tasks your system must support before accepting extra processing.
Document-based chunking: headings, functions, tables, records, and metadata
For structured engineering material, define chunk boundaries around units your application needs to retrieve, such as a Markdown section, code symbol, table row, API endpoint, or ticket record. If a unit is too large for the model constraints you have chosen, split it into clearly labeled subsections.
For engineering knowledge, retain metadata that identifies the source unit, such as a parent heading, file path, record ID, or nearby documentation reference. In GigaRAG, teams can make this project knowledge available to agents across sessions and tools through a shared central memory.
Step-by-step: choose and evaluate a chunking setup
- Inspect representative documents, including the formats and structures your application needs to retrieve.
- Check the embedding model's token limit and choose a compatible chunk-size measure, such as tokens or characters.
- Create a simple fixed-size baseline and a structure-aware alternative where document boundaries matter.
- Add overlap only to retain useful context across adjacent chunks. Don't use it as a substitute for better split points.
- Test both setups with representative queries against the actual corpus.
- Compare retrieved evidence, answer quality, processing cost, and token use before choosing a configuration.
Chunking config cheat sheet: starting settings by source type
Use the matrix as a test plan rather than a set of universal defaults. Overlap makes adjacent chunks share boundary text; inspect whether that shared text improves the retrieved material for your corpus.
Quick-reference matrix: method, best fit, starting size, overlap, and warning sign
| Method | Best fit | Starting size | Overlap | Warning sign |
|---|---|---|---|---|
| Recursive | Structured documents to test with configured separators | Set for the selected model and corpus | Test only where boundary context is needed | Inspect whether configured boundaries divide useful sections |
| Document-based | Source-native units chosen by your application | Set for the selected unit and model constraints | Test only where boundary context is needed | An oversized unit needs a clearly labeled split |
| Fixed-size | A simple comparison baseline | Example: 512 tokens | Example: 50 tokens | Sentences or ideas are divided at boundaries |
| Semantic | A method to benchmark against fixed-size splitting | Set a maximum for the selected model and corpus | Test locally | Added processing does not improve the benchmark result |
Markdown, ADRs, and runbooks: split by headings before token limits
Keep each heading with its child text, then recursively split sections that exceed your hard limit. Store the document title and heading path as metadata.
When a procedure is split, inspect whether each resulting chunk still includes the step information needed for the questions your application asks.
API references and structured records: preserve complete fields and endpoints
Make one endpoint, schema record, or table row the first chunking unit. Include its method, path, parameters, response fields, and parent service name.
If an endpoint is too large, split it within clearly named subsections and inspect the resulting chunks alongside representative API questions.
Source code: keep functions, classes, imports, and adjacent documentation intact
For source code, test boundaries based on functions, classes, or other code units alongside a fixed-size baseline. Retain identifying metadata such as file path, symbol name, language, imports, and nearby documentation when those details are useful to retrieval.
For large files, test a hierarchy in which module-level context is represented as metadata or a parent chunk rather than copied into every child chunk.
Tickets, chat exports, and agent memory: preserve task, decision, owner, and time metadata
For tickets and decision threads, test chunk boundaries that retain the task and decision context. Keep issue ID, status, author, date, repository, and linked pull request as metadata when those fields are useful to the workflow.
For long threads, test reply-group boundaries and retain the task and final decision where your retrieval questions need that context. GigaRAG can keep this project knowledge available across sessions and tools, with access controls for what each agent can use.
Chunk size, overlap, token limits, and retrieval trade-offs
[!note] Character counts and token counts are not interchangeable. Token counts depend on the tokenizer, while character-based splitting can cut words or sentences.
Checking smaller chunks for fragmented evidence
When testing smaller chunks, check the results for fragmentation—for example, whether a chunk contains an error description without the neighboring condition or resolution needed for the query.
Comparing larger chunks with smaller chunks
Compare larger chunks with smaller chunks for retrieved relevance and token use, particularly when several retrieved results must fit into the generation context.
What overlap protects and what it makes worse
Overlap makes consecutive chunks share boundary text. This can retain a sentence or procedure that crosses a boundary, so measure its effect on retrieval quality and token use in your own index.
Respect the embedding model's input limit and the generator's context budget
Set chunk size with the context constraints of the embedding and generation models in mind. Token-based splitting can help measure chunk length because language models process text as tokens; evaluate the retrieved set together with the prompt and instructions used by your application.
Benchmark semantic chunking against fixed-size splitting
Use the same corpus, query set, retrieval settings, and answer-generation setup when comparing semantic and fixed-size chunking. Record the retrieved evidence, answer results, processing cost, and token use for each configuration so the decision reflects the trade-offs in your own workflow.
Apply the cheat sheet to an agent or coding workflow
Start with a corpus inventory, not a framework default
List your source types before choosing a splitter: repositories, Markdown, API references, tickets, runbooks, and chat exports. Then pick one baseline method that preserves the units each source already uses.
Attach metadata that agents and retrievers can filter on
Store metadata beside every chunk: repository, file path, heading, language, service, owner, date, task, and version. Use those fields when your retrieval workflow needs to narrow the candidate material. GigaRAG connects agents to a central Hive Mind through MCP or the API, and its access controls can manage what each agent may use.
Test retrieval with real coding and project questions
Use questions your team actually asks, including awkward ones: "Where is this setting validated?" or "Why did we reject this approach?" Inspect the retrieved chunks, not just the final answer.
Tune from observed failures: missed context, noisy matches, slow indexing, or context overflow
Change one variable at a time and compare the retrieved material. If results lack needed context, test a larger structural unit or limited overlap; if results are noisy, test smaller chunks or metadata filters. For slow indexing or context overflow, measure chunk count, top-k, and chunk size.
Final rule: preserve meaningful units before optimizing token counts
Treat a token target as a size constraint, then inspect whether the resulting chunks retain the details needed for representative functions, endpoint records, tables, and decisions.
Frequently asked questions
What chunk size should I start with for a RAG application?
Start with a size that works within the token constraints of the models in your workflow, establish a fixed-size baseline, and evaluate it with your own documents and queries. APXML's Fixed-Size Document Chunking Strategies uses 512 tokens with 50 tokens of overlap as an example, not as a required default.
How much chunk overlap should I use?
Use overlap when sharing text at adjacent boundaries helps preserve material that crosses a split. Test the overlap value with representative queries and compare the retrieved material and token use.
What is the difference between fixed-size and recursive chunking?
Fixed-size chunking divides text by a set character or token count, regardless of content boundaries. Recursive chunking applies configured separators and size rules, so inspect its output to confirm that those rules suit your document structures.
When is semantic chunking worth the extra processing cost?
Consider it when your evaluation shows that simpler splitting loses important semantic context and the added processing is acceptable. In Is Semantic Chunking Worth the Computational Cost?, Renyi Qu, Ruixuan Tu, and Forrest Bao found no consistent performance gains that justified semantic chunking's computational costs.
Should I chunk by characters or tokens?
Token-based splitting aligns chunk size with model token limits, but requires a tokenizer and token counts vary by tokenizer. Character-based splitting is simpler, but may cut words or sentences.
How should I chunk source code for RAG?
Compare code-aware boundaries, such as functions or classes, with a fixed-size baseline. Keep identifying context such as file paths, symbol names, language, imports, and nearby documentation when it helps answer the coding questions your system receives.
How do I avoid splitting tables, API references, and Markdown sections?
Use document-aware boundaries where possible, then split oversized sections further rather than relying only on fixed-length cuts. Evaluate the resulting chunks on your actual API references, tables, and Markdown documents.
How should I evaluate chunking results?
Evaluate retrieved evidence and generated answers together on representative corpus questions. Compare configurations using the documents, queries, processing cost, and token use that matter to your application.
About GigaRAG
GigaRAG is a shared memory platform for AI agents that helps developers and teams keep project knowledge available across sessions and tools.
Connect your agents to a central Hive Mind through MCP or the API, with access controls to manage what each agent can use.
