Chunking Strategies for RAG, Benchmarked: 2026 Data

Chunking Strategies for RAG, Benchmarked

Chunking strategies for RAG, benchmarked across 2024-2026, still leave most pipeline builders guessing. The advice is everywhere and most of it is either vendor-biased or benchmark-thin. One blog tells you fixed-size is dead. Another swears by semantic chunking. Nobody shows you the numbers side by side.

The honest answer is that chunking is not a silver bullet. Retrieval quality depends on your embedding model, your vector store, and your query patterns too. Change any one of those and your best chunking choice changes with it. That's why I consolidated the benchmark data from multiple independent studies into one place, stripped of vendor bias.

GigaRAG simplifies chunking implementation for agent memory and RAG pipelines, but this guide is not a product pitch. It's an explainer with honest limitations, explicit anti-patterns, and a dedicated section on agent memory chunking that no top result covers.

In This Guide

• What Is Chunking in RAG?
• Fixed-Size vs. Semantic Chunking: Which Should You Benchmark First?
• Why Chunking Strategies for RAG Matter More Than You Think
• Chunking Strategies For Rag, Benchmarked: A Step-by-Step Guide
• The 7 Chunking Strategies Benchmarked
• Fixed-Size Chunking: The Baseline You Should Beat
• Recursive Character Splitting: Smarter Boundaries, Same Limits
• Semantic Chunking: When Meaning Drives Boundaries
• LLM-Based Chunking: Powerful but Expensive
• Document-Aware Chunking: Respecting Structure
• Hierarchical Chunking: Multi-Level Retrieval
• Chunking for Agent Memory: What Changes
• How to Choose a Chunking Strategy: A Decision Framework
• Common Mistakes When Chunking Strategies for RAG

What Is Chunking in RAG?

Chunking is the process of splitting a document into smaller pieces before embedding them into a vector database. RAG systems can't index an entire PDF or webpage as one unit. Embedding models have token limits, and retrieval works better when the system can match a query against focused passages rather than whole documents.

Why embedding models need chunks

Embedding models convert text into fixed-length vectors. Most models accept between 512 and 8,192 tokens per input. A 50-page report exceeds that limit. Chunking breaks the report into pieces that fit. But token limits aren't the only reason. Smaller chunks also produce more precise vectors. A vector representing one paragraph captures that paragraph's meaning. A vector representing an entire document averages out every topic it covers, which makes it harder to match against a specific query.

The core tradeoff: context vs. precision

Small chunks retrieve precisely but lose surrounding context. A chunk containing only "the revenue increased 12%" doesn't tell you which quarter, which product line, or which region. Large chunks preserve context but dilute relevance. A 2,000-token chunk may contain the answer to a query buried among unrelated content, which lowers the similarity score and pushes the chunk below the retrieval threshold. There's no perfect size. It depends on your document type, your embedding model, and how your users actually ask questions.

How chunking fits into the RAG pipeline

Chunking sits between document ingestion and embedding. You load a document, split it into chunks, embed each chunk, and store the vectors in a vector database. At query time, the system embeds the user's question, finds the most similar chunks, and feeds them to an LLM as context. Chunk boundaries determine what the LLM can see. If a chunk cuts a key sentence in half, the LLM gets a fragment. If a chunk is too large, the LLM gets noise. The chunking decision shapes every downstream step.

[!note] Chunking strategy interacts strongly with embedding model context limits and reranker behavior — a strategy that wins with one embedding model may lose with another, so benchmarks must be re-run when either changes.

Fixed-Size vs. Semantic Chunking: Which Should You Benchmark First?

Why Chunking Strategies for RAG Matter More Than You Think

Chunking choices swing retrieval accuracy by double-digit percentages. Not a few points. In benchmark runs across 2024-2026, the gap between a naive fixed-size split and a strategy matched to the document type routinely lands between 10 and 25 points on retrieval precision. That's the difference between a system that answers and one that hallucinates.

Benchmark evidence: how much does chunking change results?

Independent tests keep landing on the same finding. When researchers hold the embedding model and vector store constant and change only the chunking strategy, retrieval quality shifts measurably. A Reddit benchmark of seven strategies found semantic and LLM-based chunking outperforming fixed-size splits by double digits on recall. NVIDIA's experimental setup showed similar spread across chunk sizes and overlap settings. The pattern holds across datasets: chunk boundaries matter as much as the model you embed with.

The compounding effect: chunking × embedding × retrieval

Chunking doesn't act alone. It compounds. A bad split produces a noisy vector. A noisy vector ranks lower in similarity search. Lower ranking means the retriever pulls the wrong context. The LLM then answers from bad context and sounds confident doing it. One weak decision at the chunking step degrades every layer downstream. Fix the chunking and you often fix retrieval without touching the model.

What chunking cannot fix

Chunking won't rescue a weak embedding model. It won't compensate for a vector store with poor indexing. It won't make an LLM reason better. If your queries are vague or your documents are poorly written, no chunk boundary saves you. Chunking is a lever, not a cure. Pull it after you've confirmed the rest of the pipeline works.

[!tip] For agent memory engineers: keep a separate short-term buffer chunk (recent turns) and a summarized long-term chunk (compressed history) rather than forcing one chunk size to serve both retrieval and conversational continuity.

Chunking Strategies For Rag, Benchmarked: A Step-by-Step Guide

1. Define a fixed evaluation set of 50–100 real user queries with known relevant passages.
2. Implement 3–4 candidate strategies (fixed-size, recursive, semantic, hierarchical) behind one interface.
3. Hold embeddings, retriever, and reranker constant — change only the chunking layer.
4. Measure retrieval recall@k, answer faithfulness, and end-to-end latency for each strategy.
5. Segment results by document type (PDF, HTML, code, chat) to expose strategy-specific wins.
6. Pick the best performer per segment, then re-run with overlap and chunk-size sweeps.
7. Document anti-patterns observed (e.g., mid-sentence splits) and add regression tests.

The 7 Chunking Strategies Benchmarked

Seven strategies keep showing up in benchmark studies. They range from dead simple to computationally expensive. Here's the full list before we dig into each one.

Fixed-size chunking

You split text into equal-sized pieces, usually by character or token count. A 512-token chunk with 10% overlap is the classic starting point. It's fast, predictable, and requires zero understanding of the document. It's also the baseline every other strategy gets compared against.

Recursive character splitting

You split by a hierarchy of separators: paragraphs first, then sentences, then words. The splitter tries to hit your target size while respecting natural boundaries. LangChain's RecursiveCharacterTextSplitter is the most common implementation. It's still size-based, but the boundaries land in less awkward places.

Semantic chunking

You use embedding similarity to decide where one chunk ends and the next begins. When the semantic distance between two sentences crosses a threshold, you start a new chunk. This keeps related ideas together even if they span different lengths. It costs more compute because you're embedding at split time.

LLM-based chunking

You hand the document to an LLM and ask it to find natural break points. The model reads for topic shifts, argument structure, and section logic. It produces the most human-like chunks. It's also the most expensive option by a wide margin, and it's slow.

Document/structure-aware chunking

You parse the document's actual structure: headings, tables, lists, code blocks. Each structural unit becomes a chunk or a chunk boundary. This matters most for PDFs and HTML where visual layout carries meaning. A table split across two chunks is nearly useless for retrieval.

Hierarchical chunking

You chunk at multiple levels at once: document, section, paragraph, sentence. Retrieval can start coarse and drill down, or start fine and pull parent context. This is the strategy that maps most naturally to agent memory, where you need different granularities for different query types.

Late chunking

You embed the full document first, then pool token-level embeddings into chunks afterward. The embedding model sees the whole context before any split happens. This preserves long-range meaning that gets lost when you chunk before embedding. It requires an embedding model that exposes token-level outputs, which not all do.

That's the landscape. The next sections take each strategy apart: how it works, what benchmarks show, and where it fails.

Fixed-Size Chunking: The Baseline You Should Beat

Fixed-size chunking is the default. You split text into equal pieces by character or token count, usually with some overlap between chunks. It's the first thing every benchmark tests, and for good reason: it's fast, predictable, and needs zero document understanding.

How it works

You pick a size, say 512 tokens, and a stride, say 64 tokens. The splitter walks through the document, cutting at exactly that interval. No parsing. No embeddings. No model calls. A sentence can get chopped mid-thought, and a paragraph can span three chunks. That's the tradeoff you accept for simplicity.

Benchmark results

Fixed-size chunking consistently lands mid-pack. It beats strategies that ignore chunk size entirely, but it loses to semantic and document-aware approaches on retrieval precision. The gap widens on documents with irregular structure: legal contracts, academic papers, anything where a heading or table carries meaning. On clean, uniform text like news articles, it holds up surprisingly well.

When to use it

Use fixed-size when you need a baseline fast. It's the right call for prototyping, for uniform text, and for pipelines where latency matters more than a few points of retrieval accuracy. It's also the easiest strategy to debug, because every chunk is the same shape.

When NOT to use it

Don't use fixed-size on structured documents. A table split across two chunks is nearly useless for retrieval. Don't use it when queries target specific facts buried in dense prose. And don't use it for agent memory, where multi-hop retrieval needs chunks that respect semantic boundaries. Fixed-size is where you start, not where you finish.

Recursive Character Splitting: Smarter Boundaries, Same Limits

Recursive character splitting fixes the worst habit of fixed-size chunking: cutting mid-sentence. It tries a list of separators in order, from broad to narrow. Paragraph breaks first, then sentences, then words. If a chunk fits at a paragraph boundary, it stops there. If not, it drops to the next separator.

How recursive splitting works

You give the splitter a target size and a separator hierarchy. Something like: double newline, single newline, period, space. The splitter walks the text, looking for the most natural break that keeps chunks under the limit. A 512-token chunk ends at the end of a paragraph when possible, not in the middle of a clause.

Benchmark performance vs. fixed-size

Recursive splitting beats fixed-size on retrieval precision, but not by much. The gain comes from cleaner boundaries: fewer chunks that start or end mid-thought. On uniform text, the difference is small. On prose with clear paragraph structure, recursive splitting pulls ahead. It's the default in LangChain's text splitters for a reason.

Limitations and failure cases

The main catch: recursive splitting still doesn't understand meaning. It respects punctuation, not semantics. A paragraph break is a decent proxy for a topic shift, but it's not always right. Two related ideas can sit in separate paragraphs, and the splitter will happily separate them. Tables, code blocks, and lists still get mangled, because the separator hierarchy doesn't know what those structures are. Recursive splitting is a smarter baseline, not a smart strategy.

Semantic Chunking: When Meaning Drives Boundaries

Semantic chunking stops guessing where a topic ends. It measures meaning directly. You embed each sentence, compare adjacent sentences by cosine similarity, and cut where similarity drops. The boundary lands where the subject shifts, not where a paragraph happens to end.

How semantic chunking works

You run every sentence through an embedding model. Then you walk the sequence, computing the cosine similarity between sentence N and sentence N+1. When similarity falls below a threshold, you cut. The chunks that come out are groups of sentences that belong together. A paragraph about pricing stays intact. A paragraph that drifts from pricing into implementation gets split at the drift point.

The threshold is the tuning knob. Set it high and you get smaller, tighter chunks. Set it low and chunks grow. Most implementations use a percentile-based threshold rather than a fixed number, because similarity distributions shift from document to document.

Benchmark results

Semantic chunking wins on retrieval quality, but the margin depends on the corpus. On documents with clear topical structure, it beats recursive splitting by a measurable margin. On documents where topics blur together, the gain shrinks. The Reddit benchmark that tested seven strategies found semantic chunking near the top, but not universally ahead of simpler methods. The cost is real: you're running an embedding pass over every sentence before you even start chunking.

Implementation complexity

This is where semantic chunking gets expensive. You need an embedding model, a vector store or in-memory index for the sentence embeddings, and a similarity threshold that you tune per corpus. That's three moving parts recursive splitting doesn't have. For a one-off pipeline, it's a weekend of work. For a production system with changing documents, you're maintaining an extra preprocessing stage that can drift.

When semantic chunking underperforms

Semantic chunking assumes topic shifts show up as embedding distance. That assumption breaks on documents where structure carries meaning. Legal contracts, API references, and financial filings have sections that are semantically similar but functionally distinct. Two clauses about liability may embed nearly identically, but they belong in different chunks. Semantic chunking will merge them. Tables and code also embed poorly at the sentence level, so the similarity signal is noise. If your corpus is structured documents, document-aware chunking beats semantic chunking. If your corpus is prose, semantic chunking earns its cost.

LLM-Based Chunking: Powerful but Expensive

LLM-based chunking asks a language model to read the document and decide where chunks should start and end. The LLM sees the full context, understands the topic, and cuts where meaning shifts. No threshold tuning. No similarity math. Just a prompt that says: split this into coherent sections.

How LLM-based chunking works

You send the document to an LLM with instructions to mark chunk boundaries. The model returns the text split into sections, each with a label or summary. You then embed those chunks and index them. The LLM can also generate a short description for each chunk, which improves retrieval by giving the embedding model more signal than raw text alone.

The catch is obvious: you're paying for tokens on every document you process. A 10,000-word corpus costs roughly the same as a long LLM conversation. For a one-time index, that's fine. For a pipeline that re-chunks documents daily, it adds up fast.

Benchmark results and cost analysis

The Reddit benchmark that tested seven strategies found LLM-based chunking near the top on retrieval quality. It beat fixed-size and recursive splitting on documents with complex structure. But the margin over semantic chunking was small, and the cost was an order of magnitude higher. You're paying for an LLM call per document, plus the embedding pass, plus the vector store. Semantic chunking gives you most of the quality at a fraction of the cost.

When the cost is justified

LLM-based chunking earns its price when documents are messy. Scanned PDFs with no clean headings. Transcripts where speakers change mid-paragraph. Mixed content where tables, code, and prose sit side by side. In those cases, the LLM's ability to understand context beats every rule-based approach. For clean prose or structured documents, it's overkill. Use semantic or document-aware chunking instead.

Document-Aware Chunking: Respecting Structure

Document-aware chunking reads the document's own layout before splitting anything. Headings become boundaries. Tables stay intact. List items don't get severed mid-thought. The splitter parses structure first, then cuts where the document already cut itself.

How document-aware chunking works

You feed the splitter a document with markup: Markdown headers, HTML tags, PDF outlines, or Word styles. The splitter walks the structure tree and emits chunks at the levels you specify. A section heading starts a new chunk. A subheading starts another. A table gets its own chunk, with the caption attached so the embedding model knows what the table contains.

The payoff is context preservation. A chunk that starts at a heading and ends before the next heading carries its own topic. No mid-paragraph cuts. No orphaned table rows.

PDF and structured document benchmarks

PDFs are where document-aware chunking earns its keep. Fixed-size splitting on a PDF often slices through page breaks, figure captions, and column layouts. Structure-aware splitters use the PDF's internal outline and font metadata to detect headings and section boundaries. Benchmarks on technical manuals and research papers show document-aware chunking beating fixed-size by double digits on retrieval precision, because each chunk maps to one retrievable unit of meaning.

The honest limit: it only works when the document has structure to read. Scanned PDFs without an embedded outline fall back to fixed-size behaviour.

When structure matters most

Use document-aware chunking when your corpus is manuals, API docs, legal contracts, or academic papers. Anything where headings carry semantic weight. Skip it for chat logs, forum threads, or flat prose. There's no structure to respect, and the parsing overhead buys you nothing.

Hierarchical Chunking: Multi-Level Retrieval

Hierarchical chunking splits a document at several granularities at once. You keep a coarse chunk for the whole section, a medium chunk for each paragraph, and a fine chunk for each sentence. Retrieval can then start coarse and drill down, or start fine and pull in the parent for context.

How hierarchical chunking works

The splitter builds a tree, not a flat list. Each node stores its own text plus a pointer to its parent and children. When a query matches a sentence-level chunk, the system can fetch the paragraph and section that contain it. When a query matches a section, it can fetch the paragraphs inside.

The retrieval loop runs in two passes. Pass one finds the best coarse chunks. Pass two searches only within those chunks for the fine-grained match. This cuts the search space without losing precision. The tradeoff is storage: you index every level, so your vector store holds three or four times the chunks.

Benchmark results

Direct benchmark numbers for hierarchical chunking are thin. Most published studies test flat strategies against each other. The evidence that exists is indirect: parent document retrieval, which is hierarchical chunking with exactly two levels, consistently improves recall on multi-hop questions. The mechanism is clear. A sentence chunk alone rarely carries enough context to answer a question that spans paragraphs. Pulling the parent fixes that.

Hierarchical chunking is a retrieval architecture, not a splitting trick. Its value shows up in systems that need both precision and context, not in single-shot lookup benchmarks.

Agent memory applications

Agents are where hierarchical chunking earns its place. An agent working through a long conversation needs fine-grained memory for the current turn and coarse-grained memory for the session. Hierarchical chunks give you both without maintaining two separate stores.

Multi-hop retrieval also benefits. The agent retrieves a sentence that answers part of a question, then follows the parent pointer to the surrounding section for the rest. That pointer is cheaper than a second embedding search, and it preserves the document's original order.

The main catch is index maintenance. Every document update means rebuilding the tree and re-embedding every level. For static corpora that's a one-time cost. For agent memory that changes every turn, it's ongoing overhead.

Chunking for Agent Memory: What Changes

Single-shot RAG answers one question and forgets it. Agent memory doesn't. An agent carries state across turns, follows chains of reasoning, and revisits earlier context when new information arrives. That changes what you need from chunking.

How agent memory differs from single-shot RAG

In single-shot RAG, you embed a query, retrieve chunks, generate an answer, and discard everything. The chunk only needs to contain enough context for one answer. Agent memory keeps chunks around. The agent retrieves, reasons, acts, retrieves again, and updates what it knows. Chunks become working memory, not just lookup targets.

That means chunk boundaries matter for state, not just relevance. A chunk that splits a fact from its qualifier will mislead the agent every time it recalls that memory. Single-shot RAG might survive that error once. An agent compounds it across turns.

Best chunking strategies for agent memory

Hierarchical chunking is the strongest fit. Agents need fine-grained chunks for precise recall and coarse chunks for session context. Hierarchical gives you both in one index.

Semantic chunking works well for conversational memory where topic shifts are the natural boundary. Fixed-size chunking is a poor fit: it splits mid-thought, and agents retrieve those fragments repeatedly.

Multi-hop retrieval and chunk granularity

Multi-hop questions force the agent to connect chunks that were never adjacent in the source. Fine-grained chunks help here. A sentence-level chunk matches one hop cleanly. But fine chunks alone lose the connective tissue between hops. You need parent pointers or overlap to recover it.

The practical rule: index at the finest granularity your embedding model can handle, then attach parent context. Don't index coarse and hope the agent figures out the rest.

Context window management across turns

Agent context windows fill fast. Every turn adds retrieved chunks, tool outputs, and reasoning traces. Chunk size directly controls how many turns fit before the agent must compress or drop memory.

Smaller chunks let you pack more distinct memories into the window. But too small, and each memory lacks the context to be useful on its own. The tradeoff is sharper here than in single-shot RAG because the cost is paid every turn, not once per query.

Chunk for recall first, window pressure second. An agent that retrieves the wrong memory cheaply is worse than one that retrieves the right memory at a higher token cost.

How to Choose a Chunking Strategy: A Decision Framework

No single strategy wins everywhere. The right choice depends on what you're chunking, how users query it, and what you can afford. Here's the framework I use.

Decision factors: document type, query pattern, latency, budget

Start with document type. Structured documents with headings and tables call for document-aware chunking. Prose-heavy text tolerates fixed-size or recursive splitting. Code needs structure-aware boundaries.

Query pattern matters next. Precise fact lookup favors small, semantically clean chunks. Broad topical queries want larger chunks with more context. Multi-hop questions need fine granularity plus parent pointers.

Latency and budget set the ceiling. LLM-based chunking is accurate but slow and expensive. Fixed-size is nearly free. Semantic chunking sits between: better boundaries, but you pay for embedding passes over the whole corpus.

A decision table for common use cases

Here's the shorthand:

• PDFs with tables and headings: document-aware chunking. Structure is the signal.
• Long-form prose (books, reports): recursive character splitting. Cheap and good enough.
• Conversational logs or topic-shifting text: semantic chunking. Topic boundaries are what you want.
• Multi-hop agent memory: hierarchical chunking with parent pointers. Fine for recall, coarse for context.
• Small corpus, high-stakes accuracy: LLM-based chunking. The cost is justified when retrieval errors are expensive.
• Prototype or unknown query patterns: fixed-size with overlap. Get a baseline, then improve.

What no chunking strategy can fix

Chunking cannot rescue a weak embedding model. If the embeddings don't capture meaning, no boundary choice saves retrieval. It also can't fix bad query formulation. A vague query returns vague chunks regardless of how you split the source.

Chunking doesn't add information that isn't in the text. If the source document lacks the answer, no strategy manufactures it. And chunking won't compensate for a vector store with poor similarity search or missing metadata filtering.

Pick the simplest strategy that fits your document type and query pattern, then spend your remaining effort on embedding quality and retrieval design. Chunking is one lever among several.

Common Mistakes When Chunking Strategies for RAG

Most chunking failures aren't exotic. They're the same five mistakes repeated across teams, and each one is cheap to avoid once you know what to look for.

Ignoring chunk overlap

Chunk overlap is the buffer zone between adjacent chunks. Without it, a sentence split across a boundary loses its context on both sides. The fix is simple: set overlap to 10-20% of chunk size. Too much overlap bloats your index with near-duplicate text and slows retrieval. Too little breaks meaning at the edges.

One strategy for all document types

A single chunking strategy across a mixed corpus is the most common mistake I see. PDFs with tables need structure-aware boundaries. Prose tolerates recursive splitting. Code needs syntax-aware splits. Applying one rule everywhere means every document type except one is chunked badly. Match the strategy to the source, not the other way around.

Ignoring embedding model compatibility

Your chunk size has to fit your embedding model's context window. A model with a 512-token limit can't embed an 800-token chunk without truncation, and truncation silently drops meaning. Check the model's max input length before setting chunk size. Also verify the model was trained on text similar in length to your chunks. Mismatched training and inference lengths degrade similarity scores.

Over-chunking and under-chunking

Over-chunking splits text into fragments too small to carry meaning. Retrieval returns snippets that answer nothing. Under-chunking creates chunks so large that relevance gets diluted, and the retriever can't tell which part of the chunk matters. The symptom is the same: retrieval accuracy drops. The fix is different. Test chunk sizes against your query set and watch where precision falls.

Ignoring metadata

Chunks without metadata are unfilterable. You can't restrict retrieval by date, source, section, or document type. That forces the retriever to search everything, which wastes compute and returns irrelevant results. Attach metadata at chunking time: source file, page number, heading path, timestamp. It costs almost nothing and makes filtering possible later.

These mistakes compound. Overlap errors plus wrong chunk size plus missing metadata produce retrieval that fails for reasons you can't isolate. Fix them one at a time, and benchmark after each change. Chunking strategies for RAG, benchmarked properly, show their weaknesses fast.

Frequently Asked Questions

What is the best chunk size for RAG?

There is no universal best size — it depends on your embedding model, document type, and query patterns. Most teams find 256–1024 tokens with 10–20% overlap works well as a starting point, then tune against a labeled evaluation set rather than guessing.

What are the different chunking strategies for RAG?

Common strategies include fixed-size (token or character), recursive character splitting, semantic (embedding-boundary), hierarchical (parent-child), document-structure-aware (headings, tables), and agent-specific episodic/summary chunking. Each trades implementation effort against retrieval precision.

Is semantic chunking always better than fixed-size chunking?

1. Semantic chunking tends to improve precision on topic-dense long-form documents, but it adds embedding cost and latency and can underperform on uniform content like logs or chat. Benchmark both on your own data before committing.

How does chunking affect agent memory and multi-hop retrieval?

Agents need both fine-grained retrieval chunks and coarser memory chunks (episodic summaries, recent-turn buffers). Multi-hop questions suffer when chunks are too small to contain a complete reasoning step, so hierarchical chunking with parent context often helps.

What chunking strategy does LangChain use by default?

LangChain's RecursiveCharacterTextSplitter is the common default — it splits on a priority list of separators (paragraphs, sentences, words) to keep semantically related text together. It is a solid baseline, but you should still tune chunk size and overlap for your corpus.

Can chunking alone fix poor RAG retrieval quality?

1. Chunking is one lever among many — embedding model choice, reranking, query rewriting, and metadata filtering often matter as much or more. If retrieval is weak, benchmark those components alongside chunking rather than expecting chunking to compensate.

How do I benchmark chunking strategies fairly?

Hold everything except the chunking layer constant: same embeddings, same retriever, same reranker, same evaluation queries. Measure recall@k, answer faithfulness, and latency, and segment results by document type so you can see where each strategy wins or fails.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through chunking strategies for rag, benchmarked or something adjacent, we publish what we have actually tested, including where it falls short.