Retrieval quality degrades as the corpus grows, and if you build RAG pipelines or agent memory systems, you've probably felt it before the charts showed it. Your pipeline worked at 50k chunks. Recall was fine, precision was fine, users were happy. Then the corpus hit 2M chunks and the same queries started returning context that was adjacent but wrong. Answers that used to land now miss. Nobody changed the code. The data just got bigger.
This is not a bug you introduced. It's a structural property of retrieval systems, and it gets worse at predictable thresholds. The honest answer is that some of this decay cannot be reversed, only managed. What's missing from most discussions is a framework for knowing which failure mode you're hitting and what actually helps at your scale. GigaRAG was built for exactly this problem, but the mechanics below apply to any retrieval stack.
| At a glance | Details |
|---|---|
| Core problem | Recall and precision silently decay as chunk count grows |
| Key mechanisms | Embedding crowding, index approximation, context dilution, query drift |
| Typical onset | Noticeable degradation often starts around 100k–500k chunks |
| Hard limits | No fix restores perfect recall at millions of chunks |
| Most effective lever | Hybrid retrieval plus reranking and metadata filtering |
| Latency trade-off | Reranking adds tens to hundreds of ms per query |
In This Guide
- The Retrieval Quality Decay Curve
- Dense-Only Retrieval vs Hybrid Retrieval at Scale
- Why More Data Breaks Retrieval
- Retrieval Quality Degrades As The Corpus Grows: A Step-by-Step Guide
- Corpus Growth Failure Modes
- What Actually Works at Each Scale
- What You Cannot Fix
- Agent Memory and Multi-Turn Retrieval
- A Practical Diagnostic Workflow
The Retrieval Quality Decay Curve
Retrieval quality decays as the corpus grows because more chunks compete for the same top-k slots, and the signal that made a small corpus searchable gets diluted across a much larger candidate pool. The curve is steep at first, then flattens into a long, slow decline.
What the decay curve looks like
Plot recall@k on the y-axis against corpus size on the x-axis. At 10k chunks, recall@10 might sit at 0.85. By 1M chunks, it's often below 0.6. By 100M, it can drop under 0.3. The curve drops fast early, then levels off as the system approaches a floor where it's returning roughly random candidates.
Three thresholds: 10k, 1M, 100M chunks
At 10k chunks, retrieval is mostly fine. Embeddings are sparse enough that nearest neighbors are genuinely relevant. At 1M chunks, semantic crowding sets in: many chunks look similar to the query, and top-k slots fill with near-duplicates. At 100M chunks, the ANN index itself degrades. Approximate search trades accuracy for speed, and that tradeoff compounds.
Why the curve is not linear
Each order of magnitude adds a different failure mode. Going from 10k to 100k mostly adds noise. Going from 1M to 10M changes which chunks win the ranking. The mechanisms shift, so the fixes shift too.
[!note] Retrieval quality decay is not a single bug but the compounding effect of several independent mechanisms—embedding space crowding, approximate nearest neighbor index trade-offs, and context window dilution—each of which becomes dominant at different corpus sizes.
Dense-Only Retrieval vs Hybrid Retrieval at Scale
| Factor | Dense-Only | Hybrid (Dense + Sparse) |
|---|---|---|
| Recall at 1M+ chunks | Degrades as embedding space crowds | Better for rare terms and exact matches |
| Latency overhead | Lower (single index lookup) | Higher (two indexes + fusion) |
| Tuning complexity | Simpler to start | Requires weight tuning and fusion strategy |
| Failure mode | Semantic drift and near-duplicate collisions | Sparse index bloat and stale term statistics |
| Best fit | Small, homogeneous corpora | Large, heterogeneous corpora with jargon |
Why More Data Breaks Retrieval
The decay curve isn't one problem. It's three mechanisms stacking on top of each other, and each one kicks in at a different corpus size.
Semantic overlap and embedding collisions
Embeddings map chunks to points in a vector space. At 10k chunks, those points are spread out. At 1M, they crowd together. Queries that used to land near one clear answer now land in a cluster of near-duplicates. The model can't tell which one you meant, so it returns several that all look plausible. Precision drops before recall does.
ANN index degradation at scale
Approximate nearest neighbor indexes trade accuracy for speed. They skip parts of the search space to hit latency targets. At small scale, the skipped regions rarely matter. At 100M chunks, the index skips enough that genuinely relevant chunks get missed entirely. You're not ranking wrong candidates. You're never seeing them.
Top-k dilution: more candidates, same budget
You always return k chunks. At 50k corpus size, k=10 might cover most of the relevant material. At 2M, those same 10 slots get fought over by hundreds of near-relevant chunks. The relevant ones lose. Raising k helps recall but floods the context window and slows generation. The budget is fixed. The competition isn't.
[!tip] For agent memory builders: cap the number of retrieved chunks per turn and use a reranker to pick the top few, because feeding more context to the LLM often reduces answer quality and blows the latency budget—measure recall@5, not recall@50.
Retrieval Quality Degrades As The Corpus Grows: A Step-by-Step Guide
- Build a fixed evaluation set of queries with known relevant chunks, stratified by corpus region and query type.
- Measure recall@k and precision@k at your current corpus size, then re-measure after each major ingestion batch.
- Plot the decay curve: recall@k on the y-axis against corpus size on the x-axis to see where the knee occurs.
- Isolate the cause by testing with metadata filters that shrink the candidate pool to a known-good subset.
- Compare dense-only, sparse-only, and hybrid retrieval on the same evaluation set to quantify each component's contribution.
- Add a reranker and measure the recall gain against the added latency to decide if it fits your budget.
- Document the ceiling: record the corpus size at which each fix stops helping and set ingestion alerts accordingly.
Corpus Growth Failure Modes
The mechanisms from the last section don't hit all at once. They arrive in sequence, and each one has a signature. Here's how to tell which one you're dealing with.
Failure mode 1: Semantic crowding (10k–100k chunks)
Your embeddings start colliding. Two chunks about "invoice processing" sit nearly on top of each other in vector space, but one is about accounts payable automation and the other is about a specific vendor's billing portal. The model can't separate them. You'll see precision drop first: the right answer is in the top-k, but so are three wrong ones that look identical. Recall stays fine. Users complain about "off-topic" context, not missing context.
Failure mode 2: Index saturation (100k–1M chunks)
The ANN index starts making real tradeoffs. HNSW graphs get deeper, IVF partitions get coarser, and the search skips regions that used to be reachable. Now recall drops. The right chunk exists in your corpus, but the index never surfaces it. You'll see this as "I know we have that document, why didn't it come back?" The fix here isn't better ranking. It's better indexing.
Failure mode 3: Query ambiguity amplification (1M+ chunks)
Every query now matches hundreds of plausible chunks across dozens of topics. The problem isn't that the right answer is missing. It's that the query itself no longer disambiguates. "How do I handle refunds?" matches customer service policy, accounting procedures, legal compliance, and internal tooling docs. Top-k returns a grab bag. You need query reformulation or metadata filtering before retrieval even starts.
What Actually Works at Each Scale
The failure modes map to fixes. Here's what holds up in practice.
Chunking and metadata enrichment
Smaller chunks reduce semantic crowding. At 10k to 100k chunks, splitting documents into 200 to 400 token pieces with clear metadata (source, date, doc type) keeps similar chunks separable. Metadata filters cut the candidate pool before vector search runs. This works until chunks get so small they lose context.
Hybrid search: dense + sparse
Dense embeddings miss exact terms. BM25 catches them. Running both and merging results fixes recall at 100k to 1M chunks, where index saturation starts. The catch: you now run two indexes and merge two result sets. Latency goes up.
Reranking and contextual retrieval
A cross-encoder reranker scores top-k candidates against the query directly. It recovers precision that vector similarity alone loses. Contextual retrieval goes further: prepend document context to each chunk before embedding. Anthropic reported a 49% reduction in retrieval failures using this technique. It helps most at 1M+ chunks, where query ambiguity dominates.
When each technique stops helping
Chunking stops helping when chunks get too small to answer anything. Hybrid search stops helping when both indexes return the same wrong candidates. Reranking can't fix what retrieval never surfaced. Contextual retrieval adds latency per chunk and costs more at embed time. Each fix buys you headroom, not a cure.
What You Cannot Fix
Some degradation is structural. You can slow it, but you can't stop it.
The retrieval quality ceiling
Every retrieval system has a ceiling set by two things: the quality of your embeddings and the ambiguity of your queries. No amount of reranking, hybrid search, or contextual retrieval pushes you past it. At 100M chunks, even a perfect index returns candidates that are semantically adjacent but wrong. The ceiling isn't a bug. It's the limit of matching text to intent.
Why reranking cannot fully compensate
Reranking only reorders what retrieval already surfaced. If the right chunk never made it into the top-k candidates, the reranker never sees it. At scale, top-k dilution guarantees this happens more often. A cross-encoder can fix precision, not recall. You can widen top-k to give the reranker more to work with, but that costs latency and still doesn't recover chunks the index buried.
Latency vs. quality: the unavoidable tradeoff
Every fix costs time. Hybrid search doubles index lookups. Reranking adds a second model pass. Contextual retrieval adds preprocessing per chunk. At 1M+ chunks, you're choosing between fast and wrong or slow and slightly less wrong. There is no configuration that gives you both. Pick your latency budget first, then optimize quality within it.
Agent Memory and Multi-Turn Retrieval
Agent memory makes corpus growth worse because the corpus isn't static. Every turn adds new context, and the agent retrieves against a moving target.
Why agent memory amplifies retrieval degradation
A static RAG pipeline retrieves once per query. An agent retrieves many times per conversation, and each retrieval pulls from a corpus that now includes the conversation itself. Semantic overlap compounds. The agent's own outputs become candidates, crowding out the source chunks that actually answer the question. At 1M chunks, this self-interference is measurable.
Multi-turn retrieval and context window pressure
Each turn carries a context window budget. You're spending tokens on retrieved chunks, conversation history, and the agent's own reasoning. As the corpus grows, retrieval returns more candidates to fill the same top-k, and each candidate is less precise. The agent burns context window on irrelevant chunks, then has less room to reason. It's a compounding failure.
Memory persistence vs. retrieval quality
Persistent memory sounds like a fix. Store more, retrieve better. It's not that simple. Long-term memory grows the corpus without bound. Every persisted interaction becomes a chunk that competes with source documents. You're trading recall of past conversations for precision on the current query. The honest answer: persistence helps only when you prune aggressively and tag memory chunks separately from source chunks.
A Practical Diagnostic Workflow
You can't fix what you haven't measured. Start with a number, not a feeling.
Step 1: Measure recall@k at your corpus size
Build a ground truth set of 50 to 100 queries with known relevant chunks. Run them through your pipeline and compute recall@k at your current k. Record the number. That's your baseline. Without it, every fix is a guess.
Step 2: Identify your failure mode
Look at where recall drops. If relevant chunks rank below k but still appear in the index, you have a ranking problem. If they don't appear at all, you have an indexing problem. The fix differs.
Step 3: Apply the fix mapped to your threshold
Ranking problem: add reranking or hybrid search. Indexing problem: revisit chunking or metadata. Don't apply both at once. You won't know which one moved the number.
Step 4: Re-measure and set expectations
Run the same ground truth set again. If recall@k improved, keep the fix. If not, revert it. Expect diminishing returns past 1M chunks. The ceiling is real.
Retrieval quality degrades as the corpus grows, and the decay curve gives you a way to see it coming. Measure recall@k, identify the failure mode, apply the fix mapped to your threshold, and re-measure. Some of the decline is structural and won't reverse. The rest you can manage, if you know which threshold you're at.
Frequently Asked Questions
Why does retrieval quality degrade as the corpus grows?
As the corpus grows, the embedding space becomes more crowded, so semantically similar but irrelevant chunks compete for the same top-k slots. Approximate nearest neighbor indexes also trade recall for speed, and larger candidate pools increase the chance of near-duplicate or stale chunks surfacing.
At what corpus size does retrieval quality start to drop?
There is no universal threshold, but many teams notice degradation between 100k and 500k chunks, depending on embedding dimensionality, index type, and query diversity. The decay is usually gradual before it becomes obvious in user-facing metrics.
Does hybrid retrieval fix quality decay at scale?
Hybrid retrieval helps by combining semantic matching with exact term matching, which improves recall for rare or domain-specific terms. However, it adds latency and tuning complexity, and it does not eliminate the fundamental crowding problem in dense embeddings.
How can I measure retrieval quality decay in my pipeline?
Maintain a fixed evaluation set of queries with labeled relevant chunks, and measure recall@k and precision@k at regular corpus size intervals. Plotting these metrics against corpus size reveals the decay curve and the point where each fix stops helping.
What are the hard limits of retrieval at millions of chunks?
At millions of chunks, no retrieval method achieves perfect recall without unacceptable latency. Approximate indexes inherently trade recall for speed, and context windows limit how many chunks you can pass to the LLM, so you must accept a ceiling and design around it.
How does agent memory affect retrieval quality as the corpus grows?
Agent memory often accumulates redundant or outdated observations, which act as noise in the retrieval index. Without pruning or summarization, memory growth mimics corpus growth and accelerates quality decay, so memory compaction is essential for long-running agents.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through retrieval quality degrades as the corpus grows or something adjacent, we publish what we have actually tested, including where it falls short.
