Scaling RAG Past 10 Million Documents: What Works

Scaling RAG Past 10 Million Documents: What Actually Works

Scaling RAG past 10 million documents is where most pipeline builders hit a wall they didn't see coming. Your system worked fine at 100K. It held together at 1M, maybe with a few latency spikes and a recall dip you could patch. Then you cross 10M and the whole thing stops behaving: queries take seconds, recall collapses on edge cases, index rebuilds eat a weekend, and your cost per query climbs past what the product makes. Most scaling advice stops being useful around 1M documents. The honest answer is that 10M+ is a different problem, and the fixes are not incremental. Platforms like GigaRAG are built specifically for agent memory at this scale, but even with the right tools you need to know what actually breaks. This guide covers the two-tier retrieval architecture that restores speed and precision, the chunking trade-offs that change past 10M documents, and the hard limits you should not expect to engineer around.

In This Guide

• Why Scaling RAG Past 10 Million Documents Breaks Most Systems
• Stateless RAG Pipeline vs Stateful Agent Memory at 10M+ Documents
• RAG Architecture at 10M+ Documents: A Two-Tier Retrieval System
• Scaling Rag Past 10 Million Documents: A Step-by-Step Guide
• Chunking Strategy for 10M+ Document Corpora
• Vector Database Selection for Scaling RAG Past 10 Million Documents
• Agent Memory and Stateful Retrieval at 10M+ Document Scale
• Hybrid Retrieval: Sparse + Dense at Extreme Scale
• Latency, Cost, and Recall Benchmarks at 1M, 5M, and 10M+ Documents
• What You Cannot Do When Scaling RAG Past 10 Million Documents
• Common Mistakes When Scaling RAG Past 10 Million Documents
• Final Thoughts on Scaling RAG Past 10 Million Documents

Why Scaling RAG Past 10 Million Documents Breaks Most Systems

At 100K documents, your RAG pipeline hums. At 10M, the same architecture fails in ways that look random but aren't. Latency spikes, recall collapses, index rebuilds stretch into days. The problem isn't your code. It's that the assumptions baked into small-scale RAG stop holding.

What breaks first: latency, recall, or cost?

Latency breaks first. A single-node vector index searches fine at 1M vectors. Push past 5M and query time climbs from tens of milliseconds to hundreds, then seconds. Recall breaks second, and it's sneakier. Your top-k results look plausible but miss the right document entirely. Cost breaks last, but hardest: re-embedding 10M documents after a model change can run five figures in compute alone.

The 1M document wall: why most advice stops here

Most RAG guidance assumes you can hold your index in memory on one machine. That works until roughly 1M chunks. After that, you're sharding, partitioning, or paying for managed infrastructure. Tutorials don't cover this because their authors never ran a production system past that point. The advice that works at 100K, bigger chunks, denser embeddings, single-stage retrieval, actively hurts at 10M.

How agent memory changes the scaling equation

Agent memory makes scaling harder, not easier. A stateless RAG pipeline answers each query fresh. An agent carries session state, conversation history, and consolidated memory across turns. That state multiplies your retrieval surface: you're now searching not just 10M documents but also the agent's accumulated context. If your retrieval layer can't handle both, the agent degrades mid-conversation. You'll see it as answers that start coherent and drift off-topic.

[!note] At 10M+ documents, recall does not stay constant as you add data—it typically degrades unless you change retrieval strategy. Most published RAG scaling guides stop being useful around 1M documents because they assume the same architecture scales linearly.

Stateless RAG Pipeline vs Stateful Agent Memory at 10M+ Documents

RAG Architecture at 10M+ Documents: A Two-Tier Retrieval System

A two-tier retrieval system splits search into a cheap coarse filter that narrows 10M+ documents to a few hundred candidates, then a fine ranking stage that scores only those candidates for precision. Single-stage retrieval collapses at this scale because you're asking one index to be both fast and accurate, and no single index does both.

Coarse filtering: sparse retrieval and metadata partitioning

The first tier has one job: cut 10M documents down to a few hundred candidates in under 50ms. Sparse retrieval, BM25 or a learned sparse model like SPLADE, does this well because it's cheap to run and doesn't need GPU. Metadata partitioning is the other half. Partition by document type, date range, customer, or any field your queries actually filter on. A query about "Q3 revenue for enterprise accounts" should never touch the 8M documents that aren't enterprise financial records.

Here's what happens behind the scenes. You run BM25 against a sharded index, pull the top 200 to 500 candidates per shard, merge them, and pass that set forward. The coarse filter doesn't need to be perfect. It needs to not miss the right document. Recall at this stage matters more than precision.

Fine ranking: dense embeddings with cross-encoder reranking

The second tier takes those few hundred candidates and ranks them precisely. Dense embeddings capture semantic similarity that BM25 misses. A cross-encoder reranks the top 50 to 100 candidates by reading the query and document together, which is slow but far more accurate than comparing embedding vectors.

The cost math works because you're only running the expensive model on a few hundred documents, not 10M. A cross-encoder pass over 100 candidates takes tens of milliseconds. Over 10M, it would take hours.

Why single-stage retrieval fails at 10M+ documents

Single-stage retrieval asks one index to do two contradictory things. Dense ANN indexes like HNSW are fast at approximate search but degrade as the index grows past a few million vectors. Precision drops, latency climbs, and memory pressure forces you onto disk, which kills query speed. Sparse retrieval alone is fast but misses semantic matches entirely.

The two-tier pattern isn't optional at 10M+. It's the difference between a query that returns in 200ms with the right document and one that returns in 3 seconds with a plausible-looking miss.

[!tip] For agent memory engineers: keep your memory store separate from your document index. Mixing them forces you to re-embed memory on every document update and makes eviction policies impossible to reason about.

Scaling Rag Past 10 Million Documents: A Step-by-Step Guide

1. Benchmark your current pipeline at 1M, 5M, and 10M documents to find the exact failure point.
2. Replace flat vector search with hybrid retrieval (dense + sparse) and metadata filtering.
3. Introduce a stateful memory layer that persists agent context across sessions.
4. Shard or partition the index by domain, tenant, or time to keep per-query search space small.
5. Add a reranking stage that operates on a small candidate set, not the full index.
6. Instrument recall, latency, and cost per query at every scale tier before adding more documents.
7. Document what your system cannot do at 10M+ scale and set expectations with stakeholders.

Chunking Strategy for 10M+ Document Corpora

Chunking at 10M+ documents is a storage and latency problem first, a relevance problem second. Every chunk you create is a vector you must store, index, and search. At 10M documents averaging 20 chunks each, that's 200M vectors. Chunk size determines whether that fits in memory or spills to disk.

Chunk size trade-offs at scale

Small chunks, 100 to 200 tokens, give you precise retrieval but explode your vector count. Large chunks, 1,000+ tokens, keep the index smaller but bury the answer inside noise. At 10M+ scale, the index size is the binding constraint. A 200-token chunk across 10M documents means roughly 200M vectors. That's a multi-node index with real memory pressure.

The honest answer is it depends on your document type. Dense technical manuals chunk well at 300 to 500 tokens. Chat logs and support tickets need smaller chunks, 100 to 200 tokens, because the signal is sparse. Don't pick a chunk size from a blog post. Run recall tests on your own corpus at three sizes and measure.

When full-document reading beats chunking

For small, high-quality datasets, under 50K documents, full-document reading often beats chunking. You embed the entire document as one vector, and the model reads the whole thing at query time. No chunk boundary problems. No lost context.

That stops being true somewhere past 1M documents. Full-document embeddings are long, expensive to compute, and they dilute the query signal across thousands of irrelevant tokens. At 10M+ scale, full-document reading is not viable. The index is too large, and retrieval precision collapses because every document looks vaguely relevant to every query.

Overlapping chunks and metadata inheritance

Overlap between chunks, typically 10 to 20 percent, preserves context across boundaries. A sentence split mid-thought loses meaning. Overlap fixes that but increases your vector count by the overlap percentage. At 10M+ scale, 20 percent overlap on 200M vectors is 40M extra vectors. That's real money.

Metadata inheritance is non-negotiable. Every chunk must carry its source document ID, title, date, and any partition fields from the coarse filter. Without it, the fine ranking stage can't group chunks from the same document, and you'll return fragments from five different documents instead of one coherent answer.

Vector Database Selection for Scaling RAG Past 10 Million Documents

The chunking strategy you picked determines the vector count. The database you pick determines whether that count is searchable or just expensive. At 200M vectors, the choice is not academic.

Managed vs. self-hosted at 10M+ scale

Managed services, Pinecone and Qdrant Cloud, remove the operational burden but charge per vector per month. At 200M vectors, that bill is real. Self-hosted options, Milvus and Weaviate, trade your engineering time for infrastructure savings. FAISS is not a database. It's a library. You build the server, the sharding, the replication, and the monitoring yourself.

The honest answer is it depends on your team size. If you have one engineer and 200M vectors, pay for managed. If you have an infrastructure team and predictable query volume, self-host Milvus or Qdrant.

ANN index types: HNSW, IVF, and hybrid approaches

HNSW gives you fast queries and high recall but consumes memory aggressively. At 200M vectors, a pure HNSW index will not fit on one machine. IVF clusters vectors and searches only nearby clusters, trading some recall for a much smaller memory footprint. Most production systems at this scale use IVF with a coarse quantizer, or a hybrid that combines IVF for the coarse filter with HNSW inside each shard.

Index sharding and distributed retrieval

Sharding is not optional past roughly 50M vectors. You split the index across nodes by document partition, tenant, or hash. Each shard runs its own ANN search, and a coordinator merges results. The coordinator is where latency creeps in. If you merge 10 shards and each returns 100 candidates, the coordinator must deduplicate and rank 1,000 vectors before the reranker sees them.

Keep shard count low. Five shards of 40M vectors each beats 50 shards of 4M. Fewer shards means fewer network round trips and less merge overhead.

Agent Memory and Stateful Retrieval at 10M+ Document Scale

Most RAG scaling advice treats every query as a fresh start. That assumption collapses the moment an agent holds a conversation. The user asks a follow-up, the agent needs context from three turns ago, and the retrieval layer has no idea what "that" refers to. Stateful retrieval is not a feature you bolt on. It changes the architecture.

Session state and conversation history in retrieval

Session state is the working memory of an agent: the current query, the last few turns, and any entities or filters the user has locked in. At 10M+ documents, you cannot re-embed the full conversation and run it as one giant query. The embedding model will dilute the actual question. Instead, keep session state as structured data: extracted entities, active filters, and a short rolling summary. Pass that alongside the current query to the coarse filter. The coarse filter narrows the candidate set before any dense embedding runs.

Conversation history belongs outside the vector index. Store it in a fast key-value store, Redis or similar, keyed by session ID. The retrieval layer reads it, extracts the relevant constraints, and applies them as metadata filters. Do not embed conversation turns into the query vector. It pollutes recall.

Memory consolidation: what to persist, what to discard

Not every turn deserves to live forever. Memory consolidation is the process of deciding what survives the session. The rule that works in practice: persist entities, decisions, and user-stated preferences. Discard filler, greetings, and redundant restatements.

A consolidation pass runs at session end or after every N turns. It writes a compact summary to long-term memory, which is just another small collection in your vector database or a dedicated memory store. That summary gets embedded and indexed, but it is tiny compared to the main corpus. A 10M-document index does not care about 50,000 memory summaries. The retrieval layer queries both, merges results, and weights memory higher for follow-up questions.

The main catch is staleness. A preference stated three months ago may no longer hold. Version your memory entries and let newer entries override older ones on conflict.

Stateful vs. stateless retrieval: architectural implications

Stateless retrieval treats every query as independent. It scales horizontally because nothing is shared between requests. Stateful retrieval adds a dependency: the session store. That store becomes a single point of contention if you are not careful.

The fix is to keep session state small and local. Store it in a distributed cache with a short TTL, not in the vector database. The vector index stays stateless. Only the query construction step becomes stateful. That separation keeps your retrieval path horizontally scalable while giving the agent the context it needs.

The honest answer is that full stateful retrieval across 10M+ documents is not a solved problem. What works is a hybrid: stateless vector search with a thin stateful layer that rewrites queries before they hit the index. That layer is where agent memory lives.

Hybrid Retrieval: Sparse + Dense at Extreme Scale

Dense embeddings alone start missing things at 10M+ documents. They're great at semantic similarity, but they fumble exact matches: product codes, error strings, proper nouns, rare terms. BM25 catches those cold. Hybrid retrieval runs both and merges the results.

Why dense-only retrieval fails at 10M+ documents

The problem is recall collapse. As your index grows, the nearest neighbors to any query vector get crowded. A dense search returns the 100 vectors closest to your query, but at 10M+ documents, "closest" often means "semantically adjacent, factually wrong." Exact identifiers get buried. BM25 doesn't have this problem. It matches tokens directly, so a query for "error code 0x80070005" finds the document containing that string, not the 100 documents about Windows errors in general.

Implementing hybrid retrieval without doubling latency

Run both retrievers in parallel, not in sequence. BM25 and dense search hit the index at the same time, each returning top-K candidates. You merge the two candidate sets, deduplicate, and send the union to a reranker. The reranker is the only stage that sees the combined list.

The latency cost is the max of the two retrievers, not the sum. BM25 on a well-built inverted index returns in single-digit milliseconds. Dense search on HNSW returns in tens of milliseconds. Your total retrieval latency is the slower of the two, plus reranking.

Weighting sparse vs. dense scores

Don't average raw scores. BM25 and cosine similarity live on different scales. Normalize each score to a 0-1 range within its own result set, then apply a weight. A common starting point is 0.3 sparse, 0.7 dense for open-ended questions, flipped to 0.7 sparse, 0.3 dense for lookup-style queries.

The honest answer is that the right weight depends on your query mix. If users search for exact identifiers, lean sparse. If they ask natural-language questions, lean dense. Test on your own query logs. A fixed 50/50 split is a guess, not a strategy.

Latency, Cost, and Recall Benchmarks at 1M, 5M, and 10M+ Documents

Honest numbers here are ranges, not points. Your hardware, embedding model, index type, and query mix all move the needle. What follows is what I've seen in practice across managed and self-hosted setups.

Latency expectations at each scale

At 1M documents, a well-tuned HNSW index returns candidates in 30-80ms. Add reranking and you're at 150-300ms end to end. At 5M, candidate retrieval stretches to 80-150ms. Reranking 100 candidates with a cross-encoder pushes total latency to 300-600ms unless you shard the index.

At 10M+, single-node retrieval stops being predictable. Sharded indexes keep candidate retrieval at 100-200ms, but the merge step adds overhead. Plan for 500ms-1s total latency without aggressive caching. Sub-100ms at this scale means dedicated infrastructure and a smaller candidate set.

Cost-per-query breakdown

Embedding a query costs fractions of a cent. Reranking is the expensive part. A cross-encoder pass over 100 candidates runs $0.002-$0.01 per query on GPU infrastructure. At 10M+ documents, you'll rerank more candidates to hold recall, so cost climbs.

Storage dominates at rest: 10M documents at 1KB per chunk vector runs roughly 10-40GB for the index, plus metadata. Managed services charge per vector per month. Self-hosted shifts cost to RAM and engineering time.

Recall degradation and mitigation

Recall drops as the index grows. A system hitting 90% recall at 1M documents often lands at 70-80% at 10M+ without changes. The fix is not a bigger index. It's better candidate generation: hybrid retrieval, metadata filtering, and query rewriting before the dense search runs. Rerank more candidates. Accept that exact recall is gone.

What You Cannot Do When Scaling RAG Past 10 Million Documents

Some things simply stop being possible at this scale. The sooner you accept that, the less money you'll burn chasing them.

No exact-match retrieval guarantees

ANN indexes are approximate by design. HNSW, IVF, all of them trade a little recall for a lot of speed. At 10M+ documents, that trade gets sharper. You cannot promise a user that the single most relevant chunk will always surface. You can promise that a highly relevant chunk will surface, most of the time.

If you need exact nearest-neighbor search, you're looking at brute-force scanning. That's O(n) per query. At 10M documents, that's seconds per query on a GPU, minutes on CPU. Not viable. The honest answer is to design your system so exact recall doesn't matter: better candidate generation, reranking, and accepting that "good enough" is the product.

No sub-100ms latency without major investment

Sub-100ms end-to-end retrieval at 10M+ documents is possible. It is not cheap. You need sharded indexes across multiple nodes, aggressive caching, a small candidate set, and likely a dedicated GPU pool for reranking. That's infrastructure spend in the thousands per month, plus engineering time to tune it.

If you're running a single managed vector database and expecting 50ms queries at this scale, you'll be disappointed. Plan for 500ms-1s and optimize from there. The teams hitting sub-100ms built custom retrieval layers, not just bigger indexes.

No single-node solutions

A single machine, however large, runs out of RAM. A 10M-document index with metadata and HNSW graphs needs 64-128GB of RAM minimum. Add reranking models and you're past what one box handles comfortably. You can run it, but latency spikes under load and rebuilds take hours.

The honest answer: at 10M+ documents, you're running a distributed system. Multiple index shards, a coordinator, separate reranking infrastructure. That means Kubernetes or an equivalent orchestrator, health checks, and someone on call. If you don't have that team, stay below 5M documents or use a managed service that abstracts the sharding for you.

Common Mistakes When Scaling RAG Past 10 Million Documents

Most teams don't hit a wall at 10M documents because the architecture is wrong. They hit it because they kept doing things that worked fine at 100K and never questioned them.

Re-embedding too frequently

Every time your embedding model changes, you re-embed the entire corpus. At 10M documents, that's days of compute and thousands in GPU costs. Teams do this when they upgrade models, tweak chunking, or "just to be safe." Don't. Version your embeddings. Keep the old index live while you build the new one in parallel. Switch over when it's ready. Re-embedding is a migration, not a routine task.

Ignoring index rebuild times

HNSW indexes don't rebuild in minutes at this scale. A full rebuild on 10M documents can take 6-12 hours, sometimes longer. If you're adding documents daily and rebuilding nightly, you're burning compute and blocking queries. Use incremental insertion where your vector database supports it. If it doesn't, batch updates weekly and accept staleness. The alternative is a system that's down half the time.

Over-relying on a single retrieval stage

One retrieval pass, whether dense or sparse, collapses at 10M+ documents. Recall drops because the candidate set is too large for one model to rank well. The fix is two stages: coarse filtering to get from 10M to a few thousand candidates, then fine ranking to get from thousands to dozens. Teams that skip this end up with latency spikes and users complaining that search "just doesn't find things anymore." Scaling RAG past 10 million documents means accepting that one retrieval stage is not enough.

Final Thoughts on Scaling RAG Past 10 Million Documents

Scaling RAG past 10 million documents isn't a tuning problem. It's a redesign. The system that worked at 100K was built on assumptions that stop holding: single-stage retrieval, nightly re-embedding, one index on one node. Each of those breaks differently at 10M+, and the fixes aren't incremental.

The architectural shifts are consistent across every team I've talked to. Two-tier retrieval replaces one pass. Hybrid sparse-plus-dense replaces dense-only. Index sharding replaces single-node. Agent memory adds session state and consolidation, which changes what you retrieve and when. None of this is optional at scale. It's the floor.

The honest limitations still stand. No exact nearest-neighbor guarantees. No sub-100ms latency without real infrastructure spend. No single-node solution that survives contact with 10M documents. If a vendor tells you otherwise, walk away.

That's why platforms built specifically for agent memory at this scale exist. GigaRAG handles the sharding, the hybrid retrieval, and the stateful memory layer so you're not rebuilding infrastructure from scratch. It's not magic. It's just the architecture this guide describes, already assembled. For teams scaling RAG past 10 million documents, that's the difference between a quarter of engineering effort and a year of it.

Frequently Asked Questions

How many documents can a RAG system handle before it breaks?

There is no single number, but most teams hit real degradation between 1M and 5M documents with a naive pipeline. The break point depends on your index type, query patterns, and latency budget. Benchmark at each tier rather than assuming a threshold.

What is the biggest bottleneck when scaling RAG past 10 million documents?

Vector index memory and retrieval latency, not storage. As the index grows, top-k search touches more candidates and recall drops unless you add filtering or sharding. Cost per query also rises because you need more compute to maintain the same latency.

Does agent memory help with RAG at 10M+ document scale?

Yes, but not as a replacement for retrieval. Stateful agent memory reduces redundant retrieval by carrying context across turns, which lowers query volume and improves relevance. It does not fix a poorly partitioned index.

What can you not do with RAG at 10 million documents?

You cannot maintain sub-second latency with brute-force search over the full index. You cannot guarantee high recall without reranking or filtering. You cannot keep costs flat if query volume grows linearly. These are hard limits, not tuning problems.

Is it better to shard the index or use a bigger vector database?

Sharding by domain, tenant, or time usually wins because it keeps per-query search space small. A bigger single index helps with storage but does not solve latency or recall degradation. Most production systems at 10M+ use some form of partitioning.

How do you measure recall at 10 million documents?

You need a labeled evaluation set with known relevant documents per query. Measure recall@k and latency at each scale tier. Without a labeled set, you are guessing—and at 10M+ documents, guessing is expensive.

What is the cost per query for RAG at 10M+ documents?

Cost varies widely by provider and architecture, but expect it to be dominated by vector search compute and reranking, not storage. Track cost per query at each scale tier and set a budget before you grow the corpus further.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through scaling rag past 10 million documents or something adjacent, we publish what we have actually tested, including where it falls short.