Scaling RAG Past 10 Million Documents: What Works

Scaling RAG Past 10 Million Documents: What Actually Works

Scaling RAG past 10 million documents is where most pipeline builders hit a wall they didn't see coming. Your vector store hums along at 100K docs, then you cross 10M and a single query takes three seconds, recall drops to coin-flip territory, and your agent starts answering from fragments instead of facts. The honest answer is that brute-force scanning and single-index assumptions don't survive that scale. GigaRAG exists because this exact wall is real, but this guide is about the architecture, not the product. It covers a staged, practical approach: progressive narrowing with explicit latency and cost budgets per stage, a critical comparison of vector, vectorless, and hybrid retrieval, and the trade-offs you'll face when your agent memory outgrows a single index. It won't promise magic numbers, and it won't pretend there's a free lunch.

In This Guide

• What Scaling RAG Past 10 Million Documents Actually Means
• Vector-Based vs Vectorless vs Hybrid Retrieval at 10M+ Documents
• The Progressive Narrowing Pipeline: A Mental Model
• Scaling Rag Past 10 Million Documents: A Step-by-Step Guide
• Data Ingestion at 10M+ Documents
• Chunking Strategy for Massive Corpora
• Embedding Generation at Scale
• Retrieval Architecture: Vector, Vectorless, or Hybrid?
• Metadata Filtering and Namespace Isolation
• Hybrid Search and Reranking at 10M+ Scale
• Evaluation and Observability for Large-Scale RAG
• Common Mistakes When Scaling RAG Past 10 Million Documents

What Scaling RAG Past 10 Million Documents Actually Means

Scaling RAG past 10 million documents means your retrieval system stops working the way it did at 100K: latency climbs, recall drops, and costs multiply unless you change the architecture, not just the hardware.

What breaks first: latency, recall, or cost?

It depends on your index and your query load. A single vector index on one machine will hit latency walls first. HNSW graphs grow dense, and graph traversal slows as the index approaches memory limits. Recall breaks next: as the candidate space widens, approximate nearest neighbor search returns noisier neighbors. Cost breaks last but hits hardest. Embedding 10 million documents, storing the vectors, and serving queries adds up fast.

Why a single vector index stops working

A vector index is not a database. It's a graph built for fast approximate search within a bounded size. Push past that bound and you get one of two failures: the index no longer fits in RAM, so queries spill to disk and latency spikes, or the graph quality degrades and recall tanks. You can't fix either by adding more documents to the same index. You have to split the problem.

The three scaling levers: narrow, route, rerank

Every working architecture at this scale uses the same three moves. Narrow the candidate set before any vector search runs, using metadata filters or namespace isolation. Route queries to the right shard or index instead of searching everything. Rerank a small set of candidates with a precise, expensive model instead of trusting the cheap retrieval score. Miss any one of these and the other two can't save you.

[!note] At 10M+ documents, a single unsharded vector index will eventually exceed memory and degrade recall; sharding and pre-filtering are not optional optimizations but baseline requirements.

Vector-Based vs Vectorless vs Hybrid Retrieval at 10M+ Documents

The Progressive Narrowing Pipeline: A Mental Model

The fix is not a faster index. It's a pipeline that shrinks the search space at every step, spending cheap compute early and expensive compute late. Four stages, each with a hard budget.

Stage 1: Metadata filtering (cheap, fast, coarse)

Filter on structured fields before any vector math runs. Tenant ID, document type, date range, namespace. This stage should cut 10 million documents to under 500K in single-digit milliseconds. If it can't, your metadata schema is wrong.

Stage 2: Candidate retrieval (vector or hybrid)

Run vector search, BM25, or both on the filtered set. Target 50 to 200 candidates. Latency budget: 20 to 50 ms. This is where most systems stop, and it's why most systems fail at scale. The retrieval score is a rough guess, not a ranking.

Stage 3: Reranking (expensive, precise, small)

Feed those 50 to 200 candidates through a cross-encoder. It reads the query and document together, so it's far more precise than the embedding score. It's also 10 to 100 times slower per document. That's why you rerank 50, not 50,000. Budget 100 to 300 ms here.

Stage 4: Context assembly and token budgeting

Take the top 5 to 10 reranked chunks and build the prompt. Watch your token count. A 10 million document corpus doesn't matter if you stuff 50K tokens into every call. Cut chunks that don't add new information. Your LLM call is the most expensive step in the whole pipeline, so feed it the smallest context that answers the question.

[!tip] For agent memory systems, store a compact 'memory summary' vector per session alongside raw chunk vectors—this lets the agent retrieve context cheaply before drilling into full documents, cutting both latency and token cost.

Scaling Rag Past 10 Million Documents: A Step-by-Step Guide

1. Measure current p95 latency and recall at 1M docs to establish a baseline before scaling.
2. Add a metadata and lexical pre-filter layer so queries never scan the full corpus.
3. Shard the vector index by a natural key (tenant, time, domain) and route queries to relevant shards only.
4. Apply progressive narrowing: cheap filters first, ANN search second, reranking last.
5. Introduce hybrid retrieval (lexical + vector) and tune the fusion weight on a labeled eval set.
6. Set explicit per-stage latency and cost budgets, then alert when any stage exceeds them.
7. Re-embed and reindex on a schedule to prevent drift as the corpus grows.

Data Ingestion at 10M+ Documents

Ingestion is the part nobody budgets for until it's the bottleneck. At 10 million documents, you can't just point a script at a folder and wait. You need a pipeline that parallelizes, deduplicates, and keeps the index fresh without taking the query path down with it.

Parallel ingestion and backpressure

Run ingestion workers in parallel, but cap concurrency. If you blast 500 workers at your embedding API, you'll hit rate limits and burn money on retries. Use a queue with backpressure: workers pull from the queue, and the queue blocks when the downstream index or embedding service is saturated. Ten to twenty workers per embedding endpoint is a sane starting point.

Incremental updates vs. full reindexing

Full reindexing at 10 million documents is a multi-hour job. Don't do it for routine changes. Track document IDs and timestamps, and only re-embed what changed. The catch: your vector index needs to support upserts and deletes without rebuilding. Most managed vector databases do. If yours doesn't, that's a reason to switch.

Deduplication and document versioning

Duplicate documents waste embedding budget and pollute retrieval. Hash the normalized text, and skip anything you've already indexed. For versioned documents, keep the latest version in the index and tombstone the old one. Don't delete outright: you'll want the audit trail.

When ingestion lags behind query load, your index serves stale answers. Agents hallucinate on missing context. Monitor ingestion lag as a first-class metric, not an afterthought.

Chunking Strategy for Massive Corpora

Chunking at 10 million documents isn't the same as chunking at 100K. The decisions you made casually early on now multiply across every query, every embedding, and every dollar of storage.

Chunk size vs. retrieval precision

Small chunks give you precision but fragment context. Large chunks preserve context but bury the answer in noise. At scale, the trade-off sharpens: a 256-token chunk retrieves the exact sentence you need, but your agent can't tell what document it came from or why it matters. A 1,024-token chunk carries that context but drags in three paragraphs of irrelevant filler.

The honest answer is that chunk size depends on your query patterns. If users ask for specific facts, go small. If they ask for explanations or summaries, go larger. Most teams land between 300 and 800 tokens.

Parent-child chunking for context recovery

Parent-child chunking splits the difference. You embed small child chunks for retrieval precision, but each child links back to its parent document or section. When a child matches, you pull the parent into context. This gives you precise retrieval without losing surrounding context.

The cost: you store and index both parent and child, which doubles your metadata overhead.

How chunk count multiplies your index size

Ten million documents at 500 tokens per chunk with 20% overlap means roughly 12 million chunks. Every chunk is a vector, a metadata record, and a storage entry. At 1,536 dimensions, that's about 74 GB of vector data alone. Shrink your chunks to 200 tokens and you're looking at 30 million chunks, which triples your index size and your embedding bill. Chunk size isn't just a quality decision. It's a budget decision.

Embedding Generation at Scale

Embedding 10 million documents is a budget line most teams never plan for. You budget for vector storage, query latency, maybe reranking. Then the first full ingestion runs and the embedding bill arrives.

Batching and throughput optimization

Batch your embedding calls. A single request per document means 10 million round trips and 10 million rate-limit checks. Batch 100 to 500 documents per request and you cut that overhead by two orders of magnitude. Most embedding APIs cap batch size, so check your provider's limit and stay just under it.

Throughput also depends on your embedding model. A 1,536-dimension model takes longer per token than a 384-dimension one. If you don't need that much precision, a smaller model cuts both time and cost.

Embedding caching and reuse

Cache embeddings by content hash. When a document updates, only re-embed the changed chunks. Don't re-embed the whole corpus because one paragraph changed. This matters most with incremental ingestion, where 99% of your documents are unchanged between runs.

Cost per million documents: what to expect

Embedding costs vary by provider and model. At roughly $0.0001 per 1,000 tokens, embedding 10 million documents at 500 tokens each runs about $500. That's the optimistic number. Larger models, longer chunks, or re-embedding on every update can push it to $5,000 or more. The hidden cost isn't the first ingestion. It's re-embedding everything every time your chunking strategy changes.

Retrieval Architecture: Vector, Vectorless, or Hybrid?

At 10M+ documents, your retrieval architecture choice determines whether queries return in 50ms or 5 seconds. There are three real options, and each fails in specific ways.

Vector-based retrieval: strengths and degradation at scale

Vector search finds semantically similar content, which is why it works well for paraphrase-heavy queries. A single HNSW index handles a few million vectors fine. Past that, recall drops unless you increase graph connectivity, which increases memory and latency. The honest answer: vector-only retrieval at 10M+ documents needs sharding, and sharding introduces its own recall trade-offs.

Vectorless RAG: what it is and when it works

Vectorless RAG skips embeddings entirely. It uses BM25, TF-IDF, or another lexical matcher to retrieve candidates, then feeds them to an LLM. This works when your queries share vocabulary with your documents. It fails on paraphrase queries, synonyms, or anything where the wording differs. The main catch: you save embedding costs but lose semantic recall. For 10M+ documents with predictable query patterns, it's viable. For open-ended agent memory, it's not.

Hybrid search: BM25 + vectors, and why it's usually the answer

Hybrid search runs BM25 and vector similarity in parallel, then combines scores. BM25 catches exact terms and rare keywords. Vectors catch semantic matches. You get both. The trade-off is complexity: you maintain two indexes and tune a fusion parameter. In practice, hybrid search with reciprocal rank fusion beats either alone on most retrieval benchmarks. At 10M+ scale, it's the default choice unless you have a specific reason not to.

What you cannot do: brute-force scanning, single-index assumptions

You cannot brute-force scan 10 million documents per query. Latency will be seconds, not milliseconds. You cannot assume a single vector index stays fast forever. It won't. You cannot skip metadata filtering and expect hybrid search to save you. It won't. The architecture that works at 100K documents is not the architecture that works at 10M. Plan for that before you hit the wall.

Metadata Filtering and Namespace Isolation

Metadata filtering is the cheapest scaling lever you have. It cuts the candidate set before any vector math happens. At 10M+ documents, that's the difference between searching 10 million vectors and searching 50,000.

Pre-filtering vs. post-filtering

Pre-filtering applies metadata constraints before vector search. The index only scans documents that match your filter. Post-filtering runs vector search first, then discards non-matching results. Pre-filtering is faster when your filter is selective. Post-filtering preserves recall when the filter is loose. The honest answer: use pre-filtering when your metadata cuts the set by 90% or more. Otherwise, post-filtering avoids the recall hit from over-filtering.

Namespace isolation for agent memory

Namespaces partition your index by tenant, user, or agent. A query scoped to one namespace never touches another namespace's vectors. For agent memory, this means each agent's memories stay isolated. You don't search another agent's context by accident. The trade-off: cross-namespace queries get expensive, because you run the search once per namespace.

Designing metadata schemas that scale

Metadata schemas fail when they grow unbounded. Every new field adds index overhead. Keep fields to what you actually filter on: tenant ID, document type, date range, maybe source. Don't index fields you never query. Cardinality matters too. A field with 10 million unique values filters nothing. A field with 10 values filters a lot. Design for low cardinality, high selectivity.

Hybrid Search and Reranking at 10M+ Scale

Hybrid search combines BM25 keyword matching with vector similarity. BM25 catches exact terms and rare identifiers. Vectors catch semantic meaning. At 10M+ documents, neither alone is enough.

BM25 + vector: how to combine scores

The simplest approach is reciprocal rank fusion (RRF). Rank results from each retriever separately, then merge by rank position. RRF needs no score normalization, which is why it works across different scoring scales. Weighted score fusion is the alternative: normalize both scores to a shared range, then take a weighted sum. It's more tunable but more brittle. Start with RRF.

Cross-encoder reranking: when it's worth the latency

A cross-encoder reads the query and document together, not separately. That joint attention gives it far better precision than a bi-encoder. The cost is latency: a cross-encoder pass over 50 candidates adds 50 to 200 milliseconds. Worth it when precision matters more than speed. Not worth it for high-volume, low-stakes queries.

Reranking budget: how many candidates to rerank

Rerank 20 to 50 candidates. Below 20, you risk missing the right document entirely. Above 50, latency climbs without meaningful precision gains. The main catch: cross-encoder cost scales linearly with candidate count. Keep the candidate set small, and reranking stays cheap.

Evaluation and Observability for Large-Scale RAG

You cannot improve what you don't measure. At 10M+ documents, retrieval failures are silent: a query returns plausible-looking but wrong chunks, and your agent confidently hallucinates. Evaluation is the only way to catch it.

Metrics that matter: recall@k, MRR, latency p50/p99

Recall@k measures whether the right document appears in your top k candidates. MRR measures how high it ranks. Both matter. Latency p50 and p99 tell you what most users experience and what your worst users endure. Track cost per query too. A pipeline that returns great results at $0.40 per query won't survive production.

Building an evaluation set from production queries

Don't hand-craft a synthetic eval set. Sample real production queries, log the retrieved documents, and have a human or LLM judge label relevance. Aim for 500 to 1,000 labeled queries. Re-run this set on every pipeline change. Without it, you're guessing.

Observability: tracing every stage of the pipeline

Trace each query through metadata filtering, candidate retrieval, and reranking. Log how many candidates each stage passes forward. When recall drops, you'll know which stage failed. The main catch: tracing adds overhead. Sample 10% of traffic if full tracing is too expensive.

Common Mistakes When Scaling RAG Past 10 Million Documents

Most scaling failures aren't architectural mysteries. They're predictable, avoidable, and expensive. Here's what goes wrong.

Ignoring metadata filtering until it's too late

Metadata filtering is the cheapest lever you have. Narrowing a 10M-document corpus to 50K candidates before vector search costs almost nothing. Skipping it means every query pays full search cost. The fix is retrofitting metadata schemas after ingestion, which means reindexing everything. Do it first, not last.

Assuming one vector index scales forever

A single HNSW index degrades as it grows. Recall drops, latency climbs, and memory pressure forces evictions. You cannot brute-force scan all documents per query. Shard by namespace or time range before you hit the wall. Retrofitting sharding onto a live 10M-document index is a migration project, not a config change.

Underestimating embedding and storage costs

Embedding 10M documents isn't free. At 1,536 dimensions, vectors alone run into hundreds of gigabytes. Embedding API costs can dwarf your database bill. Budget for both before you start. The honest answer: if you haven't calculated cost per million documents, you don't know your scaling budget.

Treating agent memory like a static document store

Agent memory changes constantly. Conversations append, facts update, context expires. A static index goes stale within days. You need incremental updates, TTLs, and versioning. Scaling RAG past 10 million documents means treating memory as a living system, not a snapshot.

Frequently Asked Questions

How many documents can a single vector index handle before it degrades?

Most vector stores remain fast into the low millions of vectors, but performance depends on dimensionality, index type, and available memory. Past roughly 5-10 million vectors, expect to shard or move to a distributed index. Always benchmark with your own data rather than trusting vendor limits.

What is vectorless RAG and when should I use it?

Vectorless RAG retrieves using lexical search, knowledge graphs, or structured filters instead of embeddings. It excels when queries need exact matches—IDs, error codes, legal citations—or when embedding cost is prohibitive. Use it alongside vector search, not as a blanket replacement.

How do I keep latency low when scaling RAG to 10 million documents?

Adopt progressive narrowing: apply cheap metadata filters first, then ANN search on a reduced candidate set, then rerank only the top results. Shard indexes so each query touches a fraction of the corpus. Budget tens of milliseconds per stage and monitor p95, not averages.

Is hybrid retrieval worth the added complexity at scale?

Usually yes. Hybrid retrieval combines lexical precision with semantic recall, which matters more as the corpus grows and queries diversify. The cost is tuning the fusion weight and maintaining two indexes. Start hybrid once you see recall drops that pure vector search cannot fix.

How often should I re-embed documents as my corpus grows?

Re-embed when you change embedding models, when source documents change materially, or on a regular cadence (e.g., quarterly) to counter drift. Full re-embedding at 10M+ docs is expensive, so prefer incremental updates and version your embeddings.

What latency and cost budgets are realistic for a 10M-document RAG pipeline?

Aim for total retrieval latency in the low seconds: tens of milliseconds for filtering, tens to low hundreds for ANN search, and a few hundred for reranking. Cost is dominated by embedding generation and vector index memory. Treat these as starting ranges and measure your own workload.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through scaling rag past 10 million documents or something adjacent, we publish what we have actually tested, including where it falls short.