Backfills Without Taking Retrieval Offline: A Guide

How to Run Backfills Without Taking Retrieval Offline

Backfills without taking retrieval offline feel riskier than database backfills because search has to stay live the whole time. You're re-embedding documents, rebuilding vector indexes, or migrating embeddings while users are still querying against the old data. Most backfill guides cover CDC pipelines or relational tables. Almost none address vector stores or agent memory, which is exactly where RAG builders get stuck. The honest answer is that you can't eliminate every tradeoff, but you can plan around them. GigaRAG gives RAG builders a way to manage index versions and rollbacks without hand-rolling aliases and dual-write paths. This guide covers three vendor-neutral strategies: blue-green index swaps, shadow writes, and progressive backfill. You'll get concrete steps, the mistakes that cause downtime, and a plain-language section on what online backfills cannot do.

In This Guide

• What Does It Mean to Backfill a Retrieval Index?
• Blue-Green Index Swap vs Progressive Backfill
• Why Backfills Without Taking Retrieval Offline Matter for RAG
• Backfills Without Taking Retrieval Offline: A Step-by-Step Guide
• Prerequisites for a Safe Retrieval Backfill
• Strategy 1: Blue-Green Index Swap for Zero-Downtime Backfills
• Strategy 2: Shadow Writes for Incremental Backfills
• Strategy 3: Progressive Backfill for Large Vector Stores
• Common Mistakes When Backfilling Without Taking Retrieval Offline
• What You Cannot Do: Honest Limitations of Online Backfills
• When Backfilling Is the Wrong Choice
• How to Backfill Properly: A Step-by-Step Checklist
• Final Thoughts on Backfills Without Taking Retrieval Offline

What Does It Mean to Backfill a Retrieval Index?

Backfilling a retrieval index means re-embedding documents, rebuilding vector indexes, or migrating embeddings to a new model or store while search stays live. You're not starting from zero. You're filling in what's missing or outdated against an index that's already serving queries.

Backfill vs. initial index build

An initial build starts with an empty index. You embed everything, load it, then point traffic at it. Nothing is live yet, so mistakes cost time but not user trust.

A backfill runs against an index that's already answering queries. You're adding missing documents, replacing stale embeddings, or moving to a new model. The old index keeps serving while the new data lands. That's the core difference: the system is in production the whole time.

Why retrieval backfills are different from database backfills

Database backfills copy rows from one table to another. The data is exact. A row either exists or it doesn't, and you can verify it with a checksum.

Retrieval backfills regenerate embeddings. The same document produces a different vector when your chunking strategy, metadata schema, or embedding model changes. There's no checksum for "semantically close enough." You're comparing relevance, not equality. That makes verification harder and rollback less obvious.

[!note] A retrieval backfill cannot be truly zero-cost: you will temporarily use extra storage and compute for the second index, and dual writes add a small latency overhead to ingestion.

Blue-Green Index Swap vs Progressive Backfill

Why Backfills Without Taking Retrieval Offline Matter for RAG

A stale retrieval index doesn't fail loudly. It degrades quietly: search returns near-misses, agent memory recalls the wrong context, and users can't tell why answers got worse. The stakes are relevance, not availability.

The cost of stale embeddings

When your embedding model or chunking strategy changes, old vectors drift from what queries actually mean. A document about "billing disputes" still matches "payment issues," but it stops matching "chargeback reasons." Relevance drops before anyone notices. By the time users complain, the index has been wrong for weeks.

Agent memory consistency during backfills

Agent memory is worse. If an agent writes new memories to the old index while you backfill the new one, the two diverge. The agent recalls half its history. That's not a degraded search result. That's a broken conversation.

The fix is a backfill that keeps retrieval live and consistent. It's possible, but it takes planning.

[!tip] For agent memory systems, version the embedding model in the index name and store the model version alongside each vector so you can filter or re-embed only the affected memories instead of rebuilding everything.

Backfills Without Taking Retrieval Offline: A Step-by-Step Guide

1. Create a new index with the target mapping and embedding model version.
2. Enable dual writes so every new or updated document is written to both old and new indexes.
3. Backfill historical documents into the new index in batches, tracking progress by document ID.
4. Run shadow queries against the new index and compare recall and latency with the old index.
5. When parity is acceptable, switch the read alias to the new index and monitor error rates.
6. Keep the old index for a rollback window, then decommission it once stable.

Prerequisites for a Safe Retrieval Backfill

You can't backfill safely without groundwork. Three things need to exist before you touch a single embedding.

Version your embeddings and metadata

Every embedding you write needs a version tag: model name, chunking strategy, and a timestamp. Without it, you can't tell which vectors are stale. Store the version in metadata alongside each vector. When you backfill, the new index gets a new version. The old one keeps its old version. That's your rollback path.

Set up index aliases or namespaces

Your application should never point at a raw index name. It points at an alias. Backfills write to a new index, then you swap the alias. If your vector store doesn't support aliases, use namespaces. Same idea: a stable pointer that moves.

Instrument search latency and relevance

You need numbers before you backfill. Track p95 latency and a relevance sample (say, 50 queries you check by hand). Without a baseline, you can't tell if the backfill made things better or worse.

Strategy 1: Blue-Green Index Swap for Zero-Downtime Backfills

The blue-green swap is the cleanest way to rebuild a vector index without touching live search. You build a new index next to the old one, backfill it fully, then flip an alias. Search never points at a half-built index.

Build the green index in parallel

Create the new index with the same schema as the old one, plus your new embedding version tag. Give it a different name: documents_v2 next to documents_v1. Your application keeps reading from the alias, which still points at documents_v1. Nothing changes for users yet.

Backfill embeddings into the green index

Run your backfill job against documents_v2. Re-embed every document with the new model or chunking strategy, then write vectors with the new version tag. This can take hours or days for large stores. That's fine. The old index serves traffic the whole time.

Throttle the job so it doesn't starve your embedding API or vector store. You're not racing anyone.

Atomic alias swap and rollback

Once the green index has full parity, swap the alias from documents_v1 to documents_v2. Most vector stores do this as a single atomic operation. Search now hits the new index.

Rollback is the same move in reverse: point the alias back at documents_v1. Keep the old index around for at least a week. Delete it only after you've watched latency and relevance numbers hold steady.

The main catch: you pay for double storage during the backfill. For large indexes, that's real money.

Strategy 2: Shadow Writes for Incremental Backfills

Shadow writes keep the old index live while you dual-write every new embedding to the new index too. No big batch job. No cutover cliff. You just run both indexes side by side until the new one catches up.

Dual-write to old and new indexes

Point your ingestion pipeline at both indexes. Every new document gets embedded once, then written to documents_v1 and documents_v2 in the same transaction or back-to-back. Reads still hit documents_v1 only.

This works best when your backfill is mostly about new data, not re-embedding old data. Existing documents still need a one-time backfill, but that job can run slowly in the background while shadow writes handle everything new.

Verify parity before cutover

Don't trust the write path. Check it. Run a count comparison between indexes daily. Sample 100 document IDs and confirm both indexes return the same vectors and metadata. Watch for silent write failures on the new index.

Once counts match and sampled lookups agree, flip the alias. Rollback is the same alias move as the blue-green swap.

When shadow writes are overkill

Skip this if your ingestion rate is low. A handful of writes per hour doesn't justify running two indexes. Just do a blue-green swap.

Skip it too if your new index has a different schema. Shadow writes only work when both indexes accept the same payload shape.

Strategy 3: Progressive Backfill for Large Vector Stores

Progressive backfill is for when the index is too big to rebuild in one go. You backfill in batches or shards while search keeps running against the old index. Traffic shifts over gradually, not all at once.

Batch by namespace or shard

Split your index by namespace, tenant, or shard. Backfill one slice at a time. A 50 million vector index might break into 20 shards of 2.5 million each. You backfill shard 1, verify it, then move to shard 2.

This keeps memory and CPU use predictable. The old index serves reads for shards you haven't touched yet. The new index serves shards you've already backfilled. You need a routing layer that knows which shard lives where.

Throttle embedding generation

Embedding is the bottleneck. Don't fire 50 million documents at your embedding API in one burst. You'll hit rate limits, burn budget, and stall the backfill.

Set a throttle. 100 embeddings per second is a reasonable starting point for most providers. Adjust based on your rate limit and queue depth. Track failures. Retry with backoff. A backfill that dies at 80 percent because you hammered the API is worse than a slow one.

Shift traffic gradually

Don't flip the whole index at once. Route 5 percent of queries to the new index. Watch latency and relevance. If results hold, move to 25 percent, then 50, then 100.

This catches problems a blue-green swap misses. A relevance regression that only shows up under real query patterns appears at 5 percent traffic, not after a full cutover. Rollback is per-shard: point that slice back at the old index while you investigate.

The main catch is complexity. You're running a routing layer, a throttled backfill job, and partial traffic shifting all at once. For indexes under a few million vectors, a blue-green swap is simpler and just as safe.

Common Mistakes When Backfilling Without Taking Retrieval Offline

Most failed backfills aren't caused by the strategy. They're caused by skipping the boring parts: rollback plans, idempotency, and metadata checks. Here are the three that hurt most.

Skipping rollback planning

You need a way back before you start. Not after something breaks. A rollback plan means you know exactly which alias to point back at the old index, how long that takes, and who runs it.

The honest answer is that rollback is the cheapest insurance you'll buy. A blue-green swap gives you a one-command rollback: point the alias back. A progressive backfill gives you per-shard rollback. If you can't describe your rollback in two sentences, don't start the backfill.

Non-idempotent embedding writes

Your backfill job will retry. It will retry after timeouts, after rate limits, after partial failures. If writing the same document twice creates two vectors, you get duplicates. Search relevance degrades silently.

Make every write idempotent. Use a deterministic document ID as the vector ID. Upsert, don't insert. The same document maps to the same vector every time, no matter how many times the job retries.

Ignoring metadata and filter drift

Embeddings aren't the only thing that changes. Metadata filters change too. A document's category, permissions, or tenant ID may have shifted since the original index was built.

Backfill the metadata alongside the vectors. If your new index has stale filters, queries return wrong results even when the embeddings are fresh. Check filter parity before you cut over. A vector that's right but filtered wrong is still a wrong answer.

What You Cannot Do: Honest Limitations of Online Backfills

No strategy gets you a zero-latency cutover. Every approach has a moment where the alias flips, traffic shifts, or a batch lands. That moment is fast, but it isn't free.

No zero-latency cutover

An atomic alias swap takes milliseconds. It still isn't zero. During that window, in-flight queries hit the old index while new queries hit the green one. You'll see a tiny spike in p99 latency. Plan for it, don't pretend it away.

Consistency tradeoffs are unavoidable

Dual-writes mean two indexes drift apart. A document written to the new index but not yet to the old one reads differently depending on which index serves the query. You can shrink the gap with tighter write ordering. You can't remove it.

Model changes force re-embedding

Switching embedding models means every vector is wrong. There's no backfill that avoids re-embedding. You rebuild from source documents or you accept stale results. The backfill cost is the re-embedding cost.

When Backfilling Is the Wrong Choice

Backfilling isn't always the right move. Sometimes the index is small enough that a rebuild is faster and safer than a careful backfill. Sometimes the model changes so often that backfilling becomes a permanent job. And sometimes staleness just doesn't matter.

Small indexes: rebuild from scratch

If your index has a few thousand vectors, a full rebuild takes minutes. The overhead of blue-green swaps, shadow writes, and parity checks costs more than the rebuild itself. Drop the old index, build a new one, swap the alias. Done.

Frequent model changes: avoid backfills

If you swap embedding models every month, backfilling means re-embedding everything every month. That's not a backfill strategy. That's a pipeline design problem. Build re-embedding into your normal ingestion flow instead of treating it as a one-off event.

When staleness is acceptable

Some retrieval use cases don't need fresh embeddings. Internal documentation search, archived content, or low-traffic indexes can lag by days without real harm. If nobody notices stale results, don't spend engineering time fixing them.

How to Backfill Properly: A Step-by-Step Checklist

The strategies above work when you apply them in order. Here's the full workflow, compressed into a checklist you can run against any retrieval index.

Plan and version

Version your embedding model, your chunking logic, and your metadata schema before touching data. Write down the rollback point: which alias points where, which namespace holds the old index, and how to flip back. If you can't answer "how do I undo this" in one sentence, don't start.

Build and backfill

Build the new index alongside the old one. Backfill in batches, throttled to keep embedding generation under your rate limits. Use idempotent writes so a failed batch can retry without duplicating vectors. Check parity against the old index before you consider a cutover.

Verify and cut over

Run relevance checks against both indexes with real queries. Compare top-k results. When parity holds, swap the alias or flip the feature flag. Keep the old index around. Don't delete it.

Monitor and roll back

Watch search latency and relevance for the first hour after cutover. If latency spikes or results degrade, flip back to the old index. The rollback should be one alias update, not a rebuild.

Final Thoughts on Backfills Without Taking Retrieval Offline

The checklist works because it treats backfills as a process, not a single risky operation. Blue-green swaps, shadow writes, and progressive batching all share the same core: build alongside, verify, then cut over. None of them require taking search down.

The honest catch is that planning takes longer than the backfill itself. Versioning embeddings, setting up aliases, writing idempotent paths, and instrumenting latency is real work before you move a single vector. Skip it and you'll pay in rollback pain.

For RAG builders, GigaRAG handles much of this plumbing: index aliases, batched re-embedding, and rollback paths are built in rather than hand-rolled. It won't make backfills free, but it removes the parts most teams get wrong.

Backfills without taking retrieval offline are achievable. You just need the discipline to build the new index before you touch the old one.

Frequently Asked Questions

What are common backfilling mistakes?

Common mistakes include backfilling without dual writes, so new documents are missed; not versioning the embedding model, which mixes incompatible vectors; and skipping shadow queries, so recall regressions are only discovered after the swap.

What does it mean to backfill data?

Backfilling means populating a new or updated data store with historical records that were created before the new store existed. In retrieval, it usually means re-embedding and indexing documents that are already in the old index.

What should you avoid when backfilling?

Avoid writing directly to the live index, avoid deleting the old index before the new one is verified, and avoid assuming that batch order does not matter. Also avoid ignoring rate limits on your embedding provider, which can stall the backfill.

How to backfill properly?

Backfill properly by using a blue-green index pattern: create a new index, dual-write new data, backfill historical data in batches, validate with shadow queries, then swap the read alias. Keep the old index for rollback until the new one is stable.

Can you backfill a vector index without taking retrieval offline?

Yes, if you use a second index and an alias swap. The live index keeps serving queries while the new index is built and validated. The trade-off is extra storage and compute during the rebuild.

How do you handle agent memory during a backfill?

Treat agent memory like any other retrieval data: dual-write new memories to both indexes, backfill older memories in batches, and version the embedding model so you can identify which memories need re-embedding. Avoid clearing memory during the swap.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through backfills without taking retrieval offline or something adjacent, we publish what we have actually tested, including where it falls short.