Incremental Indexing and Re-Embedding Without Downtime: A Practical Guide for Agent Memory and RAG
Incremental indexing and re-embedding without downtime is the problem every backend engineer hits the week their embedding model changes. A team re-embeds 5 million documents, takes the API offline for six hours, and serves stale results for another two days while the index rebuilds. That's the tension: freshness versus cost versus downtime. The good news is you don't have to pick two and sacrifice the third. You can update individual records, swap embedding models behind a dual-index cutover, and keep the system live throughout. The honest answer is it depends, and it depends on your data volatility, your latency budget, and whether your vector store supports efficient upserts. GigaRAG handles incremental indexing natively for agent memory and RAG pipelines, but the patterns here work across Chroma, Pinecone, Weaviate, and pgvector. This guide covers a decision framework for choosing incremental indexing over live SQL or full re-embedding, implementation patterns for change tracking and cutover, and the failure modes no one else will name.
| At a glance | Details |
|---|---|
| Core idea | Update changed vectors only, not the whole index |
| Best for | High-churn data: agent memory, tool outputs, live docs |
| Main risk | Embedding drift when model versions change |
| Key requirement | Vector store with efficient upsert and delete |
| Typical latency | Seconds to minutes, not hours |
| Not a fix for | Bad chunking or stale embedding models |
In This Guide
- Why Full Re-Indexing Is the Problem You Can't Afford
- Incremental Indexing vs Full Re-Embedding: Which Should You Choose?
- What Incremental Indexing and Re-Embedding Without Downtime Actually Means
- Incremental Indexing And Re-embedding Without Downtime: A Step-by-Step Guide
- When Incremental Indexing Is the Right Call (and When It Isn't)
- Incremental Indexing for Agent Memory: Episodic Memory, Tool Outputs, and Conversation State
- Implementation Patterns for Incremental Indexing Without Downtime
- What Incremental Indexing Won't Fix
- Consistency Guarantees and Failure Recovery During Incremental Updates
- Cost Analysis: Incremental vs. Full Re-Indexing
- Putting It Together: A Checklist for Incremental Indexing Without Downtime
Why Full Re-Indexing Is the Problem You Can't Afford
Full re-indexing means re-embedding every document, then rebuilding the index from scratch. For a 5M-document corpus, that's days of compute and a bill you don't want to see.
The cost of re-embedding at scale
Embedding API calls are the hidden multiplier. Re-embedding 5M documents at $0.0001 per 1K tokens adds up fast. Compute time compounds it: hours of GPU or CPU work, plus storage churn as old vectors get replaced.
Downtime and staleness: two sides of the same coin
You either take the index offline and serve stale results, or you serve a half-built index and risk wrong answers. Neither is acceptable for agent memory.
[!note] Incremental indexing cannot fix embedding drift: if you change the embedding model or its version, all existing vectors become incompatible and a full re-embedding is required. It also cannot repair poor chunking or a vector store that lacks efficient upsert and delete operations.
Incremental Indexing vs Full Re-Embedding: Which Should You Choose?
| Factor | Incremental Indexing | Full Re-Embedding |
|---|---|---|
| Data volatility | High-churn data (agent memory, live docs) | Stable, rarely changing corpora |
| Downtime | Near-zero with dual-write or alias swap | Requires maintenance window or shadow index |
| Cost | Low: only changed chunks re-embedded | High: every chunk re-embedded |
| Embedding model change | Not supported; vectors become incompatible | Required when model or dimensions change |
| Complexity | Higher: needs change tracking and upsert logic | Lower: rebuild from scratch, simpler pipeline |
What Incremental Indexing and Re-Embedding Without Downtime Actually Means
Incremental indexing updates individual records as they change, while re-embedding replaces the embedding model or chunking strategy across all records. Both can happen without taking your index offline, but they solve different problems.
Incremental indexing vs. re-embedding: two different problems
Incremental indexing handles data changes: a new document arrives, an old one gets edited, a record gets deleted. You upsert or remove individual vectors, leaving the rest of the index untouched. Re-embedding handles model changes: you swap embedding models or alter chunk sizes, which invalidates every vector in the index.
The honest answer is these are separate operations. Incremental indexing won't fix a bad embedding model. Re-embedding won't help you track which documents changed since last Tuesday.
What "without downtime" requires from your vector store
Your vector store needs efficient upsert and delete operations, not just bulk insert. It also needs a way to serve queries against the old index while a new one builds. That means dual-index cutover or a store that supports in-place updates without blocking reads. If your store only does full rebuilds, you can't get there.
[!tip] For agent memory, treat episodic memory and tool outputs as separate namespaces with their own retention and re-embedding policies—this lets you expire short-lived tool outputs without touching long-term conversation state.
Incremental Indexing And Re-embedding Without Downtime: A Step-by-Step Guide
- Audit your data sources and classify each as high-churn (agent memory, tool outputs) or low-churn (static docs).
- Choose a vector store that supports efficient upsert and delete by ID; verify its consistency guarantees.
- Implement a change data capture (CDC) or event stream that emits document IDs and versions on every write.
- Build an embedding worker that consumes the stream, re-embeds only changed chunks, and upserts them with the same IDs.
- Use a dual-write or alias-swap pattern: write to a new index version, then atomically switch the query alias.
- Add a reconciliation job that periodically compares source and index checksums to catch missed updates.
- Monitor index freshness, query latency, and embedding drift; alert when drift exceeds a threshold.
When Incremental Indexing Is the Right Call (and When It Isn't)
The decision comes down to three variables: how often your data changes, how fast queries need to reflect those changes, and whether your embedding model is stable.
The decision tree: volatility, latency, and model stability
If data changes constantly and queries need fresh results within seconds, incremental indexing is the right call. You upsert individual records as they change, and the index stays current without a full rebuild.
If data changes constantly but queries can tolerate minutes of staleness, live SQL querying may beat re-embedding entirely. You query the source of truth directly and skip the vector index for those fields.
If your embedding model changes, incremental indexing won't help. Every vector is now wrong. You need full re-embedding.
When live SQL querying beats re-embedding
For structured fields like status, counts, or timestamps, query the database directly. Don't embed what you can filter. It's faster and always fresh.
When full re-embedding is unavoidable
Model changes, chunking strategy changes, or a vector store that lacks efficient upsert all force a full rebuild. Plan for dual-index cutover when that happens.
Incremental Indexing for Agent Memory: Episodic Memory, Tool Outputs, and Conversation State
Agent memory is not a document store. It's a stream of experiences, tool results, and dialogue turns that change constantly. Incremental indexing fits this workload better than any batch job.
Episodic memory: indexing agent experiences as they happen
Each agent action produces a record: what it tried, what happened, what it learned. You index that record immediately after the action completes. Don't wait for a batch window. The memory becomes queryable while the task is still running.
Tool outputs: indexing results without re-indexing the world
When an agent calls a search API or a calculator, the output goes into memory as a new record. You upsert just that one result. The rest of the index stays untouched. This keeps tool outputs searchable without paying to re-embed everything the agent has ever seen.
Conversation state: incremental updates to dialogue context
Dialogue context shifts with every turn. You update the conversation record in place rather than appending duplicates. One upsert per turn. The vector store holds the latest state, and old turns age out through deletes.
Implementation Patterns for Incremental Indexing Without Downtime
The patterns below work across Chroma, Pinecone, Weaviate, and pgvector. They assume you already know which records changed. The hard part is applying those changes without a visible gap.
Change tracking with SQLRecordManager and CDC
SQLRecordManager tracks which documents have been indexed by storing a hash or timestamp per record. On each run, compare your source of truth against that table. Changed records get upserted. Deleted records get removed. Nothing else is touched.
Change data capture (CDC) works the same way but reads from your database's write-ahead log. It catches changes as they happen instead of polling. CDC adds operational overhead, so use it only when latency matters.
Upsert patterns across Chroma, Pinecone, Weaviate, and pgvector
All four stores support upsert by ID. The ID is your contract. If you reuse the same ID, the store replaces the old vector. If you generate a new ID, you get duplicates.
Chroma and pgvector let you run upserts directly against the store. Pinecone and Weaviate batch them through their clients. The pattern is identical: send the ID, the new vector, and the metadata. The store handles the rest.
Dual-index cutover for embedding model migration
When you change embedding models, you can't upsert into the same index. Old vectors and new vectors live in different spaces. Build a second index with the new model in parallel. Once it's fully populated, switch your query layer to point at it. The old index stays up until the switch completes. Then you drop it.
What Incremental Indexing Won't Fix
Incremental indexing solves one problem: keeping your index in sync with changed records. It doesn't touch the other failure modes. Here's what still breaks.
Embedding model drift: you still need re-embedding
If your embedding model changes, every vector in your index is wrong. Incremental indexing only updates records that changed. It won't re-embed the 4.9 million documents that didn't. You need a full re-embedding pass or a dual-index cutover. There's no shortcut.
Bad chunking is a data problem, not an indexing problem
If your chunks split paragraphs mid-sentence or bury answers in noise, incremental indexing won't help. The vectors faithfully represent bad chunks. Fix the chunking strategy first, then re-index. Incremental updates just propagate the same mistake faster.
Vector stores without efficient upsert will fight you
Some stores treat every upsert as a delete plus insert. That's fine for ten records. It's a disaster for ten thousand. Check your store's upsert path before committing to incremental indexing. If it rewrites the whole segment on every update, you've traded downtime for write amplification.
Consistency Guarantees and Failure Recovery During Incremental Updates
An incremental update that fails mid-write leaves your index half-updated. The fix is treating every write as idempotent: running the same upsert twice produces the same result. That's the only way to recover without a full re-index.
What "consistent" means for a vector index
Consistent here means the vector index matches your source of truth at the moment a query hits it. Not eventually. Not mostly. If a document was deleted from Postgres but still returns in search results, you've got a consistency bug. Most vector stores offer no transactional guarantees across your database and the index. You build consistency yourself by tracking which records changed and replaying those changes until the index catches up.
Failure recovery: idempotent upserts and retry strategies
Upserts are naturally idempotent if you key them by record ID. A failed write leaves the old vector in place. Retry the same upsert and it overwrites cleanly. Deletes are trickier: a failed delete leaves a stale vector. Track deletes in a separate queue and replay them until confirmed. For embedding API failures mid-batch, split the batch and retry the failed half. Don't restart the whole job.
Cost Analysis: Incremental vs. Full Re-Indexing
The cost difference comes down to one number: how many records actually changed. Incremental indexing pays for changed records only. Full re-indexing pays for everything, every time.
Embedding API costs: the hidden multiplier
Embedding calls are priced per token. Re-embedding 5 million documents because 200 changed wastes 4,999,800 calls. Incremental indexing sends only the 200. The multiplier isn't the index update. It's the embedding API bill you didn't need to run.
Compute and storage churn compared
Full re-indexing rewrites every vector, burning GPU or CPU hours and triggering storage rewrites. Incremental upserts touch only changed records. Storage churn drops from the full corpus size to the delta size.
Putting It Together: A Checklist for Incremental Indexing Without Downtime
Pre-implementation checklist
- Confirm your vector store supports efficient upsert. Test it on 1,000 records before committing.
- Set up change tracking first. SQLRecordManager or CDC, not ad-hoc scripts.
- Pin your embedding model version. Record it in config.
- Define your chunking strategy in code. No manual edits.
Go-live and monitoring checklist
- Run dual-index cutover for model changes. Never swap in place.
- Make every upsert idempotent. Retry on failure.
- Monitor freshness: time from source change to index update.
- Alert on upsert latency spikes, not just error counts.
Incremental indexing and re-embedding without downtime works when change tracking, idempotent upserts, and cutover are in place. Skip any one and you'll find out during a failure.
Frequently Asked Questions
What is incremental indexing in a RAG pipeline?
Incremental indexing is the process of updating only the changed or new documents in a vector index, rather than rebuilding the entire index from scratch. It relies on change tracking and upsert operations to keep the index fresh with minimal compute and downtime.
When should I avoid incremental indexing?
Avoid incremental indexing when you change the embedding model or its version, because old and new vectors are not comparable and a full re-embedding is required. It is also a poor fit if your vector store lacks efficient upsert and delete, or if your chunking strategy is flawed—those need a rebuild, not an update.
How do I handle embedding model upgrades without downtime?
Use a shadow index: build a new index with the new model while the old index serves queries, then switch an alias once the new index is fully populated and validated. This requires a full re-embedding but avoids downtime.
Can incremental indexing work for agent memory?
Yes, agent memory is a strong use case because episodic memory, tool outputs, and conversation state change frequently. Treat each memory type as a separate namespace with its own update and retention policy to avoid re-embedding stable data.
What vector stores support efficient upsert for incremental indexing?
Many modern vector stores support upsert and delete by ID, but performance and consistency vary. Evaluate your store's documentation for upsert semantics, consistency guarantees, and whether it supports metadata filtering for namespace isolation.
How do I detect missed updates in an incremental index?
Run a periodic reconciliation job that compares checksums or timestamps between your source of truth and the index. Alert on mismatches and re-index the affected documents.
What is embedding drift and why does it matter?
Embedding drift occurs when the embedding model changes, causing new vectors to be incompatible with existing ones. It matters because similarity search degrades silently, returning irrelevant results even though the index appears healthy.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through incremental indexing and re-embedding without downtime or something adjacent, we publish what we have actually tested, including where it falls short.
