What We Changed in Our Architecture After N Queries
What we changed in our architecture after N queries started as an accident. The demo worked. The pipeline answered, retrieved, and ranked with confidence. Then real traffic hit. Latency climbed from 400ms to 3 seconds. Costs tripled in a week. Retrieval quality dropped because the same embeddings kept getting recomputed and the context window kept swallowing stale memory. Tuning helped for a day. Then the same wall, again. That's when it became obvious: this wasn't a parameter problem. It was a shape problem. Agent memory and RAG pipelines degrade in specific, predictable ways as query volume grows. GigaRAG, a platform built for exactly these builders, sees the same failure patterns across teams. This guide covers the five architectural shifts we made after N queries, what each one fixed, what each one broke, and what you cannot expect any architecture change to solve.
| At a glance | Details |
|---|---|
| Primary trigger | Latency and cost spikes after sustained query volume |
| First fix | Cache embeddings and retrieval results |
| Second fix | Separate ingestion from query-time retrieval |
| Third fix | Introduce tiered memory and context compression |
| What won't help | Only tuning model parameters or prompt wording |
| When to re-architect | When p95 latency or cost per query grows non-linearly |
In This Guide
- The Breaking Point: What N Queries Actually Does to a RAG Architecture
- Monolithic RAG Pipeline vs. Tiered Agent Memory Architecture
- What We Changed in Our Architecture After N Queries: The Five Shifts
- What We Changed In Our Architecture After N Queries: A Step-by-Step Guide
- Why This Is an Architecture Problem, Not a Tuning Problem
- Agent Memory and RAG Pipelines: The Architecture Changes No One Talks About
- What You Cannot Do: Honest Limitations of Architecture Changes
- How to Catch the Pattern Structurally Before It Catches You
- Where to Start: A Prioritized Roadmap for Your Architecture
- FAQ
The Breaking Point: What N Queries Actually Does to a RAG Architecture
The demo worked. The pipeline answered questions, cited sources, and stayed under budget. Then real users arrived.
The demo-to-production cliff
A RAG system that handles 50 queries a day can fall over at 500. Not gradually. The failure is a cliff, not a slope. Latency spikes from 300ms to 9 seconds. Your embedding bill triples overnight. Retrieval quality drops because you started cutting corners to stay alive: smaller top-k, shorter context, fewer reranking passes.
The cliff exists because naive architectures assume each query is independent. One query in, one answer out. That assumption holds in a demo. It dies the moment queries overlap, repeat, or arrive in bursts.
What N queries exposes: fan-out, cache misses, context bloat
Three bottlenecks compound as volume grows.
Fan-out is the first. One user query becomes five retrieval calls, each embedding a rewritten variant, each hitting the vector store, each returning 20 chunks. At N queries, that's 5N embedding calls and 5N vector searches. Your embedding model is now the bottleneck, not your retriever.
Cache misses follow. Naive pipelines embed every query from scratch, even when the same query arrived ten minutes ago. No memoization, no similarity check against recent queries. You pay full price for repeat work.
Context bloat compounds both. Agent memory systems that append every retrieved chunk to a running context hit token limits fast. At N queries, the context is mostly stale retrieval from earlier turns. The model spends its attention budget on noise.
Why this is an architecture problem, not a tuning problem
Tuning changes parameters. Architecture changes data flow.
You can tune batch sizes, adjust top-k, or switch embedding models. Those buy you 10-20% improvements. They don't fix fan-out, because fan-out is a structural choice: you designed the pipeline to embed and search per query variant. No parameter change removes that step.
Cache misses are the same. If your pipeline has no cache layer, tuning won't create one. Context bloat is worse: it's a state management problem. You either consolidate memory or you don't. Tuning the prompt won't reclaim a token budget that was spent on bad retrieval three turns ago.
The tell is simple. When your fixes stop being "change this number" and start being "add this component," you've crossed from tuning into architecture. That crossing is where the real work begins.
[!note] Architectural changes after N queries are about removing bottlenecks that only appear at volume; they do not fix poor retrieval relevance or bad prompt design, which should be addressed first.
Monolithic RAG Pipeline vs. Tiered Agent Memory Architecture
| Factor | Monolithic RAG Pipeline | Tiered Agent Memory Architecture |
|---|---|---|
| Latency at scale | Grows with query volume and context size | Stays bounded via caching and tiered retrieval |
| Cost per query | Rises with every additional retrieval and LLM call | Controlled by reusing cached embeddings and summaries |
| Retrieval quality | Degrades as the index grows and context dilutes | Maintained by separating hot and cold memory tiers |
| Operational complexity | Simple to start, hard to debug at scale | More components, but each has clear ownership |
| Failure mode | Cascading timeouts and cost overruns | Graceful degradation with fallback tiers |
What We Changed in Our Architecture After N Queries: The Five Shifts
After N queries, we made five architectural changes: a tiered embedding cache, query routing with sharding, consolidated agent memory, iterative retrieval with reranking, and instrumented pipeline stages. Each fixed a structural bottleneck that tuning could not.
Shift 1: From on-the-fly embedding to a tiered embedding cache
We embedded every query from scratch. Same query ten minutes apart? Full embedding cost both times. The fix was a two-tier cache: an exact-match layer for identical queries, then a semantic layer that checks cosine similarity against recent embeddings and reuses any above 0.98.
The tradeoff is staleness. If your knowledge base updates, cached embeddings can point at outdated chunks. We invalidate the semantic tier on every index refresh, which costs us a cold-start spike but keeps retrieval honest.
Shift 2: From flat vector search to query routing and sharding
One vector index for everything worked until it didn't. At roughly 10,000 queries per day, a single index meant every query scanned every namespace. We split the index by domain and routed queries to the relevant shard before search.
The catch: routing is only as good as your classifier. Misrouted queries miss relevant chunks entirely. We accept that failure mode because the latency win is real: p95 search time dropped from 800ms to 120ms on sharded indexes.
Shift 3: From stateless retrieval to consolidated agent memory
Each agent turn retrieved fresh chunks and appended them to context. After N turns, the context was mostly stale retrieval from earlier in the conversation. We added a consolidation step: after each turn, retrieved chunks get summarized and merged into a fixed-size memory block.
The tradeoff is information loss. Summarization drops details that a later turn might need. We mitigate by keeping raw chunk references alongside summaries, so the agent can re-fetch specifics when the summary looks thin.
Shift 4: From single-pass RAG to iterative retrieval with reranking
One retrieval pass, one answer. That was the design. But complex queries need follow-up retrieval: the first pass surfaces context, the model identifies gaps, a second pass fills them. We added a reranking step between passes to filter the first pass's noise.
The cost is latency. Two retrieval passes plus reranking adds 200-400ms per query. We gate it: simple queries get single-pass, complex queries get iterative. The gate is a classifier that checks query length and entity count.
Shift 5: From monolithic pipeline to observable, instrumented stages
We couldn't see where time went. The pipeline was one black box: query in, answer out. We split it into named stages (embed, route, retrieve, rerank, generate) and logged latency, token count, and cache hit rate per stage.
The tradeoff is engineering overhead. Instrumentation code is boring and easy to skip. But it's what made shifts 1 through 4 possible: we only knew embedding was the bottleneck because the stage-level logs said so.
[!tip] For RAG and agent memory builders: before adding new infrastructure, measure the cost and latency of a single query end-to-end. Often the biggest win is caching the embedding step, which is frequently the most expensive and least variable part of the pipeline.
What We Changed In Our Architecture After N Queries: A Step-by-Step Guide
- Instrument p50, p95, and p99 latency plus cost per query to identify the first bottleneck.
- Add a caching layer for embeddings and retrieval results, keyed by normalized query and context hash.
- Separate the ingestion pipeline from the query-time retrieval path so writes do not block reads.
- Introduce a tiered memory model: hot cache, warm vector store, and cold archive with different SLAs.
- Compress or summarize long context before it reaches the LLM to reduce token cost and latency.
- Add backpressure and circuit breakers so a slow dependency cannot cascade into full outages.
- Re-evaluate retrieval quality with a fixed evaluation set after each architectural change.
Why This Is an Architecture Problem, Not a Tuning Problem
Tuning changes parameters. Architecture changes data flow. The difference matters because they fail differently under load.
Tuning buys linear gains; architecture buys step changes
Tuning an embedding model's batch size from 32 to 64 gets you maybe 15% better throughput. Tuning a prompt to use fewer tokens cuts cost by a fixed percentage. These are linear gains: you get back a fraction of what you put in, and the ceiling is set by the current design.
Architecture changes move the ceiling. Sharding a vector index took our p95 search from 800ms to 120ms. That's not a 15% improvement. That's an 85% drop, and it holds as volume grows because the bottleneck itself was removed, not squeezed.
Here's the test: if a change makes the current design faster, it's tuning. If it changes what the system does, it's architecture.
The telltale signs you've outgrown tuning
You've outgrown tuning when the same fix stops working the second time. You bump the embedding cache size, latency drops for a week, then it's back. You increase the context window, quality improves, then degrades as the window fills with noise.
Three signs stand out. First, your p95 latency tracks query volume linearly no matter what parameters you adjust. Second, you're tuning the same knob repeatedly and getting diminishing returns. Third, your fixes create new problems: faster retrieval floods the context window, which slows generation.
When you see those, stop tuning. The design is the problem.
What tuning cannot fix in RAG and agent memory
Tuning cannot fix fan-out. If one agent turn triggers five retrieval calls, no embedding batch size fixes that. You need to consolidate the calls or cache the results.
Tuning cannot fix unbounded context growth. Prompt tweaks don't stop a conversation from accumulating stale chunks. You need a memory consolidation step that summarizes and drops.
Tuning cannot fix a missing stage. If you can't see where latency lives, no parameter change will find it. Instrumentation is architecture, not tuning, and it's the prerequisite for every other fix.
Agent Memory and RAG Pipelines: The Architecture Changes No One Talks About
Most RAG writing stops at retrieval quality. It ignores the two resources that actually break under load: memory and context.
Memory consolidation: what to keep, what to summarize, what to drop
Agent memory is not a log file. If you append every turn, retrieval degrades as the store fills with stale and contradictory chunks. The fix is a consolidation step that runs after each session, not during it.
Keep what changed state: decisions, user preferences, facts that contradict earlier facts. Summarize what only matters as history: the back-and-forth that led to a decision, not the decision itself. Drop what no agent will ever query again: greetings, retries, dead ends.
The honest answer is that consolidation is lossy. You will drop something useful eventually. The alternative, keeping everything, guarantees slower retrieval and noisier context for every future query.
Context window as a scarce architectural resource
Context windows look free. They are not. Every token you stuff into a prompt costs latency and money, and it dilutes attention on the tokens that matter.
Treat the window like a cache with a hard budget. Reserve a fixed slice for the current task, a smaller slice for consolidated memory, and leave headroom for the model's own reasoning. When retrieval returns more chunks than fit, rerank and truncate before assembly, not after.
The main catch: truncation is a policy decision, not a parameter. Someone has to decide what gets cut, and that decision changes per use case.
Multi-agent retrieval fan-out and its hidden costs
One agent querying a vector store is manageable. Five agents querying it in parallel is not. Each agent fans out its own retrieval calls, and the vector store sees a multiplier it was never sized for.
The hidden cost is not compute. It's duplicate work. Three agents retrieve the same chunks because they share no memory of what the others already pulled. A shared retrieval cache, keyed by embedding hash, cuts that waste before it reaches the store.
Keep in mind that shared caches create coupling. One agent's bad cache entry poisons every other agent that reads it. Version your cache keys and expire aggressively.
What You Cannot Do: Honest Limitations of Architecture Changes
Architecture changes fix how components connect. They do not fix what flows through them.
Architecture cannot fix bad embeddings or poor chunking
If your embeddings map unrelated text to nearby vectors, no amount of sharding or caching will recover that loss. The retrieval is faithfully returning garbage. Same with chunking: chunks that split a definition across two boundaries will confuse retrieval no matter how fast the pipeline runs.
The fix is upstream. Re-embed with a better model or re-chunk with domain-aware boundaries. Architecture only amplifies what the data already contains.
Architecture cannot eliminate LLM latency
A faster vector store does not make the model generate tokens faster. You can hide latency with streaming, parallelism, or caching, but the model's time-to-first-token and tokens-per-second stay fixed by the provider.
The honest answer: architecture buys you fewer model calls, not faster ones.
Architecture cannot replace domain-specific retrieval design
A generic cosine similarity search will not know that "quarterly report" and "Q3 filing" mean the same thing in your domain. That requires synonym maps, query rewriting, or a knowledge graph. None of those are architecture changes. They are retrieval design decisions you still have to make.
Architecture gives you room to run a better retrieval strategy. It does not write one for you.
How to Catch the Pattern Structurally Before It Catches You
You don't need to wait for a production outage to know your architecture is drifting. The signs show up in metrics and query shapes long before latency spikes become user-visible.
Metrics that reveal architectural debt before it's critical
Watch three numbers. First, fan-out ratio: how many retrieval calls one user query triggers. A healthy RAG pipeline sits at 1:1 or 1:2. When you see 1:8 or 1:15, you're paying for redundant lookups that caching or routing would eliminate. Second, cache hit rate on embeddings. Below 60% means you're re-embedding text you've already seen, which is pure waste. Third, p95 latency per pipeline stage. If the vector search stage is flat but the orchestration stage is climbing, the bottleneck isn't your database. It's how you're calling it.
Track these per query volume, not as daily averages. Averages hide the cliff.
Query pattern red flags in RAG and agent memory
Some query shapes are structural debt wearing a costume. Repeated near-identical queries with different phrasings mean your cache key is too strict. Queries that fan out to ten collections and then discard nine results mean your routing is absent. Agent memory that re-reads the same conversation history on every turn means you haven't consolidated state.
The pattern to catch: retrieval work that scales with session length instead of query complexity. That's the signature of unbounded context growth.
Instrumentation as an architectural requirement
You can't fix what you can't see per stage. Add a trace ID that follows every query through embedding, retrieval, reranking, and generation. Log token counts, call counts, and latency at each boundary. This isn't observability theater. It's the difference between guessing which stage broke and knowing.
Instrumentation costs a few milliseconds per query. The alternative is debugging a production incident with no data.
Where to Start: A Prioritized Roadmap for Your Architecture
You don't need to rebuild everything at once. The order matters more than the speed.
Start with observability, then cache, then shard
Observability comes first because every later decision depends on knowing where time and tokens actually go. Without per-stage traces, you'll cache the wrong thing and shard the wrong index. Add trace IDs and stage-level latency logging before you touch anything else. It's a day of work.
Then cache embeddings. It's the cheapest win available: a tiered cache keyed on normalized text cuts embedding spend and fan-out latency without changing your retrieval logic. You'll see the hit rate climb within hours.
Sharding comes last. It's the most invasive change, and doing it before you've measured query distribution means you'll shard on the wrong key and pay for a migration twice.
A decision matrix: query volume vs. architectural investment
It depends on your volume and your team size.
Under 10,000 queries per month: observability only. Don't cache, don't shard. You're not hitting the thresholds where those changes pay for themselves.
Between 10,000 and 100,000: observability plus embedding cache. Sharding is optional and usually premature.
Above 100,000: all three, in order. If you're a solo developer above 100,000 queries, skip sharding and buy a managed vector database instead. Your time is worth more than the infrastructure savings.
The first 30 days after N queries
Week one: instrument every stage. Week two: read the traces and find the top two bottlenecks. Week three: fix the biggest one. Week four: measure again and decide if the second fix is still worth it.
Don't do all five shifts from the previous section at once. Each one changes the system enough that you need a clean measurement window after it lands.
FAQ
How many queries is "N queries" before architecture needs to change?
There's no single number. It depends on your fan-out ratio, your cache hit rate, and how much context your agent memory accumulates per session. A pipeline with 1:15 fan-out breaks at a few hundred queries per day. A pipeline with 1:1 fan-out and a warm embedding cache can handle tens of thousands before architecture becomes the constraint. Watch the metrics, not the counter.
What are the first signs that my RAG pipeline needs an architecture change?
Three signs. Your p95 latency tracks query volume linearly no matter what you tune. You're adjusting the same parameter repeatedly and getting less back each time. And your fixes create new problems: faster retrieval floods the context window, which slows generation. When you see those, the design is the problem.
Can I fix N+1 query problems in RAG with better caching alone?
Sometimes. If your fan-out comes from re-embedding the same query variants, a tiered embedding cache removes most of the waste. But caching won't fix unbounded context growth or a missing routing layer. It's the cheapest first move, not the only move.
What's the difference between tuning a RAG pipeline and changing its architecture?
Tuning changes parameters: batch size, top-k, prompt length. Architecture changes data flow: adding a cache layer, sharding an index, consolidating agent memory. If a change makes the current design faster, it's tuning. If it changes what the system does, it's architecture.
How do I know if my agent memory architecture is the bottleneck?
Check whether retrieval work scales with session length instead of query complexity. If every turn re-reads the same conversation history and appends fresh chunks to a growing context, your memory has no consolidation step. The fix is architectural: summarize and merge after each turn, and keep raw chunk references for re-fetching.
What architectural changes give the biggest cost reduction in RAG pipelines?
Embedding cache first. It cuts repeat embedding spend and fan-out latency without touching retrieval logic. Sharding second, if you're above roughly 10,000 queries per day. Memory consolidation third, because it shrinks the token budget per turn. Instrumentation doesn't reduce cost directly, but it's what tells you which of the other three to do first.
Is it better to shard my vector database or use a faster embedding model?
It depends on where your latency lives. If the vector search stage is the bottleneck, sharding removes it. If the embedding stage is the bottleneck, a faster model helps more. Instrument per stage before you choose. Most teams we see hit the embedding bottleneck first.
What should I not expect from changing my RAG architecture?
Don't expect it to fix bad embeddings, poor chunking, or a retrieval strategy that doesn't understand your domain. Don't expect it to make the LLM generate tokens faster. Architecture changes how components connect. They don't fix what flows through them.
What we changed in our architecture after N queries came down to five shifts: cache embeddings, route and shard, consolidate memory, iterate with reranking, and instrument everything. The order matters. Observability first, because every later decision depends on knowing where time and tokens actually go. Cache second, because it's the cheapest win. Shard last, because it's the most invasive. And keep the honest limitation in view: architecture gives you room to run a better retrieval strategy. It does not write one for you.
Frequently Asked Questions
What breaks first in a RAG pipeline as query volume grows?
Usually the embedding and retrieval steps, because they are called on every query and scale linearly with volume. Latency and cost then compound when the LLM receives larger contexts. Caching and tiered retrieval are the first architectural fixes.
How do I know when to re-architect instead of just tuning?
When p95 latency or cost per query grows non-linearly with volume, tuning parameters or prompts will not help. That is the signal to change the architecture, such as separating ingestion from retrieval or adding memory tiers.
Does adding more vector database replicas solve scaling problems?
It can help with read throughput, but it does not address embedding cost, context bloat, or cascading failures. Replicas are a scaling tactic, not an architectural fix for inefficient query paths.
What is the role of caching in agent memory architectures?
Caching stores embeddings and retrieval results for repeated or similar queries, cutting both latency and cost. It is especially effective when queries have high overlap or when the same context is reused across sessions.
Should I use a single vector store or multiple tiers?
A single store is simpler to start, but tiered storage (hot cache, warm vector store, cold archive) lets you apply different SLAs and costs to different data. Most systems benefit from at least a hot cache and a warm store.
How does context compression affect retrieval quality?
Compression reduces token count and latency, but aggressive compression can drop relevant details. Evaluate compression against a fixed set of queries to ensure recall does not degrade below acceptable levels.
What metrics should I track after making architectural changes?
Track p50, p95, and p99 latency, cost per query, cache hit rate, retrieval recall, and error rates. These metrics show whether the changes improved scalability without harming quality.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through what we changed in our architecture after n queries or something adjacent, we publish what we have actually tested, including where it falls short.
