Query Planning and Decomposition: A Practical Guide for RAG and Agent Memory Builders
Query planning and decomposition sounds like an optimization problem for engineers building RAG pipelines and agent memory systems. It's not. It's a failure you've already shipped. A user asks "which of our two enterprise plans costs less per seat after the annual discount, and does that cheaper plan still include SSO?" Your retriever grabs chunks matching "enterprise plan cost" and "SSO", the LLM stitches them into a confident paragraph, and the answer is wrong because the discount math lived in a different document than the feature list. The system never broke the question apart.
Decomposition means splitting a compound query into sub-queries, running each against your retriever, then synthesizing the results. Planning is deciding whether that's worth doing at all. It isn't always. Decomposition adds latency, burns tokens, and sometimes returns worse results than a single well-formed query. GigaRAG gives RAG and agent memory builders the tooling to measure that trade-off directly. This guide covers when to decompose, how to do it without wrecking your latency budget, and when to skip it entirely.
| At a glance | Details |
|---|---|
| What it is | Breaking a compound query into retrievable sub-queries |
| Best for | Multi-part, comparative, or reasoning-heavy questions |
| Main trade-off | Extra LLM calls and latency vs. retrieval depth |
| Common strategies | Parallel, sequential, dependency-aware decomposition |
| Memory role | Persists sub-queries and results across agent turns |
| When to skip | Simple, single-fact lookups with clear intent |
In This Guide
- What Is Query Planning and Decomposition?
- Parallel vs Sequential vs Dependency-Aware Decomposition
- When Query Decomposition Actually Helps (and When It Hurts)
- Query Planning And Decomposition: A Step-by-Step Guide
- Decomposition Strategies: Parallel, Sequential, and Dependency-Aware
- How to Implement Query Planning and Decomposition in Your RAG Pipeline
- Integrating Query Decomposition with Agent Memory
- Failure Modes and Anti-Patterns in Query Decomposition
- Query Decomposition vs. Alternatives: Step-Back Prompting and Multi-Query Rewriting
- Common Mistakes When Implementing Query Planning and Decomposition
- How to Choose the Right Decomposition Approach for Your Use Case
What Is Query Planning and Decomposition?
Query planning decides how to answer a question. Query decomposition breaks that question into smaller sub-queries that can each be answered independently. For RAG and agent builders, planning is the strategy layer, decomposition is the tactic that executes it.
Query planning vs. query decomposition: what's the difference?
Planning asks "what do I need to know to answer this?" Decomposition asks "what smaller questions can I split this into?" The two work together but aren't the same thing.
A planner might decide a question needs three retrieval steps. Decomposition generates those three sub-queries. You can plan without decomposing, and you can decompose without an explicit planning layer. In practice, most RAG pipelines that handle compound questions do both, often in a single LLM call that returns structured sub-questions.
What is decomposition in AI?
Decomposition in AI means breaking a complex task into smaller, more manageable subtasks. It's not unique to RAG. Classical planning systems have done this for decades, and LLM-based agents use the same idea when they split a goal into steps.
For query decomposition specifically, the LLM takes a compound question like "What company acquired Instagram in 2012, and what was its stock price that year?" and produces sub-queries: "Who acquired Instagram in 2012?" and "What was Facebook's stock price in 2012?" Each sub-query retrieves more focused documents than the original would.
Why compound queries break naive RAG pipelines
A naive RAG pipeline embeds the full user query and retrieves documents by vector similarity. That works for single-fact questions. It fails when the query contains multiple facts, comparisons, or temporal constraints.
The embedding for a compound question sits in a sparse region of vector space. No single document is close to it, so retrieval returns shallow matches or misses the second half entirely. The LLM then answers confidently from whatever it got. That's how you end up with a wrong answer delivered with total certainty. Decomposition fixes this by giving each sub-query its own retrieval pass.
[!note] Query decomposition is not always beneficial — for simple single-fact questions it adds latency and token cost without improving retrieval quality, so gate it behind a complexity check.
Parallel vs Sequential vs Dependency-Aware Decomposition
| Factor | Parallel | Sequential |
|---|---|---|
| Latency | Lowest — sub-queries run at once | Highest — each step waits for the last |
| Token cost | Moderate — one planning call plus N retrievals | Higher — planning and synthesis at each step |
| Best for | Independent sub-questions (e.g. compare A and B) | Multi-hop reasoning where step 2 needs step 1's answer |
| Failure mode | Redundant or overlapping retrievals | Error compounding across steps |
| Memory fit | Sub-results stored together, easy to dedupe | Intermediate state must persist between turns |
When Query Decomposition Actually Helps (and When It Hurts)
Decomposition is not a default. It's a tool with a specific trigger condition: the query contains multiple retrievable facts that no single document is likely to hold. When that condition is absent, decomposition adds cost without adding accuracy.
Query types that benefit from decomposition
Multi-hop questions are the clearest case. "Which company acquired the startup that built the first transformer model?" requires finding the startup, then finding the acquirer. One retrieval pass can't do both. Comparative questions work too: "How does ColBERT compare to dense retrieval for multi-vector search?" splits naturally into "How does ColBERT work?" and "How does dense retrieval work?" Temporal questions with two time anchors, like "What was OpenAI's valuation before and after the 2023 funding round?", also benefit. Each sub-query targets a document that actually exists.
Query types where decomposition hurts
Single-fact lookups don't need it. "What year was BERT released?" decomposes into one sub-query, which is just the original with extra steps. Ambiguous queries get worse: if the LLM can't tell whether "bank" means a river or a financial institution, splitting it into sub-queries multiplies the ambiguity across every branch. Low-latency systems suffer most. A chatbot that must respond in under 300ms can't afford three sequential retrieval passes when one would do.
Cost and latency: the hidden tax of decomposition
Every sub-query costs tokens and time. A three-way decomposition means three embedding calls, three retrieval passes, and one synthesis call that must hold all three contexts. That's roughly 3x the retrieval latency and 2-4x the token cost of a single-pass pipeline, before you count the decomposition call itself. If the sub-queries are sequential, latency compounds: sub-query two can't start until sub-query one returns. Parallel decomposition avoids the compounding but still pays the per-branch cost.
A simple decision checklist
Ask yourself three questions before decomposing. Does the query contain more than one independently retrievable fact? Will splitting it produce sub-queries that each have a clear answer in your corpus? Can your latency budget absorb the extra retrieval passes? If the answer to any of these is no, skip decomposition. If all three are yes, decompose. The honest answer is that most production queries don't need it. The ones that do are the ones where a wrong answer costs more than the extra latency.
[!tip] For agent systems, store the decomposition plan alongside the retrieved chunks in memory — this lets the agent answer follow-up questions by reusing sub-results rather than re-running the full plan, which cuts both latency and token spend on multi-turn conversations.
Query Planning And Decomposition: A Step-by-Step Guide
- Detect whether the incoming query is compound — look for conjunctions, multiple entities, or comparison phrasing.
- Generate a plan: decompose into 2–5 atomic sub-queries, each answerable by a single retrieval pass.
- Classify dependencies — mark which sub-queries can run in parallel and which need prior results.
- Retrieve per sub-query, deduplicate overlapping chunks, and tag each result with its originating sub-query.
- Synthesize the final answer from sub-results, citing which sub-query each fact came from.
- Persist the plan and sub-results to agent memory so follow-up turns reuse them instead of re-planning.
- Log decomposition decisions and measure retrieval quality against a non-decomposed baseline.
Decomposition Strategies: Parallel, Sequential, and Dependency-Aware
Once you've decided to decompose, the next choice is how. There are three main strategies, and they differ in one thing: whether sub-queries depend on each other. That single difference drives everything else about latency, cost, and accuracy.
Parallel decomposition: independent sub-queries
Parallel decomposition splits a query into sub-queries that don't depend on each other. You run them all at once, then synthesize the answers. "Compare the pricing of Pinecone and Weaviate" becomes "What is Pinecone's pricing?" and "What is Weaviate's pricing?" Neither answer needs the other.
The good news is latency. All sub-queries hit the retriever simultaneously, so total retrieval time is roughly the slowest branch, not the sum of all branches. The catch is that parallel only works when the sub-queries are truly independent. If one answer changes what you should ask next, parallel decomposition will miss it.
Sequential decomposition: when answers build on each other
Sequential decomposition runs sub-queries one at a time, feeding each answer into the next query. "Which company acquired the startup that built the first transformer model?" decomposes into "Which startup built the first transformer model?" then "Which company acquired that startup?" The second query can't be written until the first returns.
This is the only strategy that handles multi-hop reasoning correctly. But you pay for it in latency: total time is the sum of all retrieval passes plus the synthesis call. A three-hop query means three round trips before you can answer. If any intermediate answer is wrong, every downstream query inherits the error.
Dependency-aware planning: the Instructor approach
Dependency-aware planning is sequential decomposition with a planner that decides dependencies before executing. Instead of assuming every sub-query depends on the previous one, the planner builds a dependency graph: which sub-queries can run in parallel, which must wait, and what each one needs as input.
Instructor implements this by having the LLM output a structured plan with explicit dependencies between sub-queries. The executor then runs independent branches concurrently and dependent branches in order. This gets you most of the latency benefit of parallel decomposition while still handling multi-hop queries correctly. The cost is a more complex planning step and a harder prompt to get right.
Strategy comparison: latency, cost, and accuracy
| Strategy | Latency | Cost | Accuracy on multi-hop | Best for |
|---|---|---|---|---|
| Parallel | Lowest | Higher (all branches run) | Poor | Comparative, multi-fact queries |
| Sequential | Highest | Lower (only needed branches) | Good | Multi-hop, dependent queries |
| Dependency-aware | Medium | Medium | Good | Mixed queries with partial dependencies |
Parallel is the default when sub-queries are independent. Sequential is the only option when answers build on each other. Dependency-aware planning is worth the extra complexity when you have a mix of both in the same query, which happens more often than you'd think in production.
How to Implement Query Planning and Decomposition in Your RAG Pipeline
Implementation comes down to four steps: design the decomposition prompt, run the sub-queries, synthesize the answer, and measure whether it actually helped. Each step has one main failure point. Here's how to avoid them.
Step 1: Design your decomposition prompt
The prompt is where decomposition succeeds or fails. You're asking the LLM to output sub-queries, not answers. Make that explicit.
A minimal prompt looks like this:
Break the following question into sub-questions that can be answered independently.
Return only the sub-questions, one per line.
Question: {user_query}
Two things matter. First, constrain the output format. Ask for one sub-query per line, or JSON if you need dependencies. Unstructured output breaks the executor. Second, tell the model what "good" means for your use case. If you're decomposing comparative questions, say so: "Sub-questions should each target one entity or fact."
Don't ask the model to answer anything in this step. Answers come later, from retrieval. Mixing the two roles produces sub-queries that drift toward what the model already knows instead of what your documents contain.
Step 2: Execute sub-queries against your retriever
Run each sub-query through your existing retriever. Don't build a separate retrieval path. The sub-queries are still queries: they need the same embedding model, the same vector store, the same filters.
For parallel decomposition, fire all sub-queries at once and collect results. For sequential, run one, feed the answer into the next query template, repeat. For dependency-aware, execute independent branches concurrently and dependent ones in order.
The main catch here is retrieval quality per sub-query. A sub-query like "What year was the company founded?" is short and may retrieve garbage if your retriever relies on query-document semantic overlap. If sub-query retrieval returns nothing useful, the synthesis step has nothing to work with. Check each sub-query's top-k results before passing them forward.
Step 3: Synthesize the final answer with reasoning
Synthesis is a second LLM call. You pass the original question, the sub-queries, and the retrieved context for each sub-query. The model's job is to answer the original question using only that context.
The prompt should force grounding:
Answer the original question using only the provided context.
For each sub-question, cite which context supports your answer.
If the context is insufficient, say so.
Original question: {user_query}
Sub-question 1: {sub_q1}
Context 1: {context1}
...
The "say so" clause is the important part. Without it, the synthesizer will fill gaps with parametric knowledge, and you'll get a confident wrong answer. That's the exact failure decomposition was supposed to prevent.
Step 4: Evaluate and iterate
Measure before and after. Run a fixed set of compound queries through your pipeline with decomposition on and off. Compare answer accuracy, retrieval precision per sub-query, and end-to-end latency.
If decomposition doesn't improve accuracy on your eval set, remove it. The added latency and token cost aren't worth a marginal gain. If it helps on some query types but not others, gate it: only decompose when the query matches patterns that benefit, like comparisons or multi-hop questions.
Iterate on the decomposition prompt first. Most failures trace back to sub-queries that are too vague, too narrow, or answer the wrong question. Fix the prompt before touching the retriever or the synthesizer.
Integrating Query Decomposition with Agent Memory
Most decomposition guides treat each query as a blank slate. That's the wrong model for agents. An agent that decomposes a query, retrieves, and synthesizes, then forgets everything, pays the full cost again on the next turn. Memory changes the calculus.
Why agent memory changes the decomposition calculus
Without memory, decomposition is stateless. Every follow-up question gets decomposed from scratch, even when it's a slight variation of the last one. With memory, the agent can check what it already knows before deciding whether to decompose at all.
The honest answer is that memory doesn't replace decomposition. It changes when you need it. A follow-up like "what about their pricing?" doesn't need decomposition if the agent remembers the entity from the previous turn. A genuinely new compound question still does.
Storing intermediate sub-answers for reuse
When you decompose a query, you produce sub-answers before the final synthesis. Store those. They're the most reusable artifact in the pipeline.
A sub-answer like "Acme Corp was founded in 2011" is a fact the agent can retrieve directly next time, no decomposition, no vector search, no LLM call. Store sub-answers keyed by the sub-query that produced them, with a timestamp and the source document ID.
The catch: sub-answers go stale. If the source document changes, the stored answer is wrong. Version your memory entries against the source, or set a TTL.
Using memory to avoid redundant decomposition
Before decomposing a new query, check memory for similar past queries. If the agent decomposed "compare Acme and Beta on pricing" last week, and now the user asks "Acme vs Beta pricing again," reuse the decomposition structure. Swap in fresh retrieval for any sub-queries where the underlying data may have changed.
This is a cache, not a shortcut. You still need to verify that the stored decomposition matches the new query's intent. A fuzzy match on the surface can hide a different question underneath.
Memory-aware decomposition patterns
Three patterns work in practice. First, entity persistence: keep the current subject in memory so follow-ups inherit context without re-decomposition. Second, sub-answer caching: store intermediate results as first-class memory entries. Third, decomposition templates: save the structure of a decomposition, not the answers, so similar queries skip the planning step.
Start with entity persistence. It's the simplest and fixes the most common failure: agents that treat every turn as a new conversation.
Failure Modes and Anti-Patterns in Query Decomposition
Decomposition fails in predictable ways. You'll see the same four or five patterns in production, and they're all fixable once you know what to look for.
Over-decomposition: too many sub-queries
The LLM gets enthusiastic. A question like "compare Acme and Beta on pricing and support" becomes eight sub-queries: Acme pricing, Acme support, Beta pricing, Beta support, Acme pricing history, Beta pricing history, Acme support SLA, Beta support SLA. Most of those are noise.
Each sub-query costs tokens and latency. Worse, each one pulls in context that the synthesizer has to sort through. You hit context window limits fast, and the final answer gets worse, not better, because the model drowns in marginally relevant chunks.
The fix: cap sub-queries at three or four. If the LLM wants more, tell it to merge or drop the least important ones.
Under-decomposition: missing the real question
The opposite failure is subtler. The query "did Acme raise prices after their Series B?" looks like one question. It isn't. It's two: when was Acme's Series B, and what was their pricing before and after that date.
A naive pipeline retrieves on the full query, gets a chunk about Acme's Series B, and answers "yes" or "no" without ever checking the pricing timeline. The answer is confident and wrong.
The fix: train your decomposition prompt to spot temporal and comparative cues. "After," "before," "since," "compared to" are decomposition triggers, not decoration.
Sub-query drift: when sub-questions go off-track
The LLM decomposes correctly, but one sub-query drifts. "What are Acme's enterprise pricing tiers?" becomes "what is enterprise pricing in general?" The retriever happily returns generic content about enterprise SaaS pricing, and the synthesizer weaves it in as if it were about Acme.
Drift happens because sub-queries are generated without grounding in the original query's entities. The fix is simple: require every sub-query to preserve the original entity names. If "Acme" disappears from a sub-query, reject it.
Synthesis errors: the final answer lies
Even with perfect sub-queries and perfect retrieval, synthesis can fail. The model gets three correct sub-answers and combines them into a wrong conclusion. It's not hallucinating facts. It's failing at reasoning across them.
This is the hardest failure to catch, because every intermediate step looks right. The fix is to log sub-answers alongside the final answer, then spot-check syntheses where sub-answers contradict each other or the final claim doesn't follow from them. You'll find the failure rate is higher than you expect.
Query Decomposition vs. Alternatives: Step-Back Prompting and Multi-Query Rewriting
Decomposition isn't the only way to handle complex queries. Two alternatives solve overlapping problems with different mechanics. Knowing which one you're reaching for matters, because they fail in different ways.
Step-back prompting: abstracting before answering
Step-back prompting asks the LLM to generate a more general version of the question first, then answer that. "Did Acme raise prices after their Series B?" becomes "what is Acme's pricing history?" The abstracted question retrieves broader context, then the model reasons down to the specific answer.
It's cheaper than full decomposition. One abstraction, one retrieval, one synthesis. But it loses precision. The broad question pulls in chunks that are relevant to Acme generally but not to the pricing timeline specifically. For questions where the hard part is knowing what context you need, step-back works. For questions where the hard part is combining two precise facts, it doesn't.
Multi-query rewriting: generating variations
Multi-query rewriting keeps the query at the same level of specificity but generates several phrasings of it. "Did Acme raise prices after their Series B?" becomes "Acme pricing change post-Series B," "Acme price increase timeline," "Acme Series B pricing impact."
Each variation retrieves independently, and the results merge. This helps when the retriever is sensitive to phrasing, which vector search often is. But it doesn't break the query into logical parts. A compound question rewritten five ways is still a compound question. You get better recall on the same shallow retrieval, not deeper reasoning.
When to use which technique
Use decomposition when the question has distinct sub-parts that need separate retrieval. Use step-back prompting when the question is specific but the context you need is broad. Use multi-query rewriting when retrieval quality is the bottleneck, not query structure.
The honest answer is that these compose. You can step back to find the right context, decompose within it, and rewrite each sub-query for better recall. But each layer adds latency and tokens. Start with one, measure, and add the next only if retrieval quality still lags.
Common Mistakes When Implementing Query Planning and Decomposition
Most failures aren't architectural. They're small decisions made early that compound.
Hardcoding decomposition rules instead of using the LLM
Teams write regex or keyword rules to split queries, then wonder why edge cases break. The LLM already understands the query. Let it propose sub-queries. Rules work for routing, not for decomposition.
Ignoring retrieval quality for sub-queries
A decomposed query is only as good as what each sub-query retrieves. If your retriever returns shallow chunks for "Acme pricing history," decomposition just gives you five shallow retrievals instead of one. Fix retrieval before adding planning.
No fallback when decomposition fails
The LLM sometimes produces sub-queries that are worse than the original. Or it returns nothing parseable. Without a fallback to the raw query, you've added latency and made the answer worse. Always keep the original query as a baseline path.
Not measuring the cost of decomposition
Every sub-query costs tokens and milliseconds. If you can't say what decomposition adds to your p95 latency, you're guessing. Measure it against a no-decomposition baseline on the same query set. Query planning and decomposition is a tool with a price tag. Know the price.
How to Choose the Right Decomposition Approach for Your Use Case
The decision comes down to four constraints: query complexity, latency budget, token budget, and whether you have persistent memory. Map those before you pick a strategy.
Assess your query complexity
Single-fact lookups don't need decomposition. "What's the refund policy?" retrieves fine as-is. Compound questions with comparisons, temporal logic, or multi-hop reasoning do. If a human would need to look at two or more documents to answer, decompose. If one document covers it, don't.
Match strategy to constraints
Parallel decomposition works when sub-queries are independent. It adds latency equal to the slowest sub-query, not the sum. Sequential decomposition costs more time but handles answers that build on each other. Dependency-aware planning costs the most in tokens and engineering time. Pick it only when sub-queries genuinely depend on each other's outputs.
Tight latency budget under 200ms? Skip decomposition entirely. Tight token budget? Use parallel with a cap on sub-query count. Memory available? Reuse cached sub-answers instead of re-decomposing.
Start simple, add complexity only when needed
Ship no decomposition first. Measure retrieval quality on compound queries. Add parallel decomposition where it improves answer accuracy. Add sequential only for query types that still fail. Dependency-aware planning is the last resort, not the default. Every layer adds cost. Add it when the data says you need it.
Query planning and decomposition is a tool with a specific trigger condition, not a default setting. The systems that get it right are the ones that measure the trade-off before shipping it, and the ones that know when to leave it off.
Frequently Asked Questions
What is decomposition in AI?
In AI and RAG systems, decomposition is the process of breaking a complex query or task into smaller, independently solvable sub-problems. Each sub-problem can be retrieved, reasoned about, or executed separately, then recombined into a final answer. It is most valuable when a single query spans multiple documents, entities, or reasoning steps.
What are the layers of query processing?
A typical pipeline has four layers: query understanding (parsing intent and entities), query planning (deciding whether and how to decompose), retrieval (fetching chunks per sub-query), and synthesis (combining results into an answer). Decomposition sits in the planning layer and directly shapes what the retrieval layer fetches.
What does decomposition mean in programming?
In programming, decomposition means breaking a large problem into smaller, manageable functions or modules. The same principle applies to query planning: a compound question is split into atomic sub-queries that are easier to retrieve and verify individually. The goal is the same — reduce complexity by isolating concerns.
When should I NOT use query decomposition in RAG?
Skip decomposition for simple, single-fact lookups where the intent is clear and one retrieval pass suffices. Adding decomposition there increases latency and token cost with no retrieval benefit. A lightweight complexity classifier or heuristic gate before the planner prevents this waste.
How does query decomposition interact with agent memory?
Agent memory lets you persist the decomposition plan and sub-results across turns. On a follow-up question, the agent can reuse prior sub-results instead of re-planning and re-retrieving from scratch. This reduces latency and token spend, but requires a strategy for invalidating stale memory when the underlying documents change.
What are common failure modes of query decomposition?
The most common failures are over-decomposition (splitting a simple query into unnecessary sub-queries), error compounding in sequential plans (a wrong intermediate answer poisons later steps), and redundant retrieval when sub-queries overlap. Logging plans and measuring against a non-decomposed baseline surfaces these quickly.
How do I write a query decomposition prompt?
A good decomposition prompt asks the model to output a structured list of atomic sub-queries, each answerable by one retrieval pass, and to flag dependencies between them. Include a constraint on the maximum number of sub-queries and an instruction to return the original query unchanged if it is already atomic.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through query planning and decomposition or something adjacent, we publish what we have actually tested, including where it falls short.
