Query Rewriting and Expansion: A Decision Framework for RAG

Query Rewriting and Expansion for RAG Pipelines and Agent Memory

Query rewriting and expansion gets sold hard in RAG circles, as if one more prompt will fix a retrieval layer that never worked. Builders, the people actually running these pipelines in production, know better. The honest answer is that rewriting helps some queries and actively hurts others. Short factual lookups don't need it. High-precision searches get worse when you loosen the query. And every LLM call you add costs tokens and latency you can't get back. This guide is for software engineers and ML practitioners who need a decision framework, not a list of techniques. GigaRAG helps agent memory and RAG pipeline builders implement these patterns, but the principles here are vendor-neutral. You'll get a query-type to strategy mapping, a full section on agent memory and multi-turn rewriting that most results skip, and plain numbers on what each technique costs in production. You'll also learn when to skip rewriting entirely.

In This Guide

• What Is Query Rewriting and Expansion?
• Query Expansion vs Query Rewriting: Which Do You Need?
• Query Expansion Techniques That Actually Work
• Query Rewriting And Expansion: A Step-by-Step Guide
• LLM-Based Query Rewriting Methods
• Query Rewriting for Agent Memory and Multi-Turn Conversations
• A Decision Framework for Choosing a Rewriting Strategy
• When to Skip Query Rewriting and Expansion
• Costs and Tradeoffs: Latency, Tokens, and Complexity
• Evaluating Your Query Rewriting Pipeline
• Implementation Guide: Putting It All Together
• Common Mistakes in Query Rewriting and Expansion

What Is Query Rewriting and Expansion?

Query rewriting changes the words of a user's question before it hits retrieval. Query expansion adds related terms to the original query. Both aim to improve what the retriever finds, but they do it differently.

Query rewriting vs query expansion: the key difference

Query rewriting replaces the original query with a new one. You take "how do I fix my sink" and rewrite it to "plumbing repair steps for a kitchen sink drain." The original words are gone. The retriever never sees them.

Query expansion keeps the original query and appends related terms. You take "sink repair" and expand it to "sink repair plumbing drain pipe faucet." The original words stay. The retriever sees both the original and the added terms.

Here's why the distinction matters: rewriting changes intent, expansion changes coverage. Rewriting risks losing information if the rewrite is wrong. Expansion risks adding noise if the added terms are irrelevant. You'll choose between them based on which risk you can tolerate.

Why raw user queries fail in RAG pipelines

Raw user queries fail for three reasons. First, users type short queries. The average web search is under four words. Short queries don't match the way documents are written. Your knowledge base says "troubleshooting steps for a leaking kitchen faucet cartridge." The user typed "sink broken." Embedding similarity between those two strings is low.

Second, users write conversationally. They use pronouns, implied context, and filler. A query like "what about the other one" means nothing without the previous turn. The retriever has no memory of that turn unless you build it.

Third, users don't know your vocabulary. They search for "how to stop water dripping" when your docs say "faucet aerator replacement." The concepts match, but the words don't. Embedding models help with this, but they don't close the gap entirely.

That's the problem rewriting and expansion solve. They bridge the gap between how users ask and how your documents answer.

[!note] Query rewriting changes the query, not the index; if your documents are poorly chunked or your embeddings are weak, rewriting alone will not fix retrieval quality.

Query Expansion vs Query Rewriting: Which Do You Need?

Query Expansion Techniques That Actually Work

Expansion adds terms without replacing the original query. That's the core mechanic. The retriever sees the user's words plus related terms, which widens the net. The risk is noise: add the wrong terms and you pull in irrelevant chunks. These four techniques earn their place in production.

Pseudo-relevance feedback

Pseudo-relevance feedback runs a first retrieval pass, takes the top results, and extracts terms from those documents to expand the original query. You assume the top results are relevant, then mine them for vocabulary the user didn't know to use.

Here's what happens behind the scenes: you retrieve the top 10 chunks for "sink broken," pull terms like "cartridge," "aerator," and "drain assembly" from those chunks, and run a second retrieval with the expanded query. The second pass finds chunks the first pass missed.

The main catch is the assumption. If the first retrieval pass returns garbage, you're mining garbage for expansion terms. This technique works best when your initial retrieval is already decent and you're trying to push recall higher.

Template-based expansion

Template-based expansion uses predefined patterns to generate query variants. You write templates like "What is {term}?" or "How do I fix {term}?" and fill them with terms from the original query.

This is the cheapest expansion method. No LLM call, no second retrieval pass. You're just string substitution. It works well for structured domains where you know the query patterns in advance. A support knowledge base for a hardware product, for example, sees the same twenty question shapes repeatedly.

The limitation is obvious: templates only cover patterns you've anticipated. A novel query shape gets no help.

Synonym and entity expansion

Synonym expansion maps user terms to your document vocabulary using a controlled dictionary. You maintain a mapping like "dripping" to "leaking," "broken" to "malfunctioning," "sink" to "faucet." Entity expansion does the same for named things: product names, part numbers, error codes.

This is deterministic. No model can hallucinate a synonym. The tradeoff is maintenance. Your dictionary goes stale as your knowledge base changes, and someone has to own that.

Query relaxation

Query relaxation is expansion in reverse. You remove terms from the original query to broaden the search. "How do I fix a leaking kitchen sink faucet" relaxes to "fix leaking sink faucet," then to "sink faucet."

Relaxation helps when the full query is too specific and returns nothing. You drop the least important terms until you get hits. The risk is losing the user's actual intent. Relax too far and you're searching for "sink" when the user needed "faucet cartridge replacement."

In practice, you'll combine these. Pseudo-relevance feedback for recall, synonym expansion for vocabulary gaps, relaxation as a fallback when nothing returns. The next section covers LLM-based methods, which trade determinism for flexibility.

[!tip] For agent memory systems, rewrite the current turn using the last few turns of conversation before retrieval, and store the rewritten query alongside the original so you can debug why a result was returned.

Query Rewriting And Expansion: A Step-by-Step Guide

1. Log a sample of real user queries and label each as clear, vague, conversational, or multi-part.
2. Choose one rewriting technique that matches your dominant query type, such as expansion for vague queries or HyDE for vocabulary mismatch.
3. Add the rewrite step as a separate module between the user input and the retriever, keeping the original query available.
4. Retrieve with both the original and rewritten queries, then merge and deduplicate results before reranking.
5. Measure retrieval recall and answer quality against a no-rewrite baseline on the same query set.
6. Track added latency and token cost per query, and set a budget you are willing to spend.
7. Roll out behind a flag, monitor for regressions, and disable rewriting for query types where it does not help.

LLM-Based Query Rewriting Methods

LLM rewriting trades determinism for flexibility. The model reads the query, reasons about intent, and produces a new query or set of queries. The cost is latency and tokens. These five techniques cover most production use cases.

HyDE (Hypothetical Document Embeddings)

HyDE flips the retrieval order. Instead of embedding the user's query directly, you ask the LLM to write a short hypothetical document that would answer the query. Then you embed that document and retrieve against it.

Here's why it works: user queries are short and sparse. Documents are long and dense. Embedding a query against documents means comparing a three-word phrase to a three-hundred-word chunk. HyDE generates a document-shaped query, so the embedding space matches. For "how do I fix a leaking faucet," the LLM writes a paragraph about faucet repair, and that paragraph retrieves better than the raw query.

The main catch is hallucination. The hypothetical document can contain wrong details, and those wrong details steer retrieval. HyDE works best when the query is short and the domain is one where the LLM has decent general knowledge.

Query2Doc

Query2Doc is HyDE's sibling. The LLM generates a pseudo-document, but instead of replacing the query, you concatenate the pseudo-document with the original query and embed the combined text. You keep the user's words and add the LLM's expansion.

In practice, this gives you the embedding-space benefit of HyDE while preserving the original intent. The tradeoff is a longer input to the embedding model, which costs more tokens and adds latency.

Chain-of-thought query expansion

Chain-of-thought expansion asks the LLM to reason through what the user might mean before producing the expanded query. You prompt the model to break down the query, identify missing context, and generate terms step by step.

This helps with ambiguous or multi-hop queries. "Why did my server crash" becomes a reasoning chain: the user wants causes, the causes might be memory, disk, or network, so the expanded query includes those terms. The cost is higher token usage and slower response. Use it when the query is genuinely ambiguous, not as a default.

Step-back prompting

Step-back prompting asks the LLM to generate a more general version of the query before answering. For "how do I fix error 503 on nginx," the step-back query is "what causes server errors in web servers." You retrieve on both the specific and the general query.

This works when the specific query is too narrow to retrieve well. The general query pulls in background context that helps the model answer the specific question. The risk is retrieving too much generic material that drowns out the specific answer.

Multi-query retrieval

Multi-query retrieval generates several rewritten versions of the same query, retrieves against all of them, and merges the results. The LLM produces three to five variants, each emphasizing a different interpretation or angle.

This is the highest-recall technique. Different phrasings surface different chunks, and the union covers more ground. The cost is obvious: you run retrieval multiple times and pay for multiple LLM generations. Multi-query is worth it when recall matters more than latency, like in research or exploratory search.

Query Rewriting for Agent Memory and Multi-Turn Conversations

Agents don't get clean queries. They get fragments, follow-ups, and pronouns that only make sense if you remember what came before. Single-turn rewriting treats every query like it arrived in a vacuum. That's the gap.

Why single-turn rewriting fails for agents

A user asks "what's the retention policy for EU customers?" You rewrite it, retrieve, answer. Then they ask "how do I change it?" Single-turn rewriting sees four words with no clear referent. It either rewrites badly or passes the raw query through, and retrieval returns garbage.

The fix is context persistence. Before rewriting, you pull the last N turns from the conversation store and prepend them to the query. The LLM then rewrites "how do I change it" into "how do I change the retention policy for EU customers." The rewrite is only as good as the context you feed it. If your memory store drops turns or truncates badly, the rewrite inherits those holes.

Memory-aware query expansion

Memory-aware expansion pulls terms from what the agent already knows about the user, not just from the current query. If the conversation established the user runs a PostgreSQL backend, a query about "connection pool errors" expands to include "PostgreSQL" and "pgbouncer" without the user restating them.

You need a memory store that tracks entities, preferences, and prior topics. The expansion step queries that store and injects relevant terms into the rewritten query. The catch: stale memory poisons retrieval. If the user switched from PostgreSQL to MySQL three turns ago and your memory store didn't update, you'll expand with the wrong database and retrieve irrelevant chunks. Memory freshness matters as much as memory presence.

Resolving pronouns and implicit context

Pronouns are the hard case. "It", "that", "those", "the second one" all require resolving against prior turns. Ellipsis is worse: "and for enterprise?" means "and what is the retention policy for enterprise customers?" The words aren't there at all.

You resolve these by passing the full conversation history to the rewriting LLM and instructing it to replace pronouns and fill in elided phrases. Don't try to do this with rules. Pronoun resolution needs semantic understanding of what "it" refers to, and that's an LLM job. The failure mode is silent: the LLM guesses wrong, produces a confident rewrite, and retrieval returns plausible but wrong chunks. You won't catch it without evaluating the rewrite against the conversation history.

Keep the history window bounded. Ten turns is usually enough for pronoun resolution. More than that adds token cost and lets stale context steer the rewrite.

A Decision Framework for Choosing a Rewriting Strategy

You've seen the techniques. Now you need to pick one without running a bake-off for every query. The honest answer is that most queries fall into five buckets, and each bucket has a default strategy. Start there, then adjust.

Query type to strategy mapping

Short factual queries ("what's the SLA for enterprise tier?") don't need rewriting. Pass them through. If retrieval fails, add synonym expansion before you reach for an LLM.

Ambiguous queries ("how do I handle errors?") benefit from multi-query retrieval. Generate two or three interpretations, retrieve for each, merge results. The cost is higher, but the recall gain is real.

Multi-hop queries ("which customers churned after the pricing change?") need step-back prompting. Rewrite to the broader question first ("what was the pricing change?"), retrieve, then answer the specific one.

Conversational follow-ups ("and for enterprise?") need context persistence and pronoun resolution, as covered in the previous section. No other technique applies.

High-precision queries (legal or compliance lookups) should skip rewriting entirely. HyDE hallucinates terms that pull in wrong chunks. Use the raw query with a re-ranker.

Decision tree walkthrough

Ask three questions in order. First: is the query well-formed and specific? If yes, pass it through. Don't rewrite what already works.

Second: does the query depend on prior conversation? If yes, resolve context first. If no, check whether it's ambiguous or multi-hop.

Third: is recall the problem or precision? Low recall means you're missing relevant chunks, so expand or use multi-query. Low precision means you're retrieving noise, so tighten the query or skip rewriting and re-rank instead.

That's the whole tree. Most builders overthink this. The default for 70% of queries is pass-through or synonym expansion. LLM rewriting earns its cost only on ambiguous, multi-hop, or conversational queries.

When to combine multiple techniques

Combining techniques compounds cost and latency, so do it only when one technique alone demonstrably fails. The common pair is context resolution plus multi-query: resolve pronouns first, then generate multiple interpretations of the resolved query. That handles conversational ambiguity.

Another pair is step-back plus synonym expansion for multi-hop queries over domain-specific corpora. Step-back gets the broad context, synonyms catch the domain vocabulary.

Don't stack HyDE with multi-query. Both generate synthetic text, and the combined hallucination risk outweighs the recall gain. Test each technique alone before combining. If one gets you to 90% of target recall, stop there.

When to Skip Query Rewriting and Expansion

The decision framework in the previous section already hints at this, but it deserves stating plainly: rewriting is not a default. It's a fallback for when raw retrieval fails. Most queries in production RAG systems don't need it.

Queries that don't need rewriting

Short factual queries with unambiguous terms should pass through untouched. "What's the refund policy?" doesn't need an LLM to expand it. The terms are already precise, and any expansion risks pulling in chunks about unrelated policies.

High-precision lookups are another skip case. Legal, compliance, or medical queries where a wrong chunk is worse than a missing chunk should use the raw query plus a re-ranker. Rewriting introduces terms the user didn't say, and those terms can drag in confidently wrong chunks.

Already-well-formed queries from users who know your corpus also skip rewriting. If a user types the exact product name and error code, you have the vocabulary. Don't paraphrase what's already specific.

Failure modes of over-rewriting

The most common failure is hallucinated terms. HyDE and Query2Doc generate synthetic text, and when that text contains entities or jargon that don't exist in your corpus, retrieval pulls in irrelevant chunks. You've traded precision for noise.

Another failure is latency compounding. Every LLM call adds 200 to 800 milliseconds. On a query that would have retrieved correctly in 50 milliseconds, that's a 4x to 16x slowdown for zero recall gain.

The third failure is context drift. Multi-turn rewriting that resolves pronouns incorrectly can turn "and for enterprise?" into a query about a different topic entirely. The rewrite becomes the error.

A simple pre-rewrite checklist

Before you add any rewriting step, run these checks:

• Does the raw query retrieve at least one relevant chunk in the top 10? If yes, skip rewriting.
• Is the query domain-specific with exact terms? If yes, pass through.
• Is precision more important than recall for this query type? If yes, skip expansion.
• Would a wrong answer cost more than a missing answer? If yes, don't rewrite.

If you answer yes to any of these, the raw query is your best option. Rewriting earns its place only when recall is the bottleneck and the query is genuinely ambiguous, multi-hop, or conversational.

Costs and Tradeoffs: Latency, Tokens, and Complexity

Rewriting costs real money and real milliseconds. The previous section told you when to skip it. This section tells you what you're paying when you don't.

Latency impact by technique

Template-based expansion and synonym lookup are cheap. They run in-process, add under 5 milliseconds, and don't touch an LLM. Pseudo-relevance feedback costs one extra retrieval round, so roughly double your retrieval latency.

LLM-based techniques are the expensive ones. A single rewrite call adds 200 to 800 milliseconds on most hosted models. Multi-query retrieval multiplies that: five query variants means five LLM calls, or one batched call that still takes longer than a single rewrite. HyDE sits in the middle, one generation call plus one embedding call.

The honest answer is that any LLM in the retrieval path pushes your p50 latency from tens of milliseconds to hundreds. If your users notice, they notice fast.

Token cost estimates

A single query rewrite consumes 50 to 200 tokens of input plus 50 to 150 tokens of output. That's trivial per query. At scale it compounds: 100,000 queries per day at 300 tokens per rewrite is 30 million tokens daily. On a mid-tier model that's roughly $30 to $90 per day before you count embedding costs. Multi-query retrieval multiplies this by the number of variants.

The bigger hidden cost is context. Step-back prompting and memory-aware rewriting pull conversation history into the prompt. A 10-turn conversation can add 2,000 to 5,000 tokens to every rewrite call.

When the overhead is worth it

The overhead pays for itself when recall is the bottleneck and the alternative is a wrong answer. Multi-hop queries, ambiguous conversational queries, and searches over heterogeneous corpora justify the latency. A support agent who gets the right chunk in 600 milliseconds is happier than one who gets the wrong chunk in 50.

It's not worth it for high-precision lookups, short factual queries, or any query where the raw terms already retrieve well. If your baseline recall is above 90%, you're paying for noise.

Evaluating Your Query Rewriting Pipeline

You can't improve what you don't measure. A rewriting layer that feels clever in a demo can quietly degrade retrieval in production. The only way to know is to run the numbers.

Metrics that matter: recall, precision, MRR

Recall tells you whether the right chunk came back at all. Precision tells you whether the chunks you got were mostly relevant. For a RAG pipeline, recall is usually the one that matters more: a wrong answer from a missing chunk is worse than a slightly noisy context window.

MRR (Mean Reciprocal Rank) measures where the first relevant chunk lands. If it's rank 1, you get a score of 1. If it's rank 5, you get 0.2. MRR punishes burying the answer, which is exactly what bad rewriting does. Track all three. Recall for coverage, precision for noise, MRR for position.

Ground truth challenges for LLM evaluation

The primary challenge in creating ground truth for LLM evaluation is that relevance is subjective and context-dependent. Two annotators will disagree on whether a chunk answers a query. A chunk that's relevant for one user's intent is noise for another's.

You have two options. Build a labeled set of query-to-chunk pairs by hand, which is slow but stable. Or use an LLM to judge relevance, which is fast but inherits the judge's biases. In practice, most teams start with a small hand-labeled set of 100 to 200 queries, then expand with LLM judging once the baseline is stable. Don't trust an LLM judge you haven't calibrated against human labels.

A/B testing your rewriting layer

Run the rewriting layer against a baseline of raw queries on the same traffic. Split users or queries randomly, log the retrieval results for both arms, and compare recall, precision, and MRR. A 2% MRR improvement might be noise. A 10% drop in recall is a real problem.

Keep the test window long enough to catch query diversity. A day of traffic might be all short factual lookups. Two weeks gets you the ambiguous and multi-hop queries where rewriting actually matters. And log the rewritten queries themselves. When recall drops, you need to see what the rewriter did to the input.

Implementation Guide: Putting It All Together

You've picked a strategy and measured it. Now you wire it into a pipeline that already has embeddings, a vector store, and a generation step. The good news is rewriting slots in as a preprocessing layer. The main catch is that every technique adds a failure point.

Where rewriting sits in your pipeline

Place the rewriting layer between query intake and embedding. Raw query comes in, the rewriter transforms it, the rewritten query gets embedded, and retrieval runs against the vector store. That's the whole architecture. Don't put rewriting after retrieval: by then you've already spent the retrieval budget on a bad query.

For multi-turn systems, the rewriter needs conversation history as input. Pass the last N turns alongside the current query. For agent memory, pass the relevant memory entries too. The rewriter's job is to produce a standalone query that a fresh retrieval call can answer without seeing the conversation.

Pseudocode for a basic rewriting layer

Here's a minimal implementation that works for most single-turn RAG stacks:

def rewrite_query(raw_query, history=None, memory=None):
    # Step 1: decide if rewriting is needed
    if is_well_formed(raw_query):
        return raw_query

    # Step 2: build the prompt
    prompt = build_rewrite_prompt(raw_query, history, memory)

    # Step 3: call the LLM
    rewritten = llm.generate(prompt, max_tokens=100)

    # Step 4: validate the output
    if not is_valid_query(rewritten):
        return raw_query

    return rewritten

The validation step matters more than the rewriting itself. Check that the output isn't empty, isn't a refusal, and still contains the core entities from the original query. If validation fails, fall back to the raw query. A bad rewrite is worse than no rewrite.

Caching and fallback strategies

LLM rewriting costs tokens and latency on every call. Cache the rewritten output keyed by the raw query plus a hash of the conversation context. Identical queries with identical context skip the LLM entirely. For high-traffic systems, this cuts rewriting cost by 60 to 80% on repeat queries.

Fallback works in layers. If the LLM times out, use the raw query. If the LLM returns garbage, use the raw query. If rewriting adds more than 200ms to your p95 latency, disable it for that traffic segment. The pipeline should degrade gracefully, not fail loudly.

One more thing: log every rewrite alongside the raw query and the retrieval results. When something breaks in production, you'll need to see what the rewriter did. Without logs, you're debugging blind.

Common Mistakes in Query Rewriting and Expansion

Most failures in query rewriting and expansion come from three places: rewriting queries that didn't need it, ignoring what the extra LLM call does to your p95, and dropping conversation context between turns.

Over-rewriting well-formed queries

A user types "what is the refund policy" and your rewriter turns it into "explain the terms and conditions governing monetary reimbursement for returned goods." That's not improvement. That's noise. The rewrite changed the embedding, pushed retrieval toward legal boilerplate, and made the answer worse. If the original query is specific and complete, pass it through untouched. Add a length and entity check before you call the LLM. Short queries with a clear noun phrase rarely need help.

Ignoring latency in production

Every LLM rewrite adds 200 to 800ms to your pipeline. That's fine in a batch job. It's not fine when a user is waiting on a chat response. Builders test rewriting in notebooks, see better recall, ship it, then wonder why the app feels slow. Measure the p95 before and after. If rewriting pushes you past your latency budget, cache aggressively or skip it for that traffic segment.

Breaking multi-turn context

The rewriter gets the current turn but not the history. "What about the pricing?" becomes "what about pricing" and retrieval returns generic pricing pages instead of the pricing for the plan the user just asked about. Pass conversation history into the rewrite prompt. If you don't, you're not rewriting for a conversation. You're rewriting for a search box.

Query rewriting and expansion is a tool with a narrow job. It fixes the gap between how users ask and how your documents answer. It doesn't fix bad chunking, a weak embedding model, or a knowledge base that's missing the answer entirely. Use it when recall is the bottleneck and the query is genuinely ambiguous, multi-hop, or conversational. Skip it everywhere else. The builders who get this right treat rewriting as a fallback, not a default. That's the whole playbook.

Frequently Asked Questions

What is query rewriting in RAG?

Query rewriting in RAG is the step where a user's original query is rephrased, clarified, or expanded before it is sent to the retriever. The goal is to close the gap between how users ask and how your documents are written. It is an optional preprocessing layer, not a replacement for good chunking or embeddings.

When should you skip query rewriting?

Skip rewriting when queries are already specific and keyword-rich, when your latency budget is very tight, or when your evaluation shows no recall gain over the original query. Also skip it for simple lookup queries where the user pasted an exact phrase or identifier. Rewriting adds cost, so it should earn its place with measured improvement.

What is HyDE and how does it relate to query rewriting?

HyDE, or Hypothetical Document Embeddings, asks an LLM to generate a hypothetical answer to the query and then embeds that answer for retrieval. It is a form of query expansion because the generated text often contains vocabulary closer to your documents. It can help with vague queries but adds an LLM call and may introduce hallucinated terms.

How does query rewriting fit into agent memory?

In agent memory systems, the current query often depends on earlier turns, so rewriting resolves pronouns and references before retrieval. A common pattern is to rewrite the latest turn using recent conversation history, then retrieve with both the rewritten and original forms. This keeps multi-turn retrieval coherent without storing every turn as a separate query.

Does query rewriting add too much latency?

It adds one or more LLM calls plus possibly extra retrieval passes, so latency can increase noticeably depending on your model and infrastructure. You can limit the impact by using a smaller model for rewriting, caching rewrites for repeated queries, and only rewriting query types that benefit. Always measure against a no-rewrite baseline before enabling it broadly.

What are common query rewriting techniques?

Common techniques include synonym or term expansion, multi-query generation where several paraphrases are retrieved in parallel, HyDE, query decomposition for multi-part questions, and step-back prompting that abstracts the query to a broader concept. Each targets a different failure mode, so match the technique to your query type rather than stacking all of them.

How do you evaluate whether query rewriting is working?

Compare retrieval recall and answer quality with and without rewriting on the same labeled query set. Track latency and token cost per query alongside quality so you can see the tradeoff. If rewriting does not improve recall or answer quality for your dominant query types, it is not worth the added complexity.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through query rewriting and expansion or something adjacent, we publish what we have actually tested, including where it falls short.