Retrieval Returns the Wrong Chunk: A Diagnostic Tree
You ask for the repository's release-branch rule. Your RAG system returns a chunk about branch naming, then gives a confident answer that the policy never states. Don't change chunk size, embeddings, or prompts yet.
The fastest path to a fix is to identify the stage where the expected evidence disappears.
Quick answer
Inspect the retrieval trace before changing chunking, ranking, or the model. Verify whether the returned context contains the information needed for the expected answer.
- Check whether the expected source is in the corpus and whether the returned chunk contains the needed evidence.
- If retrieval looks correct, inspect what reached the prompt before blaming generation.
What it means when retrieval returns the wrong chunk
A retrieval failure occurs when the context returned to the model does not contain the right information for the question. This differs from a generation failure: retrieval checks which documents were surfaced, while answer faithfulness checks whether the response is grounded in the retrieved context. Deasy Labs' RAG metrics guide treats these as separate checks.
Wrong, missing, stale, and unauthorized chunks are different failures
Treat the returned text and its version as part of the diagnosis.
An irrelevant chunk does not support the needed claim. A missing expected source indicates that relevant evidence was not surfaced. A stale result is an outdated version of an otherwise relevant source. An unauthorized result is content that should not be available in the caller's access context.
These symptoms call for different checks: inspect corpus versions for stale results, retrieval coverage for missing evidence, metadata and filters for exclusions, and access controls for unauthorized results.
Capture one failing query and its expected evidence
Before you change chunk size or swap embeddings, save one reproducible failure. Record the exact query, expected source and version, expected answer-bearing text, returned chunk IDs and text, rank positions, scores, metadata, active filters, and caller identity.
Keep the query unchanged. Otherwise, you can't tell which change caused the result.
Do not diagnose from the final answer alone
A confident wrong answer does not identify the failing stage by itself. Check whether the retrieved context contains the needed evidence, then assess whether the final answer is grounded in that context.
Check the retrieval trace first. If the expected evidence is in the returned context, retrieval has not yet been shown to be the failure.
Retriever failure vs. generation failure
| Factor | Retriever failure | Generation failure |
|---|---|---|
| Retrieved evidence | Expected evidence is absent, or the returned chunks do not contain the expected supporting text. | Retrieved chunks may contain the needed evidence. |
| What to inspect | Query, source presence, chunk text, metadata, filters, and ranking outputs. | Whether the response is grounded in the retrieved evidence. |
| What the symptom proves | A wrong answer alone does not identify this failure. | A wrong answer alone does not identify this failure. |
Retrieval returns the wrong chunk: Follow the diagnostic tree
Start with the saved failure, then test corpus presence, ingestion, filters, and ranking in order. Trace the result through each stage before changing retrieval settings.
Step 1: Does the expected answer exist in the active corpus and correct version?
Find the expected source in the corpus your query actually searched. Check its version, status, and timestamp. Superseded documents that remain in a corpus can produce a freshness failure, so correct corpus scope or version management before changing search.
Confirm that the active corpus includes the intended source and current version before tuning retrieval.
Step 2: Was the expected source indexed, parsed, and split without losing the answer?
Inspect the indexed text and chunk boundaries, not the original file alone. Verify that an indexed chunk contains the expected answer-bearing passage and that parsing did not omit it.
Step 3: Did query rewriting, metadata filters, or access controls exclude the source?
Log the original query, rewritten query, active filters, and effective caller identity. Replay the query while changing one constraint at a time, then compare the returned candidates. If your team uses GigaRAG for project knowledge, record the access controls and effective identity used for that run as well.
Don't bypass access controls in production just to improve recall.
Step 4: Did first-stage retrieval rank the correct chunk below the cutoff?
Search broadly enough to locate the expected chunk, then record its rank and score. Compare the candidate set with the configured cutoff to see whether the expected chunk was included. Read the chunk text as well as its similarity score: a similar-looking result may still lack the needed evidence.
Step 5: Did hybrid search or reranking demote the answer-bearing chunk?
Compare the candidate list before and after each retrieval stage, including any keyword search, vector search, fusion, or reranking your system uses. Identify where the expected chunk's position changes, then inspect that stage's scores and rules. Context precision and context recall evaluate retrieval quality, so measure them before changing the generator.
Step 6: If retrieval is correct, inspect context assembly and generation separately
If the expected chunk reaches the retrieved set, compare that set with the final prompt and then test whether the answer is grounded in the retrieved context. Correct retrieval and a grounded final answer are separate checks.
Step by step: Trace a wrong-chunk result
- Save the exact query and identify the source and passage you expected to answer it.
- Check whether that source and passage are present in the corpus, including the version you intended to use.
- Inspect returned chunk IDs and text to see whether any chunk contains sufficient evidence.
- Review active filters and the returned chunks' metadata to find exclusions or mismatches.
- Compare rank positions and available retrieval scores or reranker outputs for relevant and irrelevant chunks.
- If the right evidence was retrieved, inspect the assembled prompt before investigating how the model used it.
Using GigaRAG to diagnose wrong retrieval results
GigaRAG can serve as a shared project-knowledge system alongside a diagnostic record. Use its records as part of your evidence trail, while relying on your pipeline's own logs for the query, filters, ranks, and returned context under investigation.
[!tip] Keep each replay isolated: change one retrieval component at a time and compare the result with the saved baseline.
1. Choose the GigaRAG connection method for the test
GigaRAG connects agents to a central Hive Mind through MCP or the API. Choose the connection method used for the test and record the connection version and setup details with the diagnostic record.
2. Apply configured access controls and record the effective caller identity
Apply the access controls configured for the agent and record the effective caller identity with the test. GigaRAG provides access controls for managing what each agent can use, so the identity and permissions used in a replay belong in the diagnostic record.
3. Label shared memory separately from retrieved knowledge in the test record
Label the material in your test record by where it entered the workflow. Keep shared project knowledge, retrieved chunks, and other context fields distinct so the returned evidence can be evaluated on its own.
4. Preserve source provenance: source identifier, version or timestamp when available, returned text, and retrieval path
Save the source ID, available version or timestamp, exact returned text, and the path by which it entered the context. These details make it possible to compare the returned material with the source version you expected.
5. Replay one known failure and compare the expected source with the returned context
Run the saved query under the same identity and compare the expected source with the returned context. Check whether the returned text includes the answer-bearing passage needed for the test case, rather than relying on matching terms alone.
Keep external pipeline traces with the diagnostic record
Use the external pipeline's logs to examine chunking, embeddings, filters, reranking, prompt construction, and generation. Keep those traces with the diagnostic record so a replay can be compared against the same query, identity, and expected source.
Other methods for improving retrieval quality
Once the trace points to a likely fault, change that component first. Excessive retrieved context can reduce precision, so compare each change against saved failures rather than broadening the system indiscriminately.
[!note] A retrieved document can be the wrong version even when it covers the right topic. Freshness is a distinct failure mode from retrieving an unrelated document.
Re-chunk by document structure and answer-bearing concepts, not a universal token count
Split at headings, procedures, tables, and complete policy rules when your evaluation cases show that those boundaries keep the expected answer-bearing passage together. Validate the resulting chunks against saved failures rather than adopting one universal chunk size.
Add metadata, versioning, and narrow filters before broadening top-k
Add domain, topic, version, and access fields where your corpus supports them. Sparse metadata can contribute to irrelevant retrieval, while stale documents can return outdated material. Use filters to narrow the candidate pool before evaluating whether to expand the retrieved set.
Test hybrid retrieval and reranking with a fixed evaluation set
If your pipeline uses hybrid retrieval or reranking, evaluate those stages against saved failures with expected sources. Compare the candidate lists, ranks, and returned text before and after the stage so you can determine whether it improves retrieval for the cases you care about.
Use query rewriting only when the original and rewritten queries are both logged
When you use query rewriting, save the original and rewritten queries alongside the filters and ranks. Comparing both traces shows how the rewritten query changed the retrieved candidates.
Create regression tests for repositories, tickets, docs, and tool outputs
Keep cases that cover current and superseded documents, restricted sources, and exact technical terms. Check the retrieved context and source version before evaluating answer quality so the test distinguishes current material from superseded material.
Make retrieval debugging repeatable
Saved failures beat loose scores. Build a small evaluation set that covers expected sources, access contexts, and answer-bearing passages. Record shared project knowledge and retrieved chunks in separate fields so the retrieval result can be evaluated on its own.
Keep a small failure set with expected sources and access contexts
For each case, define the text retrieval must return and save the source version when available.
Record retrieval traces before and after every change
Save query, rewrite, filters, chunk IDs, ranks, returned text, and assembled context before and after each change. Compare both traces.
Escalate to prompt or model debugging only after retrieval and context assembly pass
Once the expected source is retrieved at the required access level and answer-bearing text survives context assembly, test prompt or model behavior. A wrong answer alone doesn't prove retrieval failed.
Frequently asked questions
How do I know whether the retriever or the LLM caused a wrong RAG answer?
Inspect the retrieved chunks and the assembled prompt. If the needed evidence is absent from retrieval, investigate retrieval; if it is present in retrieval, check whether it survived prompt assembly and how the model used it. A wrong answer alone does not establish which stage failed.
What should I log when a RAG system retrieves the wrong chunk?
Capture the exact query, expected source, returned chunk IDs and text, relevant metadata, active filters, rank positions, and any retrieval scores or reranker outputs. These artifacts help distinguish corpus, filtering, and ranking issues.
What should I check when a high-scoring chunk does not answer the question?
Read the returned chunk and compare it with the expected answer-bearing passage. A similarity score ranks candidates; it does not by itself show whether a chunk includes the text needed for the test case.
How should I investigate filters during a retrieval diagnosis?
Record the active filters and the metadata of both the expected source and returned results. Replay the saved query while changing one constraint at a time, then compare the candidate lists to identify where the expected source stops appearing.
How do stale documents create a wrong-chunk problem?
Retrieval can return a document on the right topic but from an outdated version. Check whether the returned source is the intended current version, not just whether it is relevant to the query.
Should I change chunk size first when retrieval quality is poor?
Not before inspecting the trace. First establish whether the expected evidence is in the corpus and whether returned chunk text contains it. Then use the observed failure to choose what to test.
What should I separate from retrieved evidence during a diagnosis?
Evaluate the documents or chunks returned for the query as retrieval evidence. Record shared project knowledge and other context separately so you can tell what the retrieval step actually surfaced.
When should I test reranking or hybrid search in a RAG pipeline?
Test these stages when saved traces show that relevant candidates need closer comparison or that their positions change across retrieval stages. Use a fixed set of queries with expected sources, then compare candidate lists, ranks, and returned text before and after the change.
About GigaRAG
GigaRAG is a shared memory platform for AI agents that helps developers and teams keep project knowledge available across sessions and tools.
You can connect agents to a central Hive Mind through MCP or the API. GigaRAG also has access controls to manage what each agent can use.
