How to Structure a Retrieval Eval Workflow for RAG and Agent Memory
Your coding agent gives the wrong answer after you change chunking or swap a reranker. Was the relevant passage missing, or was it retrieved and then ignored, misread, or embellished by the model? Separate retrieval tests from answer tests so the investigation starts with the relevant component.
Retrieval-only evaluation compares returned passages and their ranking with labeled relevant passages for each test query. Evaluate the generated answer separately. Separating retrieval from generation makes the next fix clearer: test the corpus snapshot, filters, embeddings, reranker, and memory scope before changing prompts. This guide covers how to define that retrieval contract, label a dataset, run fixed retrieval tests, inspect failures, and set release gates for RAG and agent memory.
How to Structure a Retrieval Eval Workflow: Process Overview
Use the same labeled cases to compare retrieval changes, then record answer and agent evaluations separately.
- Version queries with relevant passage labels and memory-scope expectations.
- Score retrieval results separately from generated answers.
- Inspect failures, change the retrieval layer, and rerun the same set.
- Factor retrieval results into the release decision.
Define the Outcome of a Retrieval Eval Workflow
A retrieval eval workflow compares the passages returned for each test query with expected labeled passages and their ranks. Score retrieval first so a retrieval miss can be investigated separately from a generation failure.
[!tip] For an agent-memory test set, include the expected project and session or durable-memory scope with each query, so a relevant passage from the wrong scope does not silently count as a success.
Separate the retrieval contract from generation and prompt behavior
Write the contract before metrics: define the evidence and allowed scope for each query. Keep the generator and prompt outside this test. For a shared-memory system such as GigaRAG, include the access controls that determine what each agent can use.
Choose the retrieval unit: document, chunk, memory record, tool result, or code symbol
Label what your system returns: a chunk, memory record, tool result, or code symbol. Document labels can hide a wrong passage.
Define what counts as relevant, partially relevant, irrelevant, stale, or out of scope
Set grades in advance: relevant supports the task, partially relevant helps, irrelevant does not, and stale conflicts with current information. Mark out-of-scope items. Filters belong in the contract.
What retrieval metrics report
VDF AI's RAG evaluation guide describes retrieval metrics as measures of whether passages were found and ranked well. Use separate answer evaluation to assess groundedness, citation accuracy, and answer correctness.
Retrieval-Only Evaluation vs End-to-End Evaluation
| Factor | Retrieval-only | End-to-end |
|---|---|---|
| What it tests | Whether relevant passages appear and rank well | Whether the generated answer passes the selected generation checks |
| Useful measures | Recall@k, Precision@k, MRR, and NDCG | Faithfulness, answer relevance, citation accuracy, and answer correctness |
| Failure it can isolate | Missed or poorly ranked context | An overall outcome; use component tests to identify the cause |
| Best use | Diagnosing and tuning retrieval changes | Checking the user-facing result after component tests |
Step 1: Build a Versioned Dataset for RAG and Agent Memory
Build a retrieval dataset from questions and labels for the passages relevant to each question. It provides a reproducible measure for comparing retrieval changes on the same cases.
Use a minimal evaluation-case schema: query, relevant IDs, scope, filters, relevance grade, and expected abstention
Store each case with query, relevant_ids, scope, required filters, and a relevance grade per item. Add expected_abstention: true when no allowed record should answer the query. Passage-level IDs matter because a document label can hide a bad chunk match.
Seed a baseline with curated and synthetic questions, then add real production failures
Start from curated source material, then generate grounded questions from chunks to cover the corpus. Synthetic questions are useful for a first baseline, but they tend to be cleaner than real traffic, so add logged failures, typos, and vague requests over time. Emergent Mind's guide to synthetic QA datasets for RAG describes source curation and quality checks rather than blind generation.
Cover factual, multi-hop, reformulated, ambiguous, out-of-scope, and permission-sensitive queries
Tag every case by type. Include direct facts, questions needing two records, alternate wording, unclear requests, and queries the system should decline. Also include users whose filters must exclude relevant-looking material. For GigaRAG projects, include cases for the access controls that manage what each agent can use.
Worked example: evaluate a coding agent retrieving repository conventions and project decisions
For "How should this repo handle API retries?", label docs/conventions.md#retries as highly relevant and decisions/ADR-014.md as relevant. Set the scope to that repository and exclude another team's runbook, even if it uses the same retry policy.
Keep dataset, corpus snapshot, chunking configuration, and relevance labels versioned together
Pin the corpus snapshot, chunk IDs, embedding settings, chunking settings, and relevance labels beside the dataset. Versioning these inputs with the evaluation set supports reproducible before-and-after comparisons of retrieval changes.
Step-by-Step: Run a Retrieval Evaluation
- Create a versioned set of queries, expected answers, relevant passage IDs, and applicable memory-scope labels.
- Run each query through the retriever alone and save its ranked passage IDs.
- Calculate Recall@k and Precision@k from the relevant and retrieved passage sets; calculate MRR and NDCG to assess ranking.
- Inspect misses and low-ranked relevant passages, grouping them by likely retrieval layer or scope condition.
- Change one retrieval component or configuration, then rerun the same dataset and compare results.
- Check relevant memory boundaries and access conditions before deciding whether the change is safe to release.
- Record generated-answer and agent evaluations in a separate report.
[!note] Recall@k measures the share of labeled relevant passages retrieved in the top k; Precision@k measures the share of the top k results that are relevant. For example, if 2 of 4 relevant passages appear in the top 5, Recall@5 is 0.5; if 3 of those 5 results are relevant, Precision@5 is 0.6.
Step 2: Run Retrieval-Only Tests and Calculate the Right Metrics
Use the pinned dataset and corpus snapshot to run the retriever without the generator. Compare the returned ranked IDs with the labels for each case and retain the results for later diagnosis.
Run the retriever without the generator: record query, filters, corpus version, ranked IDs, scores, and latency
For every case, save the query, user scope, metadata filters, corpus and index version, ranked chunk IDs, similarity or reranker scores, and latency. Keep the raw ranking. You need it when a score drops.
Recall@k = |relevant items in top k| / |all relevant items|: use it to detect missed evidence
Recall asks whether needed passages appeared at all. If a query has two labelled relevant chunks and the top five return one, Recall@5 is 1/2 = 0.5. A missed passage can't help the generator later.
Precision@k = |relevant items in top k| / k: use it to quantify noise sent downstream
Precision measures noise in the context sent downstream. Two relevant chunks in five results gives Precision@5 of 2/5 = 0.4. Low precision can waste limited context on material that looks related but isn't useful.
MRR: use the average reciprocal rank of the first relevant result when the first good result matters
For one query, use the reciprocal rank of its first relevant result. If the first good chunk is at rank two, its score is 1/2 = 0.5. MRR is the average of those reciprocal-rank scores across the evaluated queries. It matters when one strong result should appear quickly.
Use NDCG@k for graded relevance and ranking position
Use NDCG when labels have grades, such as highly relevant, relevant, and irrelevant. It evaluates ranking quality by considering both relevance and position, rewarding systems that place highly relevant results nearer the top.
Worked example: interpret one query's Recall@5, Precision@5, reciprocal rank, and NDCG@5
For the retry-policy query, suppose ranks two and four return the two labelled chunks, with grades 3 and 1. Recall@5 is 1.0, Precision@5 is 0.4, and reciprocal rank is 0.5. The result contains both labeled chunks, while the higher-graded chunk appears below the first position.
Match k to the number of contexts your production agent actually receives
Choose k to match the number of chunks this production path sends to the agent. If the path receives five chunks, report metrics at five so the evaluation reflects the context available downstream.
Step 3: Diagnose Failures by Retrieval Layer and Memory Scope
Use the saved ranking to classify each failed case before you replace an embedding model, rebuild an index, or add a reranker. The ranking and case labels provide the evidence for investigating the retrieval layer or scope condition involved.
Use a failure taxonomy: chunk boundaries, embedding mismatch, metadata-filter error, reranking error, and scope leak
Mark the first cause you can prove. A relevant chunk may be absent from the corpus, split at the wrong boundary, excluded by a filter, retrieved too low, or replaced by stale content. Different causes need different fixes.
Evaluate memory boundaries: session, user, project, repository, team, and tool-specific context
Give each eval case an allowed scope and an expected source type. Define how session, user, project, repository, team, and tool-specific context apply to that case, then label relevance accordingly. GigaRAG connects agents to a central Hive Mind through MCP or the API, with access controls to manage what each agent can use.
Treat cross-project retrieval and permission-filter leaks as severe failures, not ordinary relevance misses
A leaked chunk is not ordinary relevance noise. Treat permission and metadata filters as retrieval conditions, and include queries from users who must not see the answer. Define the escalation and release response for protected-content exposure in your project's security policy.
Create hard negatives from near-misses and regressions
Keep highly ranked, plausible but wrong chunks beside each failed query. Feedback that marks those results as not relevant can supply hard negatives for reranker work.
Change one retrieval variable at a time and rerun the fixed suite
Change chunking, filters, embeddings, or reranking one at a time, then compare the same reference dataset after each change. This makes before-and-after retrieval results easier to interpret.
Set Release Gates, Then Evaluate the Rest of the Agent
Use retrieval results as one input to a release decision. Define separate checks for the generated answer and the agent behaviors that matter in your deployment.
Compare query categories with their baselines rather than relying on one aggregate score
Compare query categories with their baselines, such as repository queries, stale-memory cases, and permission-filter cases. A versioned reference dataset helps identify which categories need attention.
Define project-specific responses to scope leaks and critical-query regressions
Agree in advance with the deployment owner on how to respond to protected-content exposure and regressions on critical queries. Document the decision, any exception, and the responsible owner.
Evaluate answer quality and agent behavior separately from retrieval
Use separate evaluations for generated answers and agent behavior. Answer checks can assess whether claims are supported by retrieved context, whether the answer addresses the question, and whether citations support the claims they accompany.
Use retrieval evaluation as a repeatable engineering loop, not a one-time benchmark
Run the versioned suite after each retrieval change. Add labeled production failures to maintain regression coverage as the system and corpus evolve.
Frequently Asked Questions
Why should retrieval evaluation be separated from generation evaluation?
Retrieval-only scores show whether the needed passages were found and ranked; generation measures whether the model used that context appropriately. Separating them helps distinguish a retrieval miss from a generation problem.
What is a good starting size for a RAG retrieval evaluation dataset?
One practical guide recommends 50 questions for a first baseline, while emphasizing that an imperfect dataset is better than no measurement. Treat that as a starting recommendation, not a universal threshold, and improve coverage as real queries become available.
What is the difference between Recall@k and Precision@k in RAG?
Recall@k is the proportion of all labeled relevant passages found in the top k results. Precision@k is the proportion of the top k results that are relevant. If 2 of 4 relevant passages are retrieved in the top 5, Recall@5 is 0.5; if 3 of the 5 results are relevant, Precision@5 is 0.6.
When should I use MRR instead of NDCG?
Use MRR when the position of the first relevant result is the main concern: it averages the reciprocal rank of that result. Use NDCG when you need to assess the ranking of multiple results, particularly when relevance is graded.
How should k be chosen for retrieval evaluation?
Match k to the number of passages the system sends to the generator. For example, if the generator receives eight passages, report Recall@8 so the metric reflects the context available to it.
Can synthetic questions replace real user queries in a RAG evaluation set?
Synthetic questions can provide a reproducible starting set grounded in the corpus, but they may not reflect real users’ short, ambiguous, or misspelled queries. Add real queries as they become available.
How do I evaluate memory retrieval for an AI agent across projects and sessions?
Include queries with expected relevant passage labels and record the applicable project and memory scope for each case. Test whether retrieval respects those labels, including session versus durable memory, freshness, permissions, and tool relevance.
Do strong retrieval metrics guarantee that an AI agent will produce a correct answer?
No. Retrieval metrics assess whether relevant passages were found and ranked. Evaluate the generated answer separately to determine whether it used the retrieved context appropriately.
About GigaRAG
GigaRAG is a shared memory platform for AI agents that helps developers and teams keep project knowledge available across sessions and tools. Connect your agents to a central Hive Mind through MCP or the API, with access controls to manage what each agent can use.
