Evaluating RAG: A Practical Guide for Builders

Evaluating RAG: A Practical Guide for Agent Memory and Pipeline Builders

Evaluating RAG gets buried under two kinds of noise: academic surveys too dense to act on, and vendor guides too biased to trust. If you're a pipeline builder or agent memory engineer, neither helps you ship. You need to know which metrics catch real regressions, which ones just look good in a dashboard, and where evaluation stops being useful altogether. GigaRAG is built for exactly that audience, but this guide isn't a pitch for it. It's tool-agnostic advice you can apply to any stack. You'll learn what to measure on the retrieval side, what to measure on the generation side, how to build a synthetic eval dataset without production traffic, and how agent memory breaks standard RAG metrics. You'll also get the part most guides skip: what evaluation cannot tell you, no matter how many metrics you run.

In This Guide

• What Is RAG Evaluation?
• Automated Metrics vs Human Evaluation for RAG
• Evaluating Retrieval Quality: Metrics That Matter
• Evaluating Rag: A Step-by-Step Guide
• Evaluating Generation Quality: Faithfulness and Relevance
• RAG Evaluation Frameworks and Tools
• Building a Synthetic Evaluation Dataset
• Evaluating RAG for Agent Memory and Multi-Step Workflows
• What RAG Evaluation Cannot Tell You
• RAG Evaluation Best Practices
• How to Choose the Right Evaluation Approach

What Is RAG Evaluation?

RAG evaluation measures how well a retrieval-augmented generation pipeline retrieves the right context and uses it to produce a correct, grounded answer. It's not one score. It's a set of checks across two linked systems: the retriever that finds relevant chunks, and the generator that turns those chunks into a response.

Most RAG failures trace to one of two places. Either the retriever pulls the wrong context, or the generator ignores the context it was given. Evaluation exists to catch both before your users do.

Why RAG evaluation is different from LLM evaluation

Evaluating a standalone LLM means asking: is this answer good? Evaluating a RAG pipeline means asking two questions at once: did we fetch the right stuff, and did the model use it faithfully?

That's the core difference. A bare LLM can hallucinate freely because nothing grounds it. A RAG system is supposed to be grounded. So evaluation has to check the grounding itself, not just the final text.

Here's what happens behind the scenes. You send a query. The retriever returns, say, five chunks from your vector store. The generator reads those chunks and writes an answer. If chunk three was irrelevant, the answer might still look fine. But it's built on sand. RAG evaluation catches that by scoring the retrieval step separately from the generation step.

The two sides: retrieval and generation

Retrieval evaluation asks whether the right documents came back. You measure this with precision, recall, nDCG, and MRR. These metrics don't care about the final answer. They only care about what the retriever returned and in what order.

Generation evaluation asks whether the answer is faithful to the retrieved context. Faithfulness means every claim in the answer traces back to a chunk you actually retrieved. Answer relevance means the response actually addresses the question asked.

The honest answer is you need both. A perfect retriever with a sloppy generator still produces wrong answers. A great generator with a broken retriever produces confident nonsense. Neither side alone tells you the pipeline works.

[!note] RAG evaluation metrics often rely on LLM-based judges, which can introduce their own biases and may not perfectly align with human judgment; always validate with human review for critical applications.

Automated Metrics vs Human Evaluation for RAG

Evaluating Retrieval Quality: Metrics That Matter

Retrieval metrics answer one question: did the right chunks come back, and did they come back in a useful order? You score this before the generator ever touches the context. That separation matters because a bad retrieval step poisons everything downstream, no matter how good your LLM is.

Context precision and context recall

Context precision measures whether the retrieved chunks that are actually relevant rank high in the returned list. It's a signal-to-noise check. If your retriever returns five chunks and only the fifth one is useful, context precision is low. The generator has to wade through four irrelevant chunks to find the one that matters.

Context recall measures whether all the relevant chunks made it into the returned set at all. If your knowledge base has three chunks that answer the query and the retriever only returns one, recall is 0.33. The generator can't use what it never received.

In practice, you want both. High precision with low recall means the retriever is picky but misses things. High recall with low precision means it grabs everything, including noise. The right balance depends on your use case. For a support bot, precision matters more: you'd rather return one good chunk than five mediocre ones. For a research assistant, recall matters more: missing a source is worse than including an extra one.

nDCG and MRR: ranking-aware metrics

nDCG (normalized discounted cumulative gain) scores the entire ranked list, not just whether the right items appeared. It rewards systems that put the most relevant chunks first and penalizes systems that bury good chunks at the bottom. It's the metric to use when order matters, which is most RAG pipelines.

MRR (mean reciprocal rank) is simpler. It asks: where does the first relevant chunk appear? If it's first, MRR is 1.0. If it's third, MRR is 0.33. MRR is useful when you only care about the top hit, like a question-answering system that reads one chunk and answers.

The main catch is that both metrics need labeled relevance judgments. Someone has to say which chunks are relevant for each query. That's the expensive part, and it's why synthetic datasets have become the default starting point.

What retrieval metrics miss

Retrieval metrics don't tell you if the answer is correct. A retriever can return perfectly relevant chunks and the generator can still hallucinate, misread, or ignore them. Retrieval metrics also don't capture semantic overlap. Two chunks can both be relevant but one might be a better fit for the specific question asked.

They also miss the user's actual experience. A chunk that's technically relevant might be poorly written, outdated, or too dense for the generator to use well. None of that shows up in precision or recall.

So retrieval metrics are necessary but not sufficient. They tell you the retriever is doing its job. They say nothing about whether the pipeline as a whole produces answers your users can trust.

[!tip] For agent memory systems, create test cases that specifically probe memory retrieval and update accuracy over multiple interactions, as standard RAG metrics may not capture these dynamics.

Evaluating Rag: A Step-by-Step Guide

1. Define clear evaluation objectives and success criteria for your RAG system.
2. Assemble a diverse test set of queries and ground-truth answers.
3. Select appropriate metrics (e.g., context precision, faithfulness) for retrieval and generation.
4. Choose evaluation tools (e.g., RAGAS, DeepEval) or build custom scripts.
5. Run evaluations and collect metric scores.
6. Analyze results to identify failure modes and areas for improvement.
7. Iterate on your RAG pipeline and re-evaluate to track progress.

Evaluating Generation Quality: Faithfulness and Relevance

Retrieval metrics tell you the right chunks showed up. Generation metrics tell you whether the LLM actually used them. A retriever can return perfect context and the generator can still ignore it, contradict it, or answer a different question entirely. You need to score the output itself.

Faithfulness and groundedness

Faithfulness asks one question: does every claim in the answer trace back to the retrieved context? If the answer says "the policy allows refunds within 30 days" and no chunk says that, the answer is unfaithful. It doesn't matter if the claim happens to be true. Faithfulness is about grounding, not accuracy.

The honest answer is that measuring this automatically is hard. You can't just check for keyword overlap. A faithful answer might paraphrase heavily. An unfaithful answer might reuse the exact words from context while adding a false claim. That's why most teams use an LLM to judge faithfulness: feed it the answer and the context, ask it to flag unsupported claims.

Groundedness is the stricter cousin. It checks whether the answer stays within the retrieved evidence at all, rather than pulling in the model's parametric knowledge. A faithful answer can still be ungrounded if it blends context with outside facts. For agent memory systems, groundedness matters more than for single-shot RAG because the model has more chances to drift.

Answer relevance

Answer relevance measures whether the response actually addresses the question asked. It's possible to produce a perfectly faithful answer that's completely irrelevant. If a user asks "how do I reset my password" and the model faithfully summarizes your security policy, faithfulness scores high and relevance scores zero.

The main catch is that relevance is subjective. Two reviewers can disagree on whether an answer is "relevant enough." LLM-as-a-judge helps here because you can define relevance criteria in the prompt: does the answer directly address the query, or does it wander?

In practice, you evaluate relevance separately from faithfulness. A pipeline can fail either way, and the fix is different. Low faithfulness means your generator needs better grounding or a stricter prompt. Low relevance means your retriever might be pulling the wrong chunks, or your generator needs clearer instructions about staying on topic.

LLM-as-a-judge: pros and cons

LLM-as-a-judge means using a language model to score your pipeline's outputs. You give it the question, the retrieved context, and the generated answer, then ask it to rate faithfulness or relevance on a defined scale. It's fast, cheap, and scales to thousands of test cases without human reviewers.

The good news is that LLM judges correlate reasonably well with human judgment on faithfulness tasks. The bad news is they have their own biases. They tend to be lenient with answers that sound confident, and they can miss subtle hallucinations that a careful human would catch. They also inherit whatever blind spots the judging model has.

So use LLM judges for fast iteration and regression testing. Use human review for the final call on quality, especially before shipping changes to production. Don't treat an LLM judge's score as ground truth. It's a signal, not a verdict.

RAG Evaluation Frameworks and Tools

You can hand-roll every metric with a few hundred lines of Python. Most teams don't. Frameworks package the metrics, the judge prompts, and the test harness into something you can run in CI. The tradeoff is you inherit someone else's assumptions about what "good" looks like.

RAGAS: metrics and tradeoffs

RAGAS is the most widely used open-source framework for RAG evaluation. It gives you faithfulness, answer relevance, and context precision out of the box, plus a few newer metrics like context recall and answer correctness. You feed it a dataset of questions, answers, and retrieved contexts, and it returns scores per metric.

The main catch is that RAGAS leans heavily on LLM-as-a-judge. Every metric runs through a language model, which means your scores inherit that model's biases and cost money per evaluation run. It also assumes a single-shot RAG setup: one question, one retrieval step, one answer. Agent memory workflows with multi-step retrieval don't map cleanly onto its default metrics.

DeepEval: modular and test-driven

DeepEval takes a different approach. It's built around pytest-style test cases: you define assertions like "faithfulness should be above 0.8" and run them as part of your test suite. That makes it a natural fit if you already run CI for your pipeline code.

It's more modular than RAGAS. You can use its metrics without adopting its whole harness, and it supports custom metrics with less boilerplate. The downside is you write more setup code. DeepEval doesn't hand you a prebuilt evaluation dataset or a turnkey dashboard. You bring your own test cases, which is fine if you have them and a problem if you don't.

LangSmith and TruLens: observability-first

LangSmith and TruLens come at evaluation from the observability side. They trace every step of your pipeline: what got retrieved, what the model saw, what it generated, how long each step took. Evaluation is a layer on top of that trace data.

LangSmith is built by the LangChain team and integrates tightly with that ecosystem. If you're already using LangChain or LangGraph, it's the path of least resistance. TruLens is more framework-agnostic and offers feedback functions you can attach to any pipeline.

The honest answer is that both are heavier than you need if you just want metric scores. They're platforms, not libraries. You're signing up for hosted infrastructure, and the evaluation features are tied to that.

How to choose without vendor lock-in

Start with what you already have. If your pipeline is in LangChain, LangSmith is the obvious first look. If you want a test suite in CI, DeepEval fits. If you want a standalone metric library with a big community, RAGAS is the default.

Keep in mind that all four frameworks let you export raw scores. The lock-in isn't the metrics themselves; it's the workflow around them. Write your evaluation dataset in a plain format like JSONL, and you can switch frameworks without redoing the hard work. The dataset is the asset. The framework is just the runner.

Building a Synthetic Evaluation Dataset

You don't need production traffic to start evaluating. You need a dataset of questions, retrieved contexts, and reference answers that you trust. A synthetic dataset gets you there in a day instead of a quarter, and it's reproducible: you can regenerate it, version it, and share it with your team.

The honest catch is that synthetic data is only as good as the documents you feed it and the model you use to generate it. If your source documents are thin or your generator model is weak, your evaluation dataset will be thin and weak too. You're measuring your pipeline against a yardstick you built yourself. That's fine for relative comparisons. It's not fine for claiming your system is production-ready.

Generating questions from your documents

Start with the documents you actually retrieve from. Not a random sample of your corpus: the documents your pipeline is supposed to find. If you're building a support bot, use your top 50 support articles. If you're building agent memory, use the conversation histories and knowledge entries your agent actually stores.

For each document, ask a language model to generate questions that the document can answer. Give it the document text and a prompt like: "Write five questions a user might ask that this document answers. Make the questions specific enough that only this document, or a near-duplicate, would be a good answer." That last clause matters. Vague questions like "What is RAG?" pull in too many documents and make retrieval scores meaningless.

You'll get better questions if you vary the type. Ask for factual questions, comparison questions, and how-to questions. A dataset of only factual lookups won't stress your pipeline the way real users will.

Creating golden answers

A golden answer is the answer you'd want your system to produce for a given question, given the retrieved context. You generate it the same way: feed the question and the source document to a language model, and ask for a concise, grounded answer that uses only information from the document.

Here's the key decision: do you generate golden answers from the document alone, or from the document plus the retrieved context your pipeline actually returns? The first tells you whether your system could answer correctly. The second tells you whether it did, given what it retrieved. You need both. Generate one set of answers from documents alone to measure retrieval quality, and another set from retrieved contexts to measure generation quality.

Don't generate golden answers with the same model you're evaluating. If your pipeline uses GPT-4 for generation and you use GPT-4 to write golden answers, you're grading your own homework. Use a different model, or better, have a human review a sample of the golden answers before you trust the dataset.

Validating your dataset

Before you run a single metric, check the dataset itself. Take 20 random question-answer pairs and read them. Are the questions answerable from the documents? Are the golden answers actually grounded in the source text? A language model will happily generate a plausible-sounding question that no document in your corpus can answer, and a golden answer that invents facts.

The cheapest validation is a coverage check: for each question, verify that the source document contains the key entities and facts in the golden answer. You can do this with string matching for a first pass, then spot-check the failures by hand.

Keep in mind that a synthetic dataset is a starting point, not a finish line. Once you have real user queries, add them to the dataset. Real queries are messier, more ambiguous, and more valuable than anything you generate. The synthetic set gets you moving. The real set keeps you honest.

Evaluating RAG for Agent Memory and Multi-Step Workflows

Standard RAG evaluation assumes a single question in, a single answer out. You retrieve some chunks, you generate a response, you score both. Agent memory breaks that assumption in ways that quietly invalidate most of the metrics you just learned.

Why agent memory breaks standard RAG metrics

An agent with memory doesn't answer one question. It answers a question in the context of a conversation that may span hours, days, or weeks. The right retrieval isn't just "which chunk matches this query" but "which chunk matches this query given what the user said three turns ago and what the agent already knows about them."

Context precision, context recall, nDCG: these all assume a clean query-document pair. They have no way to score whether the agent retrieved the right memory at the right time. A retrieval can be perfectly relevant to the literal query and still be wrong, because the agent should have retrieved something from earlier in the conversation instead.

The main catch is contamination. Agent memory systems accumulate entries over time. Old, stale, or conflicting memories sit alongside fresh ones. Standard metrics will happily score a retrieval as correct because it matches the query, even when the retrieved memory contradicts what the user said last week. You need evaluation that accounts for time and state, not just relevance.

Evaluating retrieval over time

The unit of evaluation shifts from a query to a session. You build test scenarios that unfold over multiple turns, where each turn depends on what came before. Turn one: the user says their name is Alex and they're building a support bot. Turn five: the user asks "what should I name my deployment?" The correct retrieval pulls the memory from turn one. A standard metric scores turn five in isolation and can't tell you whether the agent remembered Alex.

You also need to test forgetting. Agent memory systems have retention policies, decay functions, and eviction. The right behavior isn't always "retrieve everything." Sometimes the agent should not retrieve an old memory because it's stale or superseded. Build test cases where the correct answer is "no retrieval" or "retrieve the newer memory, not the older one." Standard metrics have no way to score a correct non-retrieval.

Time-based evaluation also means testing order sensitivity. Does the agent retrieve the same memory whether it's stored first or last? Does it handle two memories that partially overlap? These are the failure modes that show up in production, not in a single-shot benchmark.

Multi-step workflow evaluation

Multi-step workflows compound the problem. The agent retrieves, reasons, acts, retrieves again, and the quality of the final answer depends on the whole chain, not any single step. You can't score the final answer and work backwards to blame retrieval or generation. You need to instrument each step.

The practical approach is to log every retrieval and every generation in the chain, then evaluate the chain as a sequence. Did the agent retrieve the right context at step one? Did it use that context correctly at step two? Did the second retrieval depend on the first, and did that dependency hold? This is closer to tracing a program than scoring a quiz.

Keep in mind that multi-step evaluation is expensive. Each test case is a full session, not a single query. You'll run fewer of them, and you'll need to be selective about which scenarios matter. Pick the workflows your users actually run, not the ones that are easy to script. A synthetic dataset of ten realistic multi-step sessions will teach you more than a thousand single-turn queries.

What RAG Evaluation Cannot Tell You

Every metric in this guide measures something real. Retrieval quality, faithfulness, relevance: these are all worth tracking. But none of them measure what your users actually experience, and pretending otherwise is how teams ship a well-scored pipeline that people hate.

Offline vs. online evaluation

Offline evaluation runs against a fixed dataset. You generate questions, you score answers, you get a number. That number tells you how your pipeline performs on the questions you thought to ask. It tells you nothing about the questions your users will actually ask.

The gap is bigger than most teams admit. Users phrase things oddly. They ask about topics you didn't include in your synthetic dataset. They follow up, correct themselves, change direction mid-conversation. Offline metrics can't capture any of that because the dataset is frozen before you run it.

Online evaluation means watching real traffic: click-through rates, thumbs up or down, whether users rephrase the same question after getting an answer. These signals are messier and slower to collect, but they measure the thing you actually care about. The honest answer is you need both. Offline metrics catch regressions before you ship. Online metrics tell you whether the thing you shipped is good.

Metrics don't measure user trust

A response can score perfectly on faithfulness and still be useless. It can cite the right source, stay grounded, answer the literal question, and leave the user unsure whether they can rely on it. Trust is built from consistency over time, from the system admitting when it doesn't know, from not confidently citing a source that contradicts what it said yesterday.

No metric captures that. You can measure whether an answer is grounded in retrieved context. You cannot measure whether the user believes it. The only way to learn about trust is to watch users interact with the system over multiple sessions and ask them directly. That's slow, expensive, and impossible to automate fully.

This matters more for agent memory systems. An agent that retrieves the wrong memory once and answers confidently will do more damage to trust than an agent that scores lower on relevance but consistently says "I'm not sure." Your evaluation suite won't flag that difference.

Evaluation is not a one-time task

You can't evaluate a RAG pipeline once and declare it done. The documents change. The embedding model gets swapped. The chunking strategy shifts. Users start asking different questions. Every one of those changes invalidates your previous scores.

Think of evaluation as a regression suite, not a certification. You run it before every significant change, and you run it continuously in production. A pipeline that scored 0.85 on faithfulness last month might score 0.72 today because someone added a new document source that overlaps poorly with the old one. If you're not re-running, you won't know until users complain.

The main catch is that continuous evaluation costs money. LLM-as-a-judge calls aren't free, and neither is the engineering time to maintain test datasets. You'll need to decide how often to re-run based on how fast your pipeline changes. A stable pipeline with monthly document updates can get away with weekly evaluation. A pipeline where embeddings change every sprint needs it on every deploy.

RAG Evaluation Best Practices

Most evaluation failures aren't metric failures. They're process failures. Teams pick one metric, run it once, and call it done. The practices below are the ones that actually catch problems before users do.

Evaluate retrieval and generation together

Retrieval and generation are two halves of one system, but they fail in different ways. Retrieval fails silently: the right document never enters the context window, and the model answers from whatever it did get. Generation fails loudly: the right context is there, but the model ignores it or contradicts it.

If you only score generation, you'll miss retrieval failures that happen to produce plausible-sounding answers. If you only score retrieval, you'll miss a model that consistently misuses good context. Run both sides on every evaluation pass. A pipeline with 0.9 retrieval precision and 0.4 faithfulness is broken, and you need both numbers to see that.

Use multiple metrics, not one

No single metric tells the whole story. Context precision measures whether retrieved chunks are relevant. Context recall measures whether all relevant chunks were retrieved. Faithfulness measures whether the answer sticks to retrieved context. Answer relevance measures whether the answer actually addresses the question.

Each metric catches a different failure mode. A pipeline can score high on context precision while missing half the relevant documents entirely. It can score high on faithfulness while answering a question nobody asked. Run at least one retrieval metric and one generation metric on every evaluation, and add a ranking-aware metric like nDCG when ordering matters.

The tradeoff is cost. LLM-as-a-judge metrics aren't free, and more metrics means more compute. Start with two or three that map to your most common failure modes, then expand as you learn where your pipeline actually breaks.

Test with real user queries

Synthetic datasets are useful for catching regressions, but they're written by people who already know the answers. Real user queries are messier. They contain typos, ambiguous pronouns, domain shorthand, and assumptions the user never states.

Pull a sample of real queries from your production logs, score them manually, and add the interesting ones to your evaluation set. A query that broke your pipeline once is worth more than fifty synthetic questions that didn't. Re-run those real queries on every significant change. If your pipeline handles the messy ones, the clean ones will take care of themselves.

How to Choose the Right Evaluation Approach

The right evaluation approach depends on where you are in the build cycle. A prototype doesn't need the same rigor as a system about to ship, and a production system can't afford the same compute budget as a pre-launch audit. Match the method to the stage, and you'll spend evaluation effort where it actually changes decisions.

Prototyping: fast and cheap

When you're testing whether a RAG approach works at all, you need signal, not precision. Run 20 to 50 queries through your pipeline by hand. Score retrieval with a simple binary check: did the right chunk show up in the top five results? Score generation by reading the answer and asking one question: does it use the retrieved context, or does it ignore it?

Skip the full metric suite here. LLM-as-a-judge on a prototype adds latency and cost without telling you anything you can't see by reading outputs yourself. The goal is to find the failure modes that kill the idea, not to produce a benchmark number. If retrieval is wrong on half your hand-picked queries, no metric will rescue the design.

Pre-production: thorough and rigorous

This is where you build the evaluation harness you'll keep. Assemble a golden dataset of 100 to 500 question-answer pairs, mixing synthetic questions with real ones from early testers. Run the full metric stack: context precision and recall for retrieval, faithfulness and answer relevance for generation, plus nDCG if ranking order matters.

The key difference from prototyping is repeatability. Every pipeline change gets scored against the same dataset, so you can see whether a new chunking strategy or embedding model actually helps. Log the per-query scores, not just averages. Averages hide the five queries that got dramatically worse.

Production: continuous and lightweight

Once users are hitting the system, evaluation shifts from exhaustive to continuous. You can't run a 500-query benchmark on every deploy, and you shouldn't try. Sample a small set of real user queries each week, score them with your existing harness, and track the trend.

Watch for drift: embedding model updates, document corpus changes, and prompt tweaks all shift scores in ways synthetic datasets won't catch. Set a threshold for when scores drop far enough to trigger a deeper investigation. The honest answer is that production evaluation is about catching regressions early, not proving your pipeline is good. That proof never comes from a single evaluation pass, and evaluating RAG well means accepting that the work is ongoing.

Frequently Asked Questions

What are the key metrics for RAG evaluation?

Key metrics include context precision (relevance of retrieved passages), context recall (coverage of relevant information), faithfulness (factual consistency of generated answer), and answer relevance (how well the answer addresses the query). These metrics help diagnose retrieval and generation issues separately.

What tools are available for RAG evaluation?

Popular open-source tools include RAGAS, DeepEval, and TruLens. These provide implementations of common metrics and integration with LLM frameworks. You can also build custom evaluation scripts using LLM APIs for flexibility.

How do I evaluate agent memory in RAG systems?

Evaluate agent memory by testing the system's ability to retrieve relevant past interactions and update memory correctly. Metrics can include memory retrieval accuracy, relevance of retrieved memories, and consistency of responses over time. Simulate multi-turn conversations to assess memory performance.

Can I rely solely on automated metrics for RAG evaluation?

No, automated metrics are useful for continuous monitoring and regression testing, but they have limitations. They may not capture nuanced errors, bias, or context-specific issues. Combine automated metrics with human evaluation for a comprehensive assessment.

What are the limitations of RAG evaluation?

RAG evaluation cannot guarantee real-world performance, as test sets may not represent all user queries. Metrics may not capture subjective quality, and LLM-based judges can be inconsistent. Evaluation is a tool for improvement, not a certification of quality.

How often should I evaluate my RAG pipeline?

Evaluate after significant changes to the pipeline, such as updating the retrieval model or generation prompt. For production systems, run automated evaluations regularly (e.g., weekly) to detect performance drift, and conduct human evaluations periodically.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through evaluating rag or something adjacent, we publish what we have actually tested, including where it falls short.