Building a Golden Eval Set for RAG and Agent Memory

Building a Golden Eval Set: A Practical Guide for RAG and Agent Memory Builders

Building a golden eval set for RAG and agent memory is the single most reliable way to stop guessing whether your pipeline got worse overnight. Most guides fail you here. They're either locked to one vendor's dashboard or so generic they tell you to "collect good examples" and then stop. You need something you can run in CI, diff against a previous commit, and trust when it says a prompt change broke retrieval. That's what this guide delivers. It's tool-agnostic, grounded in real production traces, and honest about where golden sets stop helping. GigaRAG users will find the agent memory sections directly applicable, but the workflow works with any stack. This guide covers sourcing examples from production logs, labeling expected outputs, choosing RAG metrics like faithfulness and context relevance, handling multi-turn conversations, versioning your set, and the limitations you'll hit once you start maintaining it.

In This Guide

• What a Golden Eval Set Is (and What It Isn't)
• Golden Eval Set vs. Ad-Hoc Testing for RAG and Agent Memory
• What a Good Golden Eval Set Looks Like
• Building A Golden Eval Set: A Step-by-Step Guide
• Sourcing Examples from Production Traces
• Labeling and Validating Expected Outputs
• RAG-Specific Metrics for Your Golden Eval Set
• Building a Golden Eval Set for Agent Memory and Multi-Turn Conversations
• Maintaining and Versioning Your Golden Eval Set
• Using Your Golden Eval Set in CI and Experiments
• What Golden Eval Sets Cannot Do
• Step-by-Step Workflow: From Production Logs to CI-Integrated Golden Eval Set

What a Golden Eval Set Is (and What It Isn't)

A golden eval set is a curated collection of input-output pairs, where each expected output has been validated by a human expert or grounded in a verified source. You run your RAG pipeline or agent against it, measure how close the outputs land, and track that score over time. It's a benchmark, not a training asset.

Definition: input-output pairs validated by experts

Each entry has two parts: the input (a user query, a multi-turn conversation, a retrieval task) and the expected output (the correct answer, the right context, the appropriate action). What makes it "golden" is the validation step. Someone who knows the domain has confirmed the expected output is right. Not generated by a model and assumed correct. Checked.

The input can be a single question or a full conversation history. The output can be a final answer, a set of retrieved documents, or an intermediate state like a memory update. The format depends on what you're evaluating.

Golden eval set vs. training data vs. fine-tuning data

Training data teaches a model. Fine-tuning data adjusts a model's weights. A golden eval set does neither. It sits outside the model entirely and measures behaviour.

Here's the practical difference: training data can be noisy, duplicated, or unlabelled. A golden eval set cannot. Every example must be clean and correct, because a single bad expected output poisons your metric. You'd never fine-tune on 50 examples, but 50 well-chosen eval examples can catch a regression.

What a golden eval set cannot do

It won't tell you if your system works in production. It tells you if your system still works on the examples you chose. Those are different claims.

It also won't catch novel failures. If a user asks something you never included, the eval set is silent. And it decays: queries that mattered six months ago may not matter now. You'll need to maintain it, which means time and judgment, not just storage.

[!note] A golden eval set is not a one-time artifact; it must evolve with your application and user base to remain representative.

Golden Eval Set vs. Ad-Hoc Testing for RAG and Agent Memory

What a Good Golden Eval Set Looks Like

A good golden eval set is small, sharp, and current. You don't need thousands of examples. You need the ones that catch real regressions. Fifty well-chosen pairs beat five hundred sloppy ones.

Representative of real user queries

Your eval set should mirror what users actually ask, not what you think they ask. Pull queries from production logs, not from your imagination. If 70% of real traffic is short lookup questions, your eval set should be mostly short lookup questions. A set full of complex multi-hop queries will tell you nothing about the system your users actually hit.

Diverse across topics and difficulty

Coverage matters. Include easy queries that should never fail, medium queries that exercise retrieval, and hard queries that push the system. Spread examples across your main topics and user segments. If you support five document collections, each one needs representation. A golden eval set that only tests one topic will miss regressions everywhere else.

Expert-validated or source-grounded

Every expected output needs a reason to be trusted. Either a domain expert wrote it, or it's grounded in a specific source you can point to. Don't accept model-generated labels without human review. The whole point is a benchmark you can trust when it says something broke.

Versioned and maintainable

The set changes over time. Queries go stale, topics shift, new features need coverage. Store it in version control with clear change history. Retire examples that no longer reflect real usage. A golden eval set you can't update is a snapshot, not a benchmark.

[!tip] For agent memory systems, include multi-turn examples where the agent must recall information from earlier turns; this exposes memory failures that single-turn tests miss.

Building A Golden Eval Set: A Step-by-Step Guide

1. Collect raw production logs of user queries, retrieved contexts, and model responses, ensuring privacy compliance.
2. Sample a diverse set of queries covering common intents, edge cases, and multi-turn conversations.
3. For each query, manually or semi-automatically annotate the ideal answer and the relevant context passages.
4. Define evaluation metrics (e.g., faithfulness, context relevance) and thresholds for pass/fail.
5. Split the dataset into a development set for iteration and a held-out test set for final validation.
6. Version the dataset alongside your code and integrate it into your CI pipeline for automated regression checks.
7. Periodically review and refresh the dataset to reflect changes in user behavior and system capabilities.

Sourcing Examples from Production Traces

Production traces are the best source of golden eval examples because they're real. Users already asked these questions. The system already answered them, well or badly. You're not guessing what users want. You're mining what they actually did.

Mining production logs for real queries

Start with your logging pipeline. If you're using Langfuse, LangSmith, or a similar tracing tool, you already have query, retrieved context, and response stored per request. Export the last 30 days of traces. Filter out empty queries, duplicates, and anything under a minimum character count. You'll end up with a raw pool of real user questions.

Don't cherry-pick the interesting ones yet. Take a random sample first. That sample tells you what your traffic actually looks like. If you only pick the queries that caught your eye, you'll bias the set toward failures and edge cases, and you'll lose the boring queries that make up most of your volume.

Identifying failures and edge cases

Once you have the random sample, pull the failures separately. Look for traces where the answer was rated poorly, where the user asked a follow-up that suggests confusion, or where retrieval returned nothing useful. These are your highest-value examples. A golden eval set that catches a known failure mode is worth more than ten examples that pass every time.

Edge cases come from the same logs. Queries with unusual formatting, mixed languages, typos, very long inputs, or references to previous turns. These are the cases your system will keep getting wrong if you don't pin them down.

Balancing real failures with synthetic examples

Real failures are gold, but they're not enough. Your production logs only contain what users have tried so far. They don't contain the query that would break your system next week. Synthetic examples fill that gap. Generate them from your knowledge base: take a document, write a query it should answer, write a query it should not answer, write a query that needs two documents. The honest split is roughly 70% real traces, 30% synthetic. Real traces keep you honest about actual usage. Synthetic examples cover the gaps.

Starting with 20-30 examples for critical use cases

Don't try to build a hundred-example set on day one. Pick your two or three most critical use cases, the ones where a regression costs you real money or real trust. Build 20 to 30 examples for those. That's enough to catch a broken prompt or a bad model swap in CI. You can expand later. A small set you actually run is worth more than a large set you never finish labeling.

Labeling and Validating Expected Outputs

You have your queries. Now you need the right answer for each one. That's the label. Without a trustworthy label, your eval set measures nothing.

Expert validation vs. model-generated labels

The honest answer is: use both, but know the difference. Expert validation means a human who knows the domain reads each query and writes or approves the expected output. It's slow and expensive. Model-generated labels mean you prompt a strong LLM to produce the expected output, then spot-check it. It's fast and cheap.

For 20 to 30 examples, expert validation is feasible. Do it. For larger sets, generate labels with a strong model and have an expert review a random 20 percent. If more than one in ten labels is wrong, redo the batch.

Source-grounding prompts for label generation

When you generate labels with a model, ground them in the source documents. Don't ask the model to answer from its own knowledge. Give it the retrieved context and say: "Answer using only the text below. If the text doesn't contain the answer, say so." That constraint keeps the label tied to what your RAG system could actually retrieve. A label that requires knowledge outside your corpus is useless.

Using a separate judge model for verification

Use a different model for verification than the one that generated the label. If the same model wrote the label and then judges the system's answer against it, you get a self-confirming loop. A separate judge model, ideally from a different provider, catches errors the label generator missed. It's not perfect. But it's independent.

Handling ambiguous or subjective queries

Some queries have no single right answer. "What's the best way to X" depends on context. For these, don't write one expected output. Write a rubric: the answer must mention A and B, must not claim C, and must stay under 200 words. Judge against the rubric, not a fixed string. That's the only honest way to score subjective queries.

RAG-Specific Metrics for Your Golden Eval Set

You have labeled examples. Now you need to score your system against them. Generic metrics like BLEU or exact string match won't cut it. RAG systems fail in specific ways: they retrieve the wrong context, they ignore the context they retrieved, or they answer a question you didn't ask. You need metrics that catch each failure mode separately.

Faithfulness: is the answer grounded in retrieved context?

Faithfulness measures whether every claim in the answer is supported by the retrieved context. It's the metric that catches hallucination. A faithful answer contains nothing that isn't in the source documents you passed to the model.

Here's how it works in practice. You take the answer and the retrieved context, then ask a judge model: "Does the answer contain any claims not supported by the context?" The judge returns a score from 0 to 1. A score of 1 means every claim traces back to the context. A score of 0 means the model made things up.

The main catch is that faithfulness says nothing about whether the answer is correct or useful. A model can faithfully repeat irrelevant context. That's why you need the other metrics.

Context relevance: is the retrieved context useful?

Context relevance measures whether the retrieved chunks actually help answer the query. It scores the retrieval step, not the generation step. You ask the judge: "Is this context relevant to the query?" The score reflects how much of the retrieved material is on-topic.

This matters because retrieval quality drives everything downstream. If your retriever pulls in three irrelevant chunks and one relevant one, the generator has to work around the noise. Context relevance catches that before it poisons the answer.

Context precision and recall

Context precision and recall break context relevance into two questions. Precision asks: of the chunks you retrieved, how many were useful? Recall asks: of the useful chunks in your corpus, how many did you retrieve?

Precision matters when your retriever over-fetches. Recall matters when it under-fetches. Most RAG systems tune one at the expense of the other. You need both numbers to see the tradeoff.

Answer relevancy

Answer relevancy asks whether the answer actually addresses the query. It's different from faithfulness. A faithful answer can be irrelevant: the model stays grounded in context but answers a different question than the one asked. Answer relevancy catches that.

Score it by asking the judge: "Does this answer directly address the query?" It's the metric that catches the model going off on a tangent while technically staying grounded.

Choosing metrics based on your use case

It depends on what you're building. If you're building a customer support bot where wrong answers cost money, prioritize faithfulness and answer relevancy. If you're building a research assistant where missing sources is the bigger risk, prioritize context recall. If your retriever is slow or expensive, context precision tells you whether you're fetching more than you need.

Don't run all five metrics on every example. Pick two or three that map to your failure modes. Run those consistently. That's how you get a signal you can act on.

Building a Golden Eval Set for Agent Memory and Multi-Turn Conversations

Single-turn eval sets assume each query stands alone. Agents don't work that way. A user asks a follow-up, refers back to something said three turns ago, or expects the agent to remember a preference stated in passing. Your eval set has to test that memory, not just retrieval.

Why single-turn eval sets fail for agents

A single-turn example is a query and an answer. That's it. An agent conversation is a sequence of turns where each response depends on what came before. If you evaluate each turn in isolation, you miss the failure modes that actually break agents: forgetting context, misremembering a previous answer, or failing to update state when the user corrects something.

The honest answer is that most RAG eval sets were built for search-style retrieval. They don't capture statefulness. An agent that scores perfectly on single-turn faithfulness can still lose the thread of a conversation by turn four.

Designing multi-turn eval examples

Start with a conversation transcript, not a single query. Each example in your eval set is a full dialogue: user turn, agent turn, user turn, agent turn. The expected output is the agent's response at each step, given the full history up to that point.

Make sure you include the failure cases that matter. A user correcting the agent mid-conversation. A user referring to "the second option" without restating what the options were. A user asking the agent to recall something from ten turns earlier. These are the turns that break memory systems.

Evaluating memory retrieval quality

Memory retrieval is a separate failure mode from document retrieval. Your agent might pull the right document but the wrong memory. Or it might retrieve a stale memory that the user has since overridden.

Score memory retrieval the same way you score context relevance: ask a judge model whether the memory the agent retrieved was the right one for this turn. But you also need a second check: did the agent update its memory correctly when the user gave new information? That's a write test, not a read test. Most eval sets skip it.

Example multi-turn eval set entry

Here's a concrete entry. It's short, but it tests memory, correction, and reference resolution.

Turn 1 User: "What's the weather in Portland this weekend?" Agent: "Rain on Saturday, sun on Sunday. Highs around 65." Expected: grounded answer with source.

Turn 2 User: "Actually, I meant Portland, Maine." Agent: "Portland, Maine shows snow Saturday, clearing Sunday. Highs in the 30s." Expected: agent recognizes the correction and updates context without re-asking.

Turn 3 User: "Book me a flight there for Sunday." Agent: "To Portland, Maine, departing Sunday. What time?" Expected: agent resolves "there" to the corrected location, not the original one.

Turn 4 User: "Make it the afternoon." Agent: "Sunday afternoon to Portland, Maine. I'll search for flights after 12pm." Expected: agent retains the destination and date from previous turns.

Each turn is scored separately against the expected output, but the expected output only makes sense with the full history. That's the difference.

Maintaining and Versioning Your Golden Eval Set

A golden eval set rots if you don't maintain it. Queries drift, products change, and examples that once caught real failures become noise. The fix is boring but necessary: treat the eval set like code, with cleanup, versioning, and retirement.

Deduplication and cleanup

Production logs contain near-duplicates. Two users ask the same question with different wording, or the same user retries a failed query three times. If you don't deduplicate, your eval set overweights common queries and hides rare ones.

Cluster by embedding similarity, then keep one representative per cluster. For exact duplicates, keep the first occurrence. For near-duplicates, keep the version that best reflects real user language. Run this cleanup every time you add new examples, not once at the start.

Versioning and change tracking

Every change to your eval set should be a commit. That means storing examples in a version-controlled format: JSONL, YAML, or a database with a change log. When a metric drops after a prompt change, you need to know whether the eval set changed too.

Tag each example with a source trace ID and a date added. When you retire an example, don't delete it. Mark it retired and record why. That history is what lets you audit whether a metric improvement came from a better model or a weaker eval set.

Retiring stale or biased examples

Some examples stop being useful. A query about a deprecated feature, a product name that changed, or a policy that no longer applies. Keeping them means your eval set tests the wrong thing.

Retire an example when the expected output no longer matches what a correct system should say. That's a judgment call, so make it explicit. Review the eval set on a schedule, monthly for fast-moving products, quarterly for stable ones.

Avoiding data leakage into training

The main catch: if your eval set examples end up in your fine-tuning data, your metrics lie. The model memorizes the answers and scores well without generalizing.

Keep eval examples out of any training or fine-tuning pipeline. Store them separately, tag them clearly, and audit your data pipeline to make sure nothing crosses over. If you use production logs for both eval and training, split them at the source.

Using Your Golden Eval Set in CI and Experiments

A golden eval set only pays off when it runs automatically. Manual evaluation catches problems after users do. A CI gate catches them before merge.

Setting up a CI regression gate

Wire your eval harness into your CI pipeline so every prompt or code change triggers a full run against the golden set. The gate fails the build when any metric drops below a threshold you set. Start with faithfulness and answer relevancy, the two metrics that catch the most user-facing failures.

Keep the gate fast. A full eval run on 200 examples with a judge model can take 10 to 20 minutes. That's fine for a nightly job, too slow for every commit. Run a 20-example smoke subset on every PR, the full set on merge to main.

Comparing prompt versions over time

The eval set is your only honest way to compare prompt versions. Run both versions against the same examples, same judge model, same thresholds. The delta tells you whether the new prompt is better, worse, or a wash.

Store results per version, not just pass/fail. A prompt that improves faithfulness by 3 points but drops answer relevancy by 5 is a tradeoff you need to see, not a binary decision.

Tracking metric drift

Metrics drift even when you change nothing. Model providers update endpoints, retrieval indexes shift, user behavior changes. Track your metrics over time on a dashboard or in a simple CSV. A slow decline in context relevance often means your index is stale, not your prompt.

Alerting on regressions

Set alerts for sudden drops, not just CI failures. A 10-point faithfulness drop in one day is a provider issue or a bad deploy. A 1-point drop per week for a month is drift. Both need attention, but different responses.

What Golden Eval Sets Cannot Do

A golden eval set is a snapshot. It captures what you knew was important at the time you built it. It does not capture what you haven't seen yet, and it won't maintain itself.

Maintenance overhead is real

Every example in your set has a shelf life. User questions shift, your product changes, the underlying documents get updated. A query that was common six months ago may be rare now. A source document that was correct then may be wrong today. You'll spend real time reviewing, relabeling, and retiring examples. Budget for it. A 200-example set that nobody touches for a quarter will quietly stop measuring what you think it measures.

Overfitting to stale examples

The honest answer is that you can tune your prompts and retrieval settings to ace your eval set while getting worse in production. It happens slowly. You tweak a prompt, faithfulness goes up two points, you keep the tweak. Repeat for a month. Now your pipeline is optimized for 200 hand-picked examples, not for real users. The eval set becomes a mirror, not a test.

Cannot capture all edge cases

Your golden set is finite. Production is not. Users will ask questions you never anticipated, in phrasings you never sampled, about documents you never included. A 100% pass rate on your eval set tells you nothing about the query that arrives tomorrow. The set is a floor, not a ceiling.

Not a substitute for production monitoring

This is the one that bites. A green CI run does not mean your system is healthy in production. Users hit rate limits, retrieval indexes go down, model providers degrade. Your eval set won't catch any of that. You still need production traces, error logs, and live sampling. The eval set catches regressions in your code. Production monitoring catches regressions in the world.

Step-by-Step Workflow: From Production Logs to CI-Integrated Golden Eval Set

You've read the theory. Here's the actual sequence I use to turn raw production traces into a versioned, CI-gated eval set. It's tool-agnostic. Swap in whatever logging, storage, and CI stack you already run.

Step 1: Export production logs

Start with a query. Pull every user question, retrieved context, and final answer from your production trace store for the last 30 days. If you don't have a trace store, start logging one now. You can't build a golden eval set from memory.

import json
from datetime import datetime, timedelta

def export_traces(trace_store, days=30):
    cutoff = datetime.now() - timedelta(days=days)
    traces = trace_store.query(
        start_time=cutoff,
        fields=["user_query", "retrieved_contexts", "final_answer", "user_feedback"]
    )
    with open("raw_traces.jsonl", "w") as f:
        for t in traces:
            f.write(json.dumps(t) + "\n")
    return len(traces)

Keep user feedback if you have it. Thumbs-down traces are gold for finding failure cases.

Step 2: Filter and deduplicate

Raw logs are noisy. Drop traces with empty queries, non-English text if your product is English-only, and anything under 10 characters. Then deduplicate by normalized query text: lowercase, strip punctuation, collapse whitespace.

def normalize(q):
    return " ".join(q.lower().split())

seen = set()
deduped = []
for trace in raw_traces:
    key = normalize(trace["user_query"])
    if key not in seen and len(key) > 10:
        seen.add(key)
        deduped.append(trace)

Aim for a spread across topics and difficulty. Don't let one popular query dominate the set.

Step 3: Label and validate

For each deduped trace, you need a ground-truth answer. If the trace has a thumbs-up and the answer looks right, keep it as-is. If not, generate a label using a strong model with a source-grounding prompt: "Answer using only the provided context. If the context is insufficient, say so." Then have a second model or a human reviewer check the label against the context. This is the slow part. Budget a day per 50 examples.

Step 4: Version and store

Store the set as JSONL in git. Each line gets an ID, the query, the expected answer, the source documents, and a version number. Bump the version every time you add, remove, or relabel an example.

{"id": "eval-001", "query": "How do I reset my password?", "expected_answer": "Go to Settings > Security > Reset password.", "source_docs": ["help-center/password-reset.md"], "version": 3}

Step 5: Integrate with CI

Run the eval set on every pull request that touches retrieval, prompting, or memory logic. Fail the build if faithfulness or context relevance drops below your threshold. A 5-point drop is a real regression. A 2-point drop is noise. Set your gate accordingly.

That's the loop: export, filter, label, version, gate. Building a golden eval set is not a one-time project. It's a pipeline you run every month, and the CI gate is what keeps it honest.

Frequently Asked Questions

How many examples should a golden eval set have?

Start with 50-200 examples to cover key scenarios and edge cases. As your application matures, expand to several hundred. The goal is representativeness, not sheer size.

What metrics should I use for RAG evaluation?

Focus on faithfulness (answer grounded in context), context relevance (retrieved passages are pertinent), and context precision/recall (retrieval quality). These cover the main failure modes in RAG pipelines.

How do I evaluate multi-turn conversations and agent memory?

Include conversation histories in your eval set and design metrics that check whether the agent correctly references or updates memory across turns. For example, ask a question that requires recalling a fact from an earlier turn.

Can I use synthetic data to build a golden eval set?

Yes, synthetic data can bootstrap your dataset, but it must be validated by humans to ensure quality and representativeness. Relying solely on synthetic data risks missing real-world nuances.

How often should I update my golden eval set?

Review it quarterly or whenever you make significant changes to your model, retrieval system, or user interface. Also update when you notice new user intents or failure patterns.

What are the limitations of golden eval sets?

They can become stale, may not cover all edge cases, and can lead to overfitting if used too aggressively for tuning. They are a tool for measurement, not a guarantee of real-world performance.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through building a golden eval set or something adjacent, we publish what we have actually tested, including where it falls short.