Building a Golden Eval Set for RAG and Agent Memory

Building a Golden Eval Set for RAG and Agent Memory

Building a golden eval set is the step most RAG and agent memory builders skip, and it's the one that tells you whether your retrieval changes actually helped. Most guides don't help. They're written for generic LLM apps, or they assume you've already bought into a specific tool like Langfuse or DeepEval. If you're managing vector chunks, context windows, and memory persistence, you need ground truth that covers retrieval quality, context relevance, and recall behavior, not just answer accuracy. Platforms like GigaRAG can help teams manage eval sets and run experiments, but the process here is tool-agnostic. You'll learn how to define a golden eval set, source entries from production traces, structure them for RAG and agent memory, augment with synthetic edge cases, evaluate set quality, and maintain it over time. And you'll get the honest limits, because a golden set won't catch every failure mode.

In This Guide

• What Is a Golden Eval Set?
• Golden Eval Set vs. Ad-Hoc Testing for RAG and Agent Memory
• What a Good Golden Eval Set Looks Like
• Building A Golden Eval Set: A Step-by-Step Guide
• Sourcing Golden Eval Data from Production Traces
• Structuring Golden Eval Entries for RAG
• Structuring Golden Eval Entries for Agent Memory
• Augmenting Your Golden Eval Set with Synthetic Edge Cases
• Evaluating Your Golden Eval Set Quality
• Maintaining a Golden Eval Set
• What Golden Eval Sets Cannot Do
• Final Thoughts on Building a Golden Eval Set

What Is a Golden Eval Set?

A golden eval set is a curated collection of inputs with known-good outputs, used to measure whether changes to your RAG pipeline or agent memory actually improve quality. It's the yardstick you run every experiment against.

A regular test set checks that code works. A golden eval set checks that the system answers correctly. The difference matters because LLM outputs are fuzzy. Two runs can both "work" and produce wildly different answers. You need a fixed set of questions with verified answers to tell which run was better.

Golden eval set vs. regular test set

A regular test set lives in your CI pipeline. It asserts that functions return expected values, that endpoints respond, that nothing crashes. It tests mechanics.

A golden eval set tests judgment. Each entry pairs a query with a ground truth answer, and often with the specific chunks or context that should have been retrieved. When you change your embedding model, your chunking strategy, or your prompt, you run the golden set and compare scores. The test set tells you if the system broke. The golden set tells you if it got dumber.

Why RAG and agent memory need specialized eval sets

Generic LLM eval sets check answer quality alone. That's not enough for RAG. You also need to know whether the right chunks were retrieved, whether the context was relevant, and whether the answer stayed grounded in that context. A correct answer built on the wrong chunks is a failure waiting to happen.

Agent memory adds another layer. You need entries that test whether the agent remembers what it should across turns, forgets what it should, and recalls the right information at the right time. A golden set built for a chatbot won't catch memory regressions. You need entries designed for that job.

[!note] A golden eval set is a static snapshot of desired behavior; it cannot capture all possible user interactions or guarantee that passing it means your system is production-ready. It is a tool for measuring progress, not a substitute for live monitoring.

Golden Eval Set vs. Ad-Hoc Testing for RAG and Agent Memory

What a Good Golden Eval Set Looks Like

A golden eval set earns its name through four properties: it mirrors real user queries, it spans the full range of what your system faces, every label is verified, and every version is reproducible. Miss any one of these and your scores stop meaning anything.

Representative of real user queries

Your golden set should look like your production traffic. If 70% of real queries are short product questions and 30% are multi-hop comparisons, your eval set needs that same split. A set built from hypothetical questions you think users might ask will drift from what they actually ask. Mine your logs. Pull real queries verbatim, including the messy ones with typos, fragments, and half-formed intent. Those are the queries that break pipelines.

Diverse across topics, difficulties, and edge cases

Representativeness covers frequency. Diversity covers coverage. You need entries that span every topic your system handles, every difficulty level from trivial lookup to multi-step reasoning, and every edge case: ambiguous queries, out-of-domain questions, adversarial prompts, and queries that require information from multiple chunks. A golden set with 200 near-identical "what is X" questions will tell you nothing about whether your retriever handles negation, temporal reasoning, or conflicting sources.

Correctly labeled with verified ground truth

Every entry needs an answer you'd stake your evaluation on. For factual queries, that means checking against source documents, not trusting the first LLM output you got. For retrieval entries, it means manually confirming that the expected chunks actually contain the answer. One mislabeled entry in a 100-entry set skews your scores by a full point. In a 50-entry set, it's two points. Label errors compound when you use the set to compare models or prompts.

Versioned and reproducible

Your golden set is code. It needs a version number, a changelog, and a way to roll back. When you add 30 entries, bump the version. When you retire stale ones, bump it again. Without versioning, you can't tell whether a score change came from your pipeline or from your eval set. Store the set in git alongside your evaluation harness. Every experiment should log which version it ran against. That's the only way to compare results across time.

[!tip] For agent memory evaluation, include multi-turn scenarios where the agent must recall information from earlier in the conversation or from past sessions. This is often overlooked but critical for memory persistence.

Building A Golden Eval Set: A Step-by-Step Guide

1. Define the evaluation dimensions you care about: retrieval quality, context relevance, answer groundedness, and memory persistence.
2. Collect real user queries from logs, support tickets, or stakeholder interviews, ensuring diversity in length, complexity, and topic.
3. For each query, manually curate the ideal retrieved context and the expected answer, marking any that require multi-session memory.
4. Include adversarial and edge cases: ambiguous queries, queries with no answer in the knowledge base, and queries that test memory recall.
5. Store the dataset in a version-controlled format (e.g., JSON or CSV) with clear fields for query, expected context, expected answer, and metadata.
6. Run your RAG or agent system against the golden set, compute metrics (e.g., retrieval recall, answer correctness), and record results.
7. Review failures, update the golden set with new cases, and repeat as your system evolves.

Sourcing Golden Eval Data from Production Traces

Your production logs are the best source of golden eval data you'll ever have. Real users ask real questions, hit real failures, and leave a trail you can mine. Synthetic examples have their place, but they'll never capture the mess of actual usage.

What to log in production

You need four things from every trace: the raw query, the retrieved chunks, the final answer, and any user feedback. Log the query verbatim, typos included. Log which chunks the retriever pulled and their scores. Log the answer your system produced. And log whatever signal you have about quality: thumbs up, thumbs down, a follow-up question that suggests confusion, or a support ticket that references the interaction.

Don't log everything forever. Store traces for 30 to 90 days, then sample. A week of full traffic usually gives you enough material to start.

Identifying high-value queries and failures

Failures are worth more than successes. A query that produced a wrong answer, an empty retrieval, or a user complaint is a golden eval entry waiting to happen. Those are the cases your system got wrong, and those are exactly the cases your eval set needs to catch next time.

Look for three signals: explicit negative feedback, queries where retrieval returned nothing relevant, and answers that required a human to step in. Each one is a candidate.

Converting traces into labeled eval entries

Take the raw query and pair it with the correct answer, not the answer your system gave. That's the labeling step, and it's manual. Pull the source documents, find where the right answer actually lives, and record those chunks as expected retrieval. One trace becomes one entry: query, expected chunks, expected answer.

Avoiding data leakage

Keep your golden set separate from anything you use to tune prompts or embeddings. If you test on the same traces you trained on, your scores lie. Split traces by time: entries from June evaluate the system you ship in July. Never let an eval entry influence the system before you score it.

Structuring Golden Eval Entries for RAG

A RAG eval entry needs more than a query and an answer. Retrieval can fail while the answer still looks fine, or the answer can be wrong even when retrieval worked. You need ground truth for each stage separately, or your scores will hide the failure.

Required fields: query, expected chunks, expected answer

Every entry starts with the query, verbatim. Typos stay. Next, the expected chunks: the specific passages from your knowledge base that contain the answer. These are what your retriever should have pulled. Last, the expected answer: the correct response a user should get, written in plain language.

The expected chunks matter more than most teams admit. If you only score the final answer, a system that retrieves garbage but generates a plausible-sounding response can pass. Expected chunks let you measure retrieval quality directly.

Optional fields: rubric, difficulty, category

A rubric defines what counts as a correct answer when more than one response is acceptable. For factual queries, exact match works. For open-ended queries, you need criteria: covers the key points, cites the right source, no hallucinated details.

Difficulty and category tags help you spot patterns. If all your failures cluster in one category, you know where to focus. Tags cost little to add and pay off when you're debugging.

Separating retrieval ground truth from answer ground truth

Retrieval quality and answer quality are different things. A system can retrieve the right chunks and still write a bad answer. It can retrieve nothing useful and still stumble into a correct response by luck. If your eval entry only has one ground truth field, you can't tell which stage broke.

Store expected chunks and expected answer as separate fields. Score retrieval against the chunks. Score the answer against the answer. When a change improves one and hurts the other, you'll see it instead of watching a blended score stay flat.

Structuring Golden Eval Entries for Agent Memory

Memory eval entries look different from RAG entries. You're not testing retrieval from a knowledge base. You're testing whether the agent stored the right thing, recalled it at the right time, and used it correctly.

Multi-turn conversation entries

Each entry is a full conversation, not a single query. You need the complete turn history: what the user said, what the agent said, and what the agent should have remembered after each turn. A single-turn snapshot can't catch memory failures, because the failure only shows up two or three turns later.

Keep the conversation realistic. Real users don't say "remember my name is Sarah" in turn one and then ask "what's my name" in turn two. They mention details in passing, then reference them obliquely later. Your entries should do the same.

Expected memory state and recall behavior

After each turn, define the expected memory state: what facts the agent should have stored. Then define the expected recall behavior: when the agent should use those facts. These are separate checks. An agent can store a fact correctly and never use it, or use a fact it never stored.

The honest answer is that memory state is hard to label precisely. You're defining what a reasonable agent should remember, not what a perfect one would. Leave room for judgment.

Testing memory persistence across sessions

Persistence is the hard part. An agent can remember things within one conversation and forget everything the moment the session ends. You need entries that span multiple sessions: the user tells the agent something in session one, then returns days later and expects the agent to still know it.

These entries are expensive to build and run. They require simulating session boundaries, which most eval harnesses don't handle natively. You'll likely need custom tooling. But if your agent claims long-term memory, this is the only way to verify it.

Augmenting Your Golden Eval Set with Synthetic Edge Cases

Production traces give you real failures. They don't give you failures you haven't hit yet. Synthetic edge cases fill that gap: adversarial queries, ambiguous phrasing, out-of-domain requests. You generate them deliberately to stress-test retrieval and memory before a user does it for you.

When synthetic data is worth the effort

It depends on what you're testing. If your golden set already has 200+ real queries covering your main topics, synthetic data adds the most value on the edges: queries designed to confuse the retriever, queries with missing context, queries that look relevant but aren't. If you're still under 50 real entries, mine production traces first. Synthetic data can't replace real user behavior.

The main catch is cost. Generating good synthetic entries takes time and review. Bad synthetic entries teach your eval set to expect failures that never happen in production.

Prompt templates for generating edge cases

Start with a template that forces variety. Here's one that works:

"Generate 10 queries that would stress-test a RAG system for [domain]. Include: 3 adversarial queries that try to extract information outside the knowledge base, 3 ambiguous queries with multiple valid interpretations, 2 out-of-domain queries that sound related but aren't, and 2 queries with typos or informal phrasing."

For memory, adapt it: "Generate 5 multi-turn conversations where the user mentions a preference in passing, then references it obliquely 3 turns later. Include one conversation where the user contradicts an earlier statement."

Don't accept the first output. Generate 30, keep 10.

Validating synthetic entries before adding to the golden set

Every synthetic entry needs a human review before it enters the golden set. Check three things: the query is realistic, the expected answer is actually correct, and the entry tests something your real data doesn't already cover. If you can't verify the expected answer yourself, don't add it. A golden set with wrong ground truth is worse than no golden set.

Evaluating Your Golden Eval Set Quality

A golden set is only as good as its labels. You can have 500 entries, but if 40 of them carry wrong ground truth, every eval run reports noise. LLM-as-judge gives you a way to audit that at scale without hand-checking every entry.

Using LLM-as-judge to audit label quality

Run a judge model over your golden set and ask it to score each entry on two axes: label consistency and answer verifiability. The prompt matters. Ask "Does the expected answer follow from the query and the provided context?" not "Is this answer good?" The first question catches mislabeled entries. The second just tells you the judge's taste.

You'll find three failure patterns: entries where the expected answer contradicts the context, entries where two similar queries have conflicting labels, and entries where the label is technically right but unhelpfully vague. Fix the first two. Flag the third for human review.

Identifying ambiguous or outdated entries

Ambiguity shows up when the judge assigns middling confidence scores across multiple runs. If the same entry scores 0.8 one run and 0.4 the next, the ground truth isn't clear. Pull it or rewrite the query.

Outdated entries are harder to spot with a judge alone. You need a freshness check: does the expected answer still match what your knowledge base currently contains? If your docs changed last month, entries referencing old API names or deprecated features are now wrong. A judge can flag these if you include the current context in the audit prompt.

Active learning for prioritizing labeling

Active learning flips the order. Instead of labeling everything, you label the entries where your model is least confident. Run your current system against unlabeled candidates, find the ones with the lowest confidence or highest disagreement between retrievers, and label those first. You get the most signal per hour of labeling effort.

The honest answer is that active learning adds real value once you're past a few hundred entries. Before that, just label everything. The overhead isn't worth it.

Maintaining a Golden Eval Set

A golden eval set is a living artifact. If you build it once and never touch it again, it starts lying to you within a quarter. Your docs change, your users ask different questions, and your retrieval pipeline shifts. The eval set has to shift with it.

Deduplication and versioning

Deduplicate first. Production traces will hand you the same query in twenty phrasings, and if all twenty land in your golden set, you've overweighted one failure mode. Cluster by embedding similarity, keep the most representative phrasing, and drop the rest. A 0.85 cosine threshold catches most near-duplicates without merging genuinely different questions.

Version every change. Tag each entry with a version number and a reason: "retired, docs updated June 2025" or "added, new product category." Without versioning, you can't reproduce an old eval run, which means you can't tell whether a score drop came from your system or from your eval set.

Retiring stale or outdated entries

Entries go stale when the ground truth no longer matches reality. If your expected answer references a feature you deprecated, the entry now tests the wrong thing. Review your golden set on a schedule tied to your knowledge base update cadence. If docs ship weekly, review weekly. If quarterly, review quarterly.

Retire, don't rewrite. When an entry is stale, pull it and add a fresh one from current production traces. Rewriting old entries in place destroys the historical record and makes it impossible to compare eval runs across time.

Detecting dataset drift

Drift happens when your production queries stop looking like your golden set. Track the embedding distance between incoming production queries and your golden entries. When the median distance climbs for two weeks straight, your eval set is falling behind real usage.

The fix is to sample from the drifted queries, label them, and add them. Drift detection is the cheapest signal you have that your golden set is going stale before your eval scores tell you the same thing, usually weeks later.

What Golden Eval Sets Cannot Do

A golden eval set measures what you've already seen. It cannot tell you whether your system will handle what comes next. That's not a flaw in your process. It's a boundary you need to know before you trust the numbers.

Overfitting to your golden set

The main catch is this: every time you tune a prompt or a retrieval threshold to raise your golden score, you're fitting to the test. Do that enough and your eval set stops measuring general quality. It measures how well you've memorized the answers.

The honest answer is that some overfitting is unavoidable. You will optimize against your golden set because that's what it's for. The risk comes when you stop adding new entries and keep tuning anyway. Your score climbs while production quality stays flat.

Blind spots for novel failure modes

Golden eval sets are backward-looking by design. They capture failures you've already observed. A new failure mode, a query pattern you've never logged, a memory edge case that hasn't surfaced yet, won't appear in your eval set until it happens in production and someone labels it.

That means a perfect golden score is not a guarantee. It's a floor, not a ceiling. You still need production monitoring to catch what your eval set can't see.

Maintenance cost is real

Building the set is the cheap part. Keeping it accurate takes ongoing work: reviewing stale entries, labeling new traces, deduplicating, versioning. Budget for that before you start. A golden eval set you can't maintain will mislead you more than having no eval set at all.

Final Thoughts on Building a Golden Eval Set

Building a golden eval set is not a one-time task. It's a discipline. You source real traces, structure entries for retrieval and memory, add synthetic edge cases, audit labels, and keep the whole thing versioned. The payoff is direct: when you change a chunking strategy or a memory prompt, you know within minutes whether you made things better or worse.

The good news is you don't need a perfect set to start. Fifty well-labeled entries from real failures will tell you more than five hundred synthetic ones. Start small, run evals on every change, and add entries as you go.

Platforms like GigaRAG help here by managing eval sets and running experiments in one place, but the process works with any tooling. What matters is that building a golden eval set becomes part of how you ship, not a project you finish and shelve.

Frequently Asked Questions

What is a golden eval set?

A golden eval set is a curated collection of input queries paired with expected outputs (and often expected retrieved contexts) used to evaluate and benchmark a RAG or agent memory system. It serves as a reference standard for measuring retrieval quality, answer accuracy, and memory persistence.

How many examples should a golden eval set have?

There is no fixed number, but a practical starting point is 50–100 diverse examples that cover common cases, edge cases, and memory-related scenarios. You can expand it over time as you discover new failure modes.

How do I create a golden dataset for RAG?

Start by collecting real user queries, then manually annotate the ideal retrieved context and expected answer for each. Include queries that test retrieval precision, context relevance, and answer groundedness. Store the dataset in a structured format and version it alongside your code.

Can a golden eval set test agent memory?

Yes, but it requires designing multi-turn or multi-session scenarios where the agent must remember and use information from earlier interactions. This is more complex than single-turn evaluation but essential for memory persistence.

What are the limitations of golden eval sets?

Golden eval sets are static and cannot cover every possible user interaction or adapt to new edge cases automatically. They also may not reflect real-world distribution shifts. They are best used alongside live monitoring and user feedback.

How often should I update my golden eval set?

Review and update it regularly, such as after major system changes, when new failure modes are discovered, or on a quarterly basis. Keeping it current ensures it remains a relevant benchmark.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through building a golden eval set or something adjacent, we publish what we have actually tested, including where it falls short.