A RAG Platform Evaluation Checklist for Agent Builders

A RAG Platform Evaluation Checklist for Agent Memory and Pipeline Builders

A RAG platform evaluation checklist is what most pipeline builders reach for after the third vendor demo, and what they almost never find. The search results give you two things: a vendor's pitch dressed up as a comparison, or a wall of metrics with no pass/fail line anywhere. Neither tells you whether a platform will hold up when your agent has to remember something across thirty turns and three sessions. That's the gap this checklist fills. GigaRAG, a platform built for agent memory and RAG pipelines, gets scored against the same items as everything else. No special treatment. The checklist below is vendor-neutral and itemized, with thresholds you can apply in an afternoon. It won't tell you which platform will win in production, and the last section says exactly why. What it will give you is a scoring tool you can run against any platform, including the one you're already using.

In This Guide

• What Is RAG Evaluation?
• Vendor Tool Comparison vs. Neutral Checklist
• Evaluating Retrieval Quality
• A Rag Platform Evaluation Checklist: A Step-by-Step Guide
• Evaluating Generation Quality
• A RAG Platform Evaluation Checklist: Core Metrics
• Evaluating Agent Memory Capabilities
• A Testing Framework for RAG Platforms
• RAG Evaluation Best Practices
• What This Checklist Cannot Do
• Choosing a RAG Platform with This Checklist

What Is RAG Evaluation?

RAG evaluation measures how well a retrieval-augmented generation system performs its two jobs: finding the right context, and using that context to produce a correct answer. You score retrieval and generation separately because they fail in different ways. A system can pull perfect chunks and still write a wrong answer. It can also retrieve garbage and somehow stumble into a decent response. Treating them as one score hides which half is broken.

Why evaluation matters for agent memory

Agent memory makes this harder. A standard RAG pipeline answers one question at a time. An agent carries context across turns, stores facts, and recalls them later. When retrieval fails in an agent, the error compounds. The agent doesn't just answer one question wrong. It writes bad data into memory, then uses that bad data to make the next decision. Evaluation catches this before it cascades.

The two halves: retrieval and generation

Retrieval evaluation asks: did the system find the right chunks? You measure context precision, context recall, and hit rate. Generation evaluation asks: did the system use those chunks correctly? You measure faithfulness, answer relevancy, and hallucination rate. Both halves need their own checklist items because fixing one doesn't fix the other. A platform can ace retrieval and still generate unfaithful answers. You need scores for both before you can trust a platform with agent memory workloads.

[!note] No single checklist can guarantee production success; always validate with a proof-of-concept using your specific workloads and data.

Vendor Tool Comparison vs. Neutral Checklist

Evaluating Retrieval Quality

Retrieval quality is the first gate. If the right chunks never reach the model, generation quality doesn't matter. You can't write a faithful answer from missing context. For agent memory systems, retrieval failures are worse than wrong answers: they poison the memory store, and every later turn inherits the error.

Context precision and context recall

Context precision measures how much of the retrieved context is actually relevant. A high score means the system didn't stuff the prompt with filler chunks. Context recall measures the opposite: did the system find all the relevant chunks that exist in your knowledge base? You need both.

A system with high precision but low recall gives short, confident answers that miss critical facts. High recall but low precision burns context window on noise and drives up cost. For agent memory, precision matters more than in single-turn RAG because every irrelevant chunk competes with stored memory for space in the context window.

Pass thresholds: context precision above 0.7, context recall above 0.8. Below that, your retrieval is either missing things or burying them.

Hit rate and MRR

Hit rate is the simplest retrieval metric: what fraction of queries returned at least one relevant chunk in the top-k results? It's blunt but useful as a smoke test. If hit rate is below 0.9, you have a retrieval problem before you even look at precision or recall.

MRR (mean reciprocal rank) tells you where the first relevant chunk lands. A score of 1.0 means the right chunk was always first. A score of 0.5 means it was usually second. Lower than that and your ranking is effectively random.

MRR matters for agent memory because agents often truncate context. If the right chunk sits at position 8, it may get cut before the model sees it. Track MRR alongside hit rate: hit rate tells you if the right chunk was found, MRR tells you if it was found in time.

Checklist items for retrieval

• Context precision above 0.7 on your golden dataset
• Context recall above 0.8 on the same dataset
• Hit rate above 0.9 at top-5
• MRR above 0.6 at top-10
• Retrieval latency under 200ms at p95
• Retrieval quality stable across multi-turn conversations, not just single queries

The last item is the one generic RAG checklists miss. Run your retrieval evaluation against multi-turn traces where the agent has already stored facts. Single-turn scores don't tell you how retrieval behaves when memory is in play.

[!tip] Involve both engineering and product teams early to ensure the checklist covers technical and business requirements; this prevents costly rework later.

A Rag Platform Evaluation Checklist: A Step-by-Step Guide

1. Define your agent memory requirements: persistence, context management, multi-agent retrieval.
2. List pipeline needs: data ingestion, indexing, retrieval, generation, monitoring.
3. For each requirement, set pass/fail criteria (e.g., supports vector search with <100ms latency).
4. Score each platform against criteria using a consistent scale (e.g., 0-2 points per item).
5. Test top candidates with a proof-of-concept using your own data and queries.
6. Document limitations: what the checklist cannot assess (e.g., long-term scalability).
7. Review scores and limitations to make an informed decision.

Evaluating Generation Quality

Generation quality is the second gate. Retrieval gets the right chunks into the prompt. Generation decides whether the model uses them correctly, ignores them, or invents something new. For agent memory systems, generation failures compound. A hallucinated fact doesn't just produce one wrong answer. It can get written back into memory, retrieved later, and treated as ground truth by every downstream decision.

Faithfulness and hallucination rate

Faithfulness measures whether the generated answer is supported by the retrieved context. It's the most direct guard against hallucination. A faithful answer sticks to what the chunks actually say. An unfaithful one adds claims the context never made.

Hallucination rate is the inverse: what fraction of generated statements have no grounding in the retrieved context? You measure this with LLM-as-judge, where a separate model checks each claim in the answer against the source chunks. It's not perfect, but it catches the obvious fabrications.

For agent memory, faithfulness is the metric that matters most. An agent that recalls a fact from memory and then embellishes it is worse than an agent that says it doesn't know. Set your pass threshold at 0.9 faithfulness. Below that, you're shipping an agent that confidently makes things up.

Answer relevancy

Answer relevancy measures whether the response actually answers the question asked. A faithful answer can still be irrelevant. The model might correctly restate context that doesn't address the query.

This matters for agents because they often chain multiple steps. An irrelevant answer at step one becomes bad input for step two. The error propagates. Relevancy scores below 0.7 usually mean the model is answering a different question than the one asked, or the retrieval returned context about the wrong topic entirely.

Keep in mind: relevancy and faithfulness can trade off. A model that plays it safe and stays close to context may score high on faithfulness but low on relevancy if the context itself was off-target. Track both together.

Checklist items for generation

• Faithfulness above 0.9 on your golden dataset
• Hallucination rate below 5% across all generated answers
• Answer relevancy above 0.7
• No fabricated citations or sources in generated output
• Generation quality stable when memory is injected into the prompt
• Hallucination rate doesn't spike on multi-turn conversations

The last two items are where agent memory systems diverge from standard RAG. Test generation with and without stored memory in the prompt. Some models handle injected memory well. Others treat it as noise and start improvising. You won't know until you test it.

A RAG Platform Evaluation Checklist: Core Metrics

A RAG platform evaluation checklist scores three things: retrieval quality, generation quality, and operational cost. Score each item pass or fail against a threshold. A platform that fails retrieval metrics isn't worth evaluating further.

Retrieval metrics checklist

Retrieval is the foundation. If the platform can't pull the right chunks, nothing downstream matters. Score these first.

• Context precision above 0.7: the retrieved chunks are mostly relevant to the query
• Context recall above 0.8: the relevant chunks that exist in your store are actually retrieved
• Hit rate above 0.9: the correct chunk appears in the top-k results at least 90% of the time
• MRR above 0.6: the correct chunk ranks near the top, not buried at position 8
• Hybrid search supported: lexical and semantic retrieval combined, not one or the other
• Reranking available: a second-pass model reorders chunks before generation

The main catch: these numbers only mean something on your data. A vendor's published benchmark scores tell you nothing about how the platform handles your document types, your chunk sizes, or your query patterns. Run the numbers yourself.

Generation metrics checklist

Generation quality determines whether retrieval actually pays off. A platform can retrieve perfectly and still produce garbage if the generation side is weak.

• Faithfulness above 0.9: generated answers stay grounded in retrieved context
• Hallucination rate below 5%: fewer than 1 in 20 claims are unsupported
• Answer relevancy above 0.7: responses address the actual question
• Citation accuracy verified: generated sources point to real chunks that support the claim
• Multi-turn stability: quality doesn't degrade across a conversation with memory injected

For agent memory workloads, test generation with stored memory in the prompt. Some models treat injected memory as noise and start improvising. That's a hard fail.

Operational metrics checklist

Quality metrics get the attention. Operational metrics decide whether the platform is viable in production.

• P95 latency under 2 seconds for end-to-end query to answer
• Throughput handles your peak concurrent query load without queueing
• Cost per query tracked and predictable, not a surprise at month-end
• Observability built in: traces show which chunks were retrieved and why
• Evaluation runs can be automated, not manual one-offs

Latency matters more for agents than for chatbots. An agent making five retrieval calls per task multiplies latency. A platform that's fine at 1.5 seconds per query becomes unusable when one agent turn requires four queries.

Evaluating Agent Memory Capabilities

Generic RAG evaluation treats every query as a fresh start. Agent memory breaks that assumption. An agent carries state across turns, across sessions, and sometimes across agents. You need to test what happens when the platform has to remember.

Memory persistence and session continuity

The core question: does the platform actually persist memory, or does it fake it by re-injecting the full conversation transcript every turn? Those are different things. Re-injection works for short sessions and falls apart as context grows.

Test this directly. Run a session with 20 turns. At turn 21, ask the agent to recall a fact from turn 3. If it can't, memory isn't persisting. Then start a new session and ask the same question. If the answer is gone, you have session-scoped memory only, which may be fine for your use case or a hard fail depending on what you're building.

Checklist items:

• Memory survives across turns within a session
• Memory survives across sessions where your design requires it
• Stored memory is retrievable by explicit query, not just implicit context
• Memory updates don't require a full re-index of the vector store

Context management and windowing

Agent memory lives inside a context window. The platform's windowing strategy determines what gets dropped when the window fills. You need to know what gets evicted and whether the agent knows it was evicted.

Test with a deliberately long session. Push past the context limit. Then ask about something from early in the conversation. A good platform either summarizes and retains the key facts, or tells you it no longer has that information. A bad platform silently drops it and the agent hallucinates an answer.

• Windowing strategy is documented, not a black box
• Evicted content is summarized or flagged, not silently lost
• The agent can report what it remembers and what it doesn't

Multi-agent retrieval scenarios

If your system has multiple agents sharing a memory store, test what happens when they retrieve concurrently. Two agents writing to the same memory can overwrite each other. Two agents reading can get inconsistent snapshots.

Run a test where agent A writes a fact, agent B reads it immediately, then agent A updates it. Does agent B see the update? Does the platform handle write conflicts, or does the last write silently win?

• Concurrent reads return consistent snapshots
• Write conflicts are detected or resolved, not silently overwritten
• Memory access controls exist if agents have different permission levels

Checklist items for agent memory

The full list, consolidated:

• Memory persists across turns and sessions as your design requires
• Windowing behavior is documented and testable
• Evicted content is summarized or flagged
• The agent can report its own memory state
• Multi-agent concurrent access is handled without silent data loss
• Memory updates propagate without full re-indexing
• Forgetting mechanisms exist and work when you need them

The honest answer: most platforms treat memory as an afterthought bolted onto retrieval. Test it before you trust it.

A Testing Framework for RAG Platforms

A checklist tells you what to measure. A testing framework tells you how to measure it repeatedly, so you can catch regressions before they ship. Here's the process I use.

Building a golden dataset

You need a set of questions with known-good answers before you can evaluate anything. Start with 50 to 100 real queries from your actual use case. Don't generate them from a template. Pull them from logs, support tickets, or conversations with users.

For each query, write the correct answer and the source documents that should support it. This is tedious. It's also the only part of the process that gives you ground truth. Without it, every metric you compute is a number with no meaning.

Keep the dataset versioned. When you change chunking, embeddings, or prompts, you re-run against the same dataset. That's how you know if a change helped or hurt.

Baseline, iterate, and regression test

Run your golden dataset through the platform once. Record every metric: context precision, faithfulness, latency, cost per query. That's your baseline.

Now change one thing. A new embedding model. A different chunk size. A reranker. Re-run the dataset. Compare against baseline. If the change improves retrieval but hurts faithfulness, you know the tradeoff before you ship it.

Regression testing is the same loop, but triggered by code changes. Every time you touch the pipeline, the golden dataset runs automatically. A metric that drops below your threshold fails the build.

CI/CD integration for RAG evaluation

Evaluation belongs in your pipeline, not in a notebook you run once a quarter. Wire the golden dataset into CI/CD so every pull request that touches retrieval or generation triggers an evaluation run.

The setup: store the dataset in your repo or a versioned bucket. Add an evaluation step to your CI config. Set thresholds for the metrics you care about. Fail the build when a threshold breaks.

The catch: full evaluation runs cost money and take time. Run a fast subset on every commit, and the full dataset nightly or on release candidates. That keeps feedback quick without burning your LLM budget on every typo fix.

Don't treat CI evaluation as a gate you can ignore. If the build fails and you merge anyway, the framework is theater.

RAG Evaluation Best Practices

Metrics without process produce numbers you can't trust. These practices keep evaluation honest.

LLM-as-judge with human oversight

LLM-as-judge scales evaluation beyond what you can label by hand. You send the LLM a query, the retrieved context, and the generated answer, then ask it to score faithfulness or relevancy on a rubric. It's fast and cheap.

It's also wrong often enough to matter. Judges drift toward leniency on long answers and miss domain-specific errors a human would catch immediately. The fix: spot-check 10 to 20 percent of judge outputs against human labels. When agreement drops below your threshold, recalibrate the rubric or switch judges.

Evaluate in production, not just offline

Offline evaluation uses your golden dataset. Production evaluation uses real traffic. The two diverge more than you'd expect.

Users ask questions your dataset never anticipated. They phrase things oddly. They hit edge cases in your chunking. Offline scores can stay flat while production quality degrades. Sample live queries weekly, score them with the same metrics, and compare against your baseline. A gap between offline and production scores tells you the dataset is stale.

Track cost and latency alongside quality

A platform that returns perfect answers in nine seconds at $0.40 per query fails most production budgets. Quality metrics alone hide that.

Track three numbers for every evaluation run: quality score, latency in milliseconds, and cost per query. Set thresholds for all three. A change that improves faithfulness by 3 percent but doubles latency isn't a win. It's a tradeoff you need to see explicitly.

Don't optimize a single metric in isolation. Retrieval precision can climb while generation faithfulness drops. Cost can fall while hallucination rate rises. The evaluation only works when you read the metrics together.

What This Checklist Cannot Do

The checklist gives you a scoring framework. It does not give you certainty. Here's what it can't do.

Evaluation cannot guarantee production performance

Your golden dataset is a snapshot. Production traffic is a moving target. Users will ask questions your test set never anticipated, hit edge cases in your chunking, and phrase things in ways your evaluation queries didn't cover. A platform that scores 95 percent on your checklist can still fail on live traffic.

The honest answer is that evaluation reduces risk. It doesn't eliminate it. You'll still need production monitoring, sampled live queries, and a rollback plan. The checklist tells you which platform is most likely to work. It can't tell you which one will.

Evaluation cannot replace human judgment

Metrics catch patterns. They miss domain-specific errors that only a subject matter expert would notice. A faithfulness score of 0.9 looks fine until a clinician reviews the output and finds the answer omitted a contraindication. LLM judges have the same blind spot. They score structure and relevance, not truth in your domain.

For anything where wrong answers carry real cost, you need human review. Budget for it. The checklist narrows the field. It doesn't close the loop.

Evaluation cannot predict long-term memory degradation

Agent memory systems degrade in ways offline evaluation can't capture. Context windows fill up. Old memories get overwritten or forgotten. Retrieval quality drifts as the memory store grows. A platform that handles 50 turns cleanly may fall apart at 500.

Most evaluation datasets test short sessions. They don't simulate memory pressure over days or weeks. If your agents run long-lived sessions, you'll need to build your own stress tests. Run 1,000-turn simulations. Check whether the agent still recalls facts from turn 12. The checklist can flag platforms with memory features. It can't tell you how those features hold up under sustained load.

Choosing a RAG Platform with This Checklist

A checklist only helps if you weight it. Not every criterion matters equally for agent memory workloads. Retrieval latency might be critical for a customer-facing agent that answers in real time. Memory persistence matters more for a long-running research assistant. Start by marking each checklist item as must-have, nice-to-have, or irrelevant for your specific use case.

Weighting criteria for agent memory workloads

Three criteria matter more for agent memory than for standard RAG. First, memory persistence: can the platform retain context across sessions without manual re-injection? Second, context management: does it handle windowing and summarization automatically, or do you build that yourself? Third, multi-agent retrieval: can separate agents query shared memory without stepping on each other?

Weight these three at 30 to 40 percent of your total score. Retrieval and generation quality still matter, but a platform that scores high on standard metrics and fails on memory persistence won't work for agent workloads.

Running a pilot evaluation

Don't commit based on a checklist alone. Pick your top two platforms and run a two-week pilot. Use your own golden dataset, not the vendor's. Track the metrics from this checklist, plus cost per query and latency under load.

The pilot tells you what the checklist can't: how the platform behaves on your data, with your traffic patterns, under your memory pressure. A platform that passes the checklist but fails the pilot isn't the right choice. The checklist narrows the field. The pilot makes the call. A RAG platform evaluation checklist is only as good as the data you run through it.

Frequently Asked Questions

What is a RAG platform evaluation checklist?

It is a structured set of criteria to assess RAG platforms for specific use cases, such as agent memory and pipeline building. It helps compare platforms objectively, covering features like retrieval accuracy, latency, and memory persistence. The checklist includes pass/fail or scoring to guide decisions.

How do I evaluate agent memory in a RAG platform?

Focus on persistence (does memory survive restarts?), context management (can it handle long conversations?), and multi-agent retrieval (can multiple agents share memory?). Test with scenarios that require remembering past interactions. Assign pass/fail based on your requirements.

What are common RAG evaluation metrics?

Common metrics include retrieval precision/recall, answer relevance, faithfulness, and latency. For agent memory, add memory retention rate and context switching accuracy. Use these to score platforms, but remember metrics alone don't capture all aspects.

Can I use open-source tools for RAG evaluation?

Yes, tools like Ragas and DeepEval offer open-source RAG evaluation frameworks. They provide metrics and testing utilities. However, they may not cover agent memory specifics, so supplement with custom tests.

What are the limitations of RAG evaluation checklists?

Checklists cannot predict real-world performance under all conditions, especially for novel queries or scaling issues. They also may not capture integration challenges or cost overruns. Always complement with a proof-of-concept.

How often should I re-evaluate a RAG platform?

Re-evaluate when your requirements change significantly, such as new agent memory needs or increased data volume. Also, after major platform updates. A periodic review (e.g., annually) helps ensure continued alignment.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through a rag platform evaluation checklist or something adjacent, we publish what we have actually tested, including where it falls short.