Retrieval Incident Runbook Template for Production RAG Systems
When an agent that worked yesterday starts pulling thin context, stalling on search, or giving different answers to the same question, you don't yet know that retrieval is at fault. A retrieval incident runbook template provides a structure for documenting evidence, assigning ownership, and guiding recovery work.
A generic outage checklist won't tell you if the fault sits in ingestion, chunking, metadata filters, embeddings, vector search, reranking, or the model's use of sound context. You need telemetry, saved test queries, and controlled changes before you can diagnose relevance with confidence.
Before you begin
A wrong answer is not automatically a retrieval incident. First determine whether retrieved context is missing, irrelevant, outdated, or unavailable, and keep generation issues in scope until evidence rules them out.
- Identify the affected service and its owner.
- Confirm how to inspect queries, retrieved chunks, and index freshness.
- Check which dashboards or logs show retrieval latency and errors.
- Have a known-good query or evaluation set ready for recovery checks.
What is a retrieval incident runbook template?
A runbook gives you a bounded way to investigate when the retrieval path is broken, degraded, or returning context that doesn't support the task. Unlike a general outage checklist, it records evidence from the RAG pipeline before anyone blames the model.
A usable runbook covers detection, triage, mitigation, recovery checks, and communication. Skills Directory’s Incident Runbook Templates describes templates covering detection, triage, mitigation, resolution, and communication.
Retrieval stages to document
Write down each active stage and its versioned dependencies. Depending on the system, a query can be rewritten, converted to an embedding, matched with vectorized chunks, and filtered or reranked before generation uses retrieved material.
Weak queries, stale indexes, and fragmented chunks can fail at different points. "Bad answer" isn't enough evidence.
Minimum ownership and dependency map
Name the on-call owner for the app, ingestion job, vector store, embedding service, reranker, and source knowledge base. Add dashboards, alert links, deploy history, and each fallback path.
If GigaRAG is part of your agent setup, list its central Hive Mind and relevant access controls as dependencies. Keep the map current.
What counts as an incident versus an evaluation or content-quality task
Treat an observed retrieval regression, empty results, dependency failure, or sharp latency change as an incident when users are affected. A vague relevance concern without saved queries or a known baseline is usually an evaluation task.
Stale or inaccurate source content should usually be routed to its content owner. Keep or open an incident when user impact and your team’s incident policy call for active coordination while that content issue is contained.
Retrieval incident or generation issue?
| Factor | Retrieval issue | Generation issue |
|---|---|---|
| Retrieved context | Missing, outdated, incomplete, or irrelevant | May be relevant to the query |
| Evidence to inspect | Query, retrieved chunks, index freshness, and retrieval errors | Response compared with the retrieved context |
| Useful evaluation signal | Context relevance and retrieval effectiveness | Groundedness and answer relevance |
| Possible next investigation | Source files, query phrasing, ranking, or vector-store health | Whether the response aligns with the retrieved context |
How a retrieval incident runbook moves from alert to recovery
[!note] Use the runbook to document evidence, ownership, mitigation, recovery checks, and follow-up work. Source-content corrections and broader quality evaluation may continue after the active incident.
Detection signals: retrieval success, empty-result rate, relevance, grounding, errors, and latency
Alert on changes you can inspect: empty-result rate, retrieval errors, latency, and success rate by dependency. Keep saved queries and their expected source chunks.
Zilliz’s RAG evaluation article describes context relevance, groundedness, and answer relevance as useful signals for separating retrieval quality from response quality.
Severity levels based on user impact, blast radius, and available fallback behavior
Assign severity from user harm, not the alert alone. A vector-store outage affecting all production requests with no fallback is high severity, while poor results for a narrow metadata filter may not be.
Record affected tenants, traffic share, task criticality, and whether cached context or a degraded search path still works.
The first 15 minutes: freeze changes, compare recent deploys, preserve evidence, and assign an incident lead
Freeze nonessential deploys and index jobs. Assign one incident lead, then capture timestamps, request IDs, traces, failed queries, returned chunk IDs, filter values, model and index versions, and recent deploys before logs age out.
Check the embedding service, vector store, reranker, and source pipeline separately. Don't change three things at once. You won't know what recovered.
Recovery validation before declaring the incident resolved
Run saved queries through the restored path and compare retrieved context, errors, and latency with the pre-incident baseline. Confirm that the fallback no longer carries production traffic, watch the relevant signals after the fix, and record the rollback point.
Don't tell users a service is fixed before evidence supports it.
Step-by-step: triage a retrieval incident
- Record affected queries, users or workflows, observed symptoms, and the incident owner.
- Check service and vector-store availability, error signals, and retrieval latency.
- Inspect retrieved chunks for presence, relevance, completeness, and freshness.
- Compare the result with a known-good query or evaluation example, where available.
- Apply a reversible mitigation, such as reverting a suspect change or routing around an unhealthy dependency.
- Verify recovery through retrieval behavior and service health. Roll back the mitigation if it worsens the issue.
- Document the cause, evidence, actions, verification, and follow-up work.
Troubleshooting the main production RAG retrieval failures
[!tip] Capture the exact query alongside the retrieved chunks and final response during triage. Context relevance compares retrieved material with the query, while groundedness compares the response with that material.
Missing context: verify ingestion freshness, chunk coverage, filters, and top-k results
Start with the exact document and fact the answer missed. Check when it was ingested, whether its chunks exist in the index, which metadata filters applied, and whether it appears just below your top-k cutoff.
Don't guess from the answer. Compare returned chunk IDs with expected chunks, then widen top-k or remove a suspected filter only as a controlled test.
Irrelevant search results: inspect query intent, embeddings, metadata, chunk boundaries, and reranker behavior
Save the user query, any rewrite, its embedding model version, retrieved scores, filters, and reranked order. A vague query or query drift can send similarity search toward the wrong topic, while fragmented chunks can leave the model with weak context.
Use context relevance and groundedness as separate signals: the first evaluates the retrieved material against the query, and the second evaluates the response against that material.
Retrieval latency: separate embedding, vector search, reranking, dependency, and end-to-end latency
Trace embedding, retrieval, reranking, network, and generation stages separately so the incident record can identify where end-to-end latency is accumulating.
Compare percentiles by dependency and traffic segment. Disable an optional reranker or use a known fallback only if saved queries still return acceptable context.
Vector database failures: check availability, index state, connectivity, capacity, and fallback behavior
Check service health, authentication, network errors, index status, and capacity signals before rebuilding anything. Treat index state, capacity, data quality, and consistency as separate areas to examine, and record the evidence for each.
Use a documented fallback. Don't silently serve empty context as a normal answer.
Inconsistent AI responses: compare retrieved contexts before blaming the model
Replay the same request and compare query rewrites, retrieved chunks, scores, context order, and generation settings. Use any differences to narrow the next investigation instead of attributing response variation from the answer alone.
What a retrieval runbook cannot fix on its own
A runbook is for restoring and verifying the retrieval path. Questions about source ownership, content governance, and longer-term evaluation may require follow-up work outside the active response.
Correctness needs task-specific evaluation
Groundedness assesses whether a response aligns with retrieved context. To evaluate task-specific correctness, define the task, expected answer or source evidence, and an acceptance rule.
Use saved queries and a baseline to identify changed retrieval behavior during an incident. Questions that need broader quality measurement belong in a defined evaluation rather than an open-ended incident discussion.
When source content needs separate follow-up
Use a rollback as a reversible mitigation only after comparing the suspected change with the prior version and checking representative queries. If the remaining problem is stale, missing, or conflicting source content, route it to the owners responsible for maintaining that content.
Open a data-governance task when the fix needs source owners, review dates, retention rules, or a content rewrite.
Relevance cannot be diagnosed reliably without representative queries and baselines
One bad prompt is evidence, not a baseline. Save representative queries with expected documents or answers, then compare retrieval results across versions and traffic segments.
A stable query set and reproducible metrics provide stronger evidence of whether a change improves retrieval beyond a single example.
When to convert the incident into an evaluation, data-governance, or architecture work item
End the active incident when service is stable and immediate user harm is contained. Create follow-up work when the remaining question needs repeated measurement, source cleanup, prompt redesign, or a change to chunking, retrieval, or model architecture.
Don't keep an incident open to chase non-determinism. Keep failing queries, returned contexts, settings, and the decision record so longer-term work has evidence to act on.
Copyable retrieval incident runbook template
Use this template to assign an owner, bound user harm, preserve retrieval evidence, and plan a reversible fix. Replace bracketed fields with your vector store, tracing, agent, alerting, and ticket tools. It covers the information needed to restore and verify the retrieval path.
Runbook header: service, owner, dependencies, dashboards, alert links, and escalation contacts
Ticket: [link]. Service: [name]. Lead/owner: [name]. Dependencies: [list].
Links: [dashboards], [alerts], [traces], [status], [contacts]. Add GigaRAG if agents use its MCP or API connection.
Detection and severity checklist
Record start time, tenants, query samples, errors, empty results, and stage latency. Set SEV1 for broad failure with no safe fallback, SEV2 for major degradation, and SEV3 for limited impact.
Triage checklist and symptom-to-procedure routing
Freeze changes. Compare deploys and index jobs.
For missing context, inspect ingestion, filters, and top-k. For irrelevant results, inspect the query, embeddings, chunks, and reranker. For latency or errors, inspect each dependency.
Containment and recovery actions with verification and rollback criteria
Use the least risky fallback: a known-good index, cached results, or reduced feature. Verify that representative queries, errors, and latency recover.
Roll back only when the prior version passes the same checks.
Root-cause analysis prompts
Ask: What changed? Which stage first diverged? Did the source, index, filter, model, or traffic differ? What evidence rules out generation as the main fault?
Post-incident record: timeline, user impact, evidence, corrective actions, and test cases to retain
Save the timeline, impact, owner decisions, traces, contexts, settings, and rollback result. Create corrective actions with owners and dates. Retain failing queries plus expected documents as regression tests.
Turn each retrieval incident into a stronger operating system
A recovery is not the finish line. Retain the evidence from the incident so representative queries can be tested before later index, prompt, or source-content changes.
Convert incident evidence into regression tests and ownership decisions
Review incidents on a set cadence. Keep failing queries, retrieved contexts, expected documents, filters, and settings, then run them before changes ship.
Assign an owner for each source and index. GigaRAG can keep project knowledge available across sessions and tools, but it doesn't replace observability, evaluation, or accountable owners.
Rehearse the runbook. A quiet incident is the time to check contacts, fallbacks, dashboards, and rollback access.
Frequently asked questions
What should be included in a retrieval incident runbook?
Include service ownership, impact assessment, detection signals, triage checks, mitigation and recovery actions, rollback criteria, escalation and communication details, and a record of evidence and follow-up work.
How do you tell whether a bad RAG answer is a retrieval incident?
Inspect retrieved context. Missing, outdated, incomplete, or irrelevant chunks point toward a retrieval or source-data problem.
Compare the response with relevant retrieved context to assess whether generation is also involved.
What metrics should alert on RAG retrieval failures?
Track retrieval errors and latency alongside signals that assess context relevance and retrieval effectiveness. For end-to-end assessment, compare responses with retrieved context using groundedness and answer relevance.
How should teams assign severity to a retrieval incident?
Base severity on observed impact: how much of the service or its users are affected, whether a critical workflow is impaired, and whether the issue is limited or widespread. Record the criteria your team uses rather than relying on a label alone.
What commonly causes missing context in a RAG application?
Possible causes include an outdated index, poorly structured source documents, and weak or ambiguous query phrasing. Check the source material, index freshness, and query before attributing the failure to a specific component.
Why does a RAG system retrieve irrelevant search results?
Vague or ambiguous queries can lead to query drift, and source material may be irrelevant, outdated, or poorly structured. Inspect both the query and retrieved chunks to narrow down the cause.
How can teams investigate vector database latency or outages?
Check whether the vector database and its integrations are available, then review retrieval latency, errors, index status, capacity signals, and recent changes. Record the evidence for each suspected cause before changing the system.
How do you evaluate whether retrieved context supports an AI response?
Assess context relevance to the query and groundedness of the response against that context. Where suitable ground truth exists, compare retrieved chunks with expected relevant chunks to evaluate retrieval effectiveness.
About GigaRAG
GigaRAG is a shared memory platform for AI agents that helps developers and teams keep project knowledge available across sessions and tools.
You can connect agents to a central Hive Mind through MCP or the API. Access controls manage what each agent can use.
