RAG Failure Modes: A Pipeline-Stage Taxonomy

RAG Failure Modes: What Breaks, Why, and What You Can Actually Fix

Most rag failure modes get misdiagnosed as LLM problems. A pipeline builder sees a wrong answer and blames the model, when the real fault is a chunk that was never retrieved, an index that went stale, or a memory store that quietly overwrote yesterday's facts. If you build agent memory systems, you already know this pain. The good news is that these failures are diagnosable, once you stop looking at the model and start looking at the pipeline. This guide breaks rag failure modes into five stages: ingestion, indexing, retrieval, generation, and memory. For each stage you'll get the specific failure signature, how to confirm it, and what you can actually fix. The honest answer is that some things can't be fixed, and I'll say so plainly. For agent memory pipelines specifically, GigaRAG handles the memory-stage failures that generic RAG tooling ignores.

In This Guide

• Why RAG Systems Fail: The Pipeline-Stage View
• Retrieval Failure vs Generation Failure: How to Tell Them Apart
• Ingestion Failure Modes: Bad Data In, Bad Answers Out
• Rag Failure Modes: A Step-by-Step Guide
• Indexing Failure Modes: When Embeddings Lie
• Retrieval Failure Modes: The Most Common RAG Failure Modes
• Generation Failure Modes: Hallucination and Context Misuse
• Agent Memory Failure Modes: The Gap Nobody Covers
• A Diagnostic Checklist for RAG Pipeline Builders
• What You Cannot Fix: Honest Limits of RAG
• Final Thoughts: Building RAG Systems That Fail Less

Why RAG Systems Fail: The Pipeline-Stage View

RAG systems fail because retrieval returns the wrong context, not because the model generates bad text. Most failures trace to ingestion, indexing, or retrieval. The model only amplifies what those stages hand it.

The five stages: ingestion, indexing, retrieval, generation, memory

Every RAG pipeline has five stages. Ingestion pulls documents in and chunks them. Indexing turns chunks into embeddings and stores them. Retrieval finds relevant chunks for a query. Generation feeds those chunks to the LLM. Memory, the stage most guides skip, holds context across sessions for agent pipelines.

A failure at any stage poisons everything downstream. Bad chunks produce bad embeddings. Bad embeddings produce bad retrieval. Bad retrieval produces bad answers. The model gets blamed for all of it.

Why retrieval failures masquerade as model failures

When an LLM hallucinates, you see the symptom at the generation stage. But the cause is usually upstream. The model received irrelevant chunks, or too few, or none. It filled the gap with plausible-sounding text. That's not a model problem. It's a retrieval problem wearing a model costume.

How to use this framework for diagnosis

Start at the top. Check ingestion first: are your chunks clean and well-bounded? Then indexing: are embeddings current and domain-appropriate? Then retrieval: what's your precision and recall? Only after those three check out should you suspect generation. Memory comes last, but for agent pipelines it's often the silent killer.

[!note] RAG cannot repair incorrect, outdated, or contradictory source documents; if the underlying data is wrong, retrieval will faithfully surface wrong information. Fixing data quality is a prerequisite, not a RAG tuning problem.

Retrieval Failure vs Generation Failure: How to Tell Them Apart

Ingestion Failure Modes: Bad Data In, Bad Answers Out

Ingestion is where the pipeline meets reality. Documents arrive as PDFs with tables, scanned pages, broken HTML, or a mix of all three. If that mess isn't cleaned up before chunking, every downstream stage inherits the damage.

Unstructured or messy source documents

A PDF with a two-column layout gets parsed as one long line. A scanned page without OCR becomes an empty chunk. Headers and footers repeat in every chunk, diluting the signal. You can't retrieve what was never extracted cleanly.

Chunking strategy failures: too big, too small, wrong boundaries

Chunks over 1,000 tokens bury the answer in noise. Chunks under 100 tokens lose context. Boundaries that split a sentence or a code block produce embeddings that match nothing. The fix is testing: run your chunker on real documents, not toy examples.

Metadata loss during ingestion

If you don't capture source, date, and section headers at ingestion time, you can't filter later. Retrieval returns chunks from the wrong document or the wrong year. Metadata is the only handle you have for narrowing search.

What you cannot fix: bad source data

If the source documents are wrong, outdated, or contradictory, RAG will faithfully retrieve and repeat the errors. No chunking strategy or embedding model fixes a bad source. Clean data is a prerequisite, not a pipeline feature.

[!tip] For agent memory engineers: log every memory write and read with a timestamp and source turn, so context drift and memory overwrite become visible in traces instead of surfacing as vague answer-quality complaints.

Rag Failure Modes: A Step-by-Step Guide

1. Reproduce the failure with a fixed query and log the exact retrieved chunks.
2. Check ingestion: confirm the source document was parsed and chunked without truncation or encoding loss.
3. Check indexing: verify the chunk exists in the vector index and that embeddings are current.
4. Check retrieval: compare top-k results against a known-good chunk for the same query.
5. Check generation: insert the correct chunk manually into the prompt and see if the answer improves.
6. Check memory (for agents): inspect whether prior turns overwrote or drifted from the original context.
7. Record the stage where the failure first appears, then apply only that stage's fix and retest.

Indexing Failure Modes: When Embeddings Lie

Embeddings don't lie on purpose. They just compress text into vectors that lose information, and the loss shows up as retrieval failures you'll blame on the model. The index is where those failures get baked in.

Embedding model mismatch with domain

A general-purpose embedding model trained on web text won't know that "arrest" means something different in a cardiac context than a legal one. Your vectors cluster the wrong documents together. Test your embedding model on your actual domain before trusting it. If retrieval precision is under 0.7 on known-good queries, the model is the problem.

Stale or incomplete index updates

You add documents, but the index doesn't refresh. Or it refreshes on a schedule that lags hours behind ingestion. Users search for something that exists in the source but not in the vectors. The signature is simple: retrieval returns nothing for a query you know is in the corpus. Check index freshness before touching the model.

Missing hybrid search (BM25 + vector)

Pure vector search misses exact matches: product codes, error numbers, proper names. BM25 catches those. Vector search catches paraphrases. Run both and merge results. If you're vector-only, you're leaving recall on the table.

Index fragmentation across multiple stores

One team's docs live in Pinecone, another's in Weaviate, a third in Postgres. A query hits one store and misses everything in the others. The fix is a unified index or a federated search layer. Fragmentation is an organizational problem that masquerades as a technical one.

Retrieval Failure Modes: The Most Common RAG Failure Modes

Retrieval is where most production RAG systems break. The model looks wrong. The retriever is the problem.

Low top-k precision: irrelevant chunks returned

You ask for the top 5 chunks. Three are noise. The model then answers from noise and sounds confident doing it. Fix: add a re-ranker, or lower top-k and test precision on known queries.

Low recall: relevant chunks missed entirely

The right chunk exists in your index. The retriever never returns it. The model can't cite what it never saw. Fix: hybrid search, or check if your embedding model understands your domain.

Context window overflow from too many chunks

You stuff 20 chunks into the prompt. The model drowns. It picks the first few, ignores the rest. Fix: cap chunks at what your model actually uses, not what fits.

No retrieval grading or fallback search

You trust the retriever blindly. No score threshold, no fallback to keyword search. When vectors fail, nothing catches it. Fix: grade retrieval confidence, and fall back to BM25 when scores drop.

Diagnostic: retrieval telemetry to isolate failures

Log what you retrieved, not just what the model said. If the answer is wrong but the chunks were right, it's a generation problem. If the chunks were wrong, it's retrieval. That one log line saves hours of guessing.

Generation Failure Modes: Hallucination and Context Misuse

Retrieval can return perfect chunks and the model still gets it wrong. Generation failures are separate from retrieval failures. You need to test for them independently.

Hallucination from insufficient or conflicting context

The model fills gaps with plausible-sounding fiction. Two retrieved chunks contradict each other, and the model picks one or blends both. Fix: check for conflicting sources before generation, and flag low-confidence context.

Ignoring retrieved context entirely

The model has the right chunks in its prompt. It answers from its training data instead. This happens when context is buried or the prompt doesn't force grounding. Fix: instruct the model to cite chunks, and test with known-good context.

Citation errors and fabricated sources

The model cites a source that doesn't exist, or attributes a claim to the wrong chunk. This is the most dangerous failure mode because it looks authoritative. Fix: verify citations against retrieved chunks before returning output.

Agent Memory Failure Modes: The Gap Nobody Covers

Generic RAG guides stop at generation. Agent memory builders know the pipeline keeps running long after the first answer. Memory is a fifth stage with its own failure signatures, and they don't look like retrieval failures.

Context drift: memory degrades over long sessions

An agent stores context from turn one. By turn forty, that context has been summarized, re-summarized, and compressed. Each pass loses fidelity. The agent still sounds confident. It's answering from a distorted memory, not the original exchange. You'll see it when the agent contradicts something it said twenty turns ago.

Memory overwrite: new information erases old

The agent learns a user's new preference. It writes that to the memory store. The old preference is gone. Not updated, not versioned. Gone. This is silent data loss. The agent doesn't know it forgot something.

Stale memory retrieval: pulling outdated facts

The memory store holds a fact from three sessions ago. The world changed. The agent retrieves the old fact because it's the highest-similarity match. No timestamp check. No freshness gate. The answer is wrong in a way that's hard to spot because it was right once.

Why generic RAG fixes don't solve memory problems

Re-ranking, better chunking, hybrid search. None of these touch memory. They optimize retrieval from a static corpus. Memory is dynamic, temporal, and user-specific. You need versioning, timestamps, and decay policies. Generic RAG doesn't have those primitives.

A Diagnostic Checklist for RAG Pipeline Builders

Run this in order. Each stage catches failures the next stage would mask.

Stage 1: Check ingestion and chunking quality

Pull 50 random chunks. Count how many split a sentence mid-thought or mix two topics. If it's over 10%, fix chunk boundaries before touching anything else.

Stage 2: Validate index freshness and embedding relevance

Query ten known-good documents. Check they appear in top-k. Check the index timestamp against your last ingestion run. Stale index means every downstream metric lies.

Stage 3: Measure retrieval precision and recall

Precision: what fraction of returned chunks are relevant. Recall: what fraction of relevant chunks got returned. Track both. Precision below 0.7 means re-ranking or filtering. Recall below 0.8 means chunking or embedding problems.

Stage 4: Test generation with known-good context

Feed the model perfect context. If it still hallucinates, the problem is generation, not retrieval. If it answers correctly, your failures are upstream.

Stage 5: Audit memory store for drift and staleness

Sample stored memories. Check timestamps. Look for overwritten entries with no version history. If you can't tell when a memory was written, you can't trust it.

What You Cannot Fix: Honest Limits of RAG

Some failures are not engineering problems. They are properties of the approach.

RAG cannot fix bad source data

If your documents are wrong, outdated, or contradictory, retrieval returns wrong, outdated, or contradictory context. The model faithfully repeats it. No chunking strategy, embedding model, or re-ranker repairs a source that was never correct.

No single fix covers all failure modes

A re-ranker won't fix stale embeddings. Better chunking won't fix memory overwrite. Each stage fails differently and needs its own fix. Anyone selling one solution for every RAG problem is selling something else.

RAG does not make an LLM truthful

Retrieval narrows what the model can draw on. It does not make the model honest about what it doesn't know. Given conflicting context, the model still picks a side, sometimes the wrong one. RAG reduces hallucination. It does not eliminate it.

Final Thoughts: Building RAG Systems That Fail Less

The pipeline-stage view works because it forces you to name the failure before you fix it. Ingestion problems look different from indexing problems. Retrieval failures look different from memory failures. If you can't say which stage broke, you're guessing.

Most production rag failure modes are retrieval problems, not model problems. The second most common are memory problems, especially in agent pipelines where context drifts across sessions. Fix those two stages first. Generation fixes matter less than you think.

GigaRAG targets the memory side specifically: context retention, overwrite prevention, and stale retrieval for agent RAG pipelines. It won't fix bad source data. Nothing will. But if your failures trace to memory, it's built for that.

Run the diagnostic checklist. Track precision and recall. Audit your memory store. Most RAG systems fail less when builders stop treating every error as an LLM problem and start tracing it to the stage that actually broke.

Frequently Asked Questions

Why do RAG systems fail?

RAG systems fail at specific pipeline stages: ingestion (bad parsing or chunking), indexing (stale or missing embeddings), retrieval (irrelevant top-k results), generation (model ignores context), or memory (agent context drift). Most failures look like model problems but originate earlier in the pipeline. Diagnosing by stage prevents wasted effort tuning the wrong component.

What are the different types of failure modes?

Failure modes are commonly grouped by where they occur: data failures (missing, stale, or contradictory sources), retrieval failures (wrong or incomplete chunks), generation failures (hallucination, context neglect), and memory failures in agents (context drift, overwrite, stale retrieval). Each type has distinct signatures and fixes. Categorizing by stage keeps debugging systematic.

What are the 7 types of RAG?

RAG variants are usually described by retrieval strategy or architecture, such as naive RAG, advanced RAG, modular RAG, graph RAG, hybrid RAG, agentic RAG, and multi-hop RAG. The exact list varies by source, and the categories overlap. For debugging, the pipeline-stage view matters more than the variant label.

Is ChatGPT a RAG model?

ChatGPT is a general-purpose language model interface, not a RAG system by itself. RAG is an architecture that adds retrieval of external documents to a model's generation step. Some ChatGPT features use retrieval-like mechanisms, but the model and the RAG pattern are separate concepts.

Can RAG fix bad source data?

1. RAG retrieves and conditions on the data you provide, so incorrect, outdated, or contradictory sources will produce unreliable answers. Retrieval quality depends on source quality. Data cleaning and governance must happen before or alongside RAG tuning.

What is the most common RAG failure mode?

Retrieval failure is the most frequently reported category: the system returns chunks that are topically related but do not answer the query. This often traces to chunking strategy, embedding mismatch, or missing reranking. It is frequently misdiagnosed as a generation problem because the final answer looks wrong.

How do agent memory failures differ from standard RAG failures?

Agent memory failures involve state across turns: context drift (the agent slowly departs from the original goal), memory overwrite (new information replaces critical earlier facts), and stale memory retrieval (old, no-longer-valid context is reused). Standard RAG failures are typically single-turn retrieval or generation issues. Memory failures require tracing write and read events over time.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through rag failure modes or something adjacent, we publish what we have actually tested, including where it falls short.