The RAG Pipeline Explained for Agent Memory Builders

The RAG Pipeline Explained for Agent Memory and RAG Builders

The rag pipeline explained through a failure: you built an agent that pulls context, but it keeps retrieving the wrong chunk or forgetting what the user said three turns ago. That's not a prompt problem. It's a pipeline problem. RAG, retrieval augmented generation, is the pattern of fetching relevant external knowledge before the LLM generates a response, so the model works from grounding instead of memory alone. Most explainers stop at the diagram: boxes, arrows, done. This one goes further, into working code, honest failure modes, and what changes when the pipeline has to serve agent memory rather than one-shot chatbot Q&A. GigaRAG exists for exactly that use case, and I'll mention where it fits without pretending it solves everything. Here's what you'll get: the components and steps, a minimal code walkthrough, a clear account of what RAG cannot do, and evaluation criteria you can actually run against a pipeline.

In This Guide

• What Is a RAG Pipeline?
• Agent Memory RAG vs Standard Chatbot RAG
• RAG Pipeline Components and Steps
• The Rag Pipeline Explained: A Step-by-Step Guide
• A Minimal RAG Pipeline in Code
• RAG for Agent Memory: What Changes
• What the RAG Pipeline Cannot Do
• How to Evaluate a RAG Pipeline
• Is ChatGPT a RAG System?
• The Four Levels of RAG

What Is a RAG Pipeline?

A RAG pipeline is a system that retrieves relevant information from an external knowledge base, then feeds that information to a large language model so it can generate a grounded answer. Retrieval augmented generation, or RAG, pairs two steps: find the right context, then write with it.

The core idea: retrieval + generation

The LLM doesn't answer from memory alone. It gets a prompt packed with retrieved passages, then generates from those. The retrieval step handles what the model doesn't know. The generation step handles phrasing.

Why RAG exists: grounding LLMs in external knowledge

LLMs are frozen at training time. They can't see your documents, your codebase, or your user's history. RAG bridges that gap by fetching fresh, specific context at query time. The model still hallucinates, but it now has source material to work from. That's the whole trade: better grounding, more moving parts.

[!note] RAG augments a model with retrieved context at inference time; it does not change the model's underlying weights or guarantee factual correctness on its own.

Agent Memory RAG vs Standard Chatbot RAG

RAG Pipeline Components and Steps

A RAG pipeline is a sequence of stages. Each stage transforms data: raw text becomes chunks, chunks become vectors, vectors become searchable context, context becomes an answer. You can build each stage with different tools, but the order doesn't change.

Document ingestion and chunking

You load documents from wherever they live: PDFs, markdown files, a database, an API. Then you split them. Chunking matters because embedding models have token limits, and retrieval works better on focused passages than on whole documents. A common chunk size is 500 to 1,000 tokens with some overlap between chunks. Too small and you lose context. Too large and retrieval gets noisy.

Embedding and vector indexing

Each chunk goes through an embedding model, which turns text into a vector of numbers. Those vectors land in a vector database or index. At query time, the same embedding model converts the user's question into a vector, and the index finds the nearest neighbors. Similarity is usually cosine similarity or dot product. The index doesn't understand meaning. It just measures distance in vector space.

Query-time retrieval

The user's query gets embedded, then matched against the index. You get back the top-k chunks, typically 3 to 10. Some pipelines add a reranking step here: a cross-encoder scores each candidate against the query and reorders them. Reranking improves precision but adds latency. Whether you need it depends on your corpus size and how much noise the first pass returns.

Generation with retrieved context

The retrieved chunks get stuffed into a prompt alongside the user's question. The LLM generates an answer grounded in that context. The prompt usually says something like "Answer using only the provided context." The model still might ignore it. But when retrieval works, the answer cites real passages instead of inventing them.

[!tip] For agent memory, store metadata like timestamps, source, and confidence alongside each vector so you can filter by recency and provenance at retrieval time — this prevents stale or irrelevant memories from polluting the context window.

The Rag Pipeline Explained: A Step-by-Step Guide

1. Ingest raw sources (docs, chat logs, tool outputs) and normalize them into plain text.
2. Chunk the text into overlapping segments sized to your embedding model and context budget.
3. Embed each chunk with an embedding model and store vectors plus metadata in a vector store.
4. At query time, embed the user or agent query and retrieve the top-k most similar chunks.
5. Optionally re-rank retrieved chunks to improve precision before generation.
6. Assemble a prompt with the retrieved context and call the LLM to generate the answer.
7. For agent memory, write new interactions back into the store and tag them with timestamps and source.

A Minimal RAG Pipeline in Code

Here's a working reference implementation. It's not production-ready, but it runs end to end: load documents, embed them, retrieve, generate. You can swap any component later.

Setting up embeddings and a vector store

Start with a small set of documents. For this example, three strings stand in for whatever you'd load from files.

from sentence_transformers import SentenceTransformer
import numpy as np

docs = [
    "The agent stores user preferences in long-term memory.",
    "Short-term memory holds the current conversation turn.",
    "Retrieval pulls relevant chunks before generation."
]

model = SentenceTransformer("all-MiniLM-L6-v2")
doc_embeddings = model.encode(docs)

That's the index. A real vector store handles persistence and similarity search at scale, but a numpy array is enough to show the mechanic. Each document becomes a 384-dimensional vector. The model maps similar text to nearby points.

Retrieval function

At query time, embed the question and find the closest document vectors.

def retrieve(query, k=2):
    q_embedding = model.encode([query])
    scores = np.dot(doc_embeddings, q_embedding.T).flatten()
    top_k = np.argsort(scores)[-k:][::-1]
    return [(docs[i], scores[i]) for i in top_k]

Dot product works here because the vectors are normalized. Cosine similarity would give the same ranking. The function returns the top-k chunks and their scores, so you can see how confident the match is.

Generation call with context

Retrieved chunks get concatenated into the prompt. The LLM sees the question plus the context.

def generate(query):
    hits = retrieve(query)
    context = "\n".join([doc for doc, _ in hits])
    prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
    # Call your LLM here. Example with OpenAI:
    # response = openai.ChatCompletion.create(
    #     model="gpt-4o-mini",
    #     messages=[{"role": "user", "content": prompt}]
    # )
    return prompt

The prompt is the contract. It tells the model what it can use. If retrieval returns nothing relevant, the model still answers, just without grounding. That's the failure mode the next sections dig into.

RAG for Agent Memory: What Changes

A one-shot RAG pipeline answers a single question and forgets. An agent runs across dozens of turns, calls tools, and needs to remember what happened three turns ago. That changes the pipeline.

Short-term vs long-term agent memory

Short-term memory is the conversation window. It holds the current turn, recent turns, and tool outputs. Long-term memory is a persistent store: user preferences, past decisions, facts learned earlier. RAG serves both. Short-term context gets retrieved from the active session. Long-term context gets retrieved from a vector index that survives restarts.

Retrieval across multi-turn conversations

Each turn triggers retrieval, but the query isn't just the latest user message. It's the message plus relevant history. Query rewriting matters here: "what about the second option?" means nothing without the prior turn. The pipeline must expand or rewrite the query before embedding it.

Memory stores as a RAG backend

The vector store becomes the agent's memory backend. Write operations happen when the agent learns something worth keeping. Read operations happen at every turn. That's the loop: retrieve, act, write back, retrieve again. Most RAG explainers stop at read-only. Agent memory is read-write.

What the RAG Pipeline Cannot Do

RAG fixes grounding. It does not fix reasoning, and it does not fix retrieval itself. The honest answer is that a RAG pipeline fails in predictable ways, and you'll hit all of them if you build long enough.

Retrieval failures and silent misses

The worst failure is the one you never see. The retriever returns results, the generator produces an answer, and everyone moves on. But the right chunk was never in the top-k. Cosine similarity is not understanding. A query phrased differently from the source text can miss entirely. Chunk boundaries split the answer across two chunks, and neither one alone is enough. The generator then answers from whatever it did retrieve, which may be adjacent but wrong. No error is raised. The system just quietly returns a plausible answer built on the wrong context.

Hallucination still happens

Grounding reduces hallucination. It does not eliminate it. The generator can still ignore retrieved context, blend it with parametric knowledge, or fill gaps the chunks don't cover. When the retrieved context is thin or contradictory, the model leans on its own weights. You'll see confident statements that aren't in any retrieved document. Faithfulness metrics catch this after the fact, but at runtime the pipeline has no built-in alarm.

When not to use RAG

Don't use RAG when the knowledge fits in the context window. A 50-page manual can go straight into the prompt on models with long contexts. Don't use RAG when the answer requires reasoning across the entire corpus, not just a few retrieved chunks. Don't use RAG when the index goes stale faster than you can rebuild it. And don't use RAG when you need exact answers: a SQL query over structured data beats semantic search every time.

How to Evaluate a RAG Pipeline

There is no single best RAG pipeline. The honest answer is that the best pipeline is the one that fails least on your data, your queries, and your latency budget. You evaluate a RAG pipeline on three axes: retrieval quality, generation quality, and operational cost.

Retrieval metrics: recall, precision, MRR

Retrieval quality measures whether the right chunks show up in the top-k. Recall asks: of all relevant chunks, how many did you retrieve? Precision asks: of the chunks you retrieved, how many were relevant? MRR, mean reciprocal rank, asks: how high up did the first relevant chunk land? For agent memory, recall matters more than precision. A missed memory is worse than an extra irrelevant one, because the generator can ignore noise but cannot invent what was never retrieved.

Generation metrics: faithfulness, relevance

Generation quality measures what the model did with the retrieved context. Faithfulness asks: is every claim in the answer supported by a retrieved chunk? You check this with an LLM judge or a human audit. Relevance asks: does the answer actually address the query? A pipeline can retrieve perfectly and still generate a non-answer. Test both. Faithfulness catches hallucination. Relevance catches drift.

Operational metrics: latency, cost, index freshness

Operational metrics decide whether the pipeline survives production. Latency: how long from query to answer, including embedding, retrieval, reranking, and generation. Cost: tokens in, tokens out, plus embedding and storage fees. Index freshness: how long between a document changing and the index reflecting it. For agent memory, freshness is the one most builders ignore. A memory store that updates nightly is useless for an agent that needs what it learned five minutes ago.

Is ChatGPT a RAG System?

No. ChatGPT is not a RAG system by default. It generates from weights, not from a retrieved index. The model has no external knowledge base it queries before answering. It predicts tokens from what it learned during training, with a cutoff date.

RAG-like patterns appear when you turn on browsing or plugins. Browsing fetches web pages, inserts them into the context window, and the model answers from that inserted text. That is retrieval plus generation. Plugins that call an API and pass results back do the same thing. But the core ChatGPT product, with no tools enabled, is pure generation.

The distinction matters for builders. A RAG pipeline gives you control over what the model sees. You choose the documents, the chunks, the index. ChatGPT without tools gives you none of that. If you need grounded answers from your own data, you need a RAG pipeline, not a chat interface.

The Four Levels of RAG

RAG maturity moves in four rough stages. Most builders start at level 1 and stall there. The levels aren't official, but they map cleanly to what you'll see in production systems.

Level 1: Naive RAG

You chunk documents, embed them, index them, and retrieve the top-k chunks for a query. Then you stuff those chunks into a prompt and generate. No reranking, no query rewriting, no memory. It works for small, static knowledge bases. It breaks when queries are ambiguous or documents overlap.

Level 2: Advanced RAG with reranking

You add a reranker after initial retrieval. The retriever pulls 50 candidates, the reranker scores them for relevance, and you keep the top 5. You might also rewrite the query before retrieval. This fixes most precision problems. The cost is latency: a reranker adds a model call per query.

Level 3: Modular RAG

You break the pipeline into swappable pieces. Different retrievers for different document types. Hybrid search combining sparse and dense. A router that decides which index to query. This is where most serious production systems live. It's also where complexity starts to hurt.

Level 4: Agentic RAG with memory

An agentic RAG pipeline closes the loop. The agent retrieves, acts, writes back to memory, and retrieves again on the next turn. Memory is a first-class component, not an afterthought. The pipeline handles multi-turn context, tool outputs, and persistent user state. This is the level most agent builders need, and the level most RAG explainers never reach.

The rag pipeline explained honestly: it's a retrieval step bolted to a generation step, and the retrieval step is where most systems fail. Build the retrieval loop first. Test it against real queries. Then wire in generation. If you're building for agents, treat memory as a read-write store from day one, not a cache you bolt on later. GigaRAG is built for that loop, but the principles here apply to any stack you choose.

Frequently Asked Questions

Can you explain the RAG system in a simple way?

Think of RAG as an open-book exam for an LLM. Instead of relying only on what it memorized during training, the system first looks up relevant passages from a knowledge source, then hands those passages to the model so it can answer with grounded context.

Is ChatGPT a RAG?

ChatGPT is a general-purpose LLM application, not inherently a RAG system. However, features that let it browse or search external sources before answering do use retrieval-augmented generation techniques. The base model alone is not RAG.

Which RAG pipeline is considered the best?

There is no single best pipeline — the right design depends on your data, latency budget, and accuracy needs. What matters most is tuning chunking, retrieval quality, and re-ranking for your specific use case rather than copying a one-size-fits-all architecture.

What are the four levels of RAG?

A common framing describes four levels: naive RAG (basic retrieve-then-generate), advanced RAG (better chunking, re-ranking, query rewriting), modular RAG (composable retrieval and generation modules), and agentic RAG (an agent decides when and what to retrieve). The boundaries between levels are informal and implementations vary.

What are the main steps in a RAG pipeline?

The typical steps are ingestion, chunking, embedding, indexing, retrieval, optional re-ranking, and generation. For agent memory, you also add a write-back step so new interactions are stored for future retrieval.

What can RAG not do?

RAG cannot fix a model that reasons poorly, cannot guarantee factual accuracy if retrieval returns bad context, and cannot replace fine-tuning for teaching new skills or styles. It also adds latency and cost, so it is not always the right choice.

When should you not use RAG?

Skip RAG when your task needs no external or persistent knowledge, when the knowledge fits comfortably in the prompt, or when latency and cost budgets are extremely tight. In those cases, a well-prompted model or fine-tuning may be simpler and more effective.

About GigaRAG

GigaRAG helps GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through the rag pipeline explained or something adjacent, we publish what we have actually tested, including where it falls short.