The Open Source RAG Stack: A Builder's Guide

The Open Source RAG Stack: What Pipeline Builders Actually Need

The open source RAG stack is what pipeline builders and agent memory developers actually work with: a chain of ingestion, embedding, vector storage, orchestration, and evaluation components, all held together by code you can read and modify. Most content on the subject is either a framework roundup with no trade-offs, or a single-tool tutorial that pretends the other layers don't exist. That's not useful when you're the one who has to keep the thing running. GigaRAG is one tool in this space, built for agent memory and RAG pipeline builders who need persistent state. But the honest answer is that no single tool solves the stack. The good news is you don't need everything at once. This guide walks through each layer with specific open source options, what they cost you in complexity, and what you cannot expect them to do. You'll finish with a minimal build path you can follow this afternoon, plus a decision framework for when to skip a layer entirely.

In This Guide

• What Is the Open Source RAG Stack?
• Vector Store vs Agent Memory: What's the Difference?
• How the Open Source RAG Stack Works: Layer by Layer
• The Open Source Rag Stack: A Step-by-Step Guide
• Open Source Components for Each Layer of the RAG Stack
• Agent Memory: The Missing Layer in Most RAG Stack Discussions
• How to Build a Minimal Open Source RAG Application
• What You Cannot Do with the Open Source RAG Stack
• How to Choose the Right Open Source RAG Stack Components
• Common Mistakes When Building an Open Source RAG Stack
• Final Thoughts on the Open Source RAG Stack

What Is the Open Source RAG Stack?

An open source RAG stack is a set of self-hosted components that work together to retrieve relevant context from your own data and feed it to a language model before it generates an answer. You control every layer: how documents get split, how they get embedded, where vectors live, and how retrieval gets orchestrated.

The stack exists because LLMs don't know your data. Retrieval-augmented generation fixes that by pulling relevant chunks from a store you built, then stuffing them into the prompt. The open source version means you run the whole pipeline yourself instead of calling a hosted API that hides the internals.

The five core layers of a RAG stack

Every RAG pipeline, open source or not, breaks into five layers:

1. Ingestion and chunking. You load documents, clean them, and split them into chunks small enough to embed and retrieve.
2. Embedding generation. You run each chunk through an embedding model to turn text into vectors.
3. Vector storage and retrieval. You store those vectors in a database that supports similarity search.
4. Orchestration and prompt assembly. You wire the retrieval step to the LLM call, building the prompt with retrieved context.
5. Evaluation. You measure whether retrieval actually returns the right chunks.

Skip any layer and the pipeline breaks in ways that are hard to debug later.

Why open source matters for pipeline builders

Proprietary RAG services hide the chunking strategy, the embedding model, and the retrieval logic. That's fine until something returns wrong answers and you can't inspect why. Open source gives you the actual code. You can swap the embedding model, change the chunk size, or replace the vector store without renegotiating a contract.

The trade-off is real: you run the infrastructure. That means version upgrades, dependency conflicts, and your own monitoring. For pipeline builders who need to debug retrieval quality and tune for specific data, that control is usually worth the overhead. For a one-off prototype, it might not be.

[!note] A vector database is optimized for similarity search, not for storing or replaying conversation state; agent memory requires a separate persistence layer with its own schema and lifecycle.

Vector Store vs Agent Memory: What's the Difference?

How the Open Source RAG Stack Works: Layer by Layer

Each layer feeds the next. Chunks become vectors, vectors become search results, search results become prompt context. If one layer is weak, the answer quality drops no matter how good the LLM is.

Ingestion and chunking

You start with raw documents: PDFs, markdown files, HTML, database rows. Ingestion loads them, strips formatting, and splits the text into chunks. Chunk size matters more than most builders expect. Too small and you lose context. Too large and retrieval returns noise.

The standard starting point is 500 to 1000 tokens per chunk with some overlap, usually 10 to 20 percent. Overlap keeps a sentence that straddles a chunk boundary from getting lost. But the right size depends on your data. Legal contracts need larger chunks than FAQ entries.

Embedding generation

Each chunk goes through an embedding model that converts text into a vector, a list of numbers that captures semantic meaning. Similar text produces similar vectors. That's the whole trick behind retrieval.

Open source options like sentence-transformers, BGE, and E5 run locally. You don't pay per token and you don't send your data to a third party. The trade-off: you need a GPU or enough CPU patience for large document sets. A 7B parameter embedding model on CPU will take a while.

Vector storage and retrieval

Vectors go into a database that supports similarity search. The database RAG uses is a vector database, or a regular database with a vector extension. pgvector adds vector search to Postgres. Qdrant, Weaviate, and ChromaDB are purpose-built.

Retrieval works by converting your query into a vector, then finding the nearest neighbors in the store. You get back the top-k chunks, usually 3 to 10. The database doesn't understand your data. It just measures distance between vectors.

Orchestration and prompt assembly

Orchestration is the glue. It takes the user query, calls the embedding model, runs the vector search, and builds a prompt that includes the retrieved chunks plus instructions. Then it sends that prompt to the LLM.

Frameworks like LangChain, LlamaIndex, and Haystack handle this wiring. They also add retrieval strategies: hybrid search, reranking, metadata filtering. You can write this yourself in 50 lines of Python. The frameworks help when you need to swap components or add complexity.

Evaluation and iteration

Evaluation measures whether retrieval returns the right chunks. You need a test set of queries with known relevant documents. Run the queries, check what comes back, and score it.

Tools like RAGAS, TruLens, and DeepEval automate parts of this. They check faithfulness (did the answer use the context?) and relevance (did retrieval find the right chunks?). Without evaluation, you're tuning blind. Most builders skip this layer and then wonder why answers drift.

The honest answer is that evaluation is still hard. Building a good test set takes time, and the metrics don't catch everything. But running even a small eval set beats shipping a pipeline you've never measured.

[!tip] For agent memory, start with a simple SQLite table before reaching for a dedicated memory service — it forces you to define what state actually matters, and GigaRAG can be layered in later when you need semantic recall across sessions.

The Open Source Rag Stack: A Step-by-Step Guide

1. Install Python 3.10+ and create a virtual environment.
2. Install sentence-transformers and chromadb via pip.
3. Load your documents and split them into chunks of a few hundred tokens.
4. Generate embeddings for each chunk and store them in Chroma with metadata.
5. Write a retrieval function that embeds the query and fetches top-k chunks.
6. Pass retrieved chunks plus the question to an LLM and return the answer.
7. Add a simple memory store (e.g., SQLite) to persist conversation turns.

Open Source Components for Each Layer of the RAG Stack

You can assemble a working RAG pipeline entirely from open source components. The catch is that each layer has multiple options, and they don't all play nicely together. Here's what I'd actually use, layer by layer.

Ingestion tools: Unstructured, LlamaIndex readers, LangChain loaders

Unstructured handles the messy reality of real documents. It pulls text out of PDFs, HTML, email, and office files without you writing parsers for each format. It's the best choice when your source data is heterogeneous.

LlamaIndex readers and LangChain loaders are lighter-weight alternatives. They cover common formats like PDF, markdown, and web pages. If your data is already clean text, either works fine. Unstructured earns its place when documents are scanned, oddly formatted, or mixed.

The trade-off: Unstructured is heavier to install and slower to run. For a pile of markdown files, it's overkill.

Embedding models: sentence-transformers, BGE, E5

sentence-transformers is the easiest starting point. It wraps Hugging Face models in a simple API and runs locally. You don't pay per token.

BGE models from BAAI consistently rank near the top of retrieval benchmarks. BGE-large hits strong accuracy on standard datasets. E5 from Microsoft is another solid option, with small variants that run on modest hardware.

The honest answer is that model choice matters less than you'd think for a first build. Start with sentence-transformers and a small BGE or E5 model. Swap later if retrieval quality disappoints.

Vector stores: pgvector, Qdrant, Weaviate, ChromaDB

pgvector is the pragmatic default. If you already run Postgres, you add one extension and get vector search without a new service. It handles up to a few million vectors fine.

Qdrant and Weaviate are purpose-built vector databases. They offer better performance at scale, built-in filtering, and hybrid search. ChromaDB is the simplest option for prototyping. It runs in-process and stores everything locally.

For agent memory specifically, pgvector's transactional guarantees matter. You can update memory alongside other relational data in one transaction. Most purpose-built vector stores don't give you that.

Orchestration frameworks: LangChain, LlamaIndex, Haystack

LangChain has the largest community and the most integrations. It's also the most abstracted, which means debugging gets harder as your pipeline grows.

LlamaIndex is stronger on data ingestion and indexing. If your pipeline is document-heavy, it's a natural fit. Haystack from deepset is the most production-oriented of the three. It has cleaner abstractions and better evaluation hooks.

You can skip all three and write the wiring yourself. For a simple pipeline, that's often faster than learning a framework's abstractions.

Evaluation tools: RAGAS, TruLens, DeepEval

RAGAS focuses on RAG-specific metrics: faithfulness, answer relevance, context relevance. It's the most widely used.

TruLens adds tracing so you can see which retrieved chunks influenced the answer. DeepEval is newer but has a clean API and fast test runs.

All three require a test set of queries with known-good answers. Building that test set is the real work. The tools just automate the scoring.

Agent Memory: The Missing Layer in Most RAG Stack Discussions

Most RAG stack breakdowns stop at retrieval. You embed a query, pull relevant chunks, stuff them into a prompt, and get an answer. That's a lookup, not an agent. An agent that works across multiple turns needs to remember what it already did, what the user said three messages ago, and what it learned from earlier retrievals. That's agent memory, and it's a different problem.

Stateless retrieval vs. stateful agent memory

Stateless retrieval treats every query as if it's the first. No context carries over. You ask a follow-up question, and the system has no idea what "it" refers to. Stateful agent memory keeps a running record: conversation history, intermediate results, decisions the agent made, facts it verified. The retrieval layer still does its job. Memory sits on top of it and feeds the right context back into the prompt.

The honest answer is that most RAG tutorials ignore this entirely. They build a single-shot Q&A system and call it done. Agents need more.

Open source tools for persistent agent memory

You have a few practical options. Postgres with pgvector handles both vector search and relational storage, which means you can store conversation turns, user preferences, and retrieved facts in the same transaction. That's the simplest path to persistence.

LangGraph and LlamaIndex both offer memory abstractions. LangGraph's checkpointing saves agent state between steps. LlamaIndex has chat memory buffers and a memory module for tracking entities and facts. Mem0 is a newer option built specifically for agent memory, with an API for storing and retrieving memories across sessions.

None of these work out of the box. You wire them into your orchestration layer and decide what gets stored, when it expires, and how it's retrieved.

How GigaRAG fits into the agent memory layer

GigaRAG treats agent memory as a first-class concern rather than an afterthought bolted onto a retrieval pipeline. It's built for pipeline builders who need persistent, stateful memory without stitching together five different libraries. The trade-off is that it's opinionated: if you need fine-grained control over every memory operation, you'll still want the lower-level tools.

How to Build a Minimal Open Source RAG Application

You don't need a framework to build your first RAG app. LangChain and LlamaIndex add abstractions that help at scale, but they also hide what's actually happening. A minimal stack is four pieces: a chunking script, an embedding model, a vector store, and a query loop. You can build it in an afternoon.

Prerequisites and environment setup

You'll need Python 3.10 or newer, a machine with at least 8 GB of RAM, and an OpenAI API key or a local LLM. For this build, use sentence-transformers for embeddings, ChromaDB for vector storage, and a small local model like Llama 3.1 8B via Ollama for generation. Install with pip:

pip install sentence-transformers chromadb ollama

That's the whole dependency list. No framework required.

Step 1: Load and chunk documents

Start with plain text files. Read them in, split them into chunks of roughly 500 tokens with 50 tokens of overlap. Overlap matters: it keeps sentences from being cut in half at chunk boundaries, which is the single most common cause of bad retrieval. A simple Python loop with a sliding window does the job. Don't reach for a chunking library yet.

Step 2: Generate embeddings

Load a sentence-transformers model. all-MiniLM-L6-v2 is a good default: it's small, fast, and good enough for most document types. Run each chunk through the model to get a 384-dimensional vector. Store the chunk text and its vector together. This step is embarrassingly parallel, but for a few hundred documents a simple for loop is fine.

Step 3: Store vectors

ChromaDB runs in-process and persists to disk. Create a collection, add your chunks with their embeddings, and you're done. No server, no Docker, no config file. The main catch is that ChromaDB isn't built for high concurrency. For a single-user app or a prototype, it's the right trade-off.

Step 4: Query and generate

Embed the user's question with the same model. Query ChromaDB for the top 3 to 5 most similar chunks. Stuff those chunks into a prompt template along with the question, then send the whole thing to Ollama. The prompt should say something like: "Answer the question using only the context below. If the context doesn't contain the answer, say so."

That's the entire pipeline. It won't handle multi-turn conversations, it won't evaluate its own retrieval quality, and it won't scale past a few thousand documents without tuning. But it retrieves real documents and generates grounded answers, which is more than most tutorials deliver.

What You Cannot Do with the Open Source RAG Stack

The minimal stack works. It retrieves documents and grounds answers. It does not scale, it does not evaluate itself, and it does not remember anything between queries. Those gaps are not bugs you can patch with a better embedding model. They are structural limits of the open source components themselves.

Scale limitations and infrastructure burden

A ChromaDB instance on your laptop handles a few thousand chunks without complaint. Push past that and you'll hit real walls: index build times grow, query latency climbs, and memory usage stops being predictable. Moving to Qdrant or Weaviate helps, but now you're running a separate service. Someone has to deploy it, monitor it, upgrade it. That's infrastructure work, not RAG work.

The honest answer is that open source RAG stacks scale as far as your team's operational capacity. The software won't stop you. Your on-call rotation will.

Latency and real-time constraints

RAG adds latency at every layer: embedding generation, vector search, prompt assembly, LLM inference. With local models, a single query can take 3 to 10 seconds end to end. That's fine for document Q&A. It's not fine for chat, for agent loops that make multiple retrieval calls per turn, or for anything that needs sub-second responses.

You can cut latency with smaller models, GPU inference, or approximate nearest neighbor search. Each trade costs you accuracy. There's no free lunch here.

Evaluation is still hard

RAGAS, TruLens, and DeepEval give you metrics: faithfulness, answer relevance, context precision. What they don't give you is ground truth. You still need labeled examples of good and bad answers for your specific documents, and building that dataset is manual work. The tools measure what you tell them to measure. They won't tell you if your chunking strategy is wrong, only that retrieval scores dropped.

Agent memory is not solved out of the box

Most open source RAG stacks treat every query as stateless. The agent retrieves context, generates an answer, and forgets everything. Adding memory means building a separate persistence layer: conversation history, user preferences, task state. LangChain and LlamaIndex offer memory abstractions, but they're thin wrappers over a database you still have to design and maintain.

No open source component gives you durable, queryable agent memory without real engineering effort. That's the gap.

How to Choose the Right Open Source RAG Stack Components

The previous section listed what the stack can't do. This one is about choosing components so you hit fewer of those walls. The decision framework is simple: start minimal, add complexity only when a specific requirement forces it.

When you don't need a vector database

If your corpus is under a few hundred documents and changes rarely, skip the vector DB entirely. Load chunks into memory, embed them with sentence-transformers, and run cosine similarity with NumPy. That's it. No service to deploy, no index to maintain, no schema to design.

You need a vector database when retrieval latency matters at scale, when you need metadata filtering, or when your index must survive a process restart. Before that point, a vector DB is overhead, not infrastructure.

Matching components to use case and scale

It depends on three things: corpus size, query volume, and whether retrieval feeds an agent loop or a single-shot answer.

For document Q&A on a static corpus, ChromaDB plus a local embedding model is enough. For multi-tenant SaaS with thousands of users, you need Qdrant or Weaviate with managed infrastructure. For agent memory that persists across sessions, you need a database designed for stateful writes, not just vector search.

The honest answer is that most builders over-provision. Start with the simplest stack that works for one user, then scale components individually as bottlenecks appear.

Maintenance activity, community size, and license considerations

A component with no commits in 18 months is a liability, not a dependency. Check GitHub activity before adopting anything: commit frequency, open issue count, and whether maintainers respond. A small but active project beats a large but dormant one.

License matters more than most builders realize. ChromaDB and Qdrant are Apache 2.0. Weaviate is BSD 3-Clause. LangChain and LlamaIndex are MIT. All permissive, but check before embedding anything in a commercial product. Copyleft licenses like GPL can force you to open your own code.

Community size is a proxy for documentation quality and Stack Overflow answers. A tool with 20,000 stars but no answered questions is harder to adopt than one with 2,000 stars and an active Discord.

Common Mistakes When Building an Open Source RAG Stack

Most failed RAG projects die from the same four mistakes. None of them are exotic. All of them are fixable before you write a line of pipeline code.

Naive chunking strategies

Splitting documents into fixed 500-token chunks with no overlap is the default, and it's wrong for most corpora. Chunks that cut a sentence in half produce embeddings that don't match how users ask questions. The fix is simple: chunk by semantic boundary, not by token count. Respect paragraph breaks. Keep headings with the text they introduce. Test a few chunk sizes against real queries before committing.

Skipping evaluation

Builders ship a pipeline, eyeball three answers, and call it done. You can't tune what you don't measure. RAGAS and DeepEval exist for this. Run them early, even on a small query set. A retrieval pipeline with no evaluation is a guess with extra steps.

Underestimating orchestration complexity

LangChain makes the demo easy and the production version hard. Every abstraction hides a decision you'll need to undo later. If your pipeline is a straight line, write it yourself. Reach for a framework only when you need routing, branching, or agent loops.

Ignoring agent memory requirements

Stateless retrieval works for one-shot Q&A. It fails the moment an agent needs to remember what a user said three turns ago. If your use case involves multi-turn conversations, design for stateful memory from the start. Retrofitting it is painful.

Final Thoughts on the Open Source RAG Stack

The open source RAG stack is not a product. It's a set of decisions you make and own. That's the trade: no vendor lock-in, no per-token markup, and no one to blame when retrieval quality drops.

The honest assessment hasn't changed across this guide. Ingestion, embedding, and vector storage are solved problems for most use cases. Orchestration is where complexity hides. Evaluation is where most teams skip the work. Agent memory is where the open source defaults still fall short.

You can build a working RAG pipeline in an afternoon. You cannot build a production-grade one without evaluation, tuning, and a clear answer to the statefulness question. Those are different projects.

If you're building agent memory or a pipeline that needs persistent state across turns, GigaRAG is built for that specific gap. It handles the memory layer so you don't have to bolt it onto a stateless retrieval stack. For everything else, the open source RAG stack will serve you fine, as long as you're honest about what it won't do for you.

Frequently Asked Questions

What kind of database does RAG use?

RAG typically uses a vector database to store document embeddings and support similarity search. Common open source choices include Chroma, Qdrant, Weaviate, Milvus, and pgvector. Some stacks also use a traditional database for metadata and a separate store for agent memory.

What is the biggest open source project?

There is no single 'biggest' open source RAG project — the ecosystem is fragmented across orchestration frameworks, vector stores, and embedding models. LangChain and LlamaIndex are among the most widely used orchestration libraries, while Chroma and Qdrant are popular vector stores. The right choice depends on your latency, scale, and memory requirements.

Which AI is open-source and free?

Many open source models are free to use under permissive licenses, including Llama, Mistral, and Qwen families, as well as embedding models like sentence-transformers. 'Free' refers to the license, not the compute — you still pay for the hardware or cloud instance that runs them.

How can I build a simple RAG application?

Start with a minimal stack: Python, sentence-transformers for embeddings, Chroma as the vector store, and a simple retrieval loop that passes top-k chunks to an LLM. You do not need a framework to begin. Add orchestration and agent memory only when the simple version works.

Can I use a vector database as agent memory?

You can store embedded memories in a vector database, but it is not a complete memory system. Agent memory needs structured state, timestamps, and the ability to replay or summarize past interactions. A vector store handles semantic recall but not the full lifecycle of agent state.

What should I not expect from an open source RAG stack?

Do not expect a single framework to handle ingestion, retrieval, orchestration, and agent memory well out of the box. Most stacks require you to assemble components and write glue code. Also, do not expect open source tools to match managed services on operational polish without extra work.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through the open source rag stack or something adjacent, we publish what we have actually tested, including where it falls short.