GraphRAG with Neo4j: Honest Guide for RAG Builders

GraphRAG with Neo4j: What RAG Pipeline Builders Need to Know

GraphRAG with Neo4j gets hyped as the fix for every RAG pipeline builder's memory problem. The honest answer is more boring: it solves one specific failure, and it costs you real engineering time to get there. Vector-only retrieval stores chunks as floating points in space. It cannot tell you that a customer's ticket is related to the same account as three earlier tickets, or that a policy document amends another one written six months ago. GraphRAG adds a knowledge graph layer so your retriever can walk relationships instead of guessing from cosine similarity. That matters for agent memory, where persistence and multi-hop recall are the whole job. GigaRAG (https://www.gigarag.com) is built for exactly this use case. This guide covers what GraphRAG with Neo4j actually delivers, where it falls short, what it costs, and the mistakes that will sink your pipeline before you ship.

In This Guide

• What Is GraphRAG and How Does It Work with Neo4j?
• GraphRAG with Neo4j vs Vector-Only RAG
• Why Vector-Only RAG Fails for Agent Memory
• Graphrag With Neo4j: A Step-by-Step Guide
• Setting Up GraphRAG with Neo4j: A Practical Walkthrough
• GraphRAG Retrievers: Which One Fits Your Pipeline?
• What GraphRAG with Neo4j Cannot Do (Honest Limitations)
• Is GraphRAG More Expensive? Cost Realities for RAG Pipelines
• GraphRAG vs. Neo4j: Clarifying the Confusion
• Common Mistakes When Building GraphRAG with Neo4j
• GigaRAG: A Practical Alternative for Agent Memory Builders

What Is GraphRAG and How Does It Work with Neo4j?

GraphRAG is retrieval-augmented generation that pulls context from a knowledge graph instead of only a vector index. Neo4j stores the graph; GraphRAG is the retrieval method that queries it.

From vector-only RAG to graph-aware retrieval

Vector RAG finds passages by similarity. It matches your query to embeddings, then hands the LLM whatever chunks score highest. That works for lookup questions. It breaks when the answer depends on how things connect: who reports to whom, what entity owns what asset, which events follow which.

GraphRAG adds structure. Your data becomes nodes and relationships, not just text chunks. A query can follow a path across multiple hops instead of stopping at the nearest match. The context you send the LLM includes the relationships themselves, not just the words.

How Neo4j stores and queries knowledge graphs

Neo4j is a graph database. Nodes hold entities: people, documents, products, events. Relationships hold typed connections between them: WORKS_FOR, CITED_BY, OWNS. Properties attach to both.

Queries run in Cypher, a pattern-matching language. You describe the shape of what you want, and Neo4j returns the matching subgraph. That subgraph becomes your retrieval context. Vector indexes can live alongside the graph, so you can mix similarity search with traversal in one query.

The core GraphRAG retrieval loop

You embed the query. You find a starting node, either by vector similarity or an entity lookup. You traverse from there, following relevant relationships one or more hops. You collect the nodes and edges you touched. You format that subgraph as text or structured context. You send it to the LLM with the original question.

The difference from vector RAG is the middle step: traversal. You're not just retrieving similar text. You're walking a structure that encodes how your data actually relates.

[!note] GraphRAG is not a drop-in replacement for vector RAG: it adds an extraction and graph-maintenance layer, so it pays off mainly when queries genuinely require multi-hop or relational reasoning.

GraphRAG with Neo4j vs Vector-Only RAG

Why Vector-Only RAG Fails for Agent Memory

Vector search treats every chunk as an island. It finds passages that look like your query, but it doesn't know that one chunk references another, or that an entity in chunk A is the same entity in chunk B. For agent memory, that's a structural failure.

What vector search misses: relationships and structure

An agent needs to recall that a user said "I prefer Python" in one conversation and "my team uses Go" in another. Vector search stores both statements. It can't connect them. The relationship between preference and constraint is invisible to an embedding. You get fragments, not a memory.

Agent memory requirements: persistence, context, and recall

Agent memory needs three things. Persistence: facts survive across sessions. Context: facts stay linked to who said them, when, and why. Recall: the right facts come back when a new query arrives. Vector-only RAG handles persistence poorly and context not at all. Recall degrades as the corpus grows because similarity scores flatten.

Where graphs outperform embeddings

A graph stores the connection as a first-class object. "User prefers Python" and "team uses Go" become two nodes with a typed edge between them. Traversal finds the conflict in one hop. Embeddings can't do that. The graph doesn't replace embeddings. It adds the layer vector search was never designed to provide.

[!tip] For agent memory, start with a narrow schema covering only the entity types your agent must remember, and expand it only when retrieval evaluations show a concrete gap.

Graphrag With Neo4j: A Step-by-Step Guide

1. Define the entity and relationship schema your agent memory actually needs.
2. Extract entities and relations from source documents with an LLM pipeline.
3. Load nodes and edges into Neo4j and create constraints on entity keys.
4. Generate embeddings for nodes or chunks and store them in a Neo4j vector index.
5. Combine vector search with Cypher traversal to retrieve multi-hop context.
6. Feed retrieved subgraphs into your LLM prompt and evaluate answer quality.
7. Monitor graph growth and refresh stale entities on a schedule.

Setting Up GraphRAG with Neo4j: A Practical Walkthrough

The neo4j-graphrag-python package is the official starting point. It wraps graph construction and retrieval into a small set of Python classes, so you don't write raw Cypher for every query. You'll still need a running Neo4j instance, either local or on Aura.

Installing neo4j-graphrag-python

Install with pip. The package pulls in the Neo4j Python driver and a few dependencies for embedding and LLM integration. You'll also need an embedding model, either OpenAI's or a local one via sentence-transformers, and an LLM for answer generation. The package doesn't bundle either. That's on you.

Building a simple knowledge graph from your data

Start with a list of documents or chunks. The package provides a pipeline that extracts entities and relationships using an LLM, then writes them to Neo4j as nodes and edges. You define the node labels and relationship types upfront. A biomedical example from the repo extracts entities like genes and diseases, then links them with relationships like "treats" or "causes." The extraction step costs tokens. Every document goes through the LLM once, and the output is structured JSON that the pipeline validates before writing.

Running your first GraphRAG retrieval query

Once the graph is built, you create a retriever object. The simplest one runs a vector search over node embeddings, then expands to neighbouring nodes in the graph. You pass a query, the retriever returns a subgraph, and you feed that subgraph to your LLM as context. The retrieval is two steps: find the entry node by similarity, then traverse one or two hops to pull related nodes. That's the whole loop. The package handles the Cypher generation, but you can inspect the queries it builds if you want to tune them.

GraphRAG Retrievers: Which One Fits Your Pipeline?

The retriever is where GraphRAG lives or dies. Pick the wrong one and you get slow queries, bloated context, or answers that miss the relationships you built the graph to capture. There are three main types, and each suits a different query pattern.

Vector retrievers: fast but shallow

Vector retrievers find an entry node by embedding similarity, then stop. They're fast because they skip traversal entirely. You get the node that matches your query, maybe its immediate neighbours, and that's it. For single-hop questions like "what is this entity" or "find documents about X," that's enough. For agent memory, vector retrieval works when the agent needs a fact, not a chain of reasoning.

The catch: vector retrievers don't follow relationships. If your query requires connecting two entities through an intermediate node, a vector retriever will miss it. You'll get the right starting point and none of the path.

Graph traversal retrievers: deep but slower

Traversal retrievers start from a node and walk the graph along relationship edges. They can follow multi-hop paths, pull in related entities, and reconstruct the context a vector search can't see. For agent memory, this is where GraphRAG earns its keep. An agent asking "who worked with the person who authored this document" needs traversal, not similarity.

The cost is latency. Each hop adds query time, and the subgraph can grow fast. You'll also burn more tokens feeding a larger context to the LLM. Traversal retrievers need careful hop limits or they return too much.

Hybrid retrievers: balancing speed and depth

Hybrid retrievers combine both: vector search to find the entry point, then limited traversal to expand context. Most production GraphRAG pipelines land here. You get the speed of vector search on the front end and the relational depth of traversal on the back end, with a hop limit keeping the subgraph manageable.

What GraphRAG with Neo4j Cannot Do (Honest Limitations)

GraphRAG with Neo4j is not a magic fix. It won't turn bad data into good answers, and it won't replace the fundamentals of retrieval design. It adds a graph layer on top of your existing pipeline, which means it adds work. Here's what that work looks like.

Graph construction is still hard and manual

The graph doesn't build itself. You have to extract entities, resolve them to unique nodes, define relationship types, and write the Cypher queries that load everything into Neo4j. That's before you run a single retrieval query.

Entity resolution is the part most people underestimate. Two documents mentioning "Apple" might mean the company, the fruit, or a person's surname. Your pipeline has to disambiguate these, or your graph fills with duplicate nodes and broken relationships. LLMs can help with extraction, but they make mistakes, and you'll need a review step. There's no shortcut here.

Performance trade-offs at scale

Graph traversal gets slower as your graph grows. A two-hop query on a graph with 10,000 nodes is fast. The same query on a graph with 10 million nodes can time out if your indexes aren't right or your relationship types are too broad. You'll spend real time on query tuning.

Memory is another cost. Neo4j holds the graph in RAM for fast traversal. Large graphs mean large memory bills. You can use Neo4j Aura or another managed service, but the cost scales with graph size, not query volume.

What GraphRAG won't fix: bad embeddings, poor chunking, weak prompts

GraphRAG sits on top of your existing RAG stack. If your embeddings are weak, vector search will still return the wrong entry nodes, and traversal will walk out from the wrong starting point. If your chunking splits documents badly, the entities you extract will be fragmented and your graph will reflect that mess.

Weak prompts produce weak answers no matter how good your retrieval is. GraphRAG can hand the LLM a perfect subgraph, but if the prompt doesn't tell the model how to use that context, you'll still get a mediocre response. The graph improves retrieval. It doesn't improve generation.

Is GraphRAG More Expensive? Cost Realities for RAG Pipelines

Yes, GraphRAG costs more than vector-only RAG. The honest answer is that you're paying for graph construction, storage, and traversal on top of your existing embedding and LLM costs. Whether that's worth it depends on your query patterns.

Where GraphRAG costs more than vector RAG

Vector-only RAG has two main costs: embedding your documents and storing vectors. GraphRAG adds three more. You pay for entity extraction and resolution during graph construction, you pay for Neo4j storage (RAM-heavy, priced by graph size), and you pay for traversal queries at runtime. The construction cost is one-time per document. The storage and traversal costs are ongoing.

Token usage in graph traversal queries

Graph traversal itself doesn't burn LLM tokens. The Cypher query runs against Neo4j, not the LLM. But the subgraph you retrieve gets passed into the LLM's context window, and that's where tokens add up. A multi-hop query that pulls 50 nodes and 80 relationships can easily produce 3,000 to 5,000 tokens of context before you've added the user's question. Vector-only RAG typically passes 5 to 10 chunks, often 1,500 to 3,000 tokens. So GraphRAG queries can double your per-query LLM cost.

When the cost is worth it

It depends on what you're asking. If your queries need multi-hop reasoning across entities, relationships, and documents, the extra cost buys you answers vector-only RAG can't produce. If your queries are simple semantic lookups, you're paying for a graph you don't need. Start with vector-only RAG. Add GraphRAG when you can point to specific queries that fail without it.

GraphRAG vs. Neo4j: Clarifying the Confusion

GraphRAG and Neo4j aren't competitors. You don't choose one over the other. GraphRAG is a retrieval method. Neo4j is a database. The confusion comes from seeing both names in the same tutorials and assuming they solve the same problem.

GraphRAG is a method, Neo4j is a database

GraphRAG describes how you retrieve context: build a knowledge graph, traverse it to find related entities, and feed that subgraph to an LLM. It's a technique, like vector search or BM25. Neo4j is where the graph lives. It's a graph database that stores nodes, relationships, and properties, and runs Cypher queries against them. You can run GraphRAG on other graph stores. You can use Neo4j for plenty of things that aren't GraphRAG.

How they work together in a RAG pipeline

In practice, Neo4j is the storage and query layer. GraphRAG is the retrieval logic sitting on top. Your pipeline extracts entities and relationships from documents, writes them to Neo4j, then at query time runs a traversal to pull the relevant subgraph. That subgraph becomes context for the LLM. Neo4j handles the graph mechanics. GraphRAG handles the retrieval strategy.

Alternatives to Neo4j for GraphRAG

Neo4j is the most common choice, but it's not the only one. You can run GraphRAG on Amazon Neptune, Memgraph, or even a property graph built in Postgres with Apache AGE. The trade-off is ecosystem maturity. Neo4j has the most GraphRAG tooling and documentation. The others work, but you'll build more yourself.

Common Mistakes When Building GraphRAG with Neo4j

Most GraphRAG failures aren't the database's fault. They're design mistakes made before the first query runs. Here are the three that show up most in production pipelines.

Over-engineering the graph schema

You don't need 40 node labels and 60 relationship types on day one. Start with the entities your queries actually touch. A schema with 8 labels and 12 relationship types covers most agent memory use cases. Every extra label adds construction complexity and query latency. Add structure only when a real query pattern demands it.

Ignoring graph construction quality

The graph is only as good as the extraction that built it. If your entity resolution merges "Apple the company" with "apple the fruit," every downstream traversal returns noise. Spend time on deduplication rules and relationship validation. A small, clean graph beats a large, messy one.

Using graph traversal when vector search is enough

Not every query needs multi-hop reasoning. If the answer lives in a single document chunk, vector search finds it faster and cheaper. Reserve graph traversal for questions that require connecting entities across documents. Run both retrievers in parallel, then decide which result to use based on the query type.

GigaRAG: A Practical Alternative for Agent Memory Builders

If the last few sections made graph construction sound like work, that's because it is. GigaRAG exists for builders who want the relational recall of graphrag with neo4j without hand-rolling entity resolution, schema design, and retriever tuning. It handles graph construction and agent memory plumbing so you can focus on the queries your agents actually need to answer.

The honest trade-off: you give up fine-grained control over the graph layer. For teams that need custom Cypher patterns or unusual traversal logic, raw Neo4j remains the better fit. For everyone else, GigaRAG gets you from documents to working agent memory faster than assembling the pieces yourself.

Frequently Asked Questions

What are the limitations of GraphRAG?

GraphRAG struggles with vague or open-ended queries where no clear entity or relationship exists to traverse. It also adds extraction cost, schema design effort, and graph maintenance overhead. For simple single-fact lookups, vector-only RAG is often faster and cheaper.

What are the key differences between GraphRAG and Neo4j?

GraphRAG is a retrieval architecture that combines knowledge graphs with vector search and LLMs. Neo4j is a graph database that can serve as the storage and traversal layer inside a GraphRAG pipeline. They are complementary, not competing: GraphRAG is the pattern, Neo4j is one way to implement it.

Is graph RAG more expensive?

Typically yes, because you pay for LLM-based entity and relation extraction, plus ongoing graph updates. Vector-only RAG mainly costs embedding generation and index storage. The trade-off is worth it when multi-hop reasoning materially improves answer quality.

Does NASA use Neo4j?

Neo4j has published case studies describing use in aerospace and engineering contexts, but you should verify any specific NASA deployment claim against primary sources before citing it. Treat vendor case studies as directional, not definitive.

Can Neo4j GraphRAG work for agent memory?

Yes, it is a natural fit because agent memory is inherently relational: entities, events, and their connections over time. Storing memory as a graph lets an agent traverse prior context instead of relying only on embedding similarity. The main caveat is that you must design the memory schema deliberately.

Do I need a knowledge graph for every RAG pipeline?

1. If your queries are mostly single-fact lookups over a modest corpus, vector-only RAG is simpler and sufficient. Add a graph layer when you observe failures on multi-hop, relational, or temporal questions.

How do I get started with Neo4j GraphRAG?

Begin with a small, well-defined schema and a representative document set. Use the official Neo4j GraphRAG documentation and Python package to build a minimal pipeline, then evaluate retrieval quality before scaling. Iterate on the schema based on real query failures.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through graphrag with neo4j or something adjacent, we publish what we have actually tested, including where it falls short.