AI Memory Benchmark: GigaRAG vs Mem0, Cognee and Graphiti

What was in memory. Three open-access books published in 2025 and 2026, recent enough that an LLM is unlikely to have read them (165,000 words in all):

• Diplomacy on Time: Denmark in South Africa’s Negotiated Revolution, Peter Brückner (UJ Press, 2025)
• The Politics of Nuclear Energy in Africa, edited by Eben Coetzee (UJ Press, 2025)
• Plant Molecular Genetics and Bioengineering, edited by Walter P. Suza (Iowa State University Digital Press, 2026)

And two long chat histories from LoCoMo, a public benchmark for conversational memory: two pairs of friends talking over 19 sessions each (788 turns, 18,000 words).

The questions. 474 in all, 238 on the books and 236 on the chats. 58 of them (12%) have no answer in memory, to see whether a system admits it. The rest test one skill each: multi-step (123), fact lookup (94), time and order (79), counting and lists (64), facts that changed (32) and preferences (24). Each question is worded one of three ways: in the source’s own words (173), paraphrased (158), or sharing no words with the source except names (143).

How we kept it fair.

• A separate AI reviewer checked every question against its source. Of 544 drafts, 161 were fixed, 22 dropped and 2 duplicates removed.
• Any question the agent’s LLM could answer without memory (in two of three tries) was removed. Two models read the whole source to confirm each “not in memory” answer really is absent.
• A seeded random draw picked the final set, blind to any results.
• Every system got the same text, agent LLM, judge and server, and only its own read tools, described as in its own MCP server.
• The judge never sees which system wrote an answer.
• GigaRAG was developed on earlier question sets from these same sources; the others weren’t tuned for this test. Every question in this set is new. We changed GigaRAG once after a first run on it, which scored 96.3%; this run scored 95.9%.
• Everything is published in the question bank: every question, every system’s answer, the judge’s grade and reason, and every tool call, with CSV and JSON exports.

For every question, the same agent LLM (DeepSeek Flash) searched the system’s memory and wrote an answer. A separate judge LLM (gpt-4.1-mini) compared that answer’s meaning with a gold answer.

A correct answer scores 1, a partly correct one 0.5, and a wrong or missing one 0. Books and chats count equally. The agent could reply up to 12 times per question, each reply either a search or the final answer; we call each reply a round. Where we say we hand-checked grades, we read the answers ourselves; the headline scores always use the judge’s grades.

Is the lead real? We re-drew the 416 questions at random many times over (a bootstrap) to see how far each gap moves by chance. GigaRAG leads Cognee by 4.7 points (95% range: 2.0 to 7.5), Mem0 by 6.1 (3.1 to 9.3) and Graphiti by 17.5 (13.7 to 21.5). No range reaches zero, so no lead is down to luck.

GigaRAG is best on all six answerable skills. Counting and lists is the closest: 87.5% against Cognee’s 86.7%. About half of our misses there leave out items that appear in a different session or section. On preferences GigaRAG got all 24 right.

The last row is the one GigaRAG doesn’t win: questions whose answer isn’t in memory. Its own section, “Not in memory” questions, explains why, and shows the gap closing once the agent is given more rounds.

Every system does better on the books. GigaRAG loses the least on the chats: 95.2%, against 75.7% to 86.5% for the others.

28 of the chat questions are traps. Each asks about the wrong person: about Melanie for something Caroline did, or about Jon for something Gina did. They follow LoCoMo’s adversarial category.

The judge accepts two answers here: a bare “Not in memory”, or a correction such as “that was Melanie’s son, not Caroline’s”. To a user they are not equal. The correction answers what they meant to ask; the bare reply suggests the event never happened. So we read every answer and counted the corrections.

Every correction counted was checked against the conversation, and every one is true; no system gave a false correction. On 9 questions GigaRAG was the only system to correct the premise:

Once, the reverse happened: on pv3l0137 Graphiti named the right person and GigaRAG didn’t.

The three wording groups are different questions, 126 to 151 answerable ones each.

GigaRAG stays at 96% whatever the wording. Cognee and Mem0 slip as the question moves away from the source’s words. With no shared words, GigaRAG leads them by about 9 to 10 points, and Graphiti by 22.

Most likely, it comes down to what each system keeps when it stores text.

• GigaRAG keeps the full text of every section and chat session, adds a short summary, and searches by meaning.
• Mem0 keeps only short facts an LLM pulled out when storing. Anything left out can’t be found later.
• Graphiti searches the entities and relations it extracted; the raw text is stored but not searched.
• Cognee’s tool mostly returns a response written by a second LLM, not the source text.

58 questions have no answer in memory. The right reply is to say so. A system can fail one in two ways:

• Inventing an answer, stating a fact the source doesn’t have. This is a hallucination.
• Never answering: the agent keeps searching until it runs out of rounds. Not a hallucination, but the user gets nothing.

With the standard 12 rounds, GigaRAG ran out on 6 of these questions, and that is the whole of its gap. So we ran all 58 again, for every system, with 30 rounds. In both runs we read every answer the judge marked wrong (except empty ones). We didn’t re-read the ones it accepted, so the hand check can only raise a score.

• Every system gets 57 of 58 after the hand check, and none ran out of rounds. Every system was re-run on all 58, not just its misses, so the answers are a fresh sample.
• The six questions GigaRAG ran out on are all answered correctly, after 5 to 28 rounds.
• GigaRAG’s one miss (pq0237) misread a citation. Asked for the reference number of a 1955 US paper, it gave the book’s citation for that paper, “Drogan 2016:972”: real, but not the number asked for.
• The only invented fact is Cognee’s “China National Nuclear Corporation” on pq0228, the same as in the first run. Mem0 and Graphiti each went along with one false premise, using a true fact from elsewhere.
• The judge again marked a true GigaRAG answer wrong(pv3t0001, which said memory records no specific provisions).
• The cost is rounds: 5.8 per question for GigaRAG, against 2.7 to 4.7 for the others. Making GigaRAG stop sooner here is our next fix.

GigaRAG and Mem0 invented nothing; Graphiti invented one fact, Cognee two. Mem0’s two misses went along with the false premise using a true fact: on pl0027 it said Caroline took up the piano, which she did, but nothing links that to the drive.

The judge marked 9 of GigaRAG’s 58 answers wrong. Reading them:

• 2 were correct premise corrections (pv3l0011, pv3l0132);
• 1 was an ambiguous question (pv3m0295), asking for the middle name of “the guy who made Figure 7” in a book with eight Figure 7s. GigaRAG gave a name the book does credit;
• 6 ran out of rounds, on questions where the topic is in memory but the detail asked for isn’t.

That leaves GigaRAG 1 to 2 questions behind the others at 12 rounds, all of it from the six that ran out, which the 30-round run closes.

Tokens count everything sent to or produced by any LLM while answering. In every setup the agent LLM reads the search results and writes the answer. Cognee adds a step: on about 71% of its tool calls, a second LLM inside Cognee’s tool writes a response from what it found, and the agent reads that instead of the source. Those tokens are real spend, so they are counted.

On a typical question GigaRAG uses more tokens than Mem0 and Graphiti (24.7k against 2.7k and 12.1k), and fewer than Cognee once its second LLM is counted (about 38k).

The mean tells a different story because a few very long questions pull it up. A tenth of Graphiti’s questions use more than 177k tokens each, which puts its mean (52.7k) just above GigaRAG’s (51.6k), even though GigaRAG uses more tokens than Graphiti on 336 of 474 questions.

Why GigaRAG’s count is high. Every agent reply re-sends the conversation so far, earlier search results included. A GigaRAG search shows the agent about 17,300 characters of source text, and reading a memo returns all of it. A Mem0 search returns about 2,600 characters of short facts. That is where GigaRAG’s extra detail comes from, and where its extra tokens go.

Cost is what DeepSeek bills for those tokens at its published per-token prices for DeepSeek Flash, the model behind every LLM call while answering, Cognee’s second LLM included. It covers input, cached input and output. It leaves out the judge, ingesting the sources and embeddings, and it will differ with another model. The mean sets the total bill, so that is what is charted here.

Most of GigaRAG’s 8.6 seconds (82%) is the agent LLM writing its replies, so fewer rounds would cut time and tokens together. The search itself takes 305 ms per call on average, with the query embedded on the server’s own CPU. Mem0’s takes 335 ms, calling OpenAI’s embedding API, and Graphiti’s 1.5 s.

Mem0 is faster per question. Its store takes one call at a time, and we left out the time its calls spent queued behind other questions running in parallel (3.2 s per question). Counted in, Mem0 averages about 9.3 s.

• Mem0 is faster and much cheaper: 6.1 s per question against our 8.6 s, at about a quarter of the cost (see Tokens and cost).
• Mem0 and Graphiti use fewer tokens on a typical question: 2.7k and 12.1k against our 24.7k. GigaRAG shows the agent full passages; Mem0 returns short facts.
• Cognee and Mem0 take fewer rounds: 1.8 and 2.0 against our 2.7. On unanswerable questions the gap is wider: 2.7 and 3.8 against our 5.8.
• At 12 rounds, the others get 1 to 2 more unanswerable questions right after the hand check: 54, 54 and 53 of 58 against our 52. Every one of GigaRAG’s remaining misses there is the agent still searching when the rounds run out.
• Every system beat us on some questions. On answerable questions, Mem0 was fully right where GigaRAG wasn’t on 14, Cognee on 13 and Graphiti on 11. The other way round: 47, 43 and 98.

• We built GigaRAG and ran this benchmark, so it isn’t independent. That is why every question and answer is published.
• One run per system. Gaps under about 2 points are within run-to-run noise.
• All four systems got 62% of the questions (292 of 474) fully right, so the ranking rests on the rest.
• It compares memory products, not a plain retrieval baseline.
• The corpus is small and never changes. It doesn’t test scale, changing facts, or memory written during a conversation, which is where GigaRAG’s structure (buckets, threads, links, fact versions) is meant to matter.
• The judge is an LLM and grades premise corrections inconsistently (see False premises).
• One question (pv3m0295) is ambiguous, and one (pv3g0024), missed by every system, needs its gold answer checked. Both will be fixed in the next edition.

• Scale: the same questions with 100 or more similar books and chats added as distractors.
• Facts that change over time.
• The full LoCoMo benchmark: 10 conversations with its own questions.
• An agent writing its own memory during a long conversation and recalling it later.