Caching for AI agents and RAG: when and what to store
Knowing when to cache, and what to cache, isn't mainly a speed question. A cache reuses prior work, but reuse is appropriate only when the context and state relevant to that artifact still match.
Keep shared project knowledge separate from cached outputs. GigaRAG helps teams keep project knowledge available across sessions and tools, but availability alone does not make an earlier output reusable for a stateful task, live tool result, personalized request, or write operation.
It depends on the artifact's repeatability, the data and configuration behind it, and the harm a stale result could cause. Don't copy a universal TTL or similarity threshold. Instead, use keys, version fields, invalidation triggers, and safety gates for embeddings, vector-search results, LLM responses, and intermediate retrieval artifacts.
Quick answer
Start by caching deterministic artifacts whose dependencies are easy to identify, such as embeddings, parses, and chunks. Add retrieval or response reuse only after defining the key, freshness policy, and cases that must bypass the cache.
- Start with reusable embeddings and retrieval artifacts, scoped to the data and configuration they depend on.
- Cache generated answers only when user, conversation, permissions, prompt, and model context are accounted for.
- Treat live tool results and state-changing actions as fresh operations, not safe semantic-cache hits.
When to cache and what to cache: a decision table
Cache computed artifacts only when the same identity, permissions, data version, model setup, and task scope still apply. Don't cache live reads, writes, or ambiguous agent steps just because a similar request appeared before.
The four tests: repeatability, cost, freshness tolerance, and blast radius
Use four tests. Repeatability asks whether the same inputs should yield the same artifact, while cost asks whether avoiding the work is worth the cache lookup.
Freshness tolerance asks how long an old result remains acceptable. Blast radius asks who could be harmed by a wrong hit. A cheap miss is often safer.
Decision table: artifact, safe-to-cache conditions, key fields, expiration approach, and invalidation trigger
| Artifact | Cache when | Key fields | Expiration and invalidation |
|---|---|---|---|
| Embeddings | Normalized text is unchanged | Text hash, embedding model, preprocessing version | Purge on model or preprocessing release |
| Vector results | Corpus and access scope match | Query vector, index version, filters, tenant, permission scope | Short TTL, purge on ingestion or access change |
| LLM responses | Question is narrow, read-only, and context matches | Prompt, model, prompt version, tenant, retrieval config | Short TTL, purge on context release |
| Retrieval artifacts | Parse, chunk, or rerank inputs are stable | Document version, chunker or reranker version, scope | Purge on document or configuration change |
What should not be cached: state-changing tool calls, live operational data, authorization-sensitive outputs, and ambiguous agent actions
Don't reuse tool calls that change state, live prices or inventory, user-specific outputs, or actions whose meaning depends on an agent's current plan. Semantic caching can match queries expressed in different words, but a similar query can still need a different answer when context or state differs.
What should you cache in an AI agent or RAG pipeline?
| Factor | Reuse when | Recompute when |
|---|---|---|
| Embeddings | Source content and embedding configuration are unchanged. | Content or embedding configuration changes. |
| Vector-search results | Query, corpus state, permissions, and retrieval configuration still match. | Documents, access, or retrieval settings change. |
| LLM responses | Request context and answer validity are tightly constrained. | State, personalization, live information, or actions change the request. |
| Intermediate retrieval artifacts | Inputs and dependencies are unchanged and identifiable. | An upstream artifact or its configuration changes. |
How AI agent caching works: keys, layers, and strategies
A cache isn't one layer. You can reuse identical requests, shared prompt text, similar questions, or computed pipeline artifacts, but each choice has a different validity test.
[!note] A semantic similarity match shows that two queries are similar, not that they have identical context or need the same answer. For agents, differing context or state can make a close match unsafe to reuse.
Give each layer its own key and validity policy. An identical request, a shared prompt prefix, a semantically similar query, and a pipeline artifact rely on different conditions for safe reuse.
Exact-match, prefix/prompt, and semantic caches solve different problems
Exact-match caching uses an input string as its key and reuses a response only when that input is identical. It works well for structured data, where the same request is likely to recur unchanged.
Prompt caching reuses identical prompt prefixes, such as a long system prompt. Semantic caching embeds a query and looks for a close prior query, so it can match different wording. However, its threshold and embedding quality affect whether the reused answer fits. These layers can work together.
Cache-aside and write-through: where each pattern fits
With cache-aside, your app checks the cache, reads the source on a miss, then stores the result. The first request can therefore be slow, while a later matching request can use the stored result.
Write-through updates the cache alongside the source of truth. It can be useful where stale reads are unacceptable, but it adds work to every write and may store entries that are never read. Consider whether that trade-off fits the artifact before using it for agent workloads.
Build a safe cache key: tenant + resource version + configuration + request scope
Start keys with the boundary that must never cross: tenant:permission-scope. Then add the resource version, model or embedding version, prompt or retrieval configuration, and normalized request scope.
For a coding agent, include repository ID, commit SHA, path scope, and agent configuration when results depend on a particular code revision. These fields keep code-dependent results scoped to the revision and configuration used to produce them.
TTL versus event-driven invalidation versus versioned keys
TTL sets a maximum retention period for a cached value.
When inputs have reliable versions, include those versions in the key. Invalidate affected entries when document ingestion, permissions, or repository changes alter validity. In practice, use versions to scope reuse, events to remove affected entries after a known change, and TTL as a backstop.
Step-by-step: add cache guardrails to an agent
- List the artifacts your pipeline creates, from embeddings through final responses.
- Identify every input that can change whether each artifact remains valid.
- Build keys with relevant tenant, user or permission scope, repository and branch or commit, model and prompt versions, and retrieval configuration. For agents connected through GigaRAG, apply the same scope to the knowledge each agent is allowed to use.
- Set expiration or event-based invalidation rules based on each artifact's dependencies.
- Bypass response reuse for state-dependent tasks, live tool results, personalized requests, and write operations unless you explicitly check validity.
- Monitor hits and misses alongside stale or incorrect reuse, then adjust the policy.
What to cache in a RAG pipeline and coding agent
[!tip] For a coding agent, include the repository and branch or commit in keys for code-dependent results. For a multi-turn RAG app, include tenant, permissions, conversation context, and retrieval configuration when they affect the result.
Embeddings: cache deterministic input and include embedding model and preprocessing versions
Cache an embedding when the normalized input is unchanged. Key it with a content hash, embedding model version, and preprocessing version. Re-embed when any of those fields change.
Vector-search results: key by query representation, index or corpus version, filters, tenant, and permissions
A retrieval hit is safe only inside the same access boundary. Include the query embedding or normalized query, index or corpus version, metadata filters, tenant, permission scope, and retrieval settings such as top-k.
Don't share result lists across users just because their queries match. A permission change makes the hit unsafe.
LLM responses: reserve semantic reuse for bounded, low-risk questions with compatible context
Cache exact LLM responses when the prompt, model, system instructions, retrieved context, and user scope all match. Reserve semantic reuse for bounded, low-risk questions with a clearly defined validity policy.
The VectorQ paper, Adaptive Semantic Prompt Caching with VectorQ, by Luis Gaspar Schroeder and coauthors reports that a one-size-fits-all static threshold is insufficient across different embeddings. Keep semantic caching to bounded, low-risk questions rather than agent decisions or write requests.
Intermediate retrieval artifacts: chunking outputs, document parses, reranker results, and tool-read snapshots
Parsed documents and chunk outputs are good candidates when source bytes and parser or chunking versions are in the key. Cache reranker results with the candidate set, reranker model, and query representation.
Tool-read snapshots are different. Cache them only when the tool data has a known version or freshness bound.
Example: repository-aware coding agent keys using repo ID, commit SHA, branch, path scope, and agent configuration
A repository retrieval key might look like:
repo_id:commit_sha:branch:path_scope:permission_scope:embed_model:chunk_version:retrieval_config
Use the commit SHA as the validity anchor when a result depends on a specific repository revision. Include the branch as additional scope when it is useful to identify the workstream.
When caching hurts correctness, freshness, or performance
Why semantically similar prompts can need different answers in multi-turn and stateful agent workflows
Similarity measures aspects of wording and meaning, not the full task state. In a stateful agent workflow, a similar request can have different context or state, making reuse of an earlier answer unsafe.
Do not reuse semantic answers for agent actions, live reads, or conversations where earlier turns change the request. These tasks need a current validity check rather than a similarity match alone.
Freshness failures when documents or operational state change
Documents, permissions, tool data, and operational state can change after an entry is stored. For these dependencies, identify the change source and ensure affected retrieval results or responses are no longer selected after the change.
Use the document, index, permission, or tool-data change to target affected entries. Versioned keys prevent reuse against a changed dependency, while invalidation events remove entries promptly; TTL remains a retention backstop.
Performance failures: low reuse, expensive cache lookups, oversized values, and unnecessary write-through caching
Caching adds work too. A semantic cache introduces embedding generation and vector search in addition to the cache lookup, so measure that overhead against the reuse your workload actually achieves.
Write-through caching can slow writes and store entries that are never read.
Common mistakes: incomplete cache keys and weak invalidation controls
Cache keys should capture the versions, scope, and configuration that determine whether an artifact remains valid. When a known change affects those dependencies, use the corresponding version or invalidation path so the old result is not selected.
How to treat claimed latency or cost improvements as workload-specific measurements, not default expectations
The honest answer is: it depends on request repetition, hit quality, lookup cost, and the miss path. Measure results on the workload you intend to run. In a LinkedIn post, Shaher Emad notes that LangCache's advertised “up to” savings depend on workload and cache-hit rate and are not guaranteed.
Put caching into production with guardrails
Start with deterministic artifacts, then test retrieval and response reuse behind a feature flag
Start with embeddings, parses, and chunk outputs. Then put retrieval and response reuse behind a feature flag, where you can compare hits with fresh results before a broader release. If agents use GigaRAG for shared project knowledge, keep that knowledge layer separate from the policy that decides whether an output may be reused.
Monitor hit rate, miss latency, cache lookup overhead, stale-result reports, semantic-hit acceptance, invalidation lag, and cost per successful task
Do not evaluate a cache on hit rate alone. Track lookup overhead, miss latency, rejected semantic hits, stale-result reports, invalidation lag, and cost per successful task.
Invalidate on changes that alter validity
Invalidate on ingestion, chunking or index changes, permission updates, commits, tool-data updates, and model or prompt releases.
Keep memory separate from caching
GigaRAG helps developers and teams keep project knowledge available across sessions and tools. Treat that shared knowledge as context to retrieve and evaluate for the current task, not as proof that a previously generated output can be reused.
Frequently asked questions
What is the difference between agent memory and caching?
Caching reuses a prior artifact only when its relevant inputs and context still match. Shared agent memory keeps project knowledge available across sessions and tools; retrieve that knowledge for the current task, but do not treat it as automatic approval to reuse a prior answer.
Should I cache embeddings in a RAG application?
You can reuse embeddings for unchanged content and embedding configuration. Regenerate them when either the source content or the embedding configuration changes.
When is it safe to cache vector search results?
Reuse results only when the query, corpus state, permissions, and retrieval configuration that produced them remain valid. Refresh or invalidate them when documents, access, or retrieval settings change.
Can semantic caching return the wrong answer?
Yes. A similar query can differ in context or state, and a similarity threshold can admit a match whose answer does not fit the new request. Use semantic caching selectively, and don't rely on similarity alone for stateful tasks or live results.
What should be included in an LLM response cache key?
Include request context that can change the response, such as tenant or user scope, permissions, conversation context, prompt version, model version, and relevant retrieval configuration. For coding tasks, include the repository and branch or commit when code state matters.
How do I invalidate a cache after documents or embeddings change?
Invalidate or refresh artifacts that depend on the changed documents or embeddings. Use an expiration policy or trigger invalidation from the change event, and make sure dependent retrieval results and responses cannot be reused against the old state.
Should cached LLM responses include the model and prompt version?
Yes, when either version can change the generated response. Including them in the key or cache metadata helps prevent reuse across different generation configurations.
Which metrics should I monitor for an AI-agent cache?
Track cache hits and misses, then monitor whether reused results remain correct and fresh for their intended context. A high hit rate alone does not show that cached answers are safe to serve.
About GigaRAG
GigaRAG is a shared memory platform for AI agents that helps developers and teams keep project knowledge available across sessions and tools. Use that shared knowledge as input to the current task, with retrieval and validation appropriate to the work being performed.
You can connect agents to a central Hive Mind through MCP or the API. Access controls manage what each agent can use, helping teams govern shared project knowledge across tools and sessions.
