Retrieval Strategies Explained for RAG Pipelines

Retrieval Strategies Explained for RAG and Agent Memory

Search "retrieval strategies explained" and you'll get classroom advice for teachers, not guidance for engineers building RAG pipelines. Retrieval strategies, in the technical sense, are the rules you set for how a system finds and returns relevant information from a knowledge base. This guide covers retrieval strategies explained for RAG and agent memory: vector search, hybrid retrieval, chunking, reranking, and memory systems for agents. It does not cover educational retrieval practice. GigaRAG operationalizes several of these strategies, so I'll reference it where it's relevant, but the techniques themselves are framework-agnostic. The honest answer is that retrieval is not magic. Bad chunking, weak embeddings, or a sloppy generator will sink your pipeline no matter how clever your search strategy is. This guide covers how each retrieval strategy works, when it wins, when it fails, and a decision framework to help you pick the right one for your corpus size, query type, and latency budget.

In This Guide

• What Are Retrieval Strategies?
• Sparse vs Dense Retrieval: Which Should You Start With?
• Sparse vs. Dense Retrieval: The Core Tradeoff
• Retrieval Strategies Explained: A Step-by-Step Guide
• Hybrid Retrieval Strategies Explained
• Chunking Strategies That Make or Break Retrieval
• Reranking: The Quality Multiplier
• Retrieval Strategies for Agent Memory
• What Retrieval Strategies Cannot Do
• How to Choose a Retrieval Strategy
• Common Mistakes When Building Retrieval Strategies

What Are Retrieval Strategies?

Retrieval strategies are the methods a system uses to find and return relevant information from a knowledge base when a query arrives. In RAG and agent memory, they decide what context reaches the LLM, in what order, and at what cost.

Retrieval strategies in RAG vs. educational retrieval practice

Search "retrieval strategies explained" and most results are for teachers. That's retrieval practice: asking students to recall facts from memory to strengthen learning. It's a cognitive science technique.

RAG retrieval is different. You're not strengthening a model's memory. You're fetching text from an external store, a vector database, a document index, an agent's past interactions, and handing it to an LLM as context. The strategy is the set of decisions about how to encode the query, how to search, how to rank results, and how many to return.

The goal isn't learning. It's precision and recall under a token budget.

The retrieval pipeline: query → encode → search → rank → return

Every retrieval strategy, sparse or dense, runs the same five stages.

First, the query arrives. It might be a user question, an agent's internal state, or a tool call.

Second, the query is encoded. Sparse methods turn it into term weights. Dense methods turn it into a vector.

Third, search runs against the index. That's BM25 over an inverted index, or approximate nearest neighbor over embeddings.

Fourth, results are ranked. First-pass ranking comes from the search itself. A reranker may then reorder the top candidates.

Fifth, the top-k results are returned as context.

The strategy lives in the choices at each stage: which encoder, which index, which similarity metric, whether to rerank, what k to set. Change one and you change what the LLM sees.

[!note] Retrieval quality is usually bounded by chunking, metadata, and query formulation rather than by the embedding model itself; swapping models rarely fixes a corpus that was split badly or lacks filters.

Sparse vs Dense Retrieval: Which Should You Start With?

Sparse vs. Dense Retrieval: The Core Tradeoff

Two retrieval paradigms dominate RAG pipelines. Sparse retrieval matches exact terms. Dense retrieval matches meaning. Each wins in different conditions, and the choice shapes everything downstream.

How sparse retrieval works (BM25, TF-IDF)

Sparse retrieval treats a query as a bag of words. TF-IDF weights terms by how often they appear in a document versus how rare they are across the corpus. BM25 refines this with term saturation and document length normalization.

The index is an inverted index: a mapping from each term to the documents containing it. Search is exact matching. If the query says "postgres connection pool" and the document says "database connection pooling," sparse retrieval misses it unless you add synonyms or query expansion.

The good news is BM25 is fast, cheap, and explainable. You can point to exactly why a document matched. It needs no training data and no embedding model.

How dense retrieval works (embeddings, vector search)

Dense retrieval encodes the query and every document into vectors using an embedding model. Similarity is measured with cosine similarity or dot product. Search runs approximate nearest neighbor (ANN) over the vector index.

Here's what happens behind the scenes: the embedding model maps semantically similar text to nearby points in vector space. "Postgres connection pool" and "database connection pooling" land close together even though they share no exact terms.

The main catch is cost. You need an embedding model, a vector database, and enough compute to encode your corpus. Dense retrieval also struggles with rare terms, product codes, and exact identifiers that sparse methods handle trivially.

When to use each — and when to combine

Sparse wins for exact-match queries: SKUs, error codes, legal citations, names. It also wins when you have no embedding budget or need full explainability.

Dense wins for natural-language questions, paraphrased queries, and cross-lingual search. It wins when meaning matters more than exact wording.

In practice, most production RAG pipelines combine both. That's hybrid retrieval, and it's the next section.

[!tip] For agent memory specifically, store a short provenance record (source, timestamp, confidence) alongside each retrieved chunk so the agent can decide whether to trust or re-fetch it, rather than treating every hit as equally authoritative.

Retrieval Strategies Explained: A Step-by-Step Guide

1. Define the task: is the retriever feeding a RAG answer, an agent tool call, or long-term memory?
2. Characterize your corpus: prose, code, tables, logs, or a mix, and how exact the terms must match.
3. Pick a baseline: start with BM25 or a single dense index before adding complexity.
4. Set chunking and metadata: size, overlap, and filters (source, date, permissions) before tuning models.
5. Add a reranker over the top-k candidates to improve precision without changing the index.
6. Evaluate with a labeled query set using recall@k and answer faithfulness, not vibes.
7. Iterate: add hybrid fusion, query rewriting, or graph links only where the eval shows a gap.

Hybrid Retrieval Strategies Explained

Hybrid retrieval runs sparse and dense search in parallel, then merges the results. You get exact-term matching from BM25 and semantic matching from embeddings in one pass. It's the default for production RAG because the two signals cover each other's blind spots.

What hybrid retrieval is and why it works

The mechanism is simple. Sparse retrieval catches exact matches: product codes, error strings, names. Dense retrieval catches paraphrases and meaning-level matches. Run both, and you retrieve documents that either method alone would miss.

The cost is roughly double the compute for the retrieval stage. For most teams, that's a rounding error compared to the quality gain. A query like "how do I fix connection timeout" matches a doc titled "troubleshooting database connectivity" through dense search, while BM25 catches the doc that literally says "connection timeout" in the body.

Fusion methods: RRF, score normalization, weighted sum

Merging two ranked lists is the hard part. Scores from BM25 and cosine similarity live on different scales, so you can't just add them.

Reciprocal rank fusion (RRF) sidesteps the problem entirely. It ignores raw scores and uses rank position instead: each document gets a score of 1/(k + rank) from each list, summed across both. k is a constant, usually 60. RRF is robust because it doesn't care how confident each retriever was, only where it placed the document.

Score normalization rescales both score distributions to a common range before combining. Min-max normalization is the common choice. Weighted sum then lets you tune the blend: 0.7 dense, 0.3 sparse, or whatever your eval says works.

RRF is the pragmatic default. No tuning, no calibration, works across retrievers.

When hybrid retrieval is overkill

Don't reach for hybrid if your corpus is tiny, under a few thousand chunks. Sparse alone is fine when queries are exact-match heavy: SKU lookup, log search, legal citation. Dense alone works when you have a strong embedding model and queries are consistently natural language.

The honest answer is hybrid adds latency and infrastructure. If your eval shows single-method retrieval hitting your quality bar, skip the complexity.

Chunking Strategies That Make or Break Retrieval

Chunking is where retrieval quality is won or lost before any search runs. You can tune embeddings, add rerankers, and fuse results all day. If your chunks are the wrong size or split mid-thought, the retriever returns fragments that don't answer the query.

Fixed-size vs. semantic chunking

Fixed-size chunking splits text every N characters or tokens, usually 256 to 512. It's fast and predictable. The catch: it cuts sentences in half. A chunk that starts mid-paragraph and ends mid-thought embeds poorly because the embedding model never sees a complete idea.

Semantic chunking splits on structure: paragraphs, sections, or sentence boundaries detected by a model. Chunks stay coherent. The tradeoff is variable chunk size, which complicates storage and batching. In practice, semantic chunking wins for documentation and long-form content. Fixed-size works when your source is already uniform, like API reference pages.

Chunk overlap and why it matters

Overlap means each chunk shares a few sentences with its neighbour. It preserves context that straddles a boundary. Without overlap, a definition at the end of one chunk and its example at the start of the next get separated. The retriever returns one, not both.

Ten to fifteen percent overlap is a reasonable starting point. More than that bloats your index with duplicate text and wastes tokens.

Metadata as a retrieval filter

Metadata turns retrieval from a pure similarity search into a filtered query. Attach source, date, section, and document type to every chunk. Then filter before or after vector search: only chunks from the last 90 days, only from the API docs, only from a specific product.

This is the cheapest precision gain available. Pre-filtering cuts the candidate set before embeddings run, which also reduces latency. The main catch is you have to maintain metadata at ingestion time. Retroactive tagging is painful.

Reranking: The Quality Multiplier

Vector search returns a ranked list. That ranking is approximate. The embedding model compresses a query and a chunk into vectors, then measures cosine similarity. It's fast, but it misses nuance. A chunk about "Python exceptions" and a query about "Python exception handling best practices" score similarly even when one is a tutorial and the other is a changelog.

Reranking fixes this with a second pass. You retrieve 50 to 100 candidates cheaply, then run a more expensive model over just those candidates to reorder them. The expensive model reads the actual text of the query and the chunk together, so it catches what vector similarity misses. The result: the top 5 chunks you send to the LLM are actually the right 5.

Why first-pass retrieval is not enough

Embeddings are lossy. A 768-dimension vector cannot capture every semantic relationship. Near-misses happen. Reranking recovers precision without re-embedding your entire corpus.

Cross-encoder reranking vs. LLM-based reranking

Cross-encoders like BGE-reranker or Cohere Rerank are purpose-built for scoring query-chunk pairs. They're fast, cheap, and good enough for most pipelines. LLM-based reranking asks a model like GPT-4 to rank candidates. It's more accurate on ambiguous queries but costs more and adds latency.

Latency vs. quality tradeoffs

Reranking adds 50 to 200 milliseconds per query. For interactive chat, that's fine. For high-throughput search, you may skip it or rerank only the top 20 candidates. The honest answer: rerank when precision matters more than raw speed.

Retrieval Strategies for Agent Memory

Agent memory retrieval is not stateless RAG. In a standard RAG pipeline, every query starts fresh: encode, search, rank, return. The system has no history. An agent does. It remembers what you asked three turns ago, what it tried, and what worked. Retrieval has to account for that state.

Episodic vs. semantic memory in agents

Semantic memory is the knowledge base: facts, docs, code snippets. It's what standard RAG retrieves from. Episodic memory is the log of past interactions: what the user asked, what the agent did, whether the user accepted the result. Agents retrieve from both.

The retrieval strategy differs. Semantic memory retrieval is similarity-based: find chunks close to the query embedding. Episodic memory retrieval is often temporal or causal: find what happened last time this user asked something similar. You can't just cosine-similarity your way through an interaction log. You need filters for session ID, timestamp, and task outcome.

Recency and importance weighting

Not all memories are equal. A fact the agent learned five minutes ago matters more than one from last week. A correction the user made twice matters more than a passing comment. So agent memory systems apply weights.

Recency weighting decays the score of older memories. A simple exponential decay works: multiply each memory's retrieval score by a factor that shrinks with age. Importance weighting is harder. You can infer importance from signals: did the user repeat it, did the agent act on it, did the task succeed. Or you can ask an LLM to score memories. The second option is slower but more accurate.

Memory consolidation as a retrieval strategy

Consolidation is what happens between sessions. The agent takes raw episodic memories and compresses them into semantic ones. A long debugging session becomes a single entry: "user's Python env breaks when numpy and pandas versions mismatch; fix is to pin versions."

This is a retrieval strategy because it changes what gets retrieved later. Instead of searching through 200 raw interaction logs, the agent searches a small set of consolidated facts. Retrieval gets faster and more precise. The tradeoff: consolidation is lossy. Details get dropped. If the agent consolidated too aggressively, it can't retrieve the nuance it needs.

What Retrieval Strategies Cannot Do

Retrieval is a lookup step. It finds chunks. It does not understand them, clean them, or write the answer. When a RAG pipeline fails, the failure is usually upstream or downstream of retrieval, not in the search itself.

Retrieval cannot fix bad source data

If your knowledge base contains outdated docs, contradictory policies, or marketing fluff, retrieval will faithfully return that garbage. The retriever's job is to find the most relevant chunk. Relevance to a bad source is still bad. You fix this by curating the corpus, not by tuning the retriever.

Retrieval cannot compensate for a weak generator

A strong retriever feeding a weak LLM produces well-sourced nonsense. The generator still has to reason over the chunks, resolve conflicts, and write a coherent answer. If the model can't do that, no retrieval strategy saves it. Retrieval narrows the problem; it doesn't solve it.

Retrieval cannot guarantee zero hallucination

Even with perfect retrieval, the generator can ignore the context, misread it, or fill gaps with plausible-sounding invention. Retrieval reduces hallucination by grounding the model in real text. It does not eliminate it. You still need evaluation, guardrails, and a generator you trust.

How to Choose a Retrieval Strategy

The honest answer is it depends on four things: how big your corpus is, what your queries look like, how fast you need answers, and how much you care about precision versus recall. Get those four right and the strategy picks itself.

Decision factors: corpus size, query type, latency, quality bar

Corpus size sets the floor. Under 10,000 chunks, dense retrieval alone works fine. Over a million, you'll want sparse retrieval or hybrid to keep latency down and recall up. Query type matters just as much. Exact-match queries like error codes or product SKUs favor sparse. Natural-language questions favor dense. Latency budget decides whether you can afford a reranker. If you need answers under 200ms, skip reranking and lean on a single-pass retriever. If you can wait 500ms to a second, reranking buys you real precision gains. The quality bar is the last lever. For internal tools where a wrong answer costs a re-query, precision matters more. For research assistants where missing a relevant doc is worse, optimize for recall.

Comparison table: strategy vs. use case

A simple default stack for most teams

Start with hybrid retrieval using BM25 plus embeddings, fused with reciprocal rank fusion. Add a cross-encoder reranker on the top 20 candidates if your latency budget allows it. Chunk at 256 to 512 tokens with 10% overlap. That stack handles most corpora under a few million chunks without much tuning. You can swap in a different reranker or adjust chunk size later once you have evaluation data. Don't start with a complex setup. Start with the default, measure, then change one thing at a time.

Common Mistakes When Building Retrieval Strategies

Most retrieval failures aren't exotic. They're the same three mistakes, repeated across teams and codebases. Here's what they look like in practice and how to avoid them.

Retrieving too much context (context stuffing)

The instinct is to crank top-k up to 20 or 50 and let the LLM sort it out. That backfires. More context means more tokens, higher latency, and a generator that drowns in irrelevant passages. A model asked to answer from 15 chunks will often blend details from the wrong ones. The fix is to retrieve fewer, better chunks. Top-k of 5 to 10 with a reranker beats top-k of 50 without one, almost every time. If you're stuffing context to compensate for weak retrieval, fix the retrieval, not the prompt.

Ignoring query-document mismatch

Your embeddings were trained on general text. Your users ask questions in shorthand, with typos, or using domain jargon the model has never seen. The query "why did the build fail" retrieves nothing useful when your docs say "CI pipeline error: exit code 1." That's a mismatch between how users ask and how your corpus is written. Query expansion helps: rewrite the query, add synonyms, or use a hybrid retriever so BM25 catches the exact terms embeddings miss. Don't assume semantic search handles everything. It doesn't.

Skipping evaluation

You can't improve what you don't measure. Teams ship a retriever, eyeball a few answers, and call it done. Then recall drifts as the corpus grows and nobody notices until a user files a bug. Build a small eval set: 50 to 100 query-document pairs where you know the right answer. Run it every time you change chunk size, swap embeddings, or adjust top-k. Without that, every tuning decision is a guess. Retrieval strategies explained properly always come back to measurement.

Frequently Asked Questions

What are retrieval strategies?

In a RAG or agent context, retrieval strategies are the methods used to select relevant context from a corpus before an LLM generates an answer. They span sparse keyword search, dense vector search, hybrid fusion, graph traversal, and reranking. The choice affects both answer quality and compute cost.

What are the methods of retrieval?

The main methods are sparse retrieval (BM25/TF-IDF), dense retrieval (embedding similarity), hybrid retrieval that fuses both, graph-based retrieval over linked entities, and reranking of an initial candidate set. Many production pipelines combine two or more of these.

What are the best retrieval practice strategies?

There is no single best strategy; the right one depends on your corpus and query types. In practice, a hybrid baseline plus a reranker covers most RAG and agent-memory use cases well. Tune chunking and metadata filters before adding more retrieval stages.

Can you give me an example of retrieval-based learning?

In education, retrieval-based learning means recalling information from memory rather than re-reading it. In RAG, the analogous idea is forcing the model to ground its answer in retrieved passages instead of relying on parametric memory. Both rely on pulling the right context at the right moment.

Do I need a vector database for retrieval?

Not necessarily. A keyword index such as BM25 can be a strong baseline, especially for exact-match or compliance-heavy corpora. Vector databases become worthwhile when semantic similarity over natural-language queries is the dominant need.

How do I evaluate a retrieval strategy?

Build a labeled set of queries with known relevant passages, then measure recall@k and precision@k for retrieval, and faithfulness or groundedness for the final answer. Track these over time so changes to chunking or models can be compared fairly.

What can retrieval strategies not do?

Retrieval cannot fix a corpus that lacks the answer, cannot guarantee factual correctness on its own, and cannot resolve contradictory sources without additional logic. It also struggles when queries are ambiguous or when the needed evidence is spread across many documents.

About GigaRAG

GigaRAG helps GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through retrieval strategies explained or something adjacent, we publish what we have actually tested, including where it falls short.