What Reranking Does in RAG Pipelines

What Reranking Does and Why It Exists

What reranking does is simple to describe and easy to misjudge. You're a RAG pipeline builder or an agent memory developer, and you've hit the wall: retrieval returns twenty documents, the top five are only loosely relevant, and the LLM's answer comes back mediocre. The fix isn't a better embedding model. It's a second pass. A reranker takes those twenty documents and reorders them by true relevance before the LLM ever sees them, so the context window fills with the documents that actually matter. GigaRAG's agent memory layer leans on this same mechanism when pulling from long-term stores. Most guides stop at "use a reranker." This one goes further. You'll learn what reranking actually does, why it exists, when it fails, and how to decide if it belongs in your pipeline at all.

In This Guide

• What Is a Reranker?
• Retrieval vs Reranking: What Each Stage Does
• Why Reranking Exists: The Retrieval Problem
• What Reranking Does: A Step-by-Step Guide
• How Reranking Works in RAG
• Cross-Encoders vs. Bi-Encoders: The Trade-off That Explains Everything
• Reranking for Agent Memory Systems
• What Reranking Cannot Do
• When You Might Not Need Reranking
• Selecting a Reranking Model
• Common Mistakes When Implementing Reranking
• Final Thoughts on What Reranking Does

What Is a Reranker?

A reranker is a second-stage model that takes a small set of retrieved documents and reorders them by how precisely each one answers the query. The first-stage retriever casts a wide net for speed. The reranker then scores each query-document pair directly, putting the most relevant items at the top before the LLM sees them.

Reranker vs. retriever: two different jobs

A retriever's job is recall: find everything that might be relevant, fast. It scans thousands or millions of documents and returns a candidate set, usually 20 to 100 items. Speed matters more than precision here, so the retriever uses shortcuts. It encodes the query and each document separately, then compares them with a simple similarity score like cosine distance.

A reranker's job is precision. It receives that candidate set and asks a harder question: which of these documents actually answers the query, not just shares keywords or semantic overlap with it. The reranker looks at the query and document together, as a pair, and produces a relevance score for that specific pairing. It's slower per document, but it only has to score a handful, not the whole corpus.

The division of labour is the point. You can't afford to run a precise scorer over a million documents. You can afford to run it over 50.

The cross-encoder at the core

Most rerankers are cross-encoders. Here's what happens behind the scenes: a cross-encoder takes the query and one document, concatenates them into a single input, and passes that through a transformer model. The model attends to every token in the query against every token in the document. That joint attention is what lets it catch things a retriever misses: negation, word order, whether the document answers the question or merely mentions the same topic.

The trade-off is compute. A cross-encoder cannot pre-encode documents and cache them. Every query-document pair requires a fresh forward pass. That's why it runs on a small candidate set, not the full index.

What a reranker outputs: a reordered list

The output is simple: the same documents you passed in, sorted by score. Nothing is added. Nothing is removed. The reranker doesn't retrieve new documents and doesn't rewrite the ones it sees. It just changes the order.

That reordering matters because the LLM reads top-down and its attention is finite. A document that would have been eighth in the retriever's list might be first after reranking. The LLM now builds its answer from the strongest evidence first. The pipeline's recall hasn't changed. Its precision at the top has.

[!note] Reranking only reorders documents that were already retrieved; it cannot surface documents that the initial retrieval step missed entirely.

Retrieval vs Reranking: What Each Stage Does

Why Reranking Exists: The Retrieval Problem

Reranking exists because the first stage of a RAG pipeline is built for speed, not for judgment. The retriever's job is to shrink a million documents down to a few dozen candidates before the clock runs out. That constraint forces compromises. The reranker exists to buy back some of what those compromises cost.

Bi-encoders trade precision for speed

A bi-encoder retriever encodes the query and every document separately, then compares them with a similarity score. That separation is what makes it fast: you can pre-encode your entire corpus once, store the vectors, and never touch the model again at query time. A new query just gets encoded and compared against the stored vectors.

The cost is that the model never sees the query and document together. It can't tell whether a document answers the question or merely shares a topic. A query about "how to fix a leaking pipe" and a document about "the history of pipe manufacturing" might both score high on semantic similarity. The retriever doesn't know the difference. It wasn't built to.

That's the trade-off. You get recall: the right document is probably somewhere in the candidate set. You don't get precision: the right document isn't necessarily at the top.

The context window is a hard limit

Here's the constraint that makes precision matter. An LLM has a finite context window. You cannot feed it all 50 retrieved documents and expect it to sort through them. Even if the window technically fits, the model's attention degrades as input grows. Documents in the middle and at the bottom get less weight than documents at the top.

So you have to cut. You take the top 5 or top 10 from the retriever and hand those to the LLM. The rest get discarded.

That cut is where the damage happens. If the retriever's ordering is loose, the documents you keep might be the wrong ones. The document that actually answers the query could be sitting at position 18 in the candidate set. It never reaches the LLM. The LLM then builds an answer from weaker evidence and produces a mediocre response, not because it's a bad model, but because it never saw the right document.

Garbage in, garbage out: why order matters to the LLM

Order matters because LLMs are sequential readers. They process input left to right, top to bottom, and they weight earlier content more heavily. A document at position 1 shapes the answer more than a document at position 5. If the strongest evidence sits at the bottom of the context, the model treats it as an afterthought.

The reranker's job is to fix that ordering before the LLM reads anything. It takes the candidate set, scores each document against the query with full joint attention, and reorders the list so the most relevant items sit at the top. The LLM then reads the strongest evidence first and builds its answer from that.

The pipeline's recall hasn't changed. The same documents are in the set. What changed is which ones survive the cut and what order they're in when the LLM sees them. That's the entire point of reranking: not finding more documents, but making sure the right ones get read first.

[!tip] For agent memory systems, apply reranking when recalling past interactions or facts from a persistent store — it helps prioritize the most contextually relevant memories, but keep the memory store itself separate from the reranking step.

What Reranking Does: A Step-by-Step Guide

1. Retrieve a wider candidate set (e.g., top 50–100) with your existing retriever.
2. Choose a reranker: cross-encoder model or a hosted reranking API.
3. Score each candidate against the query with the reranker.
4. Sort candidates by reranker score and keep the top 3–10 for generation.
5. Feed the reordered top documents into your LLM prompt.
6. Measure answer quality and latency before and after to confirm the gain.
7. Tune candidate count and top-k to balance accuracy against cost.

How Reranking Works in RAG

Reranking slots into a RAG pipeline as a second scoring pass between retrieval and generation. The retriever still does its fast, approximate sweep. The reranker then takes the candidate set, scores each query-document pair with full joint attention, and reorders the list before anything reaches the LLM. Two stages, two different jobs.

Stage 1: fast retrieval with bi-encoders

Stage 1 is the retriever you already have. It encodes your query into a vector, compares it against pre-encoded document vectors, and returns the top-k matches. For most pipelines, k sits between 20 and 100. The retriever's strength is coverage: it casts a wide net and rarely misses the right document entirely. Its weakness is ordering. The document that actually answers the query might land at position 37.

That's fine. Stage 1 isn't supposed to be precise. It's supposed to be fast and to not drop the right document. You want recall here, not precision.

Stage 2: precise scoring with a cross-encoder

Stage 2 is where the reranker does its work. It takes the top-k candidates from stage 1, pairs each one with the original query, and runs them through a cross-encoder. Unlike the bi-encoder, the cross-encoder sees the query and document together in a single input. It can model the relationship between them: does this document answer the question, or does it just share vocabulary?

The reranker assigns each pair a relevance score, then sorts the list by that score. You take the top 5 or top 10 from the reordered list and hand those to the LLM. The documents that were buried at position 30 in the retriever's output can now surface to the top, because the reranker actually read them against the query.

The trade-off is compute. A cross-encoder cannot pre-encode documents. Every query-document pair has to be processed at query time, which means stage 2 is slower than stage 1 by a wide margin. That's why you rerank 50 documents, not 50,000.

What changes in your pipeline when you add reranking

Three things change. First, you add a model. The reranker is a separate component with its own weights, its own inference cost, and its own failure modes. You now have two models to maintain instead of one.

Second, you add latency. Every query now makes two passes: one through the retriever, one through the reranker. The reranker's pass is the expensive one. For a cross-encoder scoring 50 pairs, expect tens to hundreds of milliseconds depending on the model and hardware.

Third, you add a decision point. How many documents do you retrieve in stage 1? How many do you rerank? How many survive to the LLM? Each of those numbers is a knob you now have to tune, and they interact. Retrieve too few and the reranker has nothing to work with. Rerank too many and your latency budget blows out.

The pipeline's shape doesn't change. It's still retrieve, then generate. What changes is that a scoring step now sits between those two, and that step has its own cost, its own latency, and its own failure modes. You're not getting precision for free.

Cross-Encoders vs. Bi-Encoders: The Trade-off That Explains Everything

The entire reranking story comes down to one architectural difference: whether the query and document are encoded separately or together. That single choice determines speed, accuracy, and cost. Everything else is downstream.

How bi-encoders work: encode once, compare fast

A bi-encoder is two encoders running in parallel. The query goes through one. The document goes through the other. Each produces a vector, and relevance is measured by the distance between those two vectors, usually cosine similarity.

Here's why that's fast: documents can be encoded offline, once, and stored. At query time, you only encode the query itself, then run a vector search against millions of pre-computed document vectors. The comparison is a dot product. That's it.

The catch is what gets lost. Because the query and document never see each other during encoding, the model has to compress all of their meaning into fixed-size vectors independently. A query about "bank" and a document about "river banks" will produce similar vectors, because the model had no chance to notice the mismatch. Bi-encoders are fast because they sacrifice joint context.

How cross-encoders work: encode together, score precisely

A cross-encoder takes the query and document, concatenates them into a single input sequence, and runs them through a transformer together. Every token in the query attends to every token in the document. The model sees the full interaction.

That's why cross-encoders catch what bi-encoders miss. The "bank" query paired with the "river bank" document produces a low score, because the model can see the words side by side and recognize the semantic mismatch. It's not comparing two compressed vectors. It's reading the pair as a sentence.

The cost is that nothing can be pre-computed. Every query-document pair has to be processed from scratch at query time. If you have 50 candidates, that's 50 full transformer passes. If you have 50,000, you're not doing that in real time.

Why the speed/accuracy trade-off is unavoidable

The trade-off isn't a bug. It's a direct consequence of what each architecture does.

Bi-encoders trade joint context for pre-computability. That's what makes them viable for first-pass retrieval over millions of documents. Cross-encoders trade pre-computability for joint context. That's what makes them accurate enough to reorder a short list.

You cannot have both. A model that encodes query and document together cannot pre-encode documents, because the encoding depends on the query. A model that encodes them separately cannot model their interaction, because the interaction happens after encoding. The architecture forces the choice.

That's why reranking exists as a second stage rather than a replacement for retrieval. You use the fast, approximate model to narrow the field, then the slow, precise model to order the survivors. The two-stage design isn't a workaround. It's the only way to get both recall and precision without paying cross-encoder costs on the entire corpus.

Reranking for Agent Memory Systems

Agent memory changes what "relevant" means. A document corpus is static: the right answer exists somewhere, and retrieval just has to find it. A memory store is different. The right memory depends on who the agent is talking to, what it's doing right now, and what it learned three conversations ago. Reranking has to account for that.

How agent memory differs from document retrieval

In document retrieval, the query is the user's question. In agent memory, the query is often the agent's own internal state: the current task, the active context, the recent conversation. The retriever pulls candidate memories, but relevance isn't just semantic similarity. It's "does this memory help the agent act correctly in this moment?"

That's a harder scoring problem. A memory about a user's preferred API format might be semantically distant from a query about database schemas, but it's exactly what the agent needs. A cross-encoder trained on generic query-document pairs won't catch that. You need a reranker that understands task relevance, not just topical overlap.

Why reranking matters more for long-term memory

The main catch is volume. Long-term memory stores grow without bound. A document corpus might hold 10,000 chunks. An agent's memory after six months of conversations could hold 500,000. First-pass retrieval over that space returns noisier candidates, because the embedding space is denser and more semantically mixed.

Reranking becomes the filter that keeps the agent from drowning in its own history. Without it, the agent retrieves memories that are technically similar but contextually useless, and the LLM generates responses based on the wrong past. With it, the top few memories are the ones that actually shape correct behavior. The cost is latency, and for agents that's a real constraint: every reranking pass adds time to a response loop that's already slower than static RAG.

Practical considerations for memory-backed agents

Start by treating memory retrieval as a different task than document retrieval. Don't reuse a document reranker and expect it to work. You'll likely need to fine-tune on memory-specific pairs: query plus current context, scored against whether the memory improved the agent's next action.

Keep the candidate list short. Reranking 100 memories per turn is expensive and usually unnecessary. Ten to twenty is enough if first-pass retrieval is decent.

And store memory metadata separately. Timestamps, conversation IDs, user IDs. A reranker can use that metadata to boost recency or penalize stale memories, but only if it's available at scoring time. Most off-the-shelf rerankers ignore metadata entirely. That's a gap you'll have to fill yourself.

What Reranking Cannot Do

Reranking is a precision tool. It reorders what retrieval already found. It does not reach back into the document store and pull out something the retriever missed. If the right document never made it into the candidate list, no reranker on earth will surface it. That's the boundary, and it's worth stating plainly because most pipeline failures get misdiagnosed as reranking problems when they're actually retrieval problems.

Reranking cannot fix bad retrieval

The retriever sets the ceiling. Reranking can only work with what it's given. If your first-pass retrieval returns 50 documents and the correct one is number 51, reranking will confidently reorder 50 wrong documents. The output looks better. The answer is still wrong.

This matters because reranking can mask retrieval failures. You add a cross-encoder, scores improve on your eval set, and you ship. Then a user asks something your embedding model handles poorly, and the reranker happily ranks irrelevant chunks above each other. The honest answer is that reranking amplifies retrieval quality; it doesn't replace it. If recall is the problem, fix the retriever first. Check your embedding model, your index, your query formulation. Reranking comes after.

Reranking cannot fix poor chunking

Chunking happens before retrieval. If your documents are split badly, the information the user needs is scattered across chunks or buried inside one oversized chunk. Reranking scores whole chunks. It can't extract the relevant sentence from a 2,000-token block and discard the rest.

A common failure: you chunk by fixed character count, a paragraph gets split in half, and the answer spans both chunks. The retriever returns both. The reranker scores both highly. The LLM receives two fragments and has to stitch them together, which it does imperfectly. No reranking model fixes that. You fix it by rechunking with semantic boundaries or overlap. Reranking operates downstream of chunking decisions, and it inherits every mistake those decisions made.

Reranking adds latency and cost: always

There's no free reranking pass. Every candidate document you rerank costs compute and time. A cross-encoder processes query-document pairs through a full transformer forward pass. Rerank 20 candidates and you've run 20 inference calls. Rerank 100 and you've added hundreds of milliseconds to your pipeline, possibly more depending on model size and hardware.

For a static RAG system serving one query at a time, that's manageable. For an agent making multiple retrieval calls per turn, it compounds fast. You need a latency budget before you add reranking, not after. Decide how many milliseconds you can spend, then choose your reranker and candidate count to fit. If you can't fit it, skip reranking or use a smaller model. The cost is real, and it scales linearly with every document you ask the reranker to score.

When You Might Not Need Reranking

Reranking is not a default. It's a trade you make: latency and cost in exchange for precision. If you don't need the precision, don't make the trade. Here are the clearest cases where skipping reranking is the right call.

Small document collections

If your entire corpus is a few hundred documents, retrieval is already precise. A bi-encoder over 200 chunks will rarely surface irrelevant material in the top 10, because there isn't much irrelevant material to surface. The reranker's job is to separate signal from noise. When there's almost no noise, there's almost nothing to separate.

The threshold isn't exact, but in practice, collections under roughly 1,000 chunks rarely justify a reranking stage. You'll spend more time integrating the reranker than you'll save in answer quality. Test it: run your eval set with and without reranking. If the scores don't move, you have your answer.

Short context windows with high-precision retrieval

If your pipeline retrieves 3 to 5 documents and feeds all of them to the LLM, reranking adds little. The whole point of reranking is to decide which documents make the cut when the cut is tight. When everything retrieved fits in context, order matters less. The LLM sees all candidates anyway.

This changes when you retrieve 50 and keep 5. That's when ranking quality directly determines answer quality. But if your retriever already returns high-precision results and your context window holds them all, a reranker is a middleman taking a cut without adding value.

Latency-critical applications

Some systems cannot afford an extra 100 to 300 milliseconds per query. Real-time chat, voice assistants, autocomplete-style retrieval, anything where the user is waiting on a spinner. Reranking adds a full transformer pass per candidate, and that cost is non-negotiable.

The honest answer is that reranking belongs in pipelines where answer quality beats response time. If your product promise is speed, skip it. Use a better bi-encoder, tune your retrieval, or accept slightly noisier results. A reranker that makes your app feel slow is a worse trade than a retriever that occasionally returns a mediocre document.

Selecting a Reranking Model

You've decided reranking is worth the latency. Now you have to pick a model. The choice comes down to three things: how much accuracy you need, how much latency you can absorb, and whether you want to run it yourself or pay someone else to.

Open-source vs. API-based rerankers

Open-source rerankers run on your own hardware. You download the weights, serve them behind an endpoint, and pay only for compute. The main advantage is control: no per-query fees, no data leaving your infrastructure, no rate limits. The main cost is operational. You're now running another service, monitoring it, and scaling it when traffic spikes.

API-based rerankers flip that trade. You send query-document pairs to a vendor endpoint and get scores back. No infrastructure to manage, but you pay per query or per token, and your data crosses a network boundary. For teams without ML ops capacity, the API is usually the right call. For teams with strict data residency requirements or high query volume, self-hosting wins.

The honest answer is that model quality between the two is converging. The best open-source rerankers now score within a few points of commercial APIs on standard benchmarks. The decision is rarely about accuracy. It's about who runs the service.

Key evaluation criteria: accuracy, latency, cost

Accuracy is the obvious one. You measure it on your own eval set, not on a public benchmark. Build a set of 50 to 100 queries with known relevant documents, run your retriever, then run each candidate reranker over the top 20 results. Compare nDCG@5 or recall@5. The number that matters is how often the reranker puts the right document in the top 5.

Latency is the one people forget until production. A cross-encoder reranker scoring 20 candidates adds roughly 100 to 300 milliseconds per query, depending on model size and hardware. Smaller distilled models can cut that to under 50 milliseconds. Larger models can push past 500. Measure it on your hardware, not the vendor's benchmark page.

Cost breaks into two buckets. API rerankers charge per query or per token, typically fractions of a cent per query at volume. Self-hosted rerankers cost GPU hours, which are fixed whether you serve 10 queries or 10,000. The crossover point depends on your volume. Under roughly 10,000 queries per day, an API is almost always cheaper. Above that, self-hosting starts to win.

Popular reranking models compared

The field moves fast, but a few models have held up. Cohere's rerank models are the default API choice: strong accuracy, simple integration, predictable pricing. BGE-reranker from BAAI is the most widely used open-source option, with a large and base variant that trade accuracy for speed. Jina's reranker is newer and competitive, with a focus on multilingual support. Cross-encoders built on DeBERTa or MiniLM remain solid baselines if you want something small and fast.

The main catch is that model rankings shift quarterly. What was best six months ago is now mid-pack. Don't pick a reranker because a blog post from last year recommended it. Pick one because it scored well on your eval set this month.

Common Mistakes When Implementing Reranking

Most reranking failures aren't model problems. They're pipeline problems. The model does what you asked. You just asked the wrong thing.

Reranking too many documents

The most common mistake is feeding a reranker 50, 100, or 200 candidates from stage one. It feels safe. More candidates means more chances to find the right document. But cross-encoders scale quadratically with input length. Scoring 100 query-document pairs takes five times longer than scoring 20, and the accuracy gain is usually marginal.

The right number depends on your retriever's recall curve. If your bi-encoder puts the correct document in the top 20 for 95% of queries, reranking 20 is enough. Reranking 100 only helps if your retriever is weak enough that the right document regularly lands outside the top 20. And if that's true, the fix is a better retriever, not a bigger reranking batch.

Start with 20. Measure recall@20 on your eval set. If the right document is there 95% of the time or more, stop. You're done.

Ignoring latency budget

Reranking adds latency. That's not a bug, it's the cost of doing the work. The mistake is adding it without knowing what your total latency budget is.

A typical RAG query has a budget: retrieval, reranking, and generation all have to fit inside it. If your p95 target is 800 milliseconds and generation takes 500, you have 300 left. A cross-encoder scoring 20 candidates at 150 milliseconds fits. Scoring 50 at 400 milliseconds doesn't.

Measure your reranker's p95 latency on production hardware before you ship. Not the mean. The p95 is what your slowest users experience. If it doesn't fit, use a smaller model, fewer candidates, or skip reranking entirely.

Using reranking as a band-aid for bad retrieval

This is the one that costs teams months. Retrieval returns garbage, so someone adds a reranker hoping it will sort the garbage into something useful. It won't.

A reranker can only reorder what it's given. If the correct document never makes it into the candidate set, no reranker on earth can surface it. Reranking improves precision, not recall. It makes a decent top 20 into a great top 5. It cannot make a bad top 20 into a good one.

The test is simple. Run your retriever alone and check whether the right document appears in the candidate set at all. If it's missing more than 5% of the time, fix retrieval first. Better embeddings, better chunking, hybrid search. Add reranking after recall is solid.

Final Thoughts on What Reranking Does

Reranking does one thing well: it reorders a decent candidate set into a better one. That's it. It doesn't retrieve. It doesn't chunk. It doesn't fix a broken pipeline.

The decision framework is simple. If your retriever already puts the right document in the top 20 most of the time, and you have latency budget to spare, reranking will sharpen your answers. If recall is weak or your p95 is already tight, fix those first.

What reranking does is buy precision at the cost of latency and compute. The trade is worth it when your LLM is drowning in loosely relevant context. It's not worth it when retrieval is the actual problem.

For agent memory systems, the same logic applies. GigaRAG's agent memory layer benefits from reranking when pulling from long-term stores, because older memories compete with newer ones for limited context, and a reranker helps surface the ones that actually matter for the current task.

Frequently Asked Questions

How does reranking work in RAG?

In RAG, reranking takes the initial set of retrieved documents and re-scores them against the query using a more expensive but more accurate model, typically a cross-encoder. The reordered list is then passed to the LLM, so the most relevant context appears first. This two-stage approach balances speed and accuracy.

What is reranking?

Reranking is the process of reordering a list of items — usually search results or retrieved documents — based on a secondary, more precise relevance model. It exists because fast retrieval methods prioritize speed over perfect accuracy, and a second pass can correct their ordering.

What are common re-ranking techniques?

Common techniques include cross-encoder models that jointly encode query and document, late-interaction models like ColBERT, and LLM-based scoring where a language model rates relevance. Each trades off latency, cost, and accuracy differently.

What are the best reranking models for RAG?

Popular open options include cross-encoder models from the sentence-transformers family and ColBERT-style late-interaction models. Hosted options like Cohere Rerank are also widely used. The best choice depends on your latency budget, language, and whether you need self-hosting.

Does reranking add latency to a RAG pipeline?

Yes, reranking adds latency because it scores each candidate with a more computationally intensive model. The added time scales with the number of candidates, so retrieving fewer candidates or using a smaller reranker can help. Many teams find the accuracy gain worth the extra milliseconds.

When might I not need reranking?

You might skip reranking if your corpus is small, if your queries are simple keyword lookups, or if your initial retrieval already returns the correct answer in the top results. It is also less critical when latency is extremely tight and the accuracy gain would be marginal.

Can reranking fix bad retrieval?

1. Reranking can only reorder documents that were already retrieved. If the initial retrieval missed the relevant document entirely, reranking cannot bring it back. Fixing retrieval recall — through better embeddings, chunking, or query expansion — is a separate problem.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through what reranking does and why it exists or something adjacent, we publish what we have actually tested, including where it falls short.