Reranking Models Compared: NDCG, Latency, Cost

Reranking Models Compared: What RAG and Agent Memory Builders Need to Know

If you're building a RAG pipeline or agent memory system, you've likely hit the same wall: retrieval quality caps everything downstream. Reranking models compared side-by-side is the fix, but most guidance online is fragmented, vendor-slanted, or missing the numbers you actually need. This is a practical, vendor-neutral comparison built for people shipping agent memory and RAG systems, not for benchmark tourists. GigaRAG's agent memory platform uses reranking inside its retrieval stack, and this guide reflects lessons from that work, including where rerankers fail and when they're not worth the cost. You'll get a direct comparison table of eight specific models with NDCG@10, latency, and cost columns, plus a decision framework for open-source versus proprietary options, and a clear-eyed look at what reranking cannot fix.

In This Guide

• What Is a Reranker and Why Does It Matter for RAG?
• Open-Source vs Hosted Reranking Models: Which Fits Your Stack?
• How Reranking Models Work: Cross-Encoders, Multi-Vectors, and LLMs
• Reranking Models Compared: A Step-by-Step Guide
• Reranking Models Compared: Side-by-Side Benchmark Table
• Reranking for Agent Memory: What Changes
• Open-Source vs. Proprietary Rerankers: A Decision Framework
• Latency and Cost: The Numbers That Actually Matter in Production
• How to Evaluate a Reranker for Your Specific Use Case
• When Not to Use a Reranker: Limitations and Failure Modes
• Integrating a Reranker into Your RAG Pipeline
• Final Thoughts: Choosing the Right Reranker for Your Stack

What Is a Reranker and Why Does It Matter for RAG?

A reranker is a second-pass model that reorders a small set of retrieved candidates by relevance, replacing the coarse similarity score from vector search with a precise query-document match score. It sits between retrieval and generation, and it's the difference between your LLM seeing the right context or the wrong one.

Here's the problem. Vector search is fast but shallow. It compares embeddings, which compress a whole passage into one vector. That compression loses word order, negation, and exact phrasing. A query about "how to prevent memory leaks" can return passages about "memory leak symptoms" because the vectors sit close together. The embedding model can't tell the difference. A reranker can.

The two-stage retrieval pattern: recall then rerank

Production RAG pipelines almost never rely on a single retrieval pass. Stage one is recall: pull 50 to 200 candidates from the vector database using fast, approximate search. Stage two is rerank: score those candidates with a model that reads the query and document together, then keep the top 5 to 10.

The recall stage optimizes for not missing anything. The rerank stage optimizes for putting the right thing first. You need both. Recall without rerank buries good context under near-misses. Rerank without recall has nothing to work with.

Why embeddings alone underperform on precision

Embeddings are bi-encoders. The query and the document get encoded separately, then compared by cosine similarity. That's fast enough to scan millions of vectors, but the model never sees the query and document side by side. It can't catch that "not recommended" and "recommended" mean opposite things.

A reranker is typically a cross-encoder. It takes the query and document as a single input, runs them through the transformer together, and outputs a relevance score. That joint attention is what lets it catch negation, specificity, and subtle mismatches. The cost is speed: you can't run a cross-encoder over a million documents. You run it over 50.

What reranking fixes, and what it doesn't

Reranking fixes precision. It moves the right passage from position 12 to position 1, which matters because LLMs weight early context more heavily.

It doesn't fix recall. If the right document never made it into the candidate set, no reranker can recover it. And it doesn't fix a broken embedding model. Garbage in, garbage out, just reordered.

[!note] Reranking only reorders documents already retrieved by your first-stage search — it cannot fix missing documents or poor recall. If your retriever misses relevant chunks entirely, no reranker will recover them.

Open-Source vs Hosted Reranking Models: Which Fits Your Stack?

How Reranking Models Work: Cross-Encoders, Multi-Vectors, and LLMs

Rerankers come in three architectural families. Each trades precision against speed differently, and knowing which one you're running tells you what to expect at inference time.

Cross-encoder rerankers: joint query-document scoring

A cross-encoder takes the query and document, concatenates them into one input, and runs the pair through a transformer. The model attends across both sequences, so it sees how each query token relates to each document token. That joint attention is what catches negation, word order, and exact phrasing.

The output is a single relevance score. You sort candidates by that score and keep the top k.

The catch is compute. You can't precompute document representations because the document encoding depends on the query. Every query-document pair needs a full forward pass. That's fine for 50 candidates. It's not fine for a million.

Multi-vector and late-interaction rerankers

Multi-vector models split the difference. Instead of one vector per document, they store one vector per token. At query time, they compute token-level interactions between the query and document, then aggregate those into a final score.

ColBERT is the reference point here. It stores document token embeddings ahead of time, so retrieval stays fast. The late interaction step, where query tokens match against document tokens, happens only on the candidate set. That gives you cross-encoder-like precision at a fraction of the latency.

The trade-off is storage. Token-level vectors multiply your index size by the number of tokens per passage. For large corpora, that's a real cost.

LLM-based rerankers: listwise and pointwise approaches

LLM rerankers use a general-purpose language model to score or order candidates. Pointwise approaches ask the model to score each query-document pair individually. Listwise approaches hand the model the entire candidate list and ask it to reorder everything at once.

Listwise is more accurate because the model sees all candidates together and can compare them directly. It's also slower and more expensive: every rerank call consumes tokens for the full candidate list. Pointwise is cheaper but loses the comparative signal.

RankLLM and RankZephyr are the main open-source options here. They run on smaller LLMs fine-tuned for ranking, not on frontier models.

Bi-encoder rerankers and why they're usually not enough

A bi-encoder reranker encodes the query and document separately, then scores by similarity. That's the same architecture as your embedding model. Running a bi-encoder as a reranker gives you a second opinion from a slightly different model, but it doesn't fix the core problem: the query and document never interact directly.

Bi-encoders can't catch negation or word-order mismatches because those signals don't survive separate encoding. If your recall stage already uses a bi-encoder, adding another one as a reranker mostly adds latency without adding precision. Cross-encoders, multi-vector models, or LLM rerankers are the ones that actually change what gets ranked first.

[!tip] For agent memory systems, rerank only the top 20–50 candidates and cache reranker scores for repeated queries — this keeps latency low while still capturing most of the accuracy gain.

Reranking Models Compared: A Step-by-Step Guide

1. Define your retrieval budget: max docs to rerank, target latency, and cost ceiling.
2. Build a labeled eval set of 100–500 queries with known relevant docs from your domain.
3. Benchmark 3–5 candidate models on NDCG@10 and latency at your target top-k.
4. Test with your actual embedding model and chunking strategy — reranker gains depend on first-stage quality.
5. Measure end-to-end latency including network and batching overhead.
6. Run a cost projection at your expected query volume, including GPU or API spend.
7. Pick the model that meets your latency and cost budget with the best NDCG, then monitor in production.

Reranking Models Compared: Side-by-Side Benchmark Table

Here's the table. Eight models, five columns, one place. No top-ranking result for this query currently offers this, so the numbers below are pulled from published benchmarks and vendor documentation. Where a figure isn't publicly verified, it's marked.

Model comparison table

The MiniLM models are the budget option. They run on CPU, which means you can deploy them on a $5 VPS and still stay under 20ms per query. The trade-off is clear: you give up roughly 10 points of NDCG@10 compared to the top models.

How to read the table: what NDCG@10 actually tells you

NDCG@10 measures how well the reranker orders the top 10 results. It's a graded metric, so a relevant document in position 1 counts more than one in position 10. A score of 0.60 means the model gets the ordering right about 60% of the way to perfect, weighted by position.

The honest answer is that NDCG@10 differences under 0.02 rarely matter in production. A 0.59 model and a 0.61 model will disagree on a handful of queries per thousand, and those disagreements are usually on borderline cases. What matters more is whether the model holds up on your data, not on BEIR.

BEIR is a collection of 18 retrieval datasets spanning different domains. It's the standard benchmark, but it skews toward short passages and factoid queries. If you're reranking long documents or conversational memory entries, your numbers will differ.

Latency and cost columns explained

Latency in the table is per query, assuming a candidate set of 50-100 documents. That's the standard reranking batch size. If you rerank 200 candidates, latency roughly doubles for cross-encoders and more than doubles for LLM rerankers, because listwise models process the entire candidate list as one input.

Cost splits into two worlds. API models charge per search or per token, and the price is predictable but scales linearly with volume. Self-hosted models have zero marginal cost per query, but you pay for the GPU or CPU instance whether it's idle or saturated.

The MiniLM models on CPU are the only option that makes reranking essentially free at low volume. Everything else needs a GPU for reasonable latency, and that's a fixed cost you carry even when traffic is zero.

Reranking for Agent Memory: What Changes

Agent memory retrieval is not document retrieval with a different label. The target is not a passage that answers a question. It's a stored experience, preference, or decision that should shape how the agent behaves next. That changes what a reranker needs to do.

Memory retrieval vs. document retrieval: key differences

Document retrieval optimizes for topical relevance. Does this passage answer the query? Memory retrieval optimizes for contextual relevance. Does this stored item help the agent act correctly in this situation?

Three differences follow. First, memory entries are shorter and more numerous. A session might generate hundreds of entries, each a few sentences. Second, the query is often implicit. The agent isn't typing a search string. It's deciding what to recall based on the current conversation state. Third, false positives cost more. A wrong document gets filtered out by the LLM. A wrong memory gets acted on.

Why agent memory needs higher precision at lower latency

The precision bar is higher because the downstream consumer is a decision, not a reading list. If a reranker surfaces a plausible but irrelevant memory, the agent may reference it in its response. That's worse than retrieving nothing.

Latency is tighter for a different reason. Document RAG can tolerate 100ms of reranking because the user is waiting for an answer anyway. Agent memory retrieval often happens mid-turn, between tool calls or reasoning steps. Every 50ms of reranking is 50ms the agent isn't acting.

In practice, this pushes memory builders toward smaller cross-encoders like MiniLM variants, or toward skipping reranking entirely when the candidate set is already small.

Reranking strategies for long-term memory stores

The main catch is that long-term memory stores grow without bound. Reranking 100 candidates from a 10,000-entry store is fine. Reranking 100 candidates from a 10-million-entry store means your recall stage is doing most of the work, and the reranker is just polishing.

Three strategies work in practice. First, partition memory by recency, importance, or topic, then rerank within the partition. Second, use a cheap bi-encoder filter to cut candidates to 20-30 before the cross-encoder sees them. Third, cache reranking scores for frequently accessed memories, since memory relevance is more stable than document relevance.

The honest answer is that reranking matters less for memory than for document RAG, because the recall stage and the memory write policy shape results more than the reranker does. If your memory store is small or your recall is already precise, a reranker adds latency without adding much accuracy.

Open-Source vs. Proprietary Rerankers: A Decision Framework

The choice is not ideological. It's about which trade-offs you can live with in production.

When self-hosting makes sense

Self-hosting wins when data cannot leave your infrastructure. Healthcare, finance, legal, and any regulated industry where the reranker sees sensitive query-document pairs. An API call sends your query and candidate passages to a third party. A self-hosted model keeps them on your hardware.

Cost is the second driver. If you rerank more than roughly 100,000 queries per month, a self-hosted model on a single GPU is usually cheaper than per-token API pricing. The break-even point depends on your hardware and the model size, but the pattern holds: API costs scale linearly with volume, self-hosted costs plateau after the GPU is paid for.

Control is the third. You can fine-tune an open-source reranker on your own relevance judgments. Proprietary APIs don't offer that. If your domain has unusual relevance patterns, medical literature, legal precedents, internal documentation, fine-tuning matters more than the base model's benchmark score.

The catch: you own the infrastructure. Model updates, security patches, GPU failures, scaling. That's engineering time an API would have absorbed.

When an API is the pragmatic choice

APIs win when you're shipping fast and volume is low or spiky. No GPU to provision, no model to serve, no scaling to manage. Cohere Rerank and Jina Reranker both offer hosted endpoints that return scores in tens of milliseconds.

The honest answer is that most teams should start with an API. The integration cost is near zero, and you can switch later if volume or privacy demands it. Switching rerankers is easier than switching embedding models because the reranker sits at the end of the pipeline, not the foundation.

Latency is comparable for API and self-hosted when the API is in the same region. The difference shows up under load. APIs throttle or queue. Self-hosted throughput is whatever your GPU can sustain.

Hybrid approaches: local fallback + API primary

A hybrid setup covers the failure modes of both. Run a small open-source cross-encoder locally as the default. Route to a proprietary API when the local model's confidence is low, or when the query is complex enough to justify the stronger model.

This works because reranking is stateless. You can swap the model per query without breaking anything downstream. The local model handles the common cases cheaply. The API handles the hard cases where accuracy is worth the cost and the data exposure.

The main catch is complexity. You now maintain two rerankers, two sets of configuration, and a routing rule. That's only worth it if you have both volume and privacy constraints at the same time. Most teams don't.

Latency and Cost: The Numbers That Actually Matter in Production

Benchmarks tell you which model ranks best. They don't tell you what your users will feel. That's latency, and it's the number that decides whether reranking survives contact with production.

Latency budgets for interactive vs. batch retrieval

Interactive retrieval means a user is waiting. Chat, search, agent responses. Your total budget is roughly 200-400ms before the experience degrades. The reranker gets a slice of that, typically 20-80ms for cross-encoders scoring 50-100 candidates. LLM-based rerankers blow past this: RankZephyr and RankLLM can take 500ms to several seconds per query. That's fine for batch jobs. It's not fine for a chatbot.

Batch retrieval flips the constraint. You're reranking thousands of queries offline, building indexes, or processing documents overnight. Latency per query barely matters. Throughput and cost per query dominate.

Cost per 1,000 queries: API vs. self-hosted

API pricing is token-based. Cohere Rerank 3 charges per search unit, which works out to roughly $0.001-0.003 per query depending on candidate count and document length (verify at publish). Jina Reranker v2 sits in a similar range. At 100,000 queries per month, that's $100-300. At a million, $1,000-3,000.

Self-hosted cost is the GPU. A single A10 or L4 can serve a small cross-encoder like ms-marco-MiniLM at thousands of queries per second. The GPU costs $0.50-1.50 per hour on cloud, or a few thousand dollars to own. Once it's running, marginal cost per query approaches zero. The break-even against API pricing lands somewhere around 50,000-150,000 queries per month, depending on model size and hardware.

Throughput ceilings and when to parallelize

A cross-encoder on one GPU typically sustains 500-2,000 queries per second for small models, dropping to 50-200 for larger ones like bge-reranker-large. LLM-based rerankers are far slower: single-digit queries per second is common.

When you hit the ceiling, parallelize. Reranking is embarrassingly parallel because each query-document pair scores independently. Run multiple GPUs, or batch requests through an API. The honest answer is that most teams never hit the ceiling. If you're under 50 queries per second, one GPU or one API key handles it.

How to Evaluate a Reranker for Your Specific Use Case

Public benchmarks are a starting point, not a verdict. Your corpus, your queries, your latency budget: none of those appear in a leaderboard. You need to test on your own data.

Building a small labeled evaluation set

Start with 50-100 real queries from your logs. Not synthetic ones. Pull the top 20-50 candidates from your retrieval stage for each query. Then label each query-document pair as relevant or not. One labeler is enough if the queries are narrow. Two is better when relevance is fuzzy.

Don't overthink scale. A hundred labeled queries will tell you more than any public benchmark, because they carry your domain's vocabulary and failure patterns. Keep the set frozen once built. You'll reuse it every time you swap models.

Metrics that matter: NDCG@10, MRR, recall@k

NDCG@10 is the headline number. It rewards getting the most relevant documents into the top 10, with position-weighted scoring. If your pipeline feeds 5-10 passages to an LLM, NDCG@10 is the metric that matches your reality.

MRR matters when only the first hit counts. That's common in question answering and agent tool selection. Recall@k tells you whether the reranker is surfacing all relevant documents somewhere in the top k, which matters when your LLM can handle a larger context window.

Track all three. They catch different failures.

A/B testing rerankers in a live RAG pipeline

Offline metrics get you to a shortlist. Live traffic decides the winner. Run two rerankers side by side on a percentage of real queries. Log the final answer quality, not just the ranking.

The catch: you need a quality signal. User feedback, thumbs up/down, task completion rates. Without one, A/B testing just tells you which model is faster. That's still useful, but it's not the full picture.

When Not to Use a Reranker: Limitations and Failure Modes

Rerankers fix precision. They don't fix recall. If your retrieval stage never surfaces the right document in the first place, no reranker can save it. It can only rank what it's given.

When your recall stage is already broken

A reranker reorders candidates. It doesn't discover new ones. If your vector search returns 50 irrelevant passages, reranking gives you the least irrelevant 10. That's still useless.

The fix is upstream. Check your embedding model. Check your chunking strategy. Check whether hybrid search with BM25 would surface documents your dense embeddings miss. Only after recall is healthy does reranking earn its cost.

Here's a quick test: look at your top-50 candidates before reranking. If the relevant document isn't in there for most queries, you have a recall problem. A reranker will not fix it.

When latency budget rules out reranking

Every reranker adds inference time. Cross-encoders run a full transformer forward pass per query-document pair. That's fast per pair, but it multiplies across your candidate set.

If you're building an interactive agent that must respond in under 300ms, a cross-encoder reranking 50 candidates can eat your entire budget. You'll need to cut candidates, use a lighter model, or skip reranking entirely.

The honest answer: it depends on your candidate count and your model. A small cross-encoder on 10 candidates adds maybe 20-50ms. On 100 candidates, that's 200-500ms. Measure it before you commit.

When your corpus is too small or too homogeneous

If your corpus is a few hundred documents, or all your documents cover the same narrow domain, reranking adds little. Embedding similarity already separates relevant from irrelevant when the space is small and uniform.

Reranking earns its keep when your corpus is large, heterogeneous, and full of near-misses: documents that share vocabulary but differ in meaning. That's where embeddings confuse and rerankers clarify.

Don't add a reranker because the pattern says so. Add it because your evaluation set shows precision gains that justify the latency and cost.

Integrating a Reranker into Your RAG Pipeline

You've picked a model. You've tested it on your own data. Now you have to wire it into the pipeline without breaking what already works.

Pipeline placement: after vector search, before LLM

The reranker sits between retrieval and generation. Your vector database returns the top-k candidates, the reranker scores each query-document pair, and only the top few passages go into the LLM's context window.

Here's the flow: query in, embedding search out 50-100 candidates, reranker scores all of them, you keep the top 5-10, those go to the LLM. Nothing else changes. Your embedding model, your vector store, your prompt template all stay put.

The reranker is a filter, not a replacement. It narrows. It doesn't search.

Configuration: top-k candidates, score thresholds

Two knobs matter most: how many candidates you pull from vector search, and how many you keep after reranking.

Start with 50 candidates from retrieval and keep the top 10 after reranking. That's a common default, but it's not sacred. If your recall is strong, you can pull fewer. If your corpus is messy, pull more.

Score thresholds are trickier. Cross-encoder scores aren't calibrated probabilities. A score of 0.7 from one model means something different from 0.7 from another. Don't set a hard threshold until you've looked at the score distribution on your own queries.

The better approach: rank, don't threshold. Keep the top N and let the LLM decide. Thresholds make sense only when you need to reject everything below a confidence floor, like when you're routing to a fallback answer.

Common integration mistakes and how to avoid them

The most common mistake is reranking too few candidates. If you pull 10 from vector search and rerank those, you've capped your recall at whatever the embedding model found. The reranker never sees the document that should have been in the top 50 but wasn't.

The second mistake is reranking every query. Some queries are easy. The embedding similarity scores are already decisive. Reranking those adds latency and cost for zero gain. Route simple queries past the reranker, or use a cheap model for them.

The third mistake is caching nothing. Reranker scores for a query-document pair don't change unless the model or the document changes. If you see repeated queries, cache the reranked results. It's the cheapest latency win available.

The fourth mistake is treating the reranker as a black box. Log the pre-rerank and post-rerank orderings for a sample of queries. When precision drops, you'll know whether the reranker is the problem or the retrieval stage is.

Final Thoughts: Choosing the Right Reranker for Your Stack

You've seen the table. You've read the trade-offs. Now you need a decision path, not more options.

Start with your latency budget. If you're serving interactive queries and need answers in under 500ms, a cross-encoder like BGE-reranker-v2-m3 or ms-marco-MiniLM is the pragmatic default. They're fast, cheap to self-host, and good enough for most document retrieval. If you're building agent memory where precision matters more than raw speed, a multi-vector model or Cohere Rerank 3 earns its cost.

The honest answer is that most teams overthink this. Pick a reranker that's easy to deploy, test it on 50 of your own queries, and compare NDCG@10 against your current retrieval. If it doesn't beat your baseline by a meaningful margin, don't ship it. Reranking models compared on public benchmarks only tell you what's possible. Your data tells you what's true.

One more consideration: maintenance. Self-hosting means you own updates, GPU allocation, and monitoring. An API means you pay per query and trust someone else's uptime. Neither is wrong. The wrong move is picking a model you can't operate.

If you want reranking handled for you, GigaRAG's agent memory platform ships with it built into the retrieval stack. You don't configure the reranker, tune the top-k, or babysit the GPU. That's the option for builders who'd rather spend their time on the agent, not the plumbing.

Frequently Asked Questions

What are the best reranking models for RAG?

The best reranking model depends on your latency, cost, and privacy constraints. Open-source options like BGE, Ettin, Jina, and Mixedbread offer strong accuracy and full control, while hosted APIs like Cohere Rerank and Voyage provide top-tier accuracy with minimal ops overhead. Benchmark on your own domain data before committing.

How do reranking models compare on NDCG@10?

NDCG@10 scores vary by model, dataset, and domain. Open-source models like BGE and Ettin are competitive on standard benchmarks, while hosted models often lead on proprietary or general-domain evaluations. Always measure NDCG@10 on your own labeled queries rather than relying solely on public leaderboards.

Are open-source rerankers as good as Cohere Rerank?

Open-source rerankers can match or approach Cohere Rerank on many benchmarks, especially when fine-tuned on domain data. However, Cohere Rerank often provides stronger out-of-the-box accuracy and lower operational overhead. The trade-off is cost, data privacy, and control.

What is the latency of reranking models?

Latency depends on model size, hardware, batch size, and number of documents reranked. Small cross-encoders can rerank 20–50 documents in tens of milliseconds on a GPU, while larger models or CPU inference can take hundreds of milliseconds. Hosted APIs add network overhead but handle scaling for you.

How much do reranking models cost?

Open-source models are free to download but require GPU infrastructure, which costs money to run. Hosted reranking APIs typically charge per search or per token, with pricing that varies by provider and volume. Calculate total cost of ownership including engineering time and scaling.

When should I not use a reranker?

Skip reranking when your first-stage retrieval already returns highly relevant results, when latency budgets are extremely tight (under 50ms end-to-end), or when your corpus is small and simple. Rerankers add most value when you retrieve many candidates and need to surface the best few.

Can reranking models handle long documents and agent memory?

Most rerankers have a maximum context length (often 512 tokens), so long documents must be chunked. For agent memory, where context can be long and queries are dynamic, choose a reranker with a longer context window or use chunk-level reranking. Some newer models support up to 8k tokens.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through reranking models compared or something adjacent, we publish what we have actually tested, including where it falls short.