Choosing an Embedding Model for RAG: What Matters

Choosing an Embedding Model for RAG: What Actually Matters

Choosing an embedding model for RAG is a decision most agent memory and pipeline builders get wrong, and it's not their fault. The MTEB leaderboard sits at the top of every search result, so teams pick the highest-scoring model and call it done. Then retrieval quality tanks on their actual queries, and nobody can explain why.

The honest answer is that no single model is universally best. What works for a benchmark dataset often fails on your domain, your latency budget, your persistence requirements. At GigaRAG, we see this pattern constantly: builders optimize for a score instead of for their architecture.

You'll learn what embeddings actually do in RAG, which criteria matter most for agent memory and multi-turn retrieval, how to evaluate models on your own data, and what embeddings cannot solve no matter which model you pick. The end result is a five-step decision framework you can apply this afternoon, not another benchmark roundup.

In This Guide

• What Is an Embedding Model and Why It Matters for RAG
• Hosted Embedding API vs Self-Hosted Open-Source Model
• Types of Embedding Models for RAG
• Choosing An Embedding Model For Rag: A Step-by-Step Guide
• Evaluation Criteria That Actually Matter for Agent Memory and RAG Pipelines
• Comparing the Major Embedding Models for RAG
• A Decision Framework for Choosing an Embedding Model for RAG
• What You Cannot Do or Should Not Expect from Embedding Models
• When Embedding Model Choice Matters Less
• How to Evaluate Embedding Models on Your Own Data
• Final Thoughts on Choosing an Embedding Model for RAG

What Is an Embedding Model and Why It Matters for RAG

An embedding model converts text into fixed-length vectors of numbers, where semantically similar text lands close together in vector space. In RAG, this is what lets a system find the right passage from thousands of documents without scanning every word.

An embedding model takes a string of text and outputs a list of numbers, typically 768 to 3,072 of them. Those numbers aren't random. They encode meaning: two sentences about the same topic will produce vectors that sit near each other, while unrelated text produces vectors far apart. You can measure that distance with cosine similarity, and that measurement is the entire retrieval mechanism.

How embeddings turn text into vectors

Behind the scenes, the model passes your text through a transformer network trained to predict which passages are related. The final hidden layer becomes the vector. The training objective matters more than the architecture: models trained on paired data (query, relevant passage) learn to pull matching pairs together and push non-matching pairs apart. That's why a model fine-tuned on search data beats a generic language model at retrieval, even if the generic model is larger.

The output is a fixed-size vector regardless of input length. A 10-word query and a 500-word passage both become, say, 1,024 numbers. That fixed size is what makes vector databases fast: every comparison costs the same, and you can index billions of vectors with approximate nearest neighbor search.

The role of embeddings in RAG retrieval

RAG works in two stages. First, retrieval: your query gets embedded, then the vector database finds the nearest passages. Second, generation: those passages go into the LLM's context window along with the query, and the model writes an answer.

The embedding model controls the first stage entirely. If it retrieves the wrong passages, the LLM can't recover, no matter how capable it is. The generator only sees what retrieval hands it.

Why embedding quality directly impacts answer accuracy

Retrieval quality sets a ceiling on answer quality. If the correct passage isn't in the top 5 or top 10 results, the LLM answers from whatever it did receive, which means hallucination or a generic response. A better embedding model raises recall: the chance that the right passage appears in your retrieval window at all.

But keep this in perspective. Embedding quality is one lever among several. Chunking strategy, reranking, and prompt assembly often move accuracy more than swapping one embedding model for another. You'll see that trade-off in detail later. For now, the point is simpler: embeddings are the retrieval mechanism, and retrieval is where RAG either works or doesn't.

[!note] MTEB scores are useful for narrowing a shortlist, but they are measured on public benchmarks that rarely match your domain, language mix, or chunk sizes, so they should never be the deciding factor.

Hosted Embedding API vs Self-Hosted Open-Source Model

Types of Embedding Models for RAG

Not all embedding models work the same way. The type you pick changes what your retrieval can find, how much storage you need, and how fast queries run. Here's the taxonomy that matters for RAG.

Dense vs. sparse embeddings

Dense embeddings are the default. Every dimension carries some meaning, and the vector is a compressed representation of the text. They're what OpenAI, Cohere, and most open-source models produce. They handle paraphrase well: "how do I reset my password" and "password recovery steps" land close together even though they share few words.

Sparse embeddings work differently. Most dimensions are zero, and the non-zero ones map to specific terms or n-grams. Think BM25 with a learned twist. Sparse models like SPLADE excel at exact keyword matching, which dense models sometimes miss. Hybrid retrieval, combining both, often beats either alone. But it costs more to run two indexes.

Multi-vector and long-context embeddings

Standard models compress an entire passage into one vector. That's a lossy operation. A 500-word chunk about five topics becomes one point in space, and the dominant topic wins. Multi-vector models like ColBERT store one vector per token instead. Retrieval then compares query tokens against passage tokens directly, which improves precision on long or mixed-topic documents.

The trade-off is storage. ColBERT-style embeddings can take 10 to 50 times more space than a single dense vector per passage. Long-context models, by contrast, keep the single-vector format but extend the input window to 8,000 tokens or more. They're useful when your chunks are large, but they don't solve the compression problem, they just delay it.

Variable-dimension embeddings (Matryoshka)

Matryoshka models let you truncate the vector without retraining. A 3,072-dimension embedding from OpenAI's text-embedding-3 can be cut to 1,024 or 256 dimensions and still retain most of its retrieval quality. That's a real lever: smaller vectors mean faster search and lower storage costs.

The catch is that quality drops as you cut. At 256 dimensions, you'll lose recall on nuanced queries. Test the truncation on your own data before committing to it.

Code and multimodal embeddings

Code embeddings are trained on source code and docstrings, so they understand function names, variable patterns, and structural similarity. If your RAG pipeline retrieves code snippets, a general text model will underperform. Use a code-specific model.

Multimodal embeddings map text and images into the same vector space. That lets you retrieve an image from a text query, or vice versa. For most RAG pipelines built on documents, this is overkill. It matters when your knowledge base includes diagrams, screenshots, or product photos alongside text.

[!tip] For agent memory pipelines, evaluate models on multi-turn retrieval where the query depends on earlier conversation turns, not just single-shot queries; a model that looks strong on isolated queries can degrade when the query is short or ambiguous.

Choosing An Embedding Model For Rag: A Step-by-Step Guide

1. Define your retrieval constraints first: corpus size, languages, average chunk length, latency budget, and whether data can leave your infrastructure.
2. Build a small labeled evaluation set of real queries paired with the passages that should be retrieved.
3. Shortlist 3-5 candidate models that fit your constraints, mixing hosted and open-source options.
4. Embed your corpus and queries with each candidate, keeping chunking and preprocessing identical across runs.
5. Measure retrieval quality with recall@k and MRR on your evaluation set, not on public leaderboards.
6. Benchmark end-to-end latency and cost per query at your expected volume, including re-ranking if you use it.
7. Pick the model that meets your quality bar at acceptable cost, then re-test after any chunking or re-ranker change.

Evaluation Criteria That Actually Matter for Agent Memory and RAG Pipelines

MTEB leaderboards rank models on dozens of tasks, most of which have nothing to do with your retrieval workload. A model that wins on semantic textual similarity might lose on your legal documents. Here's what to measure instead.

Cost per embedding and total cost of ownership

API models charge per token. That sounds simple until you do the math on a real pipeline. Embedding 10 million tokens at OpenAI's standard rate costs real money, and you pay again every time your corpus changes. Self-hosted open-source models have no per-token fee, but you're paying for GPU hours, engineering time, and ongoing maintenance.

Total cost of ownership includes re-embedding. If you change chunking strategy, you re-embed everything. If you switch models, you re-embed everything. A model that's 2% better on MTEB but costs 5 times more to run is the wrong choice for most teams.

Latency and throughput for real-time retrieval

Agent memory systems retrieve on every turn. If your embedding step adds 200 milliseconds, that's 200 milliseconds before the LLM even starts generating. Multi-turn agents compound this: five retrieval calls per conversation turn means a full second of latency from embeddings alone.

Throughput matters when you're embedding at scale. Batch processing 50 million documents is a different workload than embedding 50 queries per second. API providers throttle you. Self-hosted models need enough GPU memory to keep up. Test both before committing.

Dimensionality and storage implications

Vector dimension drives storage cost directly. A 3,072-dimension vector takes 12KB in float32. Ten million vectors is 120GB before any index overhead. Cut to 768 dimensions and you're at 30GB. That's the difference between one database node and four.

Matryoshka models give you a dial here. You can store at full dimension and search at reduced dimension, or store truncated vectors outright. The trade-off is recall, and it's not linear. Test your queries at each truncation level.

Domain fit: general vs. specialized embeddings

General models trained on web text handle everyday language well. They struggle with domain-specific vocabulary: medical codes, legal citations, internal product names. If your corpus is specialized, a general model will retrieve the wrong passages with confidence.

Specialized models exist for biomedical text, legal text, code, and multilingual content. They're trained on domain corpora, so "myocardial infarction" and "heart attack" land close together. The catch is that specialized models often underperform on general queries. If your RAG pipeline serves mixed content, you may need two models or a hybrid approach.

Context window and long-document handling

Most embedding models cap input at 512 tokens. That's fine if your chunks are small. It's a problem if you're embedding full documents or long sections. Text beyond the limit gets truncated silently, and retrieval quality drops without any warning.

Long-context models extend this to 8,000 tokens or more. But longer input doesn't mean better embeddings. The model still compresses everything into one vector, and the dominant topic wins. For long documents, chunk first, then embed. The context window matters for how you chunk, not for how you embed.

Comparing the Major Embedding Models for RAG

Three or four models cover most RAG workloads. You don't need to track every entry on the MTEB leaderboard. You need to know which models handle your retrieval pattern, your domain, and your budget.

OpenAI text-embedding-3: pros and cons

OpenAI's text-embedding-3 family comes in two sizes: small and large. Small runs at 1,536 dimensions, large at 3,072. Both support Matryoshka truncation, so you can cut dimensions down to 256 or 512 and keep most of the recall. That's the main draw for pipeline builders who want to shrink storage without switching models.

The catch is cost and control. You pay per token, and you pay again every time your corpus changes. You can't fine-tune these models. You can't inspect them. If OpenAI changes the model behind the API, your embeddings shift and you re-embed everything. For most teams, the convenience wins. For teams with strict data residency or long-term cost pressure, it doesn't.

Cohere Embed: strengths and weaknesses

Cohere's embed-v3 models are strong on multilingual retrieval and handle long inputs well. The v3 family supports input up to 512 tokens for the base model, with a multilingual variant that covers over 100 languages. If your corpus mixes English, Spanish, and Japanese, Cohere is worth testing before you assume OpenAI is the default.

The weakness is ecosystem lock-in. Cohere's API is solid, but fewer vector databases and RAG frameworks treat it as a first-class citizen compared to OpenAI. You'll spend more time on integration glue. Pricing is per token, similar to OpenAI, so the same re-embedding cost applies.

Open-source options: BGE, E5, and sentence-transformers

BGE from BAAI is the workhorse here. The bge-large-en-v1.5 model sits near the top of MTEB for retrieval tasks and runs on a single GPU. You self-host it, so there's no per-token fee. You control the version. You can fine-tune it. The trade-off is that you own the infrastructure: GPU provisioning, batching, versioning, and monitoring.

E5 from Microsoft is the other serious contender. The e5-large-v2 model performs comparably to BGE on many retrieval benchmarks and handles asymmetric search well (short query, long passage). Sentence-transformers is the framework that makes both easy to use. It wraps hundreds of models behind one Python API, so you can swap BGE for E5 in a few lines of code.

The main catch with open-source models is the hidden engineering cost. You're not paying per token, but you're paying someone to keep the embedding service running. For a team without ML infrastructure experience, that cost is real and often underestimated.

When to consider fine-tuning an embedding model

Fine-tuning helps when your domain has vocabulary that general models get wrong. Medical codes, legal citations, internal product names. If your golden dataset shows recall@10 below 0.7 on domain queries, fine-tuning a BGE or E5 model on your own query-passage pairs can push that number up meaningfully.

Don't fine-tune as a first step. Build the pipeline with a general model, measure retrieval quality on real queries, and only fine-tune if the numbers justify it. Fine-tuning requires a labeled dataset of at least a few thousand query-passage pairs, and it doesn't fix bad chunking or weak reranking. It fixes domain vocabulary gaps. Nothing else.

A Decision Framework for Choosing an Embedding Model for RAG

Here's the framework I use when a team asks me which embedding model to pick. It's five steps, and each step eliminates options until one or two models remain. You can run through it in an afternoon.

Step 1: Define your retrieval constraints

Before you look at a single model card, write down four numbers. Your latency budget per query in milliseconds. Your corpus size in documents. Your monthly budget for embedding costs. Your storage ceiling in gigabytes.

These numbers kill most options immediately. If you need retrieval under 50ms, a 3,072-dimension model on CPU is out. If you have 50 million documents, per-token API pricing gets expensive fast. If your budget is zero, self-hosted open-source is your only path. Most teams skip this step and pick a model because a benchmark told them to. Then they discover the model doesn't fit their infrastructure.

Step 2: Match embedding type to your use case

Dense embeddings handle semantic similarity well. Sparse embeddings (BM25, SPLADE) handle exact keyword matches well. If your queries include product codes, error messages, or legal citations, you need sparse or hybrid retrieval. If your queries are natural-language questions, dense alone often works.

Agent memory adds a constraint: persistence. If your agent retrieves from memory across multiple turns, you need embeddings that stay stable across sessions. That means versioning your model and re-embedding when you change it. It also means testing whether the model handles multi-turn query reformulation without losing the original intent.

Step 3: Shortlist models by cost and latency

Take the constraints from Step 1 and apply them to the models from the previous section. OpenAI text-embedding-3-small is cheap per token but you pay on every re-embed. BGE is free per token but costs GPU time. Cohere is strong multilingual but adds integration work.

Here's the rule of thumb I use: if your corpus changes less than once a week and you have no data residency requirements, a hosted API model is fine. If your corpus changes daily or you have compliance constraints, self-host. The re-embedding cost of a hosted model on a frequently changing corpus will eat your budget faster than you expect.

Step 4: Run a custom evaluation on your own data

Don't trust MTEB. Build a golden dataset of 50 to 100 real queries from your users, with the correct passages marked. Run each shortlisted model against it. Measure recall@10 and NDCG. Also measure latency at your target throughput.

This step takes a day and saves you months of debugging a model that looked good on a leaderboard but fails on your domain. If you're choosing an embedding model for RAG in Python, the sentence-transformers library makes this evaluation script about 50 lines.

Step 5: Validate with real queries before committing

Ship the top model behind a flag. Route 10% of real traffic through it for a week. Log retrieval quality manually on a sample of queries. If recall holds and latency stays under budget, commit. If not, you still have the runner-up model ready to swap in.

This framework matters more than any model comparison table. Constraints first, then type, then cost, then evidence. In that order.

What You Cannot Do or Should Not Expect from Embedding Models

Embedding models are a retrieval component, not a retrieval system. They map text to vectors. They don't chunk your documents, rank your results, or write your prompts. Expecting them to fix those things leads to wasted evaluation cycles and a model you blame for failures that happened elsewhere.

Embeddings alone won't fix bad chunking

If your chunks split a definition across two documents, no embedding model recovers it. The vector for each half points somewhere else. Retrieval fails, and the model gets blamed.

Chunking sets the ceiling. Embeddings can only retrieve what the chunks preserve. Fix your chunk boundaries before you swap models. A 200-token chunk that cuts a procedure mid-step will fail with every embedding model on the market.

MTEB scores don't predict your domain performance

MTEB averages performance across dozens of tasks. Your domain is one task. A model that ranks third overall might rank first on legal retrieval and twentieth on medical. The average hides that.

Worse, MTEB datasets are public. Some models are trained on them, directly or indirectly. The score measures memorization as much as generalization. Your private corpus wasn't in the training data. The only number that matters is the one you measure on your own queries.

No model is universally best across all tasks

This isn't a hedge. It's a structural fact. Dense models win on paraphrase-heavy queries. Sparse models win on exact identifiers. Multilingual models trade English performance for coverage. Long-context models trade speed for span.

A model that beats everything on one benchmark loses on another. The leaderboard order changes every few months. Pick the model that fits your constraints from Step 1, not the one at the top this week.

Embedding quality matters less than reranking and prompt engineering

Here's the uncomfortable truth: swapping a mid-tier embedding model for a top-tier one often moves recall@10 by a few points. Adding a reranker moves it by ten to twenty. Fixing your prompt moves answer quality more than either.

Embeddings do coarse filtering. Reranking does fine-grained selection. The LLM does the final assembly. If your budget forces a choice, spend it on the reranker and the prompt, not the embedding model. A decent embedding model with a good reranker beats a great embedding model with no reranker, every time.

When Embedding Model Choice Matters Less

The previous section made the case that embeddings are one component among several. This section makes the practical version of that point: if you have a limited budget of time and attention, the embedding model is often the wrong place to spend it.

Chunking strategy: the overlooked lever

Chunking determines what the embedding model ever gets to see. A 500-token chunk that mixes three unrelated topics produces a vector that points at none of them cleanly. Retrieval fails no matter how good the model is.

The fix is usually boring: smaller chunks, overlap between chunks, and boundaries that respect document structure. A 150-token chunk with 20-token overlap will beat a 500-token chunk with no overlap on the same embedding model, often by a wider margin than swapping models entirely. Test your chunk size before you test a new model. It's cheaper and the effect is larger.

Reranking: where retrieval quality is won or lost

Embeddings do coarse filtering. They pull 50 or 100 candidates from a vector index. A reranker then scores those candidates against the query with a cross-encoder, which reads both texts together rather than comparing vectors. That second pass is where precision comes from.

In practice, a mid-tier embedding model with a reranker outperforms a top-tier embedding model without one on most retrieval tasks. The reranker fixes the embedding model's mistakes. The embedding model can't fix its own. If you're deciding between upgrading your embedding model or adding a reranker, add the reranker.

Prompt engineering and context assembly

The LLM sees whatever retrieval returns. If you stuff eight chunks into the context window in arbitrary order, the model answers from noise. If you order them by relevance, trim the irrelevant ones, and tell the model which chunks matter, answer quality improves without touching the embedding model at all.

Context assembly is where a lot of RAG pipelines quietly fail. The embedding model retrieves. The prompt decides what the LLM does with what was retrieved. A clear prompt with well-ordered context beats a vague prompt with perfect retrieval. Spend your iteration cycles there before you spend them on model selection.

How to Evaluate Embedding Models on Your Own Data

Leaderboards rank models on datasets you'll never query. Your users ask about your documents, your codebase, your support tickets. The only evaluation that predicts your retrieval quality is one run on your data. It takes an afternoon.

Building a small golden dataset

Start with 50 to 100 real queries. Pull them from search logs, support tickets, or questions your team actually asked. Don't invent them. For each query, mark the document IDs or chunk IDs that should be retrieved. That's your ground truth.

Fifty queries is enough to separate a bad model from a good one. It won't tell you the difference between two models within a point of each other, but that difference rarely matters in production. If you can't find 50 real queries, you don't have a retrieval problem yet.

Keep the golden set versioned. When you change chunking or add documents, re-check the labels. A golden set that drifts is worse than none.

Metrics that matter: recall@10, NDCG, and latency

Recall@10 measures whether the right chunk appears in the top 10 results. It's the metric that matters most for RAG, because your reranker or LLM only sees what retrieval returns. If the right chunk isn't in the top 10, it's gone.

NDCG accounts for position. A correct chunk at rank 1 scores higher than the same chunk at rank 9. Use it when you care about ordering, which you should if you're stuffing multiple chunks into a context window.

Latency is the third metric and the one leaderboards ignore. Measure end-to-end embedding time per query, not just model inference. Network calls to hosted APIs add 50 to 200 ms. That matters if you're doing multi-turn retrieval in an agent loop.

A simple evaluation workflow in Python

You don't need an evaluation framework. A script that loads your golden set, embeds the queries, runs a vector search, and computes recall@10 is maybe 80 lines. Use sentence-transformers for local models or the OpenAI SDK for hosted ones.

The workflow: embed all documents once, cache the vectors, then loop over queries. For each query, embed it, search the index, and check whether the labeled chunk is in the top 10. Tally the hits. Divide by total queries. That's your recall@10.

Run the same script against two or three candidate models. The one with the best recall on your data wins. If two models are within a point, pick the cheaper or faster one. Don't overthink it.

Final Thoughts on Choosing an Embedding Model for RAG

You don't need the best embedding model. You need the one that fits your retrieval constraints, your latency budget, and your domain. The teams that get stuck are the ones chasing MTEB scores instead of running a 50-query test on their own data.

The decision framework is simple. Define what you're retrieving and how fast it has to come back. Match the embedding type to that use case. Shortlist by cost and latency, not leaderboard rank. Run your own evaluation. Validate with real queries before you commit. That's it.

Keep the honest limits in view. Embeddings won't fix bad chunking. No model is universally best across every task. And if your chunking, reranking, and prompt assembly are sloppy, the embedding model choice barely moves the needle.

If you're building agent memory or a RAG pipeline, GigaRAG is built for exactly this: persistent memory, multi-turn retrieval, and a pipeline you can swap embedding models in and out of without rewriting the whole thing. It won't pick the model for you. But it makes implementing your choice, and changing it later, a lot less painful.

Frequently Asked Questions

How do I choose an embedding model for RAG?

Start from your constraints, not the leaderboard. Define corpus size, languages, latency budget, and privacy requirements, then shortlist models that fit. Evaluate the shortlist on a small labeled set of your own queries using recall@k and MRR, and confirm the winner meets your cost and latency targets.

What is the best free embedding model for RAG?

Several open-weight models are free to self-host and perform well on retrieval tasks, but the best one depends on your domain and language mix. Rather than trusting a single recommendation, benchmark two or three open models on your own data. Hosting costs and hardware requirements still apply even when the weights are free.

Does embedding model choice matter more than chunking strategy?

Often chunking, re-ranking, and prompt design move retrieval quality more than swapping between two competent embedding models. A strong model on badly chunked text will underperform a decent model on well-chunked text. Treat embedding choice as one lever among several, and re-evaluate it after you have tuned chunking.

How do I choose an embedding model for RAG in Python?

Use a library such as sentence-transformers for open models or an HTTP client for hosted APIs, and wrap both behind the same interface so you can swap them. Embed your evaluation set with each candidate, compute recall@k and MRR, and compare. Keeping the interface stable makes re-testing cheap when you change chunking or re-rankers.

What embedding dimension should I use for RAG?

Higher dimensions can improve recall but increase storage, memory, and search latency roughly linearly. Many production pipelines work well in the range of a few hundred to just over a thousand dimensions. Some models support dimension truncation, letting you trade a small amount of quality for lower cost.

Can I switch embedding models after building my RAG pipeline?

Yes, but you must re-embed the entire corpus and rebuild the index, because vectors from different models are not comparable. Plan for this by storing raw text and chunk metadata separately from vectors. Budget for re-embedding time and cost before committing to a model.

Do I need a multimodal embedding model for RAG?

Only if your corpus contains images, diagrams, or scanned pages that carry meaning text alone cannot capture. Multimodal models add complexity and cost, so use them when the retrieval task genuinely requires cross-modal search. For text-only corpora, a strong text embedding model is usually the better choice.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through choosing an embedding model for rag or something adjacent, we publish what we have actually tested, including where it falls short.