What Embeddings Actually Encode: A Practical Guide for RAG and Agent Memory Builders
If you build RAG pipelines or agent memory systems, you've probably shipped embeddings into production without fully knowing what the vector actually captures. That's not a knock. Most builders treat embeddings as a black box that turns text into numbers, and the retrieval mostly works, so the box stays closed. The problem shows up later, when retrieval returns something semantically adjacent but factually wrong, and you can't explain why without opening the box.
Here's the honest answer: embeddings are lossy compression. They keep the relationships between words and passages, and they throw away almost everything else. What's lost matters as much as what's kept, especially when you're designing retrieval thresholds or agent memory that has to decide what's worth remembering. This guide covers what embeddings actually encode, what they can't encode, and how to apply that mental model to RAG retrieval and agent memory design.
| At a glance | Details |
|---|---|
| Core idea | Vectors encode learned semantic relationships |
| What's captured | Meaning, context, similarity patterns |
| What's missing | Exact facts, logic, temporal order |
| Typical dimensions | 768 to 3072 floats |
| Key limitation | No explicit knowledge or reasoning |
| Best for | Semantic search and clustering |
In This Guide
- What Is an Embedding?
- Embeddings vs Traditional Encodings: What's the Difference?
- Encoding vs Embedding: What's the Difference?
- What Embeddings Actually Encode: A Step-by-Step Guide
- How Embeddings Actually Work
- What Embeddings Actually Encode: Semantic Relationships
- Types of Embeddings
- BERT vs Word2Vec: What Changed
- Does ChatGPT Use Embeddings?
- What Embeddings Cannot Encode: Honest Limitations for Builders
- Embeddings in RAG Pipelines: Practical Implementation
- Embeddings for Agent Memory
- Common Mistakes When Working with Embeddings
- Final Thoughts
What Is an Embedding?
An embedding is a dense vector of numbers that represents a piece of text, image, or other data in a way that captures semantic relationships. Words or phrases with similar meanings end up close together in vector space.
The vector representation
An embedding turns text into a list of floating-point numbers. A word like "king" might become a vector of 768 dimensions, each dimension holding a value between roughly -1 and 1. You can't read meaning from any single number. The pattern across all of them is what matters.
That pattern is learned, not hand-coded. A neural network trains on millions of examples and adjusts those numbers until similar concepts land near each other. "Dog" and "puppy" get vectors that point in roughly the same direction. "Dog" and "democracy" don't.
Why embeddings exist
Computers can't compare words directly. They can compare numbers. An embedding gives you a way to measure how related two pieces of text are: compute the distance between their vectors. Small distance means similar. Large distance means not.
That's the whole point. Embeddings turn fuzzy human language into geometry you can do math on.
[!note] Embeddings do not encode factual knowledge or logical rules; they only capture statistical patterns of co-occurrence and context from the training data.
Embeddings vs Traditional Encodings: What's the Difference?
| Factor | Embeddings | Traditional Encodings |
|---|---|---|
| Representation | Dense, learned vectors | Sparse, rule-based codes |
| Semantic meaning | Captures relationships and context | No inherent meaning |
| Dimensionality | Hundreds to thousands of floats | Often high-dimensional but sparse |
| Use case | Similarity search, RAG, agent memory | Categorical data, hashing, compression |
| Training | Requires large corpora and models | Deterministic, no training needed |
Encoding vs Embedding: What's the Difference?
Encoding is the process. Embedding is the output. You encode text into an embedding. The two terms get used interchangeably, but the distinction matters when you're debugging a pipeline.
Encoding as a process
Encoding takes raw input, text or otherwise, and runs it through a model that produces a vector. Tokenization happens first: the text gets split into pieces the model can process. Then the neural network does its work, layer by layer, transforming those tokens into numbers.
Embedding as the result
The embedding is the vector you get at the end. It's a fixed-length array of floats. Once you have it, the encoding step is done. You can store it, index it, compare it against other embeddings. The process is transient. The result is what you keep.
Why the distinction matters in practice
When retrieval quality drops, you need to know which part failed. Bad tokenization means the encoding step mangled the input. A bad model means the embeddings themselves don't capture what you need. If you treat them as the same thing, you'll debug the wrong layer.
[!tip] For agent memory, store embeddings alongside raw text and metadata. This lets you re-rank or filter results using exact matches or rules when pure vector similarity falls short.
What Embeddings Actually Encode: A Step-by-Step Guide
- Select a diverse set of query-document pairs from your domain.
- Generate embeddings for all queries and documents using your chosen model.
- Compute cosine similarity between each query and all documents.
- Rank documents by similarity and check if the top results are relevant.
- Analyze failure cases: are they due to missing context, ambiguity, or domain mismatch?
- Test with paraphrased queries to see if semantic meaning is captured.
- Iterate on model choice, chunking, or fine-tuning based on findings.
How Embeddings Actually Work
The previous section split encoding from embedding. Now the mechanics. Here's what happens when you run text through an embedding model.
From text to tokens
The model can't read words. It reads tokens: subword units that split "unhappiness" into something like "un", "happiness". A tokenizer maps each token to an integer ID. That ID is the model's only input. Nothing about spelling survives past this point.
The neural network's role
The token IDs feed into a transformer. Each layer applies attention: every token looks at every other token and adjusts its representation based on context. The network was trained on a task like next-token prediction. The weights it learned encode patterns about which words appear in similar contexts. Those patterns are what get projected into the vector.
The resulting vector space
The final layer outputs a fixed-length vector, typically 384 to 1536 dimensions. Each dimension is a learned feature, not a human-readable one. Similar texts land close together in this space. The distance is measured with cosine similarity. That's the whole trick: meaning becomes geometry.
What Embeddings Actually Encode: Semantic Relationships
An embedding doesn't store meaning. It stores a position in a space where similar things sit close together. The vector is a coordinate, not a definition.
Semantic similarity, not meaning
When two vectors land near each other, the model is saying: these texts appeared in similar contexts during training. That's distributional similarity, not semantic equivalence. "The cat sat on the mat" and "A dog rested on the rug" will be close. But so will "I love this" and "I hate this", because both express strong sentiment. The model learned co-occurrence patterns, not truth.
Contextual relationships
Modern embeddings capture relationships that shift with context. The word "bank" gets a different vector in "river bank" than in "bank account". This is the BERT-era improvement over static embeddings. The vector encodes how a token functions in that specific sentence, not a fixed dictionary entry.
What is lost in the encoding
The compression is lossy. Exact numbers, dates, and proper nouns blur together. Negation often survives poorly: "not good" can land near "good". The embedding keeps the shape of the relationship, not the specifics. For RAG builders, this means embeddings are good for finding relevant passages, bad for answering factual questions directly.
Types of Embeddings
Embeddings split along three axes: how sparse the vector is, whether the vector changes with context, and what unit of text gets embedded. Each choice encodes something different.
Sparse vs dense embeddings
Sparse embeddings are mostly zeros. TF-IDF and one-hot encodings fall here. Each dimension maps to a specific word or feature, so you can read the vector directly. Dense embeddings pack information into every dimension, usually 300 to 1,536 of them. You can't read them, but they capture relationships sparse vectors miss.
Static vs contextual embeddings
Static embeddings assign one vector per word, no matter the sentence. Word2Vec and GloVe work this way. "Bank" gets the same vector in "river bank" and "bank account". Contextual embeddings generate a fresh vector each time a word appears, based on surrounding tokens. BERT and most modern models do this.
Word, sentence, and document embeddings
Word embeddings represent single tokens. Sentence embeddings, from models like Sentence-BERT, encode a full sentence into one vector. Document embeddings do the same for longer passages, often by pooling or averaging token vectors. For RAG, sentence and document embeddings matter most: you retrieve chunks, not words.
BERT vs Word2Vec: What Changed
Word2Vec gave every word one fixed vector. BERT gives every word a vector that changes with its sentence. That's the whole shift, and it matters more than most people think.
Word2Vec: one vector per word
Word2Vec trains on a large corpus and learns one vector per word. "Bank" gets a single vector, whether it's a river bank or a bank account. The model can't tell the difference because it never sees the difference. It encodes statistical co-occurrence: words that appear in similar contexts get similar vectors. That's useful for similarity search, but it's a blunt instrument.
BERT: context-dependent vectors
BERT reads the whole sentence before assigning any vector. The word "bank" in "river bank" gets a different vector than "bank" in "bank account". Attention layers weigh surrounding tokens, so the vector encodes the word's role in that specific sentence, not just its dictionary meaning. This is why BERT handles polysemy, negation, and word order far better than Word2Vec.
Implications for RAG builders
If your retrieval pipeline uses static embeddings, you'll miss context-dependent matches. A query about "bank account fees" might retrieve river bank documents. Contextual embeddings fix that. The tradeoff: BERT-style models are slower and cost more to run. For RAG, the accuracy gain usually justifies the cost.
Does ChatGPT Use Embeddings?
Yes. ChatGPT uses embeddings at two levels: inside the transformer to represent tokens, and in retrieval features that search through past conversations or uploaded files. You never see the vectors, but they're doing the work.
Embeddings inside the transformer
Every token that enters ChatGPT gets converted to a vector before any attention layer runs. The model learns these token embeddings during training, then refines them through each transformer block. By the final layer, each token's vector encodes its meaning in the context of the full prompt. That's what the model reads to predict the next token.
Embeddings in ChatGPT's RAG-like features
When you upload a file or ask ChatGPT to search your past conversations, it doesn't scan raw text. It embeds your query and the stored content into the same vector space, then retrieves the closest matches. That's retrieval augmented generation under the hood. The embeddings let it find semantically similar passages even when the wording doesn't match exactly.
What Embeddings Cannot Encode: Honest Limitations for Builders
Embeddings are lossy compression. They keep what's statistically common and drop what's rare or precise. If you expect them to store facts, you'll build retrieval that fails silently.
Negation and contradiction
"Not recommended for production" and "recommended for production" sit close together in vector space. The negation word barely moves the vector. Your similarity search will retrieve the wrong passage and never know it. Don't rely on embeddings to distinguish a claim from its opposite.
Temporal and causal relationships
Embeddings capture that "launch" and "release" are related. They don't encode that A happened before B, or that B caused A. A vector for "the server crashed after the deploy" looks nearly identical to "the deploy crashed after the server." Sequence and causality are invisible to cosine similarity.
Exact values and identifiers
Numbers, SKUs, error codes, version strings. Embeddings treat "404" and "403" as similar because they're both three-digit numbers near each other in training data. But in your system, they mean completely different things. If your retrieval depends on exact values, store them as metadata and filter before you search.
Rare and out-of-vocabulary terms
Your internal API names, proprietary error messages, unusual domain jargon. If a term didn't appear often in the embedding model's training data, its vector is noise. The model will map it to something vaguely similar and retrieve irrelevant chunks. Fine-tuning helps, but it won't fix terms the model has never seen.
Embeddings in RAG Pipelines: Practical Implementation
You've seen what embeddings can't do. Now here's what to do about it. The retrieval quality you get depends on three decisions you make before a single query runs: which model, what chunk size, and where you set the similarity bar.
Choosing an embedding model
Start with a sentence transformer, not a raw BERT or Word2Vec model. Models like all-MiniLM-L6-v2 or bge-base-en-v1.5 give you document-level vectors out of the box. Check the MTEB leaderboard for retrieval-specific scores, not just semantic similarity benchmarks. A model that scores well on STS won't necessarily retrieve well.
Chunking strategy and its impact
Chunk size changes what your vectors encode. Too large and each vector averages out the specific detail you're searching for. Too small and you lose context. Start with 256 to 512 tokens per chunk with 10 to 20 percent overlap. Test retrieval on real queries before tuning further.
Similarity thresholds and retrieval quality
Cosine similarity scores are relative, not absolute. A score of 0.7 means nothing on its own. It depends on your model, your data, and your query distribution. Run a batch of known-good and known-bad queries, plot the score distributions, and set your threshold where they separate. Recheck it whenever you change models or chunking.
Embeddings for Agent Memory
Agent memory is not a database. It's a working set of past experiences the agent retrieves to inform the next action. Embeddings make that retrieval possible by turning each experience into a vector that can be matched against the current situation.
How agents use embeddings for memory
An agent stores each interaction as a vector in a memory store. When a new task arrives, it embeds the current state and runs a similarity search against past experiences. The top matches become context for the next decision. This is how agents "remember" what worked before without being explicitly programmed with rules.
Memory retrieval vs RAG retrieval
RAG retrieval pulls from a fixed corpus. Agent memory retrieval pulls from a growing, changing store of the agent's own actions and outcomes. The difference matters: in RAG, stale documents are a data problem. In agent memory, stale experiences are a reasoning problem. The agent can retrieve a memory that contradicts what it just learned.
Memory decay and consolidation
Memories should not last forever. Old experiences need to decay or the store fills with noise. Some systems apply a recency weight to similarity scores. Others consolidate: multiple similar memories get merged into a single summary vector. Both approaches trade fidelity for relevance. The honest answer is that no standard solution exists yet. You'll tune decay rates per agent and per task type.
Common Mistakes When Working with Embeddings
Most retrieval failures trace back to three decisions made before the first query runs. Fix these and you fix most of what goes wrong downstream.
Using the wrong embedding model
A model trained on product reviews won't encode legal text well. Match the model to your domain. If you're unsure, test two models on your actual queries and compare retrieval quality. Don't pick based on benchmark leaderboards alone.
Ignoring chunk size
Chunks that are too large dilute the vector. Too small and you lose context. For most RAG pipelines, 200 to 500 tokens per chunk works. Test different sizes on your own documents. The right size depends on what your queries look like.
Treating similarity scores as absolute
A cosine similarity of 0.8 doesn't mean "correct." It means "close in vector space." Different models produce different score ranges. Set thresholds by testing on your own data, not by copying numbers from a tutorial.
Final Thoughts
Embeddings encode relationships, not facts. They capture how words and chunks sit near each other in semantic space. They don't store truth, negation, time, or exact values. Builders who treat vectors as meaning will keep hitting retrieval failures that look mysterious but aren't.
The fix is treating embeddings as one component in a pipeline, not the whole answer. Chunk deliberately. Test models on your own queries. Set thresholds from your data, not a blog post. When you need factual precision, pair embeddings with structured filters or a cross-encoder rerank.
GigaRAG exists for builders doing exactly this work. It gives you the infrastructure to apply these principles in RAG pipelines and agent memory systems without hand-rolling the retrieval layer. What embeddings actually encode is a design constraint. Work with it, and your retrieval stops surprising you.
Frequently Asked Questions
How do embeddings actually work?
Embeddings map discrete items (words, sentences, images) to dense vectors in a continuous space. During training, the model learns to place similar items closer together. The vector dimensions themselves are not human-interpretable, but distances reflect semantic similarity.
Why is BERT better than Word2Vec?
BERT generates context-dependent embeddings, meaning the same word gets different vectors depending on surrounding words. Word2Vec produces static embeddings, so 'bank' has one vector regardless of whether it means a riverbank or a financial institution. This makes BERT more suitable for nuanced tasks.
Does ChatGPT use embeddings?
ChatGPT itself does not use embeddings for generating responses; it uses a transformer architecture that processes tokens directly. However, embeddings are often used in retrieval-augmented generation (RAG) systems that feed relevant context into ChatGPT or similar models.
What are different types of embeddings?
Common types include word embeddings (Word2Vec, GloVe), contextual embeddings (BERT, RoBERTa), sentence embeddings (Sentence-BERT), and image embeddings (CLIP). Each is trained on different data and objectives, capturing different aspects of similarity.
What do embeddings not encode?
Embeddings do not explicitly encode factual knowledge, logical relationships, temporal order, or exact string matches. They also lack interpretability: you cannot easily inspect a vector to understand why two items are similar.
How do I choose the right embedding model for RAG?
Consider your domain, language, and task. Evaluate models on a held-out set of query-document pairs using metrics like recall@k. Also factor in dimensionality, inference speed, and cost. Fine-tuning on domain data can significantly improve relevance.
About GigaRAG
GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through what embeddings actually encode or something adjacent, we publish what we have actually tested, including where it falls short.
