What Embeddings Actually Encode: A Practical Guide

What Embeddings Actually Encode: A Practical Guide for RAG and Agent Memory Builders

What embeddings actually encode is the question every RAG developer and agent memory builder hits the hard way: your pipeline returns a chunk about riverbanks when the user asked about financial institutions, or your agent confidently "remembers" a conversation that never happened. You check the logs, the retrieval code, the chunking. Everything looks right. The problem isn't your pipeline. It's what you assumed the vectors were doing.

Embeddings don't store meaning. They store statistical patterns from training data, compressed into dense vectors. That gap between "feels like meaning" and "actually encodes similarity" is where production systems break. GigaRAG exists because practitioners keep hitting this wall and need tooling that respects what vectors can and cannot do.

This guide covers what embeddings actually capture, what they lose, and how to design RAG pipelines and agent memory around those limits. Not theory. Field notes from systems that failed, and the fixes that made them reliable.

In This Guide

• What Is an Embedding?
• Embeddings vs Keyword Search: What Each Encodes
• What Embeddings Actually Encode: The Honest Answer
• What Embeddings Actually Encode: A Step-by-Step Guide
• How Embeddings Work: From Tokens to Vectors
• Types of Embeddings and What Each Encodes
• BERT vs Word2Vec: Why Context Changes Everything
• Do ChatGPT and LLMs Use Embeddings?
• What Embeddings Cannot Encode
• Practical Implications for RAG Pipelines
• Embeddings in Agent Memory Systems
• Final Thoughts

What Is an Embedding?

An embedding is a list of numbers that represents a piece of data — a word, a sentence, an image — as a point in a continuous vector space. Similar items land near each other. The distance between two points is a measure of how related the model thinks they are.

From raw data to vectors

You feed text into an embedding model. The model runs it through a neural network and outputs a fixed-length vector, typically 384 to 3,072 dimensions. That vector is the embedding. It is not readable by humans, but it is dense with information the model learned during training.

Dense vs. sparse representations

Older methods like TF-IDF produced sparse vectors: mostly zeros, one entry per vocabulary word. Embeddings are dense. Every dimension holds a value, and no single dimension maps to a word. The meaning is distributed across the whole vector.

The vector space metaphor

Think of a map. "Cat" and "kitten" sit close together. "Cat" and "refrigerator" sit far apart. The space itself is learned, not designed. What the model places nearby depends entirely on patterns in its training data.

[!note] Embeddings encode semantic similarity as learned from training data, not factual truth or logical structure. Two texts can be close in embedding space while one is false or contradicts the other.

Embeddings vs Keyword Search: What Each Encodes

What Embeddings Actually Encode: The Honest Answer

Embeddings encode statistical co-occurrence patterns. That's it. The model learns which words, phrases, and concepts appear in similar contexts across its training data, then places them near each other in vector space. It does not encode meaning, facts, or truth.

Statistical patterns, not semantic truth

When a model places "doctor" near "nurse," it's not because it understands medicine. It's because those words appear in similar sentences millions of times. The vector captures distributional similarity: what tends to show up around what. If your training data says "bank" appears near "river" as often as near "loan," the embedding splits the difference. The model has no opinion about which meaning is correct.

What "similarity" actually measures

Cosine similarity between two embeddings measures how closely their context patterns match. High similarity means the model saw these items in similar linguistic environments. It does not mean the items are interchangeable, causally related, or even about the same thing. A retrieval system returning high-similarity chunks is saying "these look contextually alike," not "these answer your question."

Why embeddings feel like meaning but aren't

The illusion comes from scale. With enough dimensions and enough training data, contextual patterns correlate strongly with what humans call meaning. "King" minus "man" plus "woman" lands near "queen" because those words occupy parallel relational slots in the training corpus. Impressive, but it's still pattern matching. The model never touched a crown.

[!tip] For agent memory, store the raw text alongside the embedding and retrieve by similarity, then verify with a rule or a second model before the agent acts on it. This catches the cases where similarity is high but the content is wrong or negated.

What Embeddings Actually Encode: A Step-by-Step Guide

1. Reproduce the failure with a fixed query and inspect the top-k retrieved chunks.
2. Check whether the correct chunk is in the index at all; if not, the problem is ingestion, not embeddings.
3. Compare the query embedding to the retrieved chunk embeddings; look for high similarity on the wrong content.
4. Test whether the failure involves negation, numbers, or word order, which embeddings handle poorly.
5. Try a hybrid approach: combine embedding similarity with keyword or metadata filters.
6. Re-embed a small sample with a different model and compare retrieval quality before committing.
7. Document which failure modes are embedding-related versus chunking, indexing, or prompt-related.

How Embeddings Work: From Tokens to Vectors

The pipeline is straightforward: text goes in, numbers come out. What happens in between determines what those numbers can and can't tell you.

Tokenization and input representation

Before a neural network sees your text, a tokenizer splits it into pieces: words, subwords, or characters. "Embeddings" becomes "embed" plus "dings" under a subword tokenizer. Each token gets an ID, and that ID maps to an initial vector. This first vector is arbitrary. It's just a lookup table entry. The network's job is to reshape it.

The role of neural network layers

Each transformer layer transforms the token vectors by mixing information from surrounding tokens. Attention decides which neighbors matter. Feed-forward layers apply learned transformations. After enough layers, each token's vector has absorbed context from the rest of the sequence. The final hidden state for a token, or a pooling of all token states, becomes the embedding you store and search.

Training objectives shape the encoding

What gets encoded depends on what the model was trained to predict. A model trained to predict the next token learns representations useful for generation. A model trained on contrastive pairs, where similar texts are pulled together and dissimilar ones pushed apart, learns representations tuned for retrieval. Same architecture, different objective, different vector space. When you swap embedding models in a RAG pipeline, you're not just changing quality. You're changing what the vectors point at.

Types of Embeddings and What Each Encodes

Different embedding types encode different slices of the input. Picking the wrong one means your retrieval returns results that look right but miss the point.

Word-level embeddings (Word2Vec, GloVe)

Word2Vec and GloVe produce one fixed vector per word. "Bank" gets a single vector whether it means a riverbank or a financial institution. What's encoded is co-occurrence: words that appear in similar contexts land near each other. What's lost is any sense of the current sentence. For RAG, these are mostly obsolete. You'd only reach for them when you need fast, lightweight keyword expansion and can tolerate ambiguity.

Contextual embeddings (BERT, GPT)

BERT and GPT generate a vector per token that depends on the surrounding text. The same word gets different vectors in different sentences. What's encoded is the token's role in that specific sequence, including syntactic position and nearby semantic cues. What's lost is document-level coherence. A contextual embedding of a single token tells you little about the paragraph it came from.

Sentence and document embeddings

Models like Sentence-BERT or OpenAI's text-embedding-3 pool token vectors into one fixed-length vector for a chunk, sentence, or document. What's encoded is the dominant topic and high-level meaning of that span. What's lost is fine-grained detail: specific numbers, names, and negations blur into the average. This is the default choice for RAG chunk retrieval, but chunk size directly controls what survives.

Multi-modal embeddings

CLIP and similar models map text and images into a shared vector space. A photo of a dog and the caption "a dog" land near each other. What's encoded is cross-modal correspondence learned from paired training data. What's lost is anything the training pairs didn't cover. If your corpus mixes diagrams, screenshots, and prose, multi-modal embeddings can unify retrieval, but they won't capture relationships that weren't in the training distribution.

BERT vs Word2Vec: Why Context Changes Everything

Word2Vec gives you one vector per word. BERT gives you one vector per occurrence. That single difference explains most retrieval failures you'll hit in production.

Static embeddings: one vector per word

Word2Vec trains on co-occurrence: words appearing in similar contexts get similar vectors. "Bank" gets one vector, period. It averages riverbank and financial institution into a single point in space. The model can't tell them apart because it never learned to.

Contextual embeddings: one vector per occurrence

BERT runs each token through layers of attention, so "bank" in "deposit money at the bank" gets a different vector than "bank" in "fish from the river bank." The encoding captures the token's role in that sentence, not just the word itself.

Practical implications for retrieval

For RAG, this matters at query time. A static embedding of "bank" retrieves chunks about rivers and finance equally. A contextual embedding disambiguates before retrieval starts. The honest answer: BERT isn't "better" in every case. It's slower and heavier. But for retrieval where context decides meaning, static embeddings fail silently.

Do ChatGPT and LLMs Use Embeddings?

Yes. Every transformer-based LLM, including ChatGPT, uses embeddings at multiple internal stages. The model never sees raw text. It sees vectors.

Embeddings inside the transformer

Three places, minimum. First, input tokens get mapped to embedding vectors before any processing happens. Second, the attention mechanism computes similarity between those vectors to decide which tokens matter to each other. Third, the output layer maps final hidden states back to vocabulary probabilities. The whole forward pass is vector math.

Embeddings for external retrieval

This is where the confusion starts. ChatGPT uses embeddings internally, but that's not the same as using embeddings for retrieval. When you build a RAG pipeline, you call an embedding API to convert documents and queries into vectors, then search a vector database. That's external retrieval. The model itself doesn't do that unless you wire it up.

What this means for agent memory

Agent memory systems sit in between. You store past interactions as embeddings in a vector store, then retrieve relevant ones before each new turn. The LLM's internal embeddings handle language understanding. Your external embeddings handle memory. Mixing those two up is a common architecture mistake.

What Embeddings Cannot Encode

Embeddings encode statistical co-occurrence patterns from training data. That's it. They don't encode facts, truth, causality, negation, temporal order, or logical consistency. They encode what tends to appear near what. When retrieval returns something confidently wrong, that's the gap showing.

Facts and truth

An embedding of "Paris is the capital of France" sits near "Paris is a city in France" and "London is the capital of England." All three are statistically similar. The embedding doesn't know which one is true. It knows they share words and contexts. If your training data contains false statements often enough, those false statements get embedded just as confidently as true ones.

Negation and logical relationships

"Not good" and "good" produce vectors that are close together. The model sees the shared token and similar contexts. The negation gets washed out. Same problem with "the dog bit the man" versus "the man bit the dog." Same words, opposite meaning, nearly identical vectors. Embeddings don't do logic. They do proximity.

Temporal and causal structure

"Before" and "after" are just tokens. An embedding doesn't track that event A happened before event B, or that A caused B. It tracks that A and B appear in similar documents. For agent memory, this means an embedding can retrieve "the user changed their password" and "the user forgot their password" as equally relevant, with no sense of which came first or which caused which.

Rare or unseen concepts

Embeddings are only as good as their training data. A concept that appears rarely, or never, gets a weak or nonexistent representation. Technical jargon, new product names, niche domain terms: the model either maps them to something vaguely similar or produces noise. Your RAG pipeline will retrieve confidently irrelevant chunks, and the embedding won't tell you it's guessing.

Practical Implications for RAG Pipelines

The previous section told you what embeddings can't do. Now here's what to do about it. Retrieval failures are rarely the model's fault. They're almost always a mismatch between what you asked the embedding to encode and what it can actually encode.

Chunking strategy and embedding quality

Chunk size changes what gets encoded. Small chunks (100 to 200 tokens) encode narrow, specific concepts but lose surrounding context. Large chunks (1,000+ tokens) preserve context but dilute the signal: the embedding averages across too many ideas, and retrieval returns vaguely relevant blocks. For most RAG pipelines, 300 to 500 tokens per chunk with 10 to 15 percent overlap works. Test it. Don't assume.

Choosing the right embedding model

Pick a model trained on text similar to yours. A model fine-tuned on legal documents will outperform a general-purpose model on contracts. Check the model's context window: if your chunks are 500 tokens, you need a model that handles at least that. Bigger isn't always better. A 768-dimension model tuned for your domain beats a 3,072-dimension generalist.

Similarity thresholds and retrieval tuning

Cosine similarity scores are relative, not absolute. A score of 0.7 means nothing without a baseline. Run your own queries, log the scores, and set thresholds from data. If retrieval returns irrelevant chunks at 0.8, your threshold is wrong or your embeddings are wrong. Don't tune the threshold to fix a bad embedding.

Common RAG failure modes rooted in embeddings

Three failures show up repeatedly. First, retrieval returns chunks that share words but not meaning: your embedding model is too weak for the domain. Second, retrieval misses chunks that use different words for the same concept: your model lacks semantic coverage. Third, retrieval returns confidently wrong chunks: the embedding encoded statistical similarity, not truth. Each failure points to a different fix. Diagnose before you patch.

Embeddings in Agent Memory Systems

Agent memory is where embeddings fail most visibly. An agent stores experiences as vectors, retrieves them later, and builds responses on what it finds. When retrieval is wrong, the agent doesn't just miss a fact. It confidently acts on a memory that never happened.

Episodic vs. semantic memory in vector space

Episodic memory stores specific events: what the user said last Tuesday, what the agent did, what worked. Semantic memory stores general knowledge: preferences, facts, patterns. In embedding space these blur. A vector doesn't know whether it encodes "user prefers dark mode" or "user mentioned dark mode once." You need separate stores with separate retrieval paths. Don't mix them.

Memory drift and context contamination

Memory drift happens when new embeddings shift the meaning of old ones. As you add experiences, the vector space reorganizes, and retrieval returns different results for the same query. Context contamination is worse: an agent retrieves a memory from a different user or task and treats it as current. Both failures look like the agent "forgetting" or "hallucinating." They're retrieval problems, not reasoning problems.

Architecture patterns for reliable agent memory

Three patterns hold up in practice. First, timestamp every memory and filter by recency before similarity. Second, store a memory type tag (episodic, semantic, preference) and filter on it. Third, keep a working memory separate from long-term storage: the agent's current context should not compete with its history in the same vector space. Test retrieval against known queries before trusting the memory system.

Final Thoughts

What embeddings actually encode is statistical co-occurrence, not meaning. The vector for "bank" sits near "river" and "finance" because those words appeared in similar contexts during training, not because the model understands either one. That distinction drives every failure mode in this article: irrelevant retrieval, confident wrong answers, agent memories that never happened.

The fix isn't a better embedding model. It's designing around the limitation. Filter by metadata before similarity. Keep episodic and semantic memory separate. Test retrieval against known queries. Treat similarity scores as signals, not verdicts.

GigaRAG exists for practitioners doing exactly this work. It gives you the retrieval infrastructure and evaluation tools to catch embedding failures before your users do. The honest answer is that embeddings are a blunt instrument. You can still build reliable systems with them. You just can't pretend they understand anything.

Frequently Asked Questions

How do embeddings actually work?

Embeddings map text to vectors in a high-dimensional space where similar meanings are close together. A model is trained on large text corpora to predict context, and the resulting vector for a piece of text captures its semantic neighborhood. The geometry reflects co-occurrence patterns, not explicit rules or facts.

Why is BERT better than Word2Vec?

BERT produces context-dependent embeddings, so the same word gets different vectors in different sentences. Word2Vec produces one fixed vector per word, so it cannot distinguish 'bank' in 'river bank' from 'bank' in 'savings bank'. For RAG and agent memory, context-dependent embeddings usually retrieve more relevant chunks.

Does ChatGPT use embeddings?

ChatGPT itself generates text; embeddings are a separate representation used in retrieval systems. In a RAG pipeline, embeddings are typically used to find relevant documents, and those documents are then passed to a generative model like ChatGPT. The embedding model and the generative model are usually different.

What are different types of embeddings?

Common types include word embeddings (Word2Vec, GloVe), contextual embeddings (BERT, sentence transformers), and multimodal embeddings for text, images, or audio. For RAG and agent memory, sentence or passage embeddings are most common because they represent a chunk of text as a single vector.

What do embeddings not encode?

Embeddings do not reliably encode negation, exact numbers, word order, or logical relationships. They also do not encode factual correctness; a false statement and a true one can be close in embedding space. This is why retrieval can return confidently wrong context.

Can embeddings handle negation?

Negation is a known weakness. 'The service is good' and 'the service is not good' may have similar embeddings because the overall topic and sentiment words dominate. If your RAG or agent memory depends on negation, add a keyword or rule-based check alongside similarity search.

How do I choose an embedding model for RAG?

Evaluate on your own data: take a set of queries and known relevant chunks, then measure retrieval accuracy for a few candidate models. Consider context length, language support, and whether you need multimodal input. Smaller models can be sufficient and faster; larger models may improve recall on nuanced queries.

About GigaRAG

GigaRAG is for agent memory and RAG pipeline builders. get this right. Whether you are working through what embeddings actually encode or something adjacent, we publish what we have actually tested, including where it falls short.