Embeddings Explained
A search team upgrades their embedding model for better quality, re-embeds only newly added documents with it, and leaves millions of older documents on the previous model's embeddings — search results get subtly, silently worse, with no error anywhere, because every vector is still a perfectly valid float array of the right dimension. Nothing about the failure looks like a bug; two documents just happen to compare as unrelated in vector space when they should be near-duplicates, because they were never placed in the same space to begin with.
This guide covers what an embedding vector actually represents, why cosine similarity isn't universally "the correct" metric (it's whichever metric the model's training actually optimized for), the size-versus-quality trade-off behind newer truncatable Matryoshka embeddings, and the mixed-model-version failure above traced to its actual cause. For where embeddings fit into a retrieval pipeline, see Vector Database Map; for chunking the source text before embedding it, see RAG Chunking Strategies.
What an Embedding Actually Is
An embedding is a dense vector — an array of a few hundred to a few thousand floating-point numbers — produced by a trained model such that inputs with similar meaning end up positioned close together in that vector space, under whatever distance or similarity measure the space was built for. Two sentences with completely different words but the same meaning ("cancel my subscription" and "I want to stop being billed") can land near each other in embedding space, while two sentences sharing many words but different intent land far apart.
An embedding is a lossy, compressed semantic representation — it is not a reversible encoding of the original text, and you cannot reconstruct the exact source content from the vector alone. Crucially, the vector space is specific to the exact model (and often exact model version) that produced it: embeddings from two different models are not comparable to each other at all, even if they happen to have the same number of dimensions — there is no shared coordinate system between two models' independently trained vector spaces.
Quick reference
- An embedding is a fixed-size float vector — hundreds to thousands of dimensions, depending on the model.
- Semantically similar inputs land close together in the vector space the specific model was trained to produce.
- Lossy and one-directional: you cannot reconstruct the original input from its embedding alone.
- Embeddings from different models (or different versions of the same model) are not comparable, even at identical dimensionality.
- The vector space's geometry is defined entirely by that model's training — there's no universal 'embedding space' shared across models.
- Embeddings are the substrate retrieval-augmented generation is built on — see Classic vs Graph vs Agentic RAG for how retrieval uses them end to end.
Remember this
An embedding is a model-specific compressed representation of meaning — two vectors are only meaningfully comparable if the exact same model produced both of them.
Similarity Metrics: Cosine, Dot Product, and Euclidean
Cosine similarity measures the angle between two vectors, ignoring their magnitude entirely — it answers "do these point in the same direction," which is why it's the default choice for most text embedding comparisons. Dot product is sensitive to magnitude as well as direction — appropriate when a model was specifically trained or normalized so that magnitude itself carries meaningful signal (some retrieval-tuned models are trained this way deliberately). Euclidean (L2) distance measures straight-line distance in the vector space and is also magnitude-sensitive, more common in classical clustering than in text retrieval.
The common mistake is assuming cosine similarity is universally "the correct" choice by default. The right metric is whichever one the specific embedding model's training procedure actually optimized against — many models are trained and normalized such that cosine and dot product produce identical rankings (making the choice moot), but not all are, and using the wrong metric for a model that wasn't trained with that assumption can measurably degrade retrieval quality. Check the specific model's documentation for its intended similarity metric rather than defaulting to cosine out of habit.
Quick reference
- Cosine similarity: direction only, magnitude-invariant — the common default for text embeddings.
- Dot product: direction and magnitude — correct when a model's training specifically assumed or normalized for it.
- Euclidean/L2 distance: straight-line distance, magnitude-sensitive — more common outside text retrieval.
- Many models normalize vectors to unit length during training, making cosine and dot product equivalent for that specific model — this is a property of the model, not a universal rule.
- Check the embedding model's own documentation for its intended metric — 'always use cosine' is a habit, not a correctness guarantee.
Remember this
There is no universally correct similarity metric — the right one is whichever metric the specific embedding model's training procedure actually optimized for, which cosine similarity happens to be for many but not all models.
Dimensionality and Matryoshka Embeddings
Higher-dimensional embeddings can capture more nuance, at the direct cost of more storage per vector and more compute per similarity comparison — a real trade-off at the scale of millions of vectors, not a rounding error. Traditionally, changing dimensionality meant using an entirely different, separately-trained model, since a vector's dimensions from one model aren't independently meaningful outside the full vector.
Matryoshka Representation Learning (used in newer embedding models, including some of OpenAI's text-embedding-3 family) changes this: the model is trained so that a prefix of its full-length vector is itself a valid, useful lower-dimensional embedding. Truncating a 1536-dimension vector down to its first 256 dimensions still produces a usable, if somewhat less precise, embedding from the same model — letting a team pick a smaller vector size for storage or latency-sensitive search paths and a larger one for quality-sensitive paths, from the exact same underlying embedding call, instead of maintaining two separate models.
Quick reference
- More dimensions: more nuance captured, more storage and compute cost per comparison at scale.
- Traditionally, different dimensionality meant an entirely different model — dimensions aren't independently meaningful in a standard embedding.
- Matryoshka embeddings are trained so a truncated prefix of the full vector remains a valid, usable lower-dimensional embedding.
- This lets one embedding call serve both a fast/cheap truncated-vector path and a full-precision path without maintaining two models.
- Truncation still trades some retrieval quality for the smaller size — it's a real trade-off, not a free size reduction.
Remember this
Matryoshka embeddings let a single model's output be truncated to a smaller, cheaper vector that's still usable — trading some precision for storage and latency, from one embedding call instead of two separate models.
Recover One Mixed-Model-Version Search Degradation
Trigger. A team upgrades their embedding model from version 2 to version 3 for better retrieval quality, and re-embeds only newly ingested documents with the new model — millions of existing documents keep their version-2 embeddings, both stored in the same vector index. Symptom. Search quality gets subtly worse for queries that should match older documents: a query embedded with version 3 is compared against a mix of version-3 and version-2 vectors, and the version-2 comparisons are meaningless, since the two models' vector spaces don't share a coordinate system — yet nothing errors, because every stored vector is still a valid float array of the expected dimension.
Root mechanism. Vector similarity search has no built-in way to detect that two vectors came from incompatible spaces — a dimension-matching check passes even when the vectors mean nothing relative to each other. Recovery: re-embed the entire corpus with the new model version before considering the migration complete — there is no partial-migration state that produces correct results. Verification: run a fixed evaluation set of query/expected-document pairs against the fully re-embedded index and confirm retrieval quality actually improved, rather than assuming the newer model helped just because it shipped. Prevention: version-tag every stored embedding with the exact model and version that produced it, and treat any embedding-model upgrade as an all-or-nothing full re-index, never an incremental one.
Quick reference
- Trigger: an embedding model upgrade applied only to newly ingested documents, not the existing corpus.
- Symptom: search quality silently degrades for queries touching older documents — no error anywhere.
- Root mechanism: vector similarity search cannot detect that two vectors came from incompatible model spaces on its own.
- Recovery: a full re-index with the new model is the only complete fix — there's no valid partial-migration state.
- Prevention: version-tag every stored embedding and verify a migration's completeness explicitly, rather than assuming an upgrade improved things.
Remember this
Mixing embeddings from two model versions in one index degrades search silently, because similarity search has no way to detect incompatible vector spaces on its own — the only real fix is a full, verified re-index, never an incremental one.
Key takeaway
An embedding is a model-specific, lossy vector representation of meaning — two vectors are only comparable if the same model produced both. Cosine similarity is a common default, not a universal rule; the correct metric is whatever the specific model's training actually optimized for. Matryoshka embeddings let one model's output be truncated to trade precision for size. And mixing embedding model versions in one index is a silent failure mode with no error signal — the only real fix is a full, verified re-index.
Practice (25 min): embed the same set of ten sentences with two different embedding models (or two different providers), and confirm directly that comparing a version-A query embedding against version-B document embeddings produces meaningless similarity scores, even though both are valid vectors of some dimension. Then re-embed everything with one consistent model and confirm similarity rankings become sensible again. Pass when you can state, for your own retrieval system, exactly which model and version produced every vector currently in your index.
Related Articles
Explore this topic