Understanding Vector Embeddings: Word2Vec to Gemini
Computers cannot natively process text, audio, or images as semantic concepts; they operate strictly on numerical vectors. Vector Embeddings translate high-dimensional unstructured human knowledge into dense continuous coordinate vectors where geometric proximity corresponds directly to semantic similarity.
The field of embeddings has evolved dramatically — from early static lookup vectors like Word2Vec and GloVe (2013) to dynamic contextual embeddings in modern transformer architectures like Gemini 1.5 Flash and OpenAI text-embedding-3. This guide breaks down geometric vector math, similarity metrics, dimensional reduction, and production indexing algorithms like HNSW.
Mental Model: High-Dimensional Semantic Spaces
An embedding is a numerical vector of $D$ real numbers (e.g., 768 or 1536 floating-point values) representing a concept in a $D$-dimensional continuous space. In this space, concepts with similar meanings or contexts reside close to one another, while unrelated concepts are distant.
For example, in a trained embedding space, the vector distance between "king" and "queen" closely mirrors the vector distance between "man" and "woman". Vector arithmetic can even be performed directly on semantic concepts: $\\vec{v}_{\\text{"king"}} - \\vec{v}_{\\text{"man"}} + \\vec{v}_{\\text{"woman"}} \\approx \\vec{v}_{\\text{"queen"}}$.
This continuous representation allows machine learning models and Retrieval-Augmented Generation (RAG) pipelines to search by meaning rather than exact keyword string matches. For deeper details on database indexing techniques, explore our guide on Vector Database Indexing HNSW vs IVF-Flat and embeddings explained.
Quick reference
- Embeddings map discrete tokens or text chunks into continuous D-dimensional vectors.
- Geometric proximity in embedding space corresponds directly to semantic similarity.
- Vector arithmetic preserves linear relationships (e.g., king - man + woman = queen).
- Higher dimensions (1536+) capture subtle domain nuances at the cost of higher memory.
- Embeddings serve as the fundamental representation layer for RAG and semantic search.
Remember this
Vector embeddings transform human semantic concepts into continuous geometry where distance equals meaning.
From Static Word2Vec to Contextual Transformers
The first generation of modern embeddings (Word2Vec, Skip-gram, GloVe) assigned a single static vector to each token in a vocabulary dictionary. While revolutionary in 2013, static embeddings suffered from a major flaw: polysemy. The word "bank" in "river bank" received the exact same static numerical vector as "bank" in "investment bank".
Modern Transformer architectures (such as BERT, Gemini, and Claude) replaced static lookups with Contextual Embeddings. In a Transformer, self-attention layers compute interaction weights between all words in a sequence simultaneously. As a result, the final output embedding for "bank" dynamically shifts its position in vector space based on surrounding context words.
In addition, multimodal models like Gemini 1.5 Flash project text, code, audio, and visual pixels into a shared unified embedding space. This enables cross-modal retrieval — searching an un-captioned video library using a natural language text query.
Quick reference
- Word2Vec and GloVe provided static vectors; failed to handle multi-meaning words (polysemy).
- Transformer self-attention generates dynamic contextual embeddings based on surrounding tokens.
- Multimodal models (Gemini 1.5) project text, image, and audio into a single shared space.
- Matryoshka Representation Learning (MRL) allows truncating vectors dynamically without re-training.
- Normalized embeddings simplify cosine similarity calculations to fast dot products.
Remember this
Contextual embeddings dynamically update vector coordinates based on surrounding sentence context.
Similarity Metrics: Cosine vs Dot Product vs L2 Distance
Comparing two vectors $\\vec{A}$ and $\\vec{B}$ requires defining a distance metric. The three standard mathematical metrics used in vector search are Cosine Similarity, Dot Product, and Euclidean (L2) Distance.
Cosine Similarity measures the cosine of the angle $\\theta$ between two vectors: $\\cos(\\theta) = \\frac{\\vec{A} \\cdot \\vec{B}}{\\|\\vec{A}\\| \\|\\vec{B}\\|}$. It yields a score between -1 and +1, focusing purely on vector direction rather than length. When vectors are unit-normalized ($\\|\\vec{A}\\| = 1$), Cosine Similarity equals the simple Dot Product (\\vec{A} \\cdot \\vec{B}), reducing computation from $O(D)$ square-root divisions to simple floating-point multiplications.
Euclidean (L2) Distance measures the straight-line spatial distance between two vector endpoints: $d(\\vec{A}, \\vec{B}) = \\sqrt{\\sum_{i=1}^{D} (A_i - B_i)^2}$. L2 distance is sensitive to magnitude differences, making Cosine/Dot-Product the preferred metrics for text embeddings.
Quick reference
- Cosine Similarity measures directional alignment regardless of vector magnitude.
- Dot Product equals Cosine Similarity when vectors are pre-normalized to length 1.
- Euclidean (L2) Distance measures absolute spatial distance between vector endpoints.
- Pre-normalizing vectors allows vector databases to use ultra-fast SIMD dot product instructions.
- Select the metric matching the specific training loss function of your embedding model.
Remember this
Unit-normalize your embeddings to transform expensive Cosine Similarity calculations into lightning-fast Dot Product instructions.
Production Indexing & Modern Vector Search
Performing exact brute-force K-Nearest Neighbor (k-NN) search requires computing vector distances against every document in a dataset. For 10 million 1536-dimensional vectors, a single query requires 15 billion floating-point calculations (~60GB read bandwidth), making real-time search impossible.
Production vector databases (such as pgvector, Qdrant, Pinecone, and Milvus) use Approximate Nearest Neighbor (ANN) indexing. The state of the art is HNSW (Hierarchical Navigable Small World) graphs. HNSW builds a multi-layer graph where top layers contain long-range highway links for fast coarse navigation, while bottom layers contain dense local neighbor links for precise fine-tuning.
By traversing the HNSW graph hierarchically, search latency drops from linear $O(N)$ brute-force down to logarithmic $O(\\log N)$, executing queries across millions of vectors in under 5 milliseconds.
Quick reference
- Brute-force k-NN scales linearly O(N), becoming bottlenecked at scale.
- ANN algorithms sacrifice <1% recall accuracy for 1000x faster query performance.
- HNSW graph indexing builds hierarchical skip-list layers for logarithmic O(log N) traversal.
- IVF (Inverted File Index) clusters vectors into Voronoi cells to narrow search partitions.
- Quantization (PQ/SQ) compresses 32-bit floats to 8-bit integers, saving 75% RAM.
Remember this
HNSW indexing reduces vector search latency from seconds to milliseconds by replacing linear scans with graph traversal.
Key takeaway
To test vector similarity math in your application, compute the dot product of two normalized 768-dimensional embeddings using NumPy. Verify that identical text yields 1.0, related text yields ~0.8, and unrelated text yields <0.2.
Related Articles
Explore this topic