Full-Text Search Comparison: Elasticsearch vs. pgvector
Search architecture in modern applications has expanded beyond traditional exact keyword matching to encompass semantic intent understanding powered by vector embeddings. Engineering teams building search engines face a core decision: deploy a dedicated search cluster using Elasticsearch, or use pgvector to perform vector and full-text search directly inside PostgreSQL.
Elasticsearch excels at distributed BM25 lexical keyword matching, faceted aggregations, and inverted index tokenization across massive document logs. Conversely, pgvector adds high-dimensional vector similarity indexing (HNSW and IVFFlat) directly to PostgreSQL transactional tables, simplifying infrastructure by eliminating dual-database ETL pipelines.
Mental Model: Lexical BM25 Keyword Search vs Vector Semantic Distance
Lexical full-text search algorithms (such as BM25) tokenize document text into inverted index term dictionaries. When a user searches for 'lightweight laptop', BM25 calculates Term Frequency (TF) and Inverse Document Frequency (IDF) scores for exact or stemmed word occurrences ('lightweight', 'laptop').
Vector semantic search maps text documents into dense numerical vector embeddings (e.g., 1536-dimensional arrays generated by Gemini or OpenAI models). Queries calculate geometric distance (Cosine Similarity, Dot Product, or Euclidean Distance) in high-dimensional vector space.
Vector search captures conceptual meaning: searching for 'portable computer' retrieves documents containing 'lightweight laptop' even if zero exact words match. For indexing performance deep-dives, explore top 15 vector databases and vector database indexing hnsw vs ivfflat.
Quick reference
- Lexical BM25 matches exact stemmed keywords using inverted index term frequencies.
- Vector search calculates geometric cosine distance between dense floating-point embeddings.
- Lexical search excels at SKU lookups, exact proper nouns, and part numbers.
- Vector search excels at conceptual intent, multi-lingual queries, and natural language QA.
- Choosing between systems depends on workload scale, ETL complexity, and search accuracy requirements.
Remember this
Combine lexical BM25 for exact keyword lookups with vector embeddings for conceptual search intent.
Elasticsearch Inverted Indexing & Analyzer Pipelines
Elasticsearch is built on Apache Lucene, utilizing Inverted Indexes to map tokenized terms directly to document IDs. Text fields pass through customizable Analyzer Pipelines consisting of Character Filters, Tokenizers (e.g., standard, n-gram, edge_ngram), and Token Filters (lowercase, stemming, stop-word removal).
Elasticsearch shines in high-volume, multi-node distributed environments. Features like distributed shard rebalancing, field-level doc_values aggregations, and fuzzy fuzzy-matching provide sub-100ms response times across billions of log events or product catalogs.
However, maintaining Elasticsearch requires dedicated cluster operational management, JVM garbage collection tuning, snapshot backup policies, and continuous ETL sync pipelines from primary relational databases.
Quick reference
- Inverted index term dictionaries map tokenized words to matching document IDs.
- Custom analyzer pipelines handle edge-ngram autocomplete, stemming, and synonms.
- Distributed multi-shard clusters scale horizontally across dedicated search nodes.
- Doc_values column-store data structures enable sub-second aggregation facets.
- Requires continuous CDC / ETL synchronization from primary PostgreSQL relational stores.
Remember this
Deploy Elasticsearch for dedicated, high-throughput distributed keyword search and aggregations.
PostgreSQL pgvector HNSW Indexing & Cosine Distance Queries
The pgvector extension adds a native vector data type and distance operators (<=> for Cosine, <-> for L2 distance) directly to PostgreSQL tables.
pgvector supports Hierarchical Navigable Small World (HNSW) indexes (CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops)). HNSW constructs a multi-layer graph structure in RAM, performing approximate nearest neighbor (ANN) searches in logarithmic time without evaluating every database row.
Executing vector queries inside PostgreSQL (SELECT * FROM items ORDER BY embedding <=> '[...]' LIMIT 10) allows joining vector results directly with transactional columns (WHERE tenant_id = '...' AND price < 100) in a single query execution plan without external ETL pipelines.
Quick reference
- pgvector adds native vector data types and HNSW / IVFFlat indexing to PostgreSQL.
- HNSW graph indexes deliver sub-10ms approximate nearest neighbor (ANN) vector queries.
- Joins vector similarity directly with relational SQL WHERE clauses (tenant_id, price).
- Eliminates dual-database ETL sync complexity and infrastructure maintenance overhead.
- Ensure PostgreSQL work_mem and maintenance_work_mem are sized to build HNSW graphs in RAM.
Remember this
Use pgvector inside PostgreSQL to combine vector similarity queries directly with relational SQL joins.
Hybrid Search: Reciprocal Rank Fusion (RRF) Architecture
Neither lexical search nor vector search alone yields optimal retrieval quality across all user query types. Lexical search misses conceptual synonyms, while vector search struggles with exact product part numbers.
Hybrid Search executes both BM25 lexical search and vector KNN search in parallel, combining result rankings using Reciprocal Rank Fusion (RRF).
The RRF algorithm calculates a combined relevance score for each document: RRF_Score(d) = 1 / (60 + Rank_BM25(d)) + 1 / (60 + Rank_Vector(d)). RRF elevates documents that score well across both retrieval channels, delivering superior search relevance.
Quick reference
- Hybrid Search executes lexical BM25 and vector KNN queries simultaneously.
- Reciprocal Rank Fusion (RRF) normalizes and merges ordinal rank lists without score calibration.
- Constant k=60 in RRF prevents outlier top-ranked documents from dominating the final list.
- Elasticsearch 8.x and PostgreSQL 16+ support native hybrid search rank fusion pipelines.
- Evaluated against benchmark datasets (BEIR), Hybrid RRF achieves higher nDCG@10 relevance scores.
Remember this
Merge BM25 lexical results and vector KNN rankings using Reciprocal Rank Fusion for maximum retrieval accuracy.
Key takeaway
To test PostgreSQL pgvector performance, create an HNSW index on a 1536-dimensional embedding column and verify sub-10ms execution for SELECT * ORDER BY embedding <=> '[...]' LIMIT 10.
Related Articles
Explore this topic