Semantic Caching for LLM Queries with Redis
Large Language Model (LLM) API calls — such as requesting completions from Google Gemini 1.5 Flash — introduce significant financial costs ($/token) and latency delays (500ms to 2.5s per generation). In production conversational AI agents, users frequently ask semantically identical questions using slightly different phrasing ("How do I reset my password?" vs "What is the process to change my password?").
Traditional key-value caching (like MD5 hashing of prompt strings) fails on semantically equivalent queries because raw strings differ byte-for-byte. Semantic Caching uses vector embeddings and vector database search inside Redis to evaluate semantic intent similarity. This guide details prompt embedding generation, Redis HNSW vector index setup, and cosine similarity threshold tuning.
Mental Model: Exact-Match Caching vs Semantic Embedding Vector Search
Exact-match string caching compares literal prompt strings. If a user changes a single word or punctuation mark, an exact-match cache misses, triggering an expensive LLM API call.
Semantic Caching converts incoming prompt text into a high-dimensional dense vector embedding (e.g., 768-dimensional float32 vector). Vector embeddings capture semantic meaning: prompts with similar intent map to near-identical coordinate points in vector space.
When a prompt arrives, the cache system queries a vector index in Redis using Hierarchical Navigable Small World (HNSW) vector search. If the nearest existing prompt vector has a cosine similarity score exceeding a defined threshold (e.g., ≥ 0.92), Redis returns the pre-calculated LLM response in sub-5ms, cutting API costs by up to 60%. For embedding concepts, review understanding vector embeddings and practical prompt engineering gemini 1 5 flash.
Quick reference
- Exact-match caching misses on minor phrasing differences ('How to reset PW' vs 'Forgot password').
- Semantic caching converts prompt text into dense vector embeddings that capture semantic intent.
- Redis HNSW vector search finds nearest-neighbor prompt vectors in sub-5 milliseconds.
- Cache hits bypass LLM API calls, reducing API bill costs and slashing user latency from 1.5s to 5ms.
- Stores cached LLM completion text alongside vector embeddings inside Redis Hash entries.
Remember this
Deploy Redis Vector Search semantic caching to reuse LLM responses for semantically identical prompts.
Generating Vector Embeddings for Input Prompts via Gemini API
Before querying Redis, generate a vector embedding for the incoming user prompt using an embedding model API (such as text-embedding-004).
Normalize and sanitize the prompt text by converting to lowercase and stripping extra whitespace. Pass the clean prompt string to the embedding model to receive a 768-element floating-point array.
Convert the float array into a raw binary buffer (Float32Array.buffer) before sending it to Redis to optimize memory footprint and transmission speed.
Quick reference
- Use lightweight embedding models (text-embedding-004) to compute vector representations in <20ms.
- Sanitize and normalize prompt text to improve embedding consistency across user inputs.
- Convert JavaScript Float32Array embeddings into binary Buffer objects for Redis transmission.
- Store prompt text, completion text, and creation timestamps alongside vector fields.
- Cache embeddings locally in memory for microsecond re-evaluation during continuous chat sessions.
Remember this
Compute prompt embeddings using fast embedding APIs and serialize Float32 vectors into binary buffers.
Configuring Redis Vector Search (RediSearch HNSW Indexing)
Redis supports native vector search via RediSearch indexing modules.
Create a vector index using the FT.CREATE command with HNSW (Hierarchical Navigable Small World) algorithm:
1FT.CREATE idx:prompt_cache ON HASH PREFIX 1 cache:2 SCHEMA prompt_text TEXT3 prompt_vector VECTOR HNSW 6 TYPE FLOAT32 DIM 768 DISTANCE_METRIC COSINE4 completion_text TEXT;When searching for cached prompts, execute FT.SEARCH using K-Nearest Neighbors (KNN 1 @prompt_vector $query_vector AS score). Redis returns the closest matching cached prompt and its stored LLM completion text.
Quick reference
- FT.CREATE defines RediSearch indexes over Redis HASH entries with VECTOR fields.
- HNSW algorithm provides high-recall, sub-5ms nearest-neighbor vector search.
- Configure DISTANCE_METRIC COSINE to measure angular orientation between prompt vectors.
- Execute FT.SEARCH with KNN 1 to retrieve the single closest matching cached prompt.
- Set TTL on cache entries (EXPIRE cache:key 86400) to auto-evict stale LLM completions.
Remember this
Create RediSearch HNSW vector indexes configured with COSINE distance metrics for sub-5ms KNN queries.
Cosine Similarity Threshold Tuning & Cache Hit Validation
Setting the correct Cosine Similarity Threshold is critical to avoid false-positive cache hits.
If the threshold is set too low (e.g., 0.80), Redis returns cached answers for queries with different domain contexts ("How to cancel subscription" matching "How to update subscription"). If set too high (e.g., 0.98), semantic caching behaves like exact-match caching, missing valid hits.
A cosine similarity threshold between 0.90 and 0.94 yields optimal precision and recall. For sensitive domain applications (such as medical or legal AI assistants), enforce a stricter threshold (≥ 0.96) or validate metadata labels before returning cached completions.
Quick reference
- Cosine similarity ranges from 0.0 (completely orthogonal) to 1.0 (identical vector direction).
- Target a threshold between 0.90 and 0.94 for optimal balance between cache hit rate and accuracy.
- Enforce stricter thresholds (>=0.96) for medical, legal, or high-compliance domain queries.
- Log false-positive cache hits to continuously refine vector similarity threshold parameters.
- Implement manual cache-busting endpoints for administrators when system prompts change.
Remember this
Tune cosine similarity thresholds between 0.90 and 0.94 to maximize cache hits while preventing false positives.
Key takeaway
To test Redis semantic caching, run docker run -p 6379:6379 redis/redis-stack:latest. Execute two semantically similar prompts and verify the second prompt returns in under 5ms from Redis.
Related Articles
Explore this topic