LLM Caching: Response Cache vs Semantic Cache vs Prompt Cache

CoreConceptJuly 28, 20268 min read

LLM caching sounds simple until the cached answer crosses a tenant boundary, repeats stale product policy, or hides a model regression. The hard part is not storing text. The hard part is deciding what is safe to reuse, for whom, and under which version of the prompt, data, and model.

This guide is for engineers shipping LLM-backed product features who already know basic API calls. You will learn how exact response caches, semantic caches, and prompt caches sit at different layers, then trace one support-answer request through the cache path. Pair this with AI Gateway: Routing, Cost Controls, and Observability for the control boundary and Caching Strategies: Redis, CDN, and App Cache for the broader caching vocabulary.

LLM caching map: exact response, semantic match, and prompt-prefix reuse

Three Caches, Three Different Reuse Rules

Do not treat LLM caching as one feature. A response cache reuses an exact final answer for the same normalized input and policy. A semantic cache reuses an answer when a new question is close enough in embedding space to a previous question. A prompt cache reuses stable prompt tokens inside the model-provider execution path so repeated system instructions or long context prefixes become cheaper to process.

Those caches answer different questions. Response caching asks, "Have I seen this exact safe request before?" Semantic caching asks, "Is this new request equivalent enough to reuse the old answer?" Prompt caching asks, "Can the model runtime avoid reprocessing an unchanged prefix?" Mixing those rules is how teams accidentally serve an answer that is cheap, fast, and wrong.

Choose the cache by the reuse guarantee you can actually defend.
Cache	Reusable When	Main Risk
Response cache	Normalized input, user scope, prompt, model, and data version match	Stale or cross-tenant answer
Semantic cache	Meaning is close enough for a low-risk task	Near match is not the same intent
Prompt cache	Stable prefix repeats inside provider/runtime rules	Confusing token savings with answer correctness

Response, semantic, and prompt caches reuse different parts of an LLM request

Quick reference

Response cache keys should include tenant, task, prompt version, model route, tool/data version, and relevant auth scope.
Semantic caches need a similarity threshold plus a task allowlist; they are poor fits for high-risk personalized answers.
Prompt caching is usually provider/runtime behavior; it reduces repeated-prefix processing but does not prove the final answer is safe.
Do not cache answers that depend on hidden user state unless that state is part of the key.
Measure hit rate, false-hit rate, latency saved, and cost saved separately.

Remember this

LLM caching only works when the reuse rule matches the layer: exact answer reuse, meaning-near reuse, and repeated-prefix reuse are different promises.

One Support Answer Through the Cache Path

Follow one request: support-answer receives a tenant id, user question, retrieved document ids, and prompt version. The gateway first checks whether this task is cacheable. Then it computes a normalized response-cache key. If that misses and the task allows semantic reuse, it embeds the question and asks whether a previous answer is similar enough and still valid for the same tenant, docs, and policy.

Only after cache misses does the system call the model. The response is stored with the evidence that makes future reuse defensible: route, model, prompt version, source document versions, tenant scope, expiration, and eval status if available. A cache hit should still return a trace id so support can see whether the user received a fresh generation, an exact cached answer, or a semantic reuse.

Support answer cache flow: exact hit, semantic hit, model call, store with metadata

Quick reference

Normalize whitespace and stable formatting, but do not erase facts that change meaning.
Run authorization before reading or writing tenant-scoped cache entries.
Store cache metadata next to the answer; future invalidation depends on it.
Return cache-hit type in traces, not necessarily to end users.
Cache after validation, not before; malformed or policy-denied outputs should not become reusable answers.

Unsafe cache key

1// Too broad: ignores tenant, prompt version, data version, and permissions.2const key = hash(userQuestion.toLowerCase());3const cached = await redis.get(key);4if (cached) return JSON.parse(cached);

Scoped cache key and typed hit

1type CacheHit =2  | { kind: "exact"; traceId: string; answer: string; sourceDocIds: string[] }3  | { kind: "semantic"; traceId: string; answer: string; similarity: number; sourceDocIds: string[] }4  | { kind: "miss"; traceId: string };5 6const key = hash({7  tenantId,8  task: "support-answer",9  normalizedQuestion,10  promptVersion: "support-v4",11  modelRoute: "default-answer",12  sourceDocVersion: "docs-2026-07-28",13  authScope,14});

Remember this

A useful LLM cache hit carries evidence: which tenant, prompt, model route, source data, and cache rule made the reuse valid.

Invalidation Is a Trust Boundary

Classic cache invalidation is already hard; LLM cache invalidation adds meaning and trust. If a product policy changes, a cached answer can become wrong even when the original question has not changed. If a user's permission changes, a cached answer can become unauthorized even when the source document still exists. If a prompt changes from support-v3 to support-v4, the answer may no longer match the behavior you evaluated.

Use versions as part of the key where correctness matters, and use expiration where the data changes independently. For tenant support answers, include document version and authorization scope. For low-risk public FAQ answers, a short TTL plus source-version stamp may be enough. For personalized billing, account, security, or medical-like advice, prefer no semantic cache and consider exact response caching only when every relevant state field is in the key.

Quick reference

Invalidate by source version when documentation, policy, price, or entitlement data changes.
Invalidate by prompt version when instructions, output schema, refusal rules, or citation rules change.
Invalidate by model route when provider/model behavior changes enough to require a new eval baseline.
Use short TTLs for volatile facts and explicit purge hooks for high-impact policy updates.
Keep semantic cache off by default for personalized, financial, legal, security, or irreversible workflows.

Remember this

Invalidation is not housekeeping in LLM systems; it is the rule that prevents old text from crossing a new trust boundary.

Semantic Cache Hits Need Evidence, Not Hope

Semantic caching is tempting because users ask the same thing many ways. "How do I reset my password?" and "I forgot my login, what now?" can often share an answer. But embeddings measure closeness, not legal equivalence, user authorization, freshness, or intent safety. A high similarity score is a hint, not proof.

Use semantic caching where a false hit is cheap and recoverable: public docs, low-risk help content, broad explanations, or internal developer Q&A with citations. Avoid it when small wording changes alter the answer: refunds, account-specific access, compliance, incident response, contract terms, or anything that changes from "can I" to "should I" based on user state. If you enable it, sample semantic hits and grade them as part of the eval loop, just like prompt changes.

Semantic cache detail: similarity is only one gate before answer reuse

Quick reference

Set thresholds per task; one global cosine threshold is fake precision.
Require same tenant, policy, data version, and auth scope before a semantic answer can hit.
Prefer returning citations with semantic hits so users and reviewers can inspect the evidence.
Track false-hit rate by task; a high hit rate with low correctness is just fast harm.
Route uncertain matches to generation or review instead of forcing reuse.

Remember this

A semantic cache is a decision system: similarity proposes reuse, but task risk, scope, freshness, and eval evidence decide whether reuse is allowed.

Recover a Stale Cached Answer

Trigger. The refund policy changes from 14 days to 7 days, but cached support-answer entries were keyed only by normalized question and tenant. Symptom. Users still receive the old 14-day answer after the docs update. Root mechanism. The cache key did not include source document version, and the invalidation hook only refreshed the vector index, not the answer cache.

Recovery: purge entries tied to the old refund docs, add sourceDocVersion to exact and semantic cache metadata, and force regeneration for refund-policy questions until the new answers pass review. Prevention: make every cacheable task declare its invalidation sources, run a cache-hit audit during policy releases, and add one regression case that fails if a cached answer survives a source-version change.

Failure detail: stale refund answer survives because source version is missing

Quick reference

Diagnose with a trace: cache kind, key fields, source docs, prompt version, and generated-at time.
Purge by metadata, not only by key prefix; old answers may share several question phrasings.
Warm caches only from reviewed answers for high-impact support policies.
Replay the incident query against exact hit, semantic hit, and forced miss paths.
Prevention passes when a source-version bump makes old entries unreachable or expired.

Remember this

A stale LLM answer is usually a missing ownership field in the key or metadata; recovery starts by proving which version the cached text came from.

Key takeaway

Cache LLM systems by the promise you can defend. Response caches are safest when the key contains every state field that changes the answer. Semantic caches need task risk, thresholds, scope checks, and eval sampling. Prompt caches save repeated-prefix work inside the model path, but they are not an answer-correctness guarantee.

Practice (30 min): wrap one LLM support endpoint with a cache layer. Start with exact response caching only. Expected success: the second identical request returns { kind: "exact" } with the same answer and a new trace id. Intentional failure: change the source document version and verify the old answer no longer hits. Recovery: regenerate and store a new entry with the new sourceDocVersion. Pass when exact hit, forced miss after version change, and stale-entry purge are all observable in traces.