Caching 101: Latency, Layers, and Eviction
Caching stores a copy of frequently read data in a faster place so you avoid repeating expensive work. Done well, it cuts latency, shields databases from repeat traffic, and lowers cloud spend. Done poorly, it serves stale secrets and hides real bugs.
This primer maps why caches matter, the latency ladder from CPU to network, seven cache layers, hit vs miss flow, and eviction choices (LRU, LFU, FIFO, TTL, random). For Redis patterns and multi-level strategies, see our deeper [caching strategies](/blog/caching-strategies-redis-cdn) guide.
Running example: a product detail page — first visitor pays for Postgres; everyone after should hit Redis or the CDN when the data is still fresh.
Why caching matters
A cache hit is microseconds to low milliseconds; a database or cross-region API call is tens to hundreds of milliseconds. Five payoffs show up in production: lower latency, less load on backends, better spike handling, snappier UX, and lower bills from fewer origin reads.
Cache what is read often and changes slowly. Skip shared caches for secrets and highly personal data unless keys are scoped and encrypted — a shared Redis key is not a vault.
Quick reference
- Latency: cache µs–ms vs DB/API tens–hundreds of ms.
- Load: one origin read can serve thousands of hits.
- Scale: absorb spikes without linearly scaling Postgres.
- Cost: fewer billed DB and egress calls.
- Security: do not park PII or tokens in shared caches casually.
Remember this
I can name five reasons teams cache — and when not to put sensitive data in a shared store.
The latency hierarchy
Speed is a ladder, not a single number. CPU L1/L2 is nanoseconds. RAM (Redis, Memcached) is tens of microseconds. SSD is ~0.1–1ms; HDD is ~5–10ms. Network calls to APIs or remote DBs often land in 10–500ms.
Concrete contrasts help interviews and design docs: Redis hit ~0.5ms vs Postgres ~50ms; CDN edge ~10ms vs origin ~200ms; browser cache on repeat visits under ~100ms vs a cold 1–2s page. You do not need every layer — pick the layer whose latency budget matches the user path.
Quick reference
- CPU cache: 1–3ns — hardware, not your app code.
- RAM (Redis/Memcached): ~10–100µs typical.
- SSD / HDD: ~0.1–1ms vs ~5–10ms.
- Network / remote DB: often 10–500ms.
- Design to the budget: interactive UI ≠ batch analytics.
Remember this
I can order CPU → RAM → SSD → HDD → network by typical access time.
Seven types of caches
Caches stack from the browser inward. Browser holds static assets. CDN / edge (Cloudflare, Akamai) serves near the user. Application caches API or computed results. Database buffer/query caches live inside the engine. In-memory stores like Redis or Memcached sit beside the app. OS page cache speeds disk. CPU L1/L2/L3 is the innermost hardware layer.
Most product teams deliberately own browser headers, CDN rules, and Redis. OS and CPU caches still matter — they explain why warm boxes feel faster — but you configure them less often.
Quick reference
- Browser — HTML/CSS/JS/images via Cache-Control.
- CDN / edge — geography + origin shield.
- Application — process or sidecar memoization.
- Database — buffer pool / query result cache.
- In-memory — Redis, Memcached for shared hot keys.
- OS + CPU — automatic; measure, rarely micro-manage.
Remember this
I can place browser, CDN, app, DB, in-memory, OS, and CPU caches on the request path.
Hit vs miss: the request path
User → application → cache. On a hit, return fast. On a miss, fetch from database/API/slow storage, store the result (usually with a TTL), then return. That populate-on-miss path is cache-aside — the default for most APIs.
When to use: read-heavy product catalogs, search snippets, weather-like APIs. When to be careful: inventory counts and money — short TTLs, write-through, or skip cache until you have invalidation.
Quick reference
- Hit = serve from cache; miss = load origin then populate.
- Cold start and thundering herd are miss storms — use locks or jittered TTLs.
- Fallback: if cache is down, still serve from origin.
- Measure hit rate; below ~70% on a hot path usually means wrong keys or TTLs.
Remember this
I can trace user → app → cache hit/miss → origin → store → response.
Eviction: what leaves when memory is full
Caches are finite. LRU drops what has not been touched longest. LFU drops lowest access counts. FIFO drops oldest inserts. TTL expires entries by clock regardless of popularity. Random is simple and sometimes good enough under uniform access.
Many systems combine TTL with LRU: expire stale data and, under pressure, evict cold keys. Wrong policy looks like thrashing — hot keys bouncing in and out — or stale pages living forever.
Quick reference
- LRU — default mental model for web hot keys.
- LFU — good when frequency beats recency.
- FIFO — predictable; ignores popularity.
- TTL — freshness contract; pair with invalidation on writes.
- Random — cheap; works when access is fairly uniform.
Remember this
I can contrast LRU, LFU, FIFO, TTL, and random eviction and when each fits.
Best practices and where teams use caches
Cache frequently read, rarely written data. Set TTLs you can defend. Invalidate or update on writes when freshness matters. Watch hit rate and origin latency. Fail open to the database. Never treat a shared cache as a place for unscoped secrets.
Common homes: static assets, public API responses, media, search results, catalogs and prices, sessions (carefully keyed), ML inference outputs, and email templates. The slogan that sticks: right data, right place, right time.
Quick reference
- Prefer cache-aside + TTL for most CRUD reads.
- Invalidate on write for catalogs that must feel instant.
- CDN for public static; Redis for shared dynamic hot keys.
- Sessions: short TTL + secure cookies; avoid dumping full PII blobs.
- Practice: pick one endpoint, add Redis with TTL, graph hit rate for a day.
Remember this
I can list safe cache use cases and the operational habits that keep caches trustworthy.
Caching is not one Redis box — it is a stack from CPU to CDN, plus a policy for what expires and what gets evicted. Start with the latency budget for your path, pick one or two layers, and measure hit rate before adding more.
Your homework: take one read-heavy endpoint, add a Redis (or in-process) cache with a deliberate TTL, force a miss and a hit in logs, then write down which eviction story you would use if memory filled up. Next, deepen with [Caching Strategies: Redis, CDN, and Multi-Level Caches](/blog/caching-strategies-redis-cdn).
Related Articles
Explore this topic