Hot Partitions and Hotspot Mitigation

A hot partition (or hotspot) is a single shard receiving disproportionate load — traffic, data volume, or both — relative to its peers in the same cluster. It's a distinct failure mode from simple under-provisioning: adding more total cluster capacity does nothing for a hot partition, because the problem isn't that the cluster has too little capacity overall, it's that the key distribution routes far more work to one specific shard than to the others.

This is why aggregate cluster metrics — average CPU, average request latency across all shards — actively hide the problem. A cluster running at a healthy 40% average CPU can have one shard pegged at 100% and fifteen others sitting at 20%; the average looks fine right up until the hot shard falls over, at which point everyone sharing that shard is affected, whether or not they generated the load.

Hot partitions and single points of failure are related but different: a SPOF is a component with no redundant alternative; a hot partition has redundant peer shards that are simply idle while one shard absorbs load none of them are positioned to help with.

Natural key skew is the most common cause: real-world traffic follows a power-law distribution far more often than a uniform one — one customer, one product, one user ends up generating orders of magnitude more load than a typical key, and if the partitioning scheme maps keys to shards deterministically (as most do, for query locality), that one key's shard absorbs all of that disproportionate load by design, not by accident.

Monotonic or time-based partition keys create a different, self-inflicted hotspot: if writes are partitioned or sorted by a strictly increasing key (a timestamp, an auto-incrementing ID), every new write lands on whichever partition currently owns the newest range — the entire write stream funnels through one "current" shard while every older shard goes cold. This is a design mistake independent of traffic patterns; it happens even under perfectly uniform customer behavior.

Low virtual-node count in a consistent-hashing scheme produces the same symptom from a different mechanism — with too few points per physical shard on the hash ring, one shard can end up owning a disproportionately large arc purely from hash luck, independent of any real-world traffic skew at all.

Cluster-wide averages cannot detect a hotspot by construction — they're exactly the metric a hot outlier hides behind. Detection requires per-partition metrics: request rate, CPU, and latency broken out by shard, not aggregated across all of them. A dashboard showing sixteen separate shard lines, or an alert that fires on the maximum shard's load rather than the average, catches what a single cluster-wide number never will.

Once a specific shard is identified as hot, the next question is which key is responsible — a per-shard breakdown of request or byte volume by tenant/key, sorted descending, usually makes the cause obvious immediately: one tenant at 85% of a shard's traffic is a very different problem than sixteen tenants evenly sharing it. Without this breakdown, the fix gets guessed at rather than targeted.

Alert on the ratio between the hottest and coldest shard, not just an absolute threshold — a cluster where the hottest shard runs 5x the load of the coldest is unhealthy regardless of whether the hottest shard's absolute numbers still look acceptable in isolation.

Key salting splits one hot key into several sub-keys (append a random or hashed suffix), spreading what was one shard's worth of load across multiple shards, then fans reads back out and aggregates the results. This works well for high-write keys like counters or event streams where the write can be split arbitrarily and reconciled later.

Dedicated isolation gives a known large key its own shard rather than making it share capacity with smaller, unrelated keys — this is the direct fix for the noisy-neighbor pattern, where an oversized tenant's load spills onto other tenants purely because they happen to share physical resources. Caching in front of a hot shard absorbs read load specifically, the same caching discipline applied at the shard level instead of the whole-service level. Targeted rate limiting or throttling, scoped specifically to the identified noisy key, protects shared capacity for everyone else without punishing the whole cluster — the same backpressure principle applied per-key instead of per-service.

Rebalancing partition boundaries, or raising virtual-node count in a consistent-hashing scheme, addresses hash-placement-caused hotspots specifically — it does nothing for genuine traffic skew, which needs salting or isolation instead. Match the mitigation to the actual cause identified in detection, not to whichever technique is easiest to implement.

Trigger. A mid-sized customer, tenant-4471, launches a major marketing campaign. Their event ingestion volume jumps roughly 40x within a few hours — real, legitimate traffic, not an attack or a bug. Their tenant_id hashes to shard-07, as it always has. Symptom. Shard-07's CPU pegs at 98% and write latency for tenant-4471 climbs into the seconds. Within the hour, two smaller, completely unrelated tenants who happen to share shard-07 — tenant-1102 and tenant-2290 — start seeing their own write latency degrade too, despite generating no unusual traffic of their own.

Root mechanism. The hash-based partitioning scheme had no per-tenant capacity accounting and no notion that a single tenant might need isolation — every tenant, regardless of size, shared physical shard resources with whichever other tenants happened to hash nearby. There was no salting mechanism (this workload's writes were structured as one stream per tenant, not an arbitrarily-splittable counter), and no rate limit scoped to an individual tenant, so tenant-4471's legitimate spike consumed shard-07's entire capacity budget, and the tenants sharing it paid for it without ever being warned.

Evidence. Per-shard dashboards show shard-07 alone at 98% CPU while the other fifteen average 22%; a per-tenant breakdown on shard-07 shows tenant-4471 at 91% of that shard's request volume, up from a typical 15%. Recovery: apply an emergency rate limit to tenant-4471's write path to protect the shared shard, then migrate them to a dedicated, isolated shard sized for their new volume. Prevention: monitor per-tenant load continuously (not just per-shard), define a size/volume threshold above which a tenant is proactively moved to dedicated capacity before an incident forces it, and build per-tenant rate limiting as a standing operational control, not an emergency-only tool.

Treat per-shard and per-key monitoring as a first-class, always-on practice, not a tool reached for only after an incident — a hotspot detected from a customer complaint is a hotspot detected far too late. Alert on the ratio between the hottest and coldest shard, and maintain a per-key breakdown capability so the cause is identifiable within minutes, not hours.

Define a proactive threshold — by volume, by tenant tier, by whatever metric predicts outsized load — above which a key gets migrated to dedicated, isolated capacity before it becomes a shared shard's problem, rather than waiting for an incident to force the decision under pressure. Build salting and per-key rate limiting as standing operational tools available immediately, not techniques invented for the first time during an active outage.

Match the mitigation to the actual cause: rebalance or raise virtual-node count for hash-placement issues, salt or isolate for genuine traffic skew, and never assume a monotonic partition key is safe for a write-heavy workload without checking it against this exact failure mode first.