Single Points of Failure: Finding the Hidden Ones

A single point of failure is any one component whose failure takes down a capability the system is supposed to provide, with no other path around it. The word "single" describes the failure domain, not the org chart box — a SPOF can be a server, but it can just as easily be one NAT gateway, one DNS provider, one TLS certificate nobody is watching the expiry of, one shared config service, or one engineer who is the only person who knows how the deploy pipeline actually works.

Teams often eliminate the SPOF they can see — the single app server — and declare victory, without asking what every replica of that server still depends on. If three redundant API replicas all route outbound traffic through one NAT gateway, or all read from one un-replicated feature-flag service, the visible redundancy is real but incomplete: the system still has exactly one throat to choke, just one layer down from where people looked.

Finding SPOFs is a dependency-tracing exercise, not a headcount of servers: for every capability the system must keep providing, list every component a request actually touches, then ask "if this one disappears right now, does the capability survive?" This is closely related to, but distinct from, availability vs reliability vs durability — a SPOF audit finds where a failure can originate; those three properties define what survives it.

Start from one capability — "a customer can complete checkout" — and list literally everything a request touches to make that true: DNS resolution, TLS certificate validity, load balancer, app replicas, the database and its connection pool, the cache, the message broker, the NAT gateway or egress path for any outbound call, the secrets/config store, and any third-party API in the critical path. Most SPOF audits stop at the app tier and never reach the network and infrastructure layer underneath it.

For every item on that list, ask the same question: if this disappears right now, does checkout still work? A component with only one instance is a candidate; a component with multiple instances that all share one deeper dependency (one AZ, one region, one account, one certificate) is still a candidate, just disguised as redundant. Include operational dependencies too — a deploy or rollback process that only one person knows how to run is a SPOF with a very long, very human recovery time.

This exercise is worth doing on paper before it's tested in production, because production is where a hidden SPOF gets discovered by an incident instead of a review.

Redundancy removes a SPOF only when the replicas fail independently — the failure of one must not make the failure of the others more likely. Three app replicas in three separate availability zones, behind a load balancer that itself has multi-AZ redundancy, are close to independent: a single AZ's networking incident takes out one replica's zone, not all three. Three replicas in one AZ behind a single-instance load balancer are not independent at all — they share the AZ's power, network, and the load balancer as one more SPOF layered on top.

Active-active redundancy (all replicas serving live traffic, load balanced across them) is verified continuously by production traffic itself — if a replica were broken, requests would already be failing against it. Cold standby (a backup that only activates during failover) removes a SPOF on paper but not in practice until failover has actually been tested; an untested standby is a SPOF with a false sense of security attached, because the first time it's needed is also the first time anyone finds out if it works.

Redundancy also has to match the failure domain of the risk it targets: process redundancy protects against a process crash, AZ redundancy protects against an AZ outage, and region redundancy protects against a regional event — buying more of one kind does not substitute for the kind the actual risk requires. When a downstream dependency itself becomes the slow or unavailable link, the backpressure and load-shedding mechanisms there apply just as much to a degraded redundant path as to a single one.

A correlated failure happens when components that look independent all fail together because they share a hidden common cause — the same cloud region, the same certificate authority, the same config deploy, the same library version carrying the same bug, or the same on-call engineer who happens to be unreachable during the one incident that needs them. Redundancy assumes independence; a correlated failure is exactly the case where that assumption quietly breaks.

The classic version: a bad config or feature-flag change gets pushed to "all replicas" simultaneously specifically because they are treated as identical, redundant units. The redundancy that was supposed to isolate failures becomes the mechanism that guarantees they happen everywhere at once. The same pattern shows up with certificate expiry (one cert, every replica trusts it, every replica breaks the same second), and with a shared upstream dependency like a single payment provider or a single third-party auth service.

Mitigating correlated failure means asking a different question than the SPOF checklist: not "is there only one instance," but "what do all these instances have in common that could break them at the same moment" — staged rollouts, canarying config changes, and diversifying where genuinely feasible (separate cloud accounts, separate cert issuance, separate credential sets) all target this specific risk.

Trigger. An AZ hosting one of three checkout API replicas has a networking incident. Two replicas, in two other AZs, remain fully healthy and reachable. Symptom: checkout fails for essentially all traffic, not the expected one-third — every replica, healthy or not, returns payment-provider timeouts.

Root mechanism. All three replicas' outbound calls to the payment provider route through a single NAT gateway, provisioned in the AZ that just had the incident, because it was cheaper to run one NAT gateway than one per AZ and nobody re-evaluated that choice after the API tier became multi-AZ. The app tier's redundancy was real; the egress path underneath it was a SPOF the team had already "fixed" once (by adding AZ redundancy to the app tier) without checking whether the fix actually reached every dependency.

Evidence. Payment-provider timeout logs correlate exactly with the NAT gateway's AZ, not with any individual replica's AZ — all three replicas show the identical failure signature at the identical timestamp. Recovery: manually reroute egress through a NAT gateway in a healthy AZ, or fail over DNS/routing to bypass the broken path. Prevention: provision one NAT gateway per AZ so each replica's egress path shares its own AZ's failure domain, and add a scheduled check that verifies every declared-redundant component's actual dependency graph, not just its instance count.

Not every SPOF is worth eliminating immediately — some are cheap to accept if the recovery time is short and the blast radius is small. Prioritize using blast radius (how much of the system breaks) multiplied by probability (how often the trigger actually occurs), not by how visible or embarrassing the component looks in an architecture diagram.

For high blast-radius, non-trivial-probability SPOFs — shared egress paths, shared certificates, a single region for a global product — invest in real redundancy with independent failure domains and test it with a deliberate failure (kill the component in a controlled way and confirm the system survives). For low blast-radius or very low-probability SPOFs, an explicit accepted-risk decision with a documented, tested recovery plan (RTO/RPO) is often more honest and cheaper than chasing full redundancy everywhere.

Finish every SPOF review with the same question the finding phase started with, applied to the fix itself: does the redundancy just added share a deeper dependency with anything else declared redundant? A SPOF audit that stops one layer too early just moves the incident, it doesn't prevent it.