Fault Tolerance vs High Availability

Fault tolerance means the system continues operating correctly through a component failure, with no visible interruption to in-flight work — the failure is masked before it reaches the caller. High availability means the system, as a whole, is reachable and serving requests for a very high percentage of time — but it does not promise that every individual request survives every individual failure. A system can fail over in under a second, count as "available" for the month at 99.99%, and still have dropped every request that was in flight during that second.

The difference is what happens to the specific request or transaction that was active at the moment of failure. Fault tolerance masks that failure from the caller — the request completes as if nothing happened, usually because redundant, synchronized components picked up the exact in-progress work. High availability just means the next request, and the one after that, get served quickly — the system as a measured whole stays up, even if a handful of individual requests did not survive the transition.

Both are legitimate, valuable properties. The problem is choosing a mechanism that provides one while a stakeholder assumes it provides the other — a payment system built for high availability but assumed to be fault-tolerant will surprise someone the first time a failover drops an in-flight charge. This is a sharper, request-level version of the distinction in availability vs reliability vs durability: availability there asks whether the service can be used now, and high availability here is that same question measured over time.

Fault tolerance requires redundant components that are actively, synchronously participating in the same unit of work — not just standing by to take over afterward. State machine replication (via Raft or Paxos) has every replica apply the same operations in the same order, so any single replica's crash mid-operation loses nothing: another replica already has the same state and can answer the next request as if nothing happened. Synchronous replication commits a write to multiple nodes before acknowledging it, so a single node's crash immediately after commit does not lose the write.

Idempotent retries provide a weaker, cheaper form of fault tolerance at the client/caller boundary: if the caller can safely retry the exact same operation and get the exact same result, a mid-flight node crash is masked by the retry succeeding against a different, healthy replica — the caller never has to know a failure happened. This only works if the operation is genuinely safe to repeat, which usually means an idempotency key tied to the specific attempt.

All of these mechanisms share a cost: they require doing the work more than once, synchronously, before answering the caller — which is real latency, paid on every request, not just during a failure.

High availability is built from health checks, load balancing across replicas, and automated failover — mechanisms that detect a failed component and route around it quickly, so the system as a whole resumes serving new requests without a human paging in. Health checks (liveness and readiness) let a load balancer stop routing to an unhealthy instance within seconds; multi-AZ or multi-region deployment ensures the failure of one physical location doesn't take the whole system down with it.

Asynchronous replication with automated failover is the most common HA pattern for stateful systems: a primary serves writes, replicas lag slightly behind, and if the primary fails, one replica is promoted to primary within a bounded window. This is fast and keeps the system available — but any write acknowledged by the old primary and not yet replicated to the promoted replica is gone. That's availability without fault tolerance for that specific write: the service survived; that write did not.

Measuring high availability means tracking uptime/reachability (often as "nines") over a window, and it is a legitimate, widely useful target — most systems do not need every write to be fault-tolerant, only the system as a whole to keep serving.

A system can have both: synchronous quorum replication (fault tolerance for each write) combined with multi-AZ deployment and automated leader election (high availability for the service as a whole) — this is exactly what systems like properly configured Raft-based stores provide, at the cost of the latency every fault-tolerant write pays. A system can have HA without fault tolerance — the common async-replication-plus-failover pattern above — trading some in-flight data loss for much lower everyday write latency.

The less obvious case is a system with fault tolerance but poor availability: a strict-quorum store that refuses to accept writes at all when it cannot reach a majority is fault-tolerant for every write it does accept, but during a network partition it may become fully unavailable rather than serve a possibly-inconsistent answer — correct, but not "available" during that window. This is the same trade-off CAP theorem describes: consistency-preserving fault tolerance and availability compete directly during a partition.

Most systems don't need uniform treatment — apply fault tolerance to the specific operations where losing an in-flight one is unacceptable (payment capture, order placement), and accept HA-only, faster mechanisms everywhere else (session data, view counts, most reads).

Trigger. The payment service's primary database crashes mid-transaction while processing a card charge. Automated failover promotes the most caught-up replica within 2.8 seconds — well inside the team's availability SLO. Symptom: the payment service resumes serving new requests almost immediately (availability metrics look fine), but the specific charge that was in flight at the moment of the crash is gone — no record of it exists on the promoted replica, and the customer's card was never actually charged, though the client had already shown a "processing" spinner.

Root mechanism. The database used asynchronous replication with a replication lag of roughly 400ms under normal load. The crash happened after the primary acknowledged the write to the application but before that write had replicated to any standby. The promoted replica was, correctly, the most caught-up one available — it simply never received this specific write. The system's availability recovered in under 3 seconds; that specific transaction did not recover at all, because nothing about the HA mechanism was designed to preserve it.

Evidence. The application log shows a write acknowledged by the primary at T; the promoted replica's write-ahead log shows its last received entry at T - 380ms; the charge is present in neither the new primary nor the payment provider's own confirmed-charge log (the provider call itself failed when the primary crashed mid-flow). Recovery: reconcile against the payment provider's own transaction log, and only re-attempt charges genuinely absent everywhere, using an idempotency key. Prevention: for payment capture specifically, upgrade to synchronous (quorum) replication or an idempotent retry protocol that survives node failure — accepting the latency cost only on this specific, loss-intolerant operation, not the entire service.

Ask, for each operation: if this specific request is in flight when a node crashes, is losing it acceptable? If no — payment capture, order placement, anything with an external, hard-to-reverse side effect — build fault tolerance for that operation specifically: synchronous replication, quorum writes, or an idempotency protocol the caller actually uses on retry.

If yes — most reads, session refreshes, analytics events, anything cheap to lose or safely re-derived — high availability alone is the right, cheaper choice: async replication, fast automated failover, and a good uptime SLO. Forcing fault-tolerant guarantees onto operations that don't need them pays synchronous-replication latency for no real benefit.

State the chosen guarantee explicitly per operation in design docs and SLAs — "available" and "fault-tolerant" are not interchangeable words, and the gap between them is exactly where incidents like the one above happen.