Fail-Fast vs Fail-Safe Systems

Fail-fast answers when: the moment a fault is detected — an invariant violated, a dependency unreachable, input that doesn't match its contract — stop immediately instead of continuing on with a possibly-corrupt state. The alternative, continuing to operate on bad data or an unverified assumption, tends to produce a worse failure later, further from its cause and harder to diagnose.

Fail-safe answers what: given that the system has to stop or degrade, which specific state minimizes harm? Critically, "safe" is domain-specific, not universal. A fire-alarm-triggered door lock is fail-safe when it releases — people must be able to escape, even though that is a fail-open outcome from a pure access-control point of view. A wire-transfer approval gate is fail-safe when it denies — an unauthorized approval is the harm to prevent, so its safe failure state is fail-closed, the opposite direction.

These are not opposites, and a system should usually be both: fail fast (stop the instant the fault is detected, don't let it propagate) into the safe state for that specific system (which might be halt, deny, degrade, or — for physical safety — open). The dangerous failure mode is neither of these: continuing to run, slowly, while doing the wrong and unsafe thing — the same trap graceful degradation has to guard against when a fallback path covers the wrong failure modes.

Invariant checks and assertions at function or module boundaries turn a silent, propagating bug into an immediate, loud one: if a precondition is violated, throw right there instead of letting corrupted state flow three layers deeper before something finally breaks in a way that's hard to trace back. Strict input validation at the boundary — reject a malformed request immediately rather than coercing it into something "close enough" — is the same principle applied to external input.

Circuit breakers that trip on the first sustained failure signal, rather than retrying indefinitely, are a fail-fast mechanism at the network layer: once a dependency looks unhealthy, stop sending it work immediately instead of queuing requests behind a slowly dying service. Crash-only software takes this furthest: a process that detects an unrecoverable internal inconsistency simply exits and lets its supervisor restart it clean, rather than attempting in-process recovery that risks leaving hidden corrupted state behind.

All of these share the same shape: detect the fault as close to its source as possible, and stop there — the cost of a fast, loud failure is almost always smaller than the cost of a slow, silent one.

Before wiring any failure default, write down explicitly what harm this specific system is protecting against — that answer determines which direction is safe. For an authorization gate, the harm is an unauthorized action succeeding, so the safe default is fail-closed: deny, and require an explicit successful check before allowing anything. For a physical safety system like an electronic door lock, the harm is trapping someone during an emergency, so the safe default is fail-open: the latch releases on power or system failure.

This is why borrowing a resilience pattern from one domain into another without checking direction is dangerous. "Default to allow when the dependency is unreachable" is a completely reasonable choice for a recommendation service or a non-critical feature flag — it protects availability, and the cost of being wrong is small. Applied to a permission check, the exact same pattern inverts the safety property the gate exists to provide.

Once the safe direction is chosen, implement it as the default the code falls into on any uncertain outcome — not just the failure paths someone thought to test. A permission check that fails closed on a timeout but accidentally fails open on a malformed response has not actually implemented fail-safe; it has implemented "fail-safe for the failure modes I imagined."

The strongest designs use both properties together: detect the fault immediately (fail-fast) and land in the direction that minimizes harm for this specific system (fail-safe). For the wire-transfer gate, that means the permission check times out quickly (300ms, not 30 seconds), the timeout is caught right there rather than propagating an ambiguous pending state upward, and the fallback is an immediate, explicit deny — not a retry loop that eventually gives up into an unclear default.

The worst outcome is neither fail-fast nor fail-safe: a system that keeps running for a long time, in a state nobody would call safe. A permission check with a 30-second timeout that eventually times out and then defaults to allow is fail-slow (users and downstream systems wait a long time not knowing the outcome) and fail-unsafe (the eventual default is wrong) at once — combining the worst of both failure axes.

Evaluate any critical path against both questions independently: does it stop fast enough when something goes wrong, and does it land somewhere that actually protects what this system is supposed to protect? A design can pass one and fail the other, and only checking one of the two axes is how the failure story below happened — the same per-operation discipline fault tolerance vs high availability argues for when a single blanket architecture decision hides a gap on one specific path.

Trigger. During a routine deploy, the permission service becomes intermittently slow, and calls from the transfer-approval gate start timing out at roughly a 12% rate. Symptom: an internal audit weeks later finds several wire transfers approved during that window with no corresponding successful permission-check record — transfers that should have required explicit authorization went through without it.

Root mechanism. The client code wrapping the permission-service call had been "hardened for availability" months earlier by an engineer following a general resilience pattern used elsewhere in the codebase for non-critical services: catch the timeout, log a warning, and default to allowing the request rather than failing the whole endpoint. That pattern is correct for a recommendation call. Applied here, it silently inverted the gate's required fail-closed default into a fail-open one — the code compiled, passed its happy-path tests, and looked identical in a code review to a dozen other "resilient" call sites.

Evidence. The transfer-approval audit log shows entries where permissionCheckResult: "TIMEOUT" and approved: true appear together — a combination that should be structurally impossible in a fail-closed design. Recovery: freeze the affected accounts, manually re-verify and reverse any transfer that lacks a genuine ALLOW record. Prevention: an explicit test asserting isAuthorized returns false on every non-success path (timeout, malformed response, network error, not just the one case someone thought to test), and a security review rule that any dependency-failure fallback on an authorization path must be reviewed for direction, not just resilience.

Default to fail-fast almost everywhere: an invariant violation, a malformed input, or a dependency that's clearly unhealthy should stop the current operation immediately rather than continue on uncertain footing. The cost of a fast, clear failure is nearly always lower than a slow, ambiguous one.

Decide the fail-safe direction explicitly, per path, from the specific harm being protected against — never by copying a fallback pattern from a different call site without checking whether the harm being protected against is the same. Authorization, payment authorization, and anything that gates an irreversible action should default fail-closed on any uncertain outcome; availability-focused, low-stakes features can reasonably default toward staying up in a degraded form.

Write an explicit test for the failure default on every security-sensitive path — "what does this return when the dependency times out, errors, or returns something malformed" — and treat a passing test suite that never exercises those branches as untested, not verified.