Backpressure in Distributed Systems

When a producer is faster than a consumer, the gap between them has to go somewhere: it either accumulates in a buffer, or it has to slow the producer down. An unbounded buffer looks like it solves the problem — no requests are rejected, nothing errors — but it only delays the failure and changes its shape. Latency grows unboundedly as the queue backs up, and if the buffer lives in process memory, the failure eventually becomes an out-of-memory crash instead of a graceful slowdown.

Backpressure is an explicit signal from a slow consumer back to its producer saying "stop, or slow down" — turning an invisible, unbounded accumulation into a visible, boundable one. The producer that receives this signal gets to make a decision: block, queue up to a known bound, retry later, or reject the request outright. None of those outcomes is free, but all of them are bounded and observable, unlike an in-memory queue quietly growing until the OS kills the process.

The goal is not to eliminate the mismatch between producer and consumer speed — that mismatch is often real and temporary. The goal is to make it visible and bounded instead of hidden inside a buffer that has no ceiling.

TCP flow control is the canonical low-level example: every ACK carries a receive-window size telling the sender exactly how many more bytes the receiver can currently buffer. The sender is mechanically prevented from sending more than that — backpressure here is built into the protocol, not bolted on.

Reactive streams (RxJS, Project Reactor, Java Flow) generalize the same idea to application code: a consumer calls request(n) to pull exactly n items, and the producer is not allowed to push more until asked. This inverts the naive "producer pushes as fast as it can" model into a pull-based one, where slowness downstream mechanically throttles upstream.

Bounded queues with blocking or rejecting put are the simplest mechanism for in-process pipelines: a queue with a fixed capacity either blocks the producer thread until space frees up, or immediately rejects with a signal the producer must handle. HTTP 429 Too Many Requests with Retry-After is the application-boundary version — an explicit, client-visible instruction to slow down, instead of the server silently queuing the request internally.

Backpressure preserves all the work by making the producer wait or retry — nothing is lost, but latency for everyone rises while the system catches up. Load shedding takes the opposite trade: when capacity is exceeded, it deliberately drops or rejects a portion of incoming work immediately, so that the work it does accept keeps its normal latency instead of degrading for everyone.

They are not competing solutions — they answer different questions. Backpressure answers "how do I keep the producer from overwhelming me" and is usually the first line of defense. Load shedding answers "what do I do once I'm already at capacity and slowing down further isn't acceptable," and it usually kicks in after backpressure's queue is already full.

A well-designed pipeline uses both in sequence: bounded queue accepts up to capacity (backpressure), and once truly full, the system sheds — rejecting new requests outright, or dropping low-priority ones first — rather than letting the queue's blocking behavior turn every caller's latency into the slowest possible outcome.

A pipeline with backpressure at only one hop still fails — the mismatch simply relocates to whichever hop still has an unbounded buffer. If the worker pool applies backpressure toward the queue, but the queue itself is unbounded, the queue absorbs the entire mismatch silently and grows anyway. Backpressure has to propagate end to end: each stage must be bounded, and each stage's "I'm full" signal must reach the stage feeding it, all the way back to the system's edge.

This is why reactive-stream libraries chain request(n) through every operator instead of only at the final subscriber, and why a well-built HTTP gateway forwards a 429 from a downstream service as a 429 to its own caller instead of quietly retrying forever or buffering the request itself. Breaking the chain at any single hop reintroduces the exact unbounded-buffer risk the rest of the pipeline was built to avoid.

For asynchronous pipelines specifically, this usually means: bound the queue at every hop, size each bound from that hop's acceptable worst-case latency, and make each hop's full-signal cause the previous hop to slow down or shed — not silently retry into its own unbounded buffer.

Trigger. The payment provider that the worker pool calls for every order slows from ~80ms to ~4s during a provider-side incident. The worker pool's per-order processing time rises roughly 50x, so its effective consumption rate from the internal order queue drops by the same factor. Symptom: within minutes, API pod memory climbs steadily; 20 minutes in, pods start getting OOM-killed, taking every order still sitting in the in-memory queue down with them, and clients see a mix of hung connections and 502s.

Root mechanism. The order queue between the API and the worker pool was an unbounded in-memory array — no capacity limit, no backpressure signal to the API when the worker pool fell behind. The queue absorbed the entire producer/consumer mismatch silently, converting a downstream latency incident into an upstream memory-exhaustion crash, and destroying orders that had already been accepted and acknowledged to the client.

Evidence. The queue-depth metric (once added post-incident) would show unbounded, near-linear growth starting exactly at the payment-provider latency spike, with no plateau until the OOM kill. Recovery: restart the API with an emergency queue cap and start returning 429 past that cap; reconstruct lost orders from the client-side order confirmation timestamps and payment-provider records where possible. Prevention: bound the queue's capacity, return 429 { Retry-After } once full instead of silently accepting, alert on queue depth as a first-class metric (not just process memory after the fact), and give the queue durability (a persisted log, not just an in-memory array) so a crash does not also mean data loss for already-accepted work.

Bound every internal queue by default — an unbounded queue is a memory-exhaustion incident with a delay timer on it, not a safety margin. Size the bound from the hop's acceptable worst-case latency (queue depth × per-item processing time), not from a round number, and treat queue depth as a first-class alerting metric alongside latency and error rate.

Decide what happens at the bound from what the operation needs: request-response paths (an order API a user is waiting on) usually want to reject fast with 429/503 and Retry-After so the caller can retry or fail visibly, rather than block and let the caller's own timeout fire uselessly. Fire-and-forget or durable-queue-backed paths can often afford to block the producer briefly, since nothing user-facing is waiting synchronously.

For pipelines with multiple hops, verify propagation explicitly: trace one request through every queue and confirm each hop is bounded and that hitting one hop's bound produces a signal the previous hop actually handles — not a silent retry into its own unbounded buffer.