Latency vs Throughput Explained

When a user clicks Pay, they care about latency: the wall-clock time from request start to response. When the business asks whether Black Friday traffic fits, they care about throughput: how many successful checkouts per second the system can sustain.

A single worker that finishes each checkout in 100 ms can complete about 10 requests per second if it handles one at a time. Ten workers in parallel can raise throughput while keeping the same per-request latency—until a shared dependency (database, lock, CPU, connection pool) saturates. After that point, queues grow, latency rises, and throughput stops climbing.

Running example. Subject: checkout-api POST /checkout. Success: p99 latency ≤ 300 ms at a stated concurrency, with error rate near zero. Intentional failure: push concurrency past the knee so p99 blows up while average latency still looks acceptable.

Latency is the time a single request spends in the system: queue wait + service time + downstream waits. For POST /checkout, that includes auth, inventory check, payment call, and response write. Network RTT, TLS, and client parsing sit outside the server's view unless you measure end-to-end.

Averages lie about user experience because latency distributions are usually right-skewed: most requests are fine, a few are very slow, and those slow ones pull the mean up without describing any typical request. Report p50 (median), p95, and p99. Service objectives almost always target a high percentile (“p99 under 300 ms”), not the mean.

Tail latency compounds across fan-out. If checkout calls five dependencies and each has a 1% chance of being in its own slow tail, the chance that the overall request hits at least one slow dependency grows quickly. That is why a healthy-looking p50 can still produce an unacceptable checkout p99.

Throughput is the rate of completed work—successful POST /checkout responses per second, not attempted connections and not bytes on the wire unless that is the metric you chose. A system can accept traffic faster than it completes work; that gap becomes a queue and later becomes latency.

Raising throughput usually means more parallelism (replicas, workers, shards) or less work per request (cache hits, smaller payloads, batching). Batching is the classic trade: wait to fill a batch, raise items processed per second, and increase the wait each item sees. That may be correct for a settlement job and wrong for interactive checkout.

Always pair throughput with an error budget. “5,000 req/s” that includes 20% timeouts is not capacity—it is a failure mode producing garbage numbers. Sustainable throughput is the highest rate you can hold while meeting the latency SLO and error SLO.

In a stable system (queues do not grow without bound), Little's Law relates concurrency, arrival rate, and latency:

L = λ × W

L is the average number of requests in the system (in flight). λ (lambda) is the arrival rate (throughput when stable). W is the average time each request spends in the system (latency).

For checkout-api: if you sustain 200 successful checkouts per second and each spends 0.05 s (50 ms) in the system on average, you need about 200 × 0.05 = 10 concurrent in-flight requests. If latency jumps to 200 ms while you try to keep 200 req/s, concurrency must rise to about 40. Connection pools, thread pools, and memory must absorb that jump—or the system queues harder and latency rises again.

Little's Law does not require a particular scheduler. It is a long-run identity for stable systems. Use it for capacity sanity checks: size pools from target_throughput × p99_latency with headroom, and treat rising concurrency at fixed offered load as a latency-regression alarm.

Trigger. The team enables request batching on the inventory service “to improve performance,” packing up to 50 SKU checks per batch with a 40 ms collection window. Mean inventory latency drops. Symptom. Checkout p99 climbs from ~180 ms to over 900 ms during peak, while dashboards showing average latency still look green. Conversion drops; on-call pages on timeout rate, not on “slow average.”

Root mechanism. Batching raised inventory throughput and lowered average service time per SKU, but each checkout now waits for the batch window plus the slowest item in the batch. Queueing at the batcher adds delay exactly when arrival rate rises—the knee of the latency–throughput curve. Mean latency improved; the user-visible percentile did not.

Recovery. Disable or shrink the batch window for the interactive path; move batching to async reconciliation jobs. Re-run load at peak concurrency and gate release on p99 + error rate, not mean. Prevention. Separate interactive and batch workloads; set SLOs on percentiles; load-test to the knee and refuse to run production past it.

Set the latency budget from the user journey first. Checkout, search-as-you-type, and payment confirmation are latency-bound: a missed p99 is a failed experience even if the cluster is “busy and productive.” Then maximize sustainable throughput inside that budget—scale out, cache, shed load, or reduce work per request without crossing the percentile SLO.

Optimize for throughput first when the consumer is a batch pipeline, analytics ingest, or overnight settlement: wall-clock for one item matters less than items processed per hour, and controlled batching is a feature. Still measure latency so you know queue depth and failure SLOs; “infinite queue” is not free throughput.

If you must choose under incident pressure: protect latency for the interactive path (shed, degrade, serve stale where safe) before chasing raw req/s. A system that completes fewer checkouts correctly beats one that accepts everything and times out.