Capacity Planning for Beginners: Traffic, Bottlenecks, and Headroom

Capacity planning starts with a unit of work, not a server count. For PAY-204, the unit is one successful POST /checkout: validate cart, reserve inventory, read customer state, create payment intent, write an order row, publish a fulfillment event, and return a response. Each step consumes a different constrained resource: CPU on the API tier, database connections and write IOPS in Postgres, memory and network in Redis, provider rate limits at the payment edge, and consumer throughput behind the queue.

The map matters because one system can look over-provisioned at the top while a dependency is already saturated underneath. Ten idle API pods do not help if the Postgres pool has only 80 safe connections, or if the payment provider allows 300 requests per second and campaign traffic needs 500. Capacity planning asks where work queues, where it can be shed, and which component owns the business promise.

Quick choice grid

Most bad plans fail because they use averages. If checkout normally handles 1.2 million orders per day, that is only about 14 orders per second as a daily average. But Friday's campaign may place 25% of daily orders into a three-hour window, and ads may make the first fifteen minutes much sharper than the full campaign window. Capacity should be planned for the peak arrival rate that the system must absorb, not for the arithmetic average over a calm day.

For a beginner-friendly first pass, convert business traffic into requests per second, then multiply by fan-out. If 90,000 checkouts can arrive in three hours, the window average is about 8.3 checkouts per second. If the first fifteen minutes may receive 30% of that traffic, the burst is 30 checkouts per second. If each checkout performs 2 database writes, 3 database reads, 1 Redis write, 1 payment call, and 1 event publish, every checkout becomes multiple operations on different resources.

This estimate is intentionally simple. The point is not fake precision; the point is to make assumptions visible early enough to challenge them. Write the input numbers beside the plan so product, marketing, and engineering can correct the same model before infrastructure is changed.

After traffic is estimated, capacity planning becomes a bottleneck search. A component's rough maximum throughput is its safe concurrency divided by average service time. If the checkout API can safely run 200 concurrent requests and each request spends 250 ms of active API time, the API tier has about 800 requests per second of CPU-side capacity before other limits. That does not mean checkout can process 800 payments per second; it only means the API tier is unlikely to be first.

The same math exposes tighter limits. If the Postgres pool has 80 safe connections and each checkout transaction occupies a connection for 120 ms, the database transaction path has about 666 transactions per second before saturation under perfect conditions. If the payment provider caps the merchant at 300 calls per second, the provider becomes the hard synchronous ceiling even if every internal tier looks healthy. The first bottleneck is the smallest safe limit along the required path.

Headroom is the gap between planned peak and that safe limit. Running a critical checkout path at 95% of its measured capacity is not planning; it is betting that traffic, garbage collection, noisy neighbors, retries, provider jitter, and deploys will all behave perfectly at the same time. Beginners should budget explicit headroom and state it as a policy, for example: planned peak must stay below 60-70% of the measured safe capacity for synchronous checkout.

A spreadsheet can point to the likely bottleneck, but it cannot prove the system's real behavior. A useful beginner load test does not need to mimic the entire internet. It needs the same unit of work, realistic dependency behavior, enough warm-up time, and the metrics that reveal saturation: request rate, latency percentiles, error rate, database connection usage, queue depth, CPU, memory, provider throttles, and retry volume.

For PAY-204, run a staged test: warm up at normal traffic, hold the planned campaign peak, then step beyond it until one resource bends. The goal is not to impress anyone with a high number. The goal is to find the first clear failure signal and compare it to the estimate. If the spreadsheet says the payment provider quota should fail first but the database pool saturates first, the map or assumptions are wrong and the plan must change.

Keep test data isolated. Do not call the real payment provider with live charges, do not pollute production analytics, and do not run an unbounded stress test against shared infrastructure. The test should be realistic enough to exercise the path and controlled enough to stop before it causes unrelated damage.

Trigger. The campaign starts, checkout latency rises, and the on-call engineer sees API CPU at 75%. They double the API pod count because the autoscaler dashboard is the most visible graph. Symptom. Pod CPU drops, but checkout p99 keeps climbing, payment timeouts increase, and the order queue grows even faster than before.

Root mechanism. The API tier was not the first bottleneck. More pods created more concurrent database transactions and more simultaneous payment calls, which pushed the Postgres pool and payment quota closer to saturation. Retries then amplified the traffic: a slow payment call caused clients or workers to retry, and each retry consumed the same scarce provider quota again. Scaling the unconstrained tier increased pressure on the constrained tier.

Evidence. The timeline shows API CPU dropping at the same time database pool usage hits its cap, provider 429 responses rise, and retry rate doubles. Recovery: cap checkout concurrency at the API edge, disable aggressive retries, shed non-critical coupon validation, and request a temporary provider quota raise. Prevention: autoscaling policies should use bottleneck-aware signals, and launch reviews should include a dependency capacity table before campaign traffic is admitted.

Once the bottleneck is known, the decision is conditional. If CPU is the bottleneck and the code is reasonably efficient, scale out the stateless API tier. If database connection time is the bottleneck, reduce transaction duration, cache reads, batch writes, add replicas for read-only paths, or move non-critical writes behind a queue. If an external quota is the bottleneck, request capacity early, reduce fan-out, rate-limit admission, or add a graceful fallback. Scaling everything at once is expensive and often hides the real constraint.

A capacity plan should also name the stop rule. For PAY-204, that might be: admit checkout traffic while p95 latency is below 800 ms, payment throttles stay below 0.5%, database pool usage stays below 70%, and oldest fulfillment event age stays below 30 seconds. If any threshold fails for five minutes, disable recommendations and coupon enrichment first, then queue low-priority fulfillment, then rate-limit new checkout attempts with a clear retry-after response.

The best plan is not the one that predicts the exact number. It is the one that makes assumptions visible, proves the narrowest safe path, and defines what the system will stop doing before the core business promise breaks.