Sharding vs Partitioning vs Replication

Partitioning divides one large table into smaller physical pieces — by range, list, or hash — while it all still lives in one database with one primary for writes. Sharding divides an entire dataset across multiple independent databases, each a separate failure domain with its own storage and often its own primary. Replication makes copies of the same data, on separate nodes, for availability and read capacity — it doesn't divide anything, it duplicates.

A large real system commonly runs all three together: a dataset sharded across twenty databases by tenant ID, where each shard's largest table is itself partitioned by month, and each shard is also replicated for failover. Asking "should we shard or replicate?" is often the wrong question — the right one is "which axis does our measured bottleneck actually sit on?"

The mechanism looks similar — both divide data by a key — but the scope is the entire distinction. A partitioned table still has one primary; a query that needs data from two partitions is one local query the planner prunes automatically. A sharded dataset has no single primary at all; a query that needs data from two shards means the application (or a routing layer) issuing two separate queries to two separate databases and combining the results itself — there is no query planner spanning shards.

That's why partitioning is usually the earlier, cheaper move: it solves "this one table is too large to scan efficiently" without introducing cross-database routing, rebalancing, or the loss of cross-row transactions that sharding brings. Reach for sharding only when write throughput or total storage exceeds what one database (even a well-partitioned one) can hold — see Database Scaling Techniques for the bottleneck-matching decision in full.

Leader-follower (the most common) has one node accept all writes and stream them to followers, which serve reads. Failover requires promoting a follower to leader — and anything written to the old leader but not yet replicated at the moment of failure is at risk of loss, exactly the trade-off Active-Active vs Active-Passive Architecture covers in depth.

Multi-leader lets more than one node accept writes — usually one leader per data center, replicating to the others. This avoids a single node bottlenecking writes globally, at the cost of needing conflict resolution when the same record is written in two places before either replication catches up. Leaderless (Dynamo-style) has no leader at all: any replica can accept a write, and reads/writes use configurable quorums to stay consistent enough — the exact mechanism Quorum Reads and Writes walks through, including sloppy quorums and read repair. Choosing between these three is really choosing how you want to handle the moment two writes to the same data disagree — a leader-follower system prevents the disagreement by construction (one writer), while multi-leader and leaderless systems accept it can happen and define how to reconcile it after the fact.

A sharding key decides which shard owns each row, and it is the single hardest-to-reverse decision in a sharded system — resharding an entire production dataset to a new key is a project measured in weeks, not a config change. The key needs three properties together: even distribution (no single value should own a disproportionate share of the data or traffic), query alignment (most queries should be answerable from one shard, not a scatter-gather across all of them), and stability (a row's key should rarely need to change, since that means moving it between shards).

A plausible-looking key can still fail in production: sharding a multi-tenant SaaS product by tenant_id looks even in a demo with equal-sized tenants, and then one enterprise customer's usage grows to dwarf every other tenant combined — that single tenant's shard now carries a wildly disproportionate load, the exact hot-partition failure Hot Partitions and Hotspot Mitigation traces in depth. Test candidate keys against your actual data's distribution and your actual query patterns before committing, not against a demo dataset with uniformly-sized entities.

Start from what you've actually measured, in this order: if one table's scans are slow, partition it — no new failure domain, no routing layer. If read capacity or availability is the limit, add replicas to whatever you have, sharded or not. If total write throughput or storage genuinely exceeds one well-partitioned, well-replicated database, shard — and only then does the sharding-key decision above become unavoidable.

A mature large-scale system typically looks like: data sharded across N databases by a carefully chosen key, each shard's largest tables partitioned by date or another natural range, and each shard replicated (leader-follower being the common default) for failover and read scaling. None of the three replaces the other two — they stack, each solving the specific bottleneck it targets.