Kafka Consumer Groups Explained
A consumer service scales from two instances to eight expecting throughput to climb accordingly — instead, six of the eight sit idle while the same two partitions do all the work. The topic has exactly two partitions, and a consumer group can never assign a partition to more than one consumer at a time. Scaling read parallelism in Kafka has a hard ceiling most teams don't hit until the exact moment they need the headroom.
This guide covers what a consumer group actually coordinates, the real mechanical difference between the old stop-the-world rebalance and the newer cooperative-sticky protocol, and the offset-commit choice that decides whether a crash loses work or duplicates it. Then it traces a real rebalance storm — a slow consumer repeatedly kicked from and rejoining a group — to its root cause. For what 'exactly-once' can and can't mean for the messages a consumer processes, see Exactly-Once Delivery: Myth vs Reality; for how Kafka compares to other brokers, see Kafka vs RabbitMQ vs SQS.
What a Consumer Group Actually Coordinates
A consumer group is a named set of consumers that jointly consume a topic, and Kafka's one hard rule is that each partition is assigned to exactly one consumer within the group at any moment. This is the entire mechanism for parallelism: a topic with 8 partitions can have up to 8 consumers in one group each processing a distinct partition in parallel — a 9th consumer in that group would sit completely idle, because there's no partition left to give it.
Different consumer groups reading the same topic are fully independent — each group tracks its own committed offsets per partition, so a billing group and an analytics group can both read every message on the same topic at their own pace, one slow and one fast, without affecting each other. This is what makes "how many consumer groups do we have on this topic" and "how many partitions does this topic have" two completely different scaling questions.
Quick reference
- One partition, one consumer within a group, at any given moment — this is Kafka's fundamental parallelism ceiling.
- More consumers than partitions in one group means the excess consumers are idle, not a smaller share of work.
- Different consumer groups reading the same topic are fully independent, each with its own offset position.
- Partition count is a capacity-planning decision made before you need more read parallelism than it allows — repartitioning an existing topic is a separate, disruptive operation.
- A single consumer process can run multiple threads each with their own consumer instance to consume multiple partitions from one process, but the one-partition-one-consumer group rule still holds per consumer instance.
Remember this
A consumer group's maximum read parallelism is fixed by partition count — adding the ninth consumer to an 8-partition group's topic doesn't add capacity, it adds an idle process.
Rebalancing: Eager vs Cooperative Sticky
A rebalance reassigns partitions among a group's consumers, and it's triggered whenever the group's membership or the topic's partition count changes — a consumer joins, a consumer leaves cleanly, a consumer's heartbeat times out (crash, GC pause, or anything blocking past max.poll.interval.ms), or partitions are added to the topic. The group coordinator (a designated broker) manages this.
The original, still-common eager rebalance protocol is stop-the-world: every consumer in the group revokes all its partitions first, the coordinator reassigns everything from scratch, and only then does processing resume — meaning even a group of 50 consumers pauses entirely because one consumer joined or left. The newer cooperative-sticky assignor (KIP-429) only revokes and reassigns the specific partitions that actually need to move, letting every unaffected consumer keep processing the whole time — a materially different operational experience for a large group, and worth explicitly confirming which assignor a given consumer group is actually configured with, since eager remains a valid and still-deployed default in many client configurations.
Quick reference
- Triggers: consumer joins, consumer leaves cleanly, heartbeat/session timeout, or partition count change.
- Eager: all partitions revoked from all consumers, then reassigned from scratch — full-group pause every time.
- Cooperative-sticky: only the specific partitions that need to move are revoked — unaffected consumers keep working.
- session.timeout.ms controls how fast a crashed consumer is detected; max.poll.interval.ms bounds how long processing one poll's batch may take before being considered stuck.
- Verify which assignor is actually active in your client config — many teams assume cooperative-sticky is already the behavior when eager is still configured.
Remember this
Eager rebalancing pauses the entire group for any membership change; cooperative-sticky reassigns only the partitions that actually need to move — confirm which one your consumers are actually configured with.
Offsets and Commit Strategies
A consumer group's progress is tracked as a committed offset per partition, stored in Kafka's internal __consumer_offsets topic — this is what lets a consumer resume from where it left off after a restart or rebalance, instead of reprocessing the entire topic. How and when that offset gets committed is the single choice that decides your failure behavior.
Auto-commit (enable.auto.commit=true) periodically commits the latest offset returned by poll() on a timer, regardless of whether your code actually finished processing those records — a crash between receiving a batch and finishing its processing, right after an auto-commit fires, silently skips that unprocessed work forever; the consumer resumes past it on restart. Manual commit after processing (enable.auto.commit=false, explicit commitSync()/commitAsync() once your handler completes) instead guarantees a message is only marked done after your code actually finished with it — a crash mid-processing means that batch gets redelivered on restart, which is safe only if your processing is idempotent, exactly the pattern Exactly-Once Delivery covers in depth.
Quick reference
- Auto-commit on a timer risks silently skipping unfinished work on a crash — an at-most-once risk most teams don't intend.
- Manual commit-after-processing risks redelivering already-processed work on a crash — an at-least-once behavior that needs an idempotent consumer.
- There is no commit strategy that gives exactly-once on its own — see Exactly-Once Delivery for why, and for the idempotency pattern that makes at-least-once safe.
- commitSync() blocks until the broker acknowledges; commitAsync() doesn't block but needs its own error-handling callback — mixing them carelessly can commit offsets out of order.
- Committing too eagerly (before processing) trades correctness for throughput; committing too late (batching many records before one commit) trades a larger reprocessing window for fewer commit round trips.
Remember this
Auto-commit can silently skip unfinished work on a crash; manual commit-after-processing can redeliver it instead — pick manual commit plus an idempotent consumer unless you've deliberately decided at-most-once is acceptable.
Recover One Rebalance Storm
Trigger. A consumer's message handler occasionally makes a slow downstream call, and one particular batch pushes total processing time past max.poll.interval.ms before the next poll() call. Symptom. The group coordinator considers that consumer dead, triggers a rebalance, and reassigns its partitions to other members — but moments later the original consumer finishes its slow batch, calls poll() again, discovers it's been evicted, and rejoins the group, triggering a second rebalance to hand partitions back. If the slow-call pattern repeats, this cycles continuously, and the group spends more time rebalancing than consuming.
Root mechanism. max.poll.interval.ms is a promise about how long your code takes between poll calls, not about how long the broker connection stays alive — a slow handler breaks that promise even though the consumer process is still healthy and connected. Recovery: raise max.poll.interval.ms to accommodate the real worst-case processing time for a batch, or reduce max.poll.records so each batch is smaller and finishes faster. Verification: monitor the group's rebalance rate directly (Kafka exposes this as a consumer-group metric) and confirm it drops to near-zero after the fix, not just that throughput looks better. Prevention: move genuinely slow per-record work (a slow external API call, a large computation) off the poll-processing thread entirely, or paginate it, so no single batch's processing time is at the mercy of one slow downstream call.
Quick reference
- Trigger: a slow handler pushes one batch's total processing time past max.poll.interval.ms.
- Symptom: repeated evict-then-rejoin cycles — a rebalance storm, not a single one-off rebalance.
- Root mechanism: the interval bounds time between poll() calls, which depends on your code's processing time, not connection health.
- Recovery: raise the interval to real worst-case processing time, or shrink batch size so it's less likely to be exceeded.
- Prevention: bound or offload genuinely slow per-record work so no single batch's duration depends on one slow external call.
Remember this
A rebalance storm is usually max.poll.interval.ms catching a real slow-handler problem repeatedly — the fix is bounding processing time per batch, not raising the timeout until the symptom disappears.
Key takeaway
A consumer group's parallelism ceiling is its partition count, not its consumer count. Eager rebalancing pauses an entire group for any membership change; cooperative-sticky reassigns only what actually needs to move. Auto-commit risks silently skipping unfinished work on a crash; manual commit-after-processing risks redelivering it instead, which is safe only with an idempotent consumer. A rebalance storm is almost always a real processing-time problem exceeding max.poll.interval.ms repeatedly, not a Kafka misconfiguration to paper over.
Practice (25 min): run a 3-partition topic with a 2-consumer group and confirm one consumer handles two partitions while the other handles one. Add a third consumer and confirm each now owns exactly one partition, then add a fourth and confirm it sits idle. Configure a handler with an artificial slow path that occasionally exceeds max.poll.interval.ms, and watch the group's rebalance-rate metric spike. Fix it by bounding per-record processing time, and confirm the rebalance rate returns to zero. Pass when you can explain, for your own consumer group, exactly what triggered its most recent rebalance.
Related Articles
Explore this topic