Exactly-Once Delivery: Myth vs Reality

Two generals command armies on separate hills and must attack the enemy in the valley at the same time, or lose. They can only communicate by sending a messenger through enemy territory, where the messenger might be captured. General A sends "attack at dawn." To be sure General B got it, A needs an acknowledgment. But B can't be sure A received that acknowledgment either — so B would need an ack for the ack, forever. No finite number of messages lets both generals be simultaneously certain the other will attack. This isn't a failure of cleverness; it's a proof that no protocol over an unreliable channel can guarantee both sides have common knowledge of a single event.

Map this onto message delivery: a producer sends a message and wants to know the consumer processed it exactly once. The consumer's acknowledgment can be lost exactly like the messenger — so the producer must choose between two failure modes: retry without a confirmed ack (risking a duplicate if the first message actually arrived) or don't retry (risking the message never arrives at all). Every real messaging system picks the first: at-least-once, with duplicates as the accepted cost of never silently losing a message.

Since exactly-once delivery is off the table, the practical goal is exactly-once effect: a message can arrive twice, but processing it twice produces the same result as processing it once. The mechanism is a processed-message record, checked and written in the same transaction as the side effect itself — not as a separate best-effort step.

This is the same idempotent-consumer pattern that makes an outbox relay's at-least-once delivery safe. The dedup key should be something the message actually carries — an event id, an idempotency key from the original request — not something derived from content that could coincidentally repeat (like a customer id alone, which would wrongly block a second legitimate charge for the same customer).

Kafka's exactly-once semantics (EOS) are real, not marketing — but they cover a specific, bounded scope: the path from producing a message, through the topic, to a consumer that reads, processes, and produces further messages entirely within Kafka. An idempotent producer attaches a producer ID and sequence number to each message so the broker can reject a retried duplicate at the point of write. Kafka transactions let a consume-transform-produce step commit its output messages and its consumer-offset update atomically — either both happen or neither does, so a crash mid-step can't half-apply the transform.

What EOS does not cover is any side effect outside Kafka's own transactional boundary. Calling a payment API, writing to an external database, or sending an email are not part of the Kafka transaction — Kafka has no way to make an external HTTP call roll back. EOS guarantees the Kafka-internal bookkeeping is exactly-once; it says nothing about the external side effect a consumer triggers while processing a message.

Trigger. Consumer reads InvoiceApproved for invoice-4471 from a Kafka topic, calls the external payment processor, which charges the card and returns success. Before the consumer commits its offset, the process crashes. Symptom. Kafka redelivers invoice-4471 to the next consumer instance, which has no record the charge already happened, and calls the payment processor again — the customer is now charged twice.

Root mechanism. The offset commit is inside Kafka's transactional boundary; the payment API call is not. Kafka's EOS guarantees are working exactly as designed — no message was lost, and Kafka's own bookkeeping is consistent — the duplicate happened entirely outside anything Kafka's transaction could cover. Recovery: the payment processor's own idempotency-key support (most major processors accept one) rejects the second charge attempt as a duplicate of the first, provided the consumer passed the same key both times. Verification: query the payment processor for invoice-4471's idempotency key and confirm exactly one successful charge recorded, despite two API calls. Prevention: always pass a stable, message-derived idempotency key to any external call made during message processing — the fix lives at the consumer and the external API, not inside Kafka.