Distributed Transactions: Microservice Saga Pattern
Transitioning from monolithic database architectures to distributed microservices breaks traditional ACID database transactions. When an e-commerce order workflow requires updating Inventory, Charging Credit Cards, and Reserving Shipping Slots, executing these operations across independent microservices cannot rely on a single database BEGIN ... COMMIT block.
While traditional Two-Phase Commit (2PC) protocols lock database records across network boundaries, 2PC creates severe availability bottlenecks and single points of failure. The Saga Pattern resolves distributed transaction consistency by executing a sequence of local transactions, paired with Compensating Transactions that undo previous steps if a downstream microservice fails. This guide details Saga architecture, Choreography vs Orchestration trade-offs, compensating action idempotency, and state machine recovery.
Mental Model: Monolithic 2-Phase Commit (2PC) vs Saga Event-Driven Compensations
Monolithic ACID transactions lock database table rows until all updates complete. In microservices, locking remote database tables during 2PC causes distributed deadlocks and holds open long-lived connection pools.
Saga Pattern Architecture decomposes a global transaction into a series of independent Local Transactions ($T_1, T_2, \dots, T_n$).
Each local transaction updates its local microservice database and emits a message or event. If a downstream step fails (e.g., Payment Rejected at step $T_3$), the Saga executes a series of Compensating Transactions ($C_2, C_1$) in reverse order, undoing side-effects to restore data consistency. For architectural trade-offs, review saga vs two phase commit and building event driven microservices nats jetstream.
Quick reference
- Decomposes global multi-service workflows into a sequence of isolated local transactions.
- Eliminates distributed 2PC database row locks to maintain high microservice availability.
- Executes compensating transactions (C_n) in reverse order when downstream failures occur.
- Maintains Eventually Consistent data states across distributed database stores.
- Powers complex multi-step workflows at Amazon, Uber, Netflix, and Shopify.
Remember this
Implement the Saga pattern to manage multi-service distributed transactions with eventual consistency.
Choreography-Based Sagas vs Orchestrator-Driven State Machines
Sagas can be coordinated using two distinct communication topologies:
1. Choreography (Event-Driven): Microservices listen to domain events and execute local transactions autonomously. Service A emits OrderCreated, Service B consumes it, updates inventory, and emits InventoryReserved. Choreography offers loose coupling but becomes difficult to trace when workflows exceed 5+ steps.
2. Orchestration (Centralized State Machine): A dedicated Saga Orchestrator service explicitly commands microservices to execute local transactions (e.g. ExecutePayment()) and tracks global workflow progress in an orchestration state machine DB.
Quick reference
- Choreography relies on pub/sub domain events; ideal for simple 2-to-3 step workflows.
- Orchestration uses a central coordinator (Temporal, AWS Step Functions) for complex workflows.
- Orchestrators provide centralized visibility into active, pending, and failed Saga states.
- Prevents cyclic dependency traps inherent in large-scale event choreography networks.
- Orchestration simplifies error handling by centralizing compensating transaction triggers.
Remember this
Use Choreography for simple workflows and Orchestration state machines for complex multi-step sagas.
Designing Idempotent Compensating Transactions & Backward Recovery
Compensating transactions are NOT simply database rollbacks — they are forward-executing corrective actions (e.g., issuing a refund credit rather than deleting a payment record).
Because network retries may deliver compensating commands multiple times, compensating actions MUST BE IDEMPOTENT:
1async function compensatePayment(paymentId: string): Promise<void> {2 // 1. Idempotency Check: Verify if refund was already issued3 const existingRefund = await db.refunds.findUnique({ where: { paymentId } });4 if (existingRefund) return; // Safe duplicate skip!5 6 // 2. Issue Refund & Record Transaction7 await stripe.refunds.create({ payment_intent: paymentId });8 await db.refunds.create({ data: { paymentId, status: 'REFUNDED' } });9}Quick reference
- Compensating actions execute forward-corrective logic (e.g. issuing refund credits).
- Must be fully idempotent to handle duplicate retry delivery safely without double-refunding.
- Never delete historical database records; append corrective compensation logs instead.
- Saga step failures trigger backward recovery (C_n -> C_1) automatically.
- Pivot Transactions demarcate the point-of-no-return after which compensating is impossible.
Remember this
Design idempotent compensating transactions that record append-only corrective actions during failures.
Handling Out-of-Order Events & Idempotency Keys with Event Sourcing
In asynchronous messaging networks, messages can arrive out-of-order (e.g. OrderCancelled arrives before OrderCreated).
To handle out-of-order Saga events, microservices maintain a Saga Execution Log in local storage using Idempotency Keys (e.g. order_id + sequence_number). If a cancellation event arrives for an un-created order, the microservice records a pre-cancellation tombstone state. When the creation event eventually arrives, the microservice inspects the tombstone and aborts execution immediately.
Quick reference
- Saga Execution Logs track sequence numbers to detect out-of-order event arrivals.
- Pre-cancellation tombstones handle race conditions where cancel events precede create events.
- Combines Event Sourcing with Saga state machines for complete auditability.
- Outbox Pattern ensures local DB updates and event publishing occur in a single transaction.
- Guarantees data consistency across asynchronous microservices despite network reordering.
Remember this
Use Saga Execution Logs and tombstones to protect distributed transactions against out-of-order events.
Key takeaway
To test the Saga pattern, implement a simple 2-step Orchestrator using Node.js or Go. Simulate a failure in Step 2 and verify that Step 1 compensating actions execute cleanly.
Related Articles
Explore this topic