Distributed Consensus with HashiCorp Raft in Go
In distributed storage systems, keeping data consistent across multiple independent server nodes in the presence of network partitions, hardware crashes, and message drops is a fundamental challenge. If multiple server nodes accept conflicting write requests simultaneously, the cluster degrades into catastrophic data corruption (Split-Brain scenario).
Raft is a consensus algorithm designed for fault-tolerant distributed systems, offering equivalent safety to Paxos while being significantly easier to understand and implement. HashiCorp's hashicorp/raft library is the production-grade Go implementation powering Consul, Vault, and NATS JetStream. This guide details Raft state machine replication, leader election quorums, log compaction, and custom Finite State Machine (FSM) implementation in Go.
Mental Model: Split-Brain State Risks vs Raft Replicated Finite State Machines
Without consensus protocols, distributed server nodes risk accepting conflicting state updates independently during network partitions.
Raft Consensus Architecture decomposes distributed agreement into three well-defined sub-problems: Leader Election, Log Replication, and Safety.
Nodes in a Raft cluster exist in one of three roles: Leader, Follower, or Candidate. All state mutation requests flow strictly through a single elected Leader node. The Leader appends command entries to its local Write-Ahead Log (WAL) and replicates log entries across a majority quorum ($N/2 + 1$) of Follower nodes before applying updates to its Finite State Machine (FSM). For consensus protocol background, review deep dive raft consensus algorithm and building distributed task queues celery redis.
Quick reference
- Decomposes consensus into distinct Leader Election, Log Replication, and Safety sub-problems.
- Guarantees linearizable state machine consistency across N nodes despite up to (N-1)/2 node failures.
- Strict single-leader model simplifies log entry ordering and conflict resolution.
- Requires a strict majority quorum ($N/2 + 1$) to commit log entries and elect new leaders.
- Powers distributed state engines in HashiCorp Consul, Vault, CockroachDB, and Etcd.
Remember this
Use Raft consensus to guarantee linearizable state machine consistency across distributed Go clusters.
Leader Election, Randomized Heartbeat Timeouts, & Quorum Majority
Followers maintain an Election Timeout (randomized between 150ms–300ms). If a Follower receives no heartbeat RPCs from the active Leader before the timeout expires, it transitions to Candidate mode:
1. Increments current Term number ($T$).
2. Votes for itself and broadcasts RequestVote RPCs to all cluster peers.
3. If it receives votes from a majority quorum ($N/2 + 1$), it ascends to Leader and immediately begins broadcasting periodic AppendEntries heartbeats.
Randomizing election timeouts prevents Split Votes, where multiple Candidates request votes simultaneously and fail to achieve a majority.
Quick reference
- Randomized election timeouts (150ms–300ms) prevent candidate split vote ties during elections.
- RequestVote RPCs verify candidate log completeness (up-to-date term and index) before granting votes.
- Requires majority quorum ($N/2 + 1$) votes to prevent dual-leader split-brain scenarios.
- Heartbeat AppendEntries RPCs maintain leader authority and suppress follower election timers.
- Monotonically increasing Term numbers resolve outdated leader authority instantly.
Remember this
Rely on randomized election timeouts and majority quorums to elect leaders cleanly without split votes.
Replicated Write-Ahead Logs & State Machine (FSM) Application
To implement custom storage engines in Go, developers implement the raft.FSM interface:
1type KeyValueFSM struct {2 mu sync.RWMutex3 data map[string]string4}5 6func (f *KeyValueFSM) Apply(log *raft.Log) interface{} {7 var cmd Command8 json.Unmarshal(log.Data, &cmd)9 10 f.mu.Lock()11 defer f.mu.RUnlock()12 if cmd.Op == "SET" {13 f.data[cmd.Key] = cmd.Value14 }15 return nil16}When a client submits a write (raft.Apply(payload, timeout)), the Leader writes the log to disk, replicates it to Followers via AppendEntries, and invokes FSM.Apply() only after a quorum confirms storage.
Quick reference
- raft.FSM interface requires implementing Apply(), Snapshot(), and Restore() methods in Go.
- Leader commits log entries to state machines only after receiving majority quorum ACKs.
- Uncommitted log entries are safely overwritten if a leader crashes prior to quorum commit.
- In-memory data structures (maps, trees) acquire mutex locks during FSM Apply() execution.
- Guarantees identical state machine execution order across all cluster node replicas.
Remember this
Implement Go's raft.FSM interface to bind custom storage logic to Raft replicated logs.
Cluster Membership Changes, Joint Consensus, & Log Compaction Snapshots
Over time, the Raft Write-Ahead Log grows indefinitely, consuming disk space and making node recovery slow. Raft solves this using Log Compaction & Snapshots:
1. FSM Snapshot: FSM.Snapshot() serializes the current in-memory state into a compact binary stream.
2. Log Truncation: Once the snapshot is safely persisted, Raft truncates older committed log entries up to index $K$.
3. InstallSnapshot RPC: When a newly added or recovering follower node lags far behind the leader, the leader sends an InstallSnapshot RPC to stream the full snapshot directly, bringing the node up-to-date in seconds.
Quick reference
- Log compaction truncates historical committed log entries to bound disk usage.
- FSM.Snapshot() streams in-memory data state to persistent storage asynchronously.
- InstallSnapshot RPC streams full state snapshots to catch up severely lagging followers.
- Joint Consensus configuration changes add or remove nodes safely without stopping writes.
- BoltDB / Badgerv4 log stores provide durable disk persistence for Raft WAL indexes.
Remember this
Use Raft FSM snapshots and InstallSnapshot RPCs to compact logs and onboard lagging follower nodes.
Key takeaway
To test HashiCorp Raft in Go, clone github.com/hashicorp/raft and run go test -v ./.... Instantiate a 3-node in-memory Raft cluster in Go unit tests.
Related Articles
Explore this topic