Here's a question that sounds simple: you have a key-value store running on one server. That server dies. How do your users keep reading and writing?
The obvious answer is "run three copies." But now you've made the problem worse. If every copy accepts writes independently, three machines that were supposed to be identical slowly drift into telling three different stories. Which one is true?
This is the fundamental problem of distributed systems, and the answer is a beautiful idea called State Machine Replication. I wanted to stop reading about it and actually build it — so I wrote machina: three Java processes that elect a leader among themselves, replicate a log of commands, and survive you killing any one of them mid-flight. No frameworks, no libraries doing the hard parts. Just the JDK, Gson, and raw HTTP.
The full source lives here.
Part 1: The Deceptively Simple Recipe
A deterministic state machine is any program with a guarantee: same commands in,
applied in the same order, produces the same state out. A key-value store is one — feed it
set(x=1) then set(y=x+1) and you always end up with {x:1, y:2},
every time, on every machine.
That determinism unlocks the trick:
1. Take N copies of the same deterministic state machine.
2. Feed every copy the same commands, in the same order.
3. The copies can never disagree.
You don't replicate state. You replicate the log of commands — and the state takes care of itself.
This reframes the entire problem. We don't need to keep three databases in sync — we need one shared, agreed-upon ordered list of operations: the replicated log. The log is the hero. Whoever controls the order of that log controls reality.
Same log everywhere → same state everywhere. The log is the source of truth.
So the real question becomes: who decides what goes into the log, and in what order?
Part 2: Why Everyone Can't Be in Charge
Your first instinct might be "let clients send writes to all three nodes." Now imagine two client
requests hit the cluster at the same moment. Node 1 applies set(x=1) first, Node 2 applies
set(x=2) first. Both followed the rules. Both are wrong.
Ordering requires a single decision-maker: a leader. All writes go through the leader, it stamps them into its log in one total order, and everyone else copies that order. Simple.
Except now the leader is a single point of failure — which was the whole thing we were trying to fix. So the system needs the ability to pick a new leader while it's running, without human intervention, without two leaders existing at once.
And that's genuinely hard, because networks lie. A leader that's alive but slow is indistinguishable from a dead one. If two nodes both decide the old leader is gone, you get a split brain: two leaders, two logs, two versions of the truth.
Part 3: Raft's Three Ideas That Make It Work
Raft (Diego Ongaro & John Ousterhout, 2014) was designed to be understandable, and it shows. Machina implements its heart with three mechanisms:
1. Terms
A monotonically increasing integer acts as a logical clock. Every message carries a term. Anything from an old term is stale and gets rejected. Time is divided into terms, each with at most one leader.
2. Randomized timeouts
Each node waits a random 1.5–3s before deciding the leader is dead. Randomness breaks symmetry: some node fires first, others concede before firing themselves. Split elections become rare accidents instead of guaranteed disasters.
3. Majority quorum
A candidate needs votes from a majority (2 of 3) to win. Two majorities always overlap in at least one node — so two competing leaders can never both collect a majority. Math, not luck, prevents split brains.
Every node cycles through three roles: FOLLOWER → CANDIDATE → LEADER. Followers do
nothing but listen. When silence stretches past their timeout, they promote themselves, increment the
term, and ask for votes. Win a majority? You're the leader, and you spend your reign sending
heartbeats to keep everyone else quiet.
Part 4: Inside Machina
The Shape of the Thing
Three processes on one machine (in production they'd be separate boxes — nothing in the design cares): each runs an HTTP server on ports 7001–7003 exposing four endpoints:
GET /status # JSON snapshot: role, term, leader, log size
POST /set # a write — only the leader accepts it
POST /requestvote # Raft vote solicitation during elections
POST /heartbeat # leader's "I'm alive" + log sync
The whole brain of the node is one synchronized tick() method, fired every 500ms:
synchronized void tick() {
// been too quiet for too long? stage a coup.
if ((role == Role.FOLLOWER || role == Role.CANDIDATE)
&& System.currentTimeMillis() - lastActivityMs > electionTimeoutMs) {
startElection();
return;
}
if (role == Role.LEADER) {
sendHeartbeats(); // remind everyone who's boss
}
}
// the randomness that prevents dueling elections
private static long freshTimeout() {
return ThreadLocalRandom.current().nextLong(1500, 3000);
}
The Election
When the timeout fires, the node declares candidacy: bump the term, vote for itself, fire off
VoteRequest(term, id) to both peers, and count.
if (req.term() < currentTerm) {
grant = false; // candidate is living in the past
} else {
if (req.term() > currentTerm) {
currentTerm = req.term(); // future term? update ours
votedFor = -1; // fresh term, fresh vote
}
grant = votedFor == -1 || votedFor == req.candidateId();
// ↑ one vote per term. this line IS the safety guarantee.
}
Notice how small the safety rule is: each node votes at most once per term. Since a majority of 2-of-3 must overlap with any other majority of 2-of-3, and a node that already voted in the current term refuses again, it is mathematically impossible for two candidates to gather a majority in the same term. Split brain: prevented by arithmetic.
One node's timer fires first → it collects 1 self-vote + 1 peer vote → majority of 3 → leader.
The Write Path
A client POSTs {"key":"lang","value":"java"} to /set. Non-leaders bounce it
back with the leader's ID (so the client can retry against the right node). The leader appends to its
log:
log.add(new LogEntry(currentTerm, log.size() + 1, key, value));
// entry stamped with (term, index) — its address in history
Then, on the next tick (≤500ms later), the leader's heartbeat carries the entire log to every follower, which does something almost brutally simple:
if (hb.term() >= currentTerm) {
currentTerm = hb.term();
role = Role.FOLLOWER; // any valid heartbeat tames a rogue candidate
leaderId = hb.leaderId();
lastActivityMs = System.currentTimeMillis();
log.clear();
log.addAll(hb.entries()); // my log := leader's log. no negotiation.
}
Full Raft would ship only the missing suffix and reconcile conflicts entry-by-entry. Machina ships everything and clobbers. For a 3-node demo it costs nothing and buys something valuable: the follower sync logic is five lines you can hold in your head. The log converges because there is literally nothing to converge toward except the leader's version.
Part 5: Don't Take My Word For It — Break One Live
Below is the same cluster, running for real in your browser. Same rules as the Java code above: nodes time out, campaign, vote, and replicate. Press kill leader and watch the survivors elect a replacement. Send a write, kill the leader, and check the new leader still has it.
Or run the real thing on your machine — one script boots all three JVMs:
$ ./run.sh
[machina] building...
[machina] node 1 on port 7001 pid=40211
[machina] node 2 on port 7002 pid=40212
[machina] node 3 on port 7003 pid=40213
$ sleep 3 && curl -s localhost:7002/status
{"nodeId":2,"port":7002,"role":"leader","currentTerm":1,"leaderId":2,"logSize":0}
Nobody told node 2 it was special. Its random timer simply fired first after startup, it collected a majority, and the cluster quietly organized itself. Now write through the leader, then assassinate it:
$ curl -s -X POST localhost:7002/set -d '{"key":"king","value":"node2"}'
{"logged":true,"index":1}
$ kill $(lsof -ti:7002) # goodbye, leader
$ sleep 4 && curl -s localhost:7001/status
{"nodeId":1,"port":7001,"role":"leader","currentTerm":2,"leaderId":1,"logSize":1}
# term incremented 1→2, leadership moved 2→1,
# and logSize=1 — the write survived the coup.
That moment — where the survivors inherit the dead leader's log and pick a new king, all in under four seconds with zero configuration — is the entire promise of consensus in one terminal session.
Part 6: What Machina Doesn't Do (Yet)
Being honest about tradeoffs is half of systems engineering. Machina is Raft-shaped, not Raft-complete:
Whole-log shipping
Every heartbeat carries the full log. Fine at 10 entries; quadratic traffic at a million. Real Raft sends the missing suffix and backfills.
No persistence
currentTerm, votedFor, and the log live in memory. A node reboot
forgets its vote — which in real Raft can cause double-voting in one term and break safety.
Votes ignore log freshness
Real Raft refuses to vote for candidates whose log is stale. Machina doesn't check — a laggy old node can win and clobber fresher entries.
Fire-and-forget writes
/set acks after appending locally, before any follower confirms. True Raft
commits only once a majority holds the entry — that's the durability contract.
None of these are bugs — they're scope. Each one is also the natural next milestone: persist the term/vote/log to disk, add commit-index tracking, then switch votes to require up-to-date logs. Each upgrade maps one-to-one onto a paragraph of the Raft paper, which makes it a great syllabus.
Why This Project Was Worth Building
I could recite what Raft does before I built this. I couldn't have told you why the vote-per-term rule implies election safety, or why the heartbeat doubles as a log-reconciliation primitive, or how quickly a "dead" leader detection actually feels at 1.5-second timeouts.
Distributed consensus is one of those topics where reading gives you vocabulary and building gives you understanding. Three hundred lines of Java bought me the second thing.
If you want to go deeper: run machina, kill leaders in different orders, unplug two nodes and watch elections stall (no majority!), then read the paper and find the exact sentences that describe what you just observed. It's the best way to read it.
• In Search of an Understandable Consensus Algorithm — the original Raft paper
• raft.github.io — visualizations and the full spec
• etcd/raft — the production-grade implementation powering Kubernetes