FiveFamilies

Numbers from the laptop (i7-1270P, Docker Desktop on WSL2, a Samsung PM9A1 NVMe), five nodes in containers, each result committed with the code that produced it. The design notes behind them, including what didn't work, are in the repository.

How long a succession takes

The Don is killed with SIGKILL under 200 writes a second, ten times per election timeout. The gap is the widest silence between two acknowledged writes, as a client sees it.

election timeoutclients waited, medianrangenew Don chosen after
150ms204 ms181 ms – 249 ms174 ms
300ms388 ms340 ms – 436 ms351 ms
600ms720 ms645 ms – 853 ms694 ms
1s1,306 ms1,041 ms – 2,005 ms1,270 ms

How long a dead service stays listed

An instance registered with a 2 s lease is killed, and a client watching the registry times how long it still sees it. A new Don gives every lease its full time again, so losing the Don as well adds an election.

what diedmedianminmaxkills
the instance1,785 ms1,476 ms2,067 ms10
the instance and the Don3,445 ms3,114 ms3,747 ms10

Real faults, checked

Kills, pauses, partitions, slow links, membership changes and leadership handovers on a real five-node cluster, with every client operation recorded and the whole history checked for linearizability afterwards.

runfaultsoperationslinearizablestates equal
20261015-seed2-3m27 in 180 s41,208yesyes
20261016-seed3-3m-members27 in 180 s48,902yesyes
20261016-seed4-3m-membership-heavy32 in 180 s49,766yesyes

Bugs planted on purpose

The same checks against builds with a known bug switched on, to show what the net catches on a real cluster and what it doesn't. The simulator catches every one of them; these runs were 90 seconds each.

bugoperationslinearizablestates equalcaught
anyvote2,002Illegalnoyes
figure820,026yesyesnot in 90 s
prevterm23,129yesyesnot in 90 s
staleterm21,504yesyesnot in 90 s