Skip to content
Terminal Ledger Desk
Directory

Reg Sci Disaster Recovery Outage Rca

Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover

When a matching-engine blade suffers an uncorrectable ECC memory fault mid-session, LMAX Disruptor + Aeron Raft replicated state machines elect a bit-exact follower in milliseconds with zero lost fills.

By Marcus Thorne, CQFReview not recorded · Updated
Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover — Terminal Ledger Desk 2026 quantitative & supervisory data card
Terminal Ledger Desk (SEC Regulation SCI, Active-Active Exchange Failover & Outage Post-Mortems): Kernel-bypass 3/5-node Raft quorums, 16us p99 RPO=0 commits, and xxHash64 book guards.Source: Terminal Ledger Desk Institutional Research DeskLicense: CC-BY-4.0

Traditional enterprise databases replicate state by locking rows, writing WAL disk pages, and shipping 2-Phase Commit transactions (5 to 25 milliseconds per trade)—1,000x too slow for a central limit order book matching 300,000 orders per second! Modern exchanges (Nasdaq, LMAX, MEMX, Euronext Optiq, institutional FX/crypto venues) achieve RPO = 0 (zero lost fills) at 8 to 25 microseconds replication latency by pairing Deterministic Single-Threaded State Machines (LMAX Disruptor Framework) with kernel-bypass Raft / Multi-Paxos Log Consensus (Aeron Cluster).

Why Replicating Input Commands Beats Replicating Output Book Mutations#

Connecting the mechanical-sympathy architecture of our SEC Reg SCI & Outage Post-Mortem Hub, suppose a single incoming Marketable Limit Sweep matches against 42 resting price levels, modifying 85 order structs and generating 85 ExecutionReport messages:

  • State-Mutation Shipping (Slow): Serializing all 85 mutated order nodes + 42 price levels across the network consumes ~18 KB and stalls the matching loop.
  • Deterministic Replicated State Machine (RSM — Fast): Ship ONLY the 64-byte sequenced inbound command ({LogIndex, LeaderTerm, HardwareTimestampNs, OrderPayload}) across the 3-node Raft cluster before emitting outbound ExecutionReports! Because all 3 nodes (1 Leader + 2 Hot Followers) execute the exact same single-threaded C++/Java matching code on the exact same sequenced byte stream with zero wall-clock gettimeofday, rand, or thread races, all 3 in-memory order books stay 100% bit-for-bit identical at all times!
Three-node kernel-bypass Aeron Raft matching engine cluster showing Sequenced Command Multicast, Fastest-Follower 2/3 Quorum Commit in 12us, and Term-Fenced Gateway Egress
Three-node kernel-bypass Aeron Raft matching engine cluster showing Sequenced Command Multicast, Fastest-Follower 2/3 Quorum Commit in 12us, and Term-Fenced Gateway EgressVerified Exhibit

Pipelined Kernel-Bypass Raft (Aeron Cluster / RDMA): 12-Microsecond Quorum Math#

How does a 3-node Raft cluster (Node_0 Leader, Node_1 Follower, Node_2 Follower) inside one colocation hall (HA Trading Failover Architecture) commit an order in ~12 microseconds?

  1. Pipelined Batch Streaming (No Stop-and-Wait): Node_0 never waits for LogIndex #1001 ACK before sending #1002; it streams commands continuously over kernel-bypass UDP Multicast or Solarflare ef_vi / RDMA to Node_1 and Node_2 simultaneously.
  2. Majority Quorum (Floor(N/2) + 1 = 2 of 3): As soon as whichever faster follower (MIN(RTT_Node1, RTT_Node2)) appends LogIndex #1001 to its pinned NVMe/Optane/DRAM ring and returns an AppendPosition ACK (~6 us wire RTT!), CommitIndex advances!
  3. Tail-Jitter Masking: Notice the magic of a 3-node or 5-node quorum: if Node_1 experiences a transient 400-microsecond SSD wear-leveling hiccup, Node_0 + Node_2 (2/3 majority!) keep committing uninterrupted at 12 microseconds!
Quantitative Distribution · Quorum Size (Q)
Quorum Size (Q)Max Simultaneous Node Failures
013-Node In-Hall Raft (Standa…2 of 3 nodes025-Node In-Hall Raft (Critic…3 of 5 nodes035-Node + Async Remote DR (S…3/5 Local + Asy…
Cluster TopologyQuorum Size (Q)Max Simultaneous Node FailuresSplit-Brain Safe?Typical p99 Latency
2-Node Active/Passive (Naive)Needs external witness!0 if sync (2/2), or split-brain risk!No (Classic split-brain trap!)12 us (Fragile)
3-Node In-Hall Raft (Standard)2 of 3 nodes1 complete node crashYes (Mathematically disjoint)14 – 22 us
5-Node In-Hall Raft (Critical)3 of 5 nodes2 simultaneous node crashesYes (Allows rolling maintenance + 1 crash!)18 – 28 us
5-Node + Async Remote DR (SCI)3/5 Local + Async WAN2 Local OR 2-hr Regional DRYes (TermId + STONITH)18 – 28 us

Consensus Cluster Sizing & Quorum Physics Comparison Matrix#

Comparing 2-node Master/Slave (dangerous!), 3-node Raft, 5-node Raft, and Cross-Region Geo-Paxos for matching engines.

Formula or model from the articleNot independently verified
Quorum_Majority(N) = FLOOR( N / 2 ) + 1 [Requires Odd N in {3, 5} so Failure_Tolerance F = (N - 1) / 2]
  • Tail-Latency Shielding of 3-Node Raft Over Single Synchronous Pair:
  • Single Sync Follower Commit Time = Tnet + Tfollower1 (100% exposed to Follower1 p99.9 jitter!)
  • 3-Node Raft Quorum Commit Time = Tnet + MIN( Tfollower1, Tfollower_2 )!
  • If P(Single Follower Stall > 100us) = 0.001 (0.1%), then P(Both Stall Simultaneously) = (0.001)^2 = 0.000001 (1 in 1,000,000)!

Preventing Split-Brain Dual Leaders: Hardware Leases & Monotonic LeadershipTermId#

What prevents the deadliest distributed-systems bug—where Node_0 suffers a network partition, Node_1 + Node_2 elect Node_1 as new Leader (Term = 8), while partitioned Node_0 (Term = 7) accidentally releases a stale fill?

  • Strict Output Gating After Quorum Commit: Node_0 never transmits an outbound ExecutionReport (35=8) or ITCH trade print until CommitIndex >= Command_LogIndex is acknowledged by a 2/3 majority!
  • Monotonic LeadershipTermId Fencing on Gateways: Every outbound packet from the matching cluster carries (LeadershipTermId, LogSeqNum). The moment FIX/OUCH gateways and Multicast A/B feed handlers observe Term = 8 from Node_1, they cryptographically/logically fence off and drop any late packet carrying Term < 8!
p99 commit latency and fault tolerance across 1-Node Unreplicated (unsafe), 3-Node In-Hall Kernel-Bypass Raft, 5-Node Metro Campus Raft, and Traditional Disk SQL 2-Phase Commit
p99 commit latency and fault tolerance across 1-Node Unreplicated (unsafe), 3-Node In-Hall Kernel-Bypass Raft, 5-Node Metro Campus Raft, and Traditional Disk SQL 2-Phase CommitVerified Exhibit

Determinism Killers That Corrupt Follower Order Books Silently#

One non-deterministic line of code inside a matching engine causes Follower #2 order book to diverge silently from the Leader until failover day explodes! Enforce continuous Rolling State Checksum Verification (XOR / xxHash64 of every symbol book included in periodic Raft log heartbeats) and ban:

  • Calling clock_gettime or std::chrono::system_clock::now inside matching logic (pass leader timestamp inside the sequenced command header!).
  • Iterating over unordered hash maps (std::unordered_map / Java HashMap with randomized hash seeds!).
  • Non-IEEE-754 extended x87 floating-point rounding (use 64-bit scaled fixed-point integers exclusively!).

Compare hot Raft failover against cold replay in deterministic event-log replay standby boot and model unhedged failover risk in our Terminal Margin VaR & ES Calculator.

Frequently asked questions

Why is a 5-node Raft cluster preferred over a 3-node cluster for Tier-1 national exchanges?

In a 3-node cluster (F = 1), taking 1 node offline for a BIOS/firmware patch leaves ZERO fault tolerance—if a second node crashes during maintenance, the market halts! A 5-node cluster (Quorum = 3, F = 2) can have 1 node undergoing maintenance and STILL survive an unexpected hardware crash with zero downtime!

Why does iterating over std::unordered_map or Java HashMap break Raft state-machine determinism?

Modern C++ and Java hash tables randomize per-process hash seeds (to prevent hash-flooding DoS) or depend on memory pointer addresses! If a Mass Cancel or Auction Uncross iterates over a hash map in different order on Node 1 vs Node 2, tie-breaking and outbound ExecID sequence numbers diverge permanently!

How fast do clients reconnect or switch when Raft elects Follower #1 as the new Leader in 35 milliseconds?

Clients NEVER reconnect to the matching engine directly! Clients stay connected via persistent TCP/FIX/OUCH sessions to stateless Front-End Order Gateways, which subscribe to the internal Aeron/Raft cluster bus and seamlessly route the next microsecond packet to the newly elected Term-8 Leader!

How does Raft avoid running out of disk/memory as the replicated command log grows to 500 million messages intraday?

Periodic off-thread Snapshot + Log Compaction: a background worker serializes a point-in-time binary snapshot of the deterministic order book at LogIndex = 100,000,000 (via copy-on-write fork or double-buffered ring), allowing older log segments to be archived to WORM storage.

Publication record

Published
13 November 2026
Updated
14 November 2026
Author
Marcus Thorne, CQF
Review
Review not recorded
Automated fact check
Passed in the stored pipeline record
Source URLs
3 recorded

AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.

Academic & Institutional Attribution

Cite This Research & Embed Live Data Card

Open Attribution License (CC BY 4.0)

Standard academic & institutional research paper citation format.

Marcus Thorne, CQF. (2026, November 14). Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/raft-paxos-consensus-replicated-state-machines-in-matching-clusters
Explore Reg Sci Disaster Recovery Outage Rca Hub (20) →

Terminal Ledger Desk Data, Benchmarks & Contributor Records

Raft & Paxos Replicated State Machines in Matching Engines | Terminal Ledger Desk