Reg Sci Disaster Recovery Outage Rca
Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover
When a matching-engine blade suffers an uncorrectable ECC memory fault mid-session, LMAX Disruptor + Aeron Raft replicated state machines elect a bit-exact follower in milliseconds with zero lost fills.

Traditional enterprise databases replicate state by locking rows, writing WAL disk pages, and shipping 2-Phase Commit transactions (5 to 25 milliseconds per trade)—1,000x too slow for a central limit order book matching 300,000 orders per second! Modern exchanges (Nasdaq, LMAX, MEMX, Euronext Optiq, institutional FX/crypto venues) achieve RPO = 0 (zero lost fills) at 8 to 25 microseconds replication latency by pairing Deterministic Single-Threaded State Machines (LMAX Disruptor Framework) with kernel-bypass Raft / Multi-Paxos Log Consensus (Aeron Cluster).
Why Replicating Input Commands Beats Replicating Output Book Mutations#
Connecting the mechanical-sympathy architecture of our SEC Reg SCI & Outage Post-Mortem Hub, suppose a single incoming Marketable Limit Sweep matches against 42 resting price levels, modifying 85 order structs and generating 85 ExecutionReport messages:
- State-Mutation Shipping (Slow): Serializing all 85 mutated order nodes + 42 price levels across the network consumes
~18 KBand stalls the matching loop. - Deterministic Replicated State Machine (
RSM— Fast): Ship ONLY the64-bytesequenced inbound command ({LogIndex, LeaderTerm, HardwareTimestampNs, OrderPayload}) across the 3-node Raft cluster before emitting outboundExecutionReports! Because all 3 nodes (1 Leader + 2 Hot Followers) execute the exact same single-threaded C++/Java matching code on the exact same sequenced byte stream with zero wall-clockgettimeofday,rand, or thread races, all 3 in-memory order books stay 100% bit-for-bit identical at all times!
Pipelined Kernel-Bypass Raft (Aeron Cluster / RDMA): 12-Microsecond Quorum Math#
How does a 3-node Raft cluster (Node_0 Leader, Node_1 Follower, Node_2 Follower) inside one colocation hall (HA Trading Failover Architecture) commit an order in ~12 microseconds?
- Pipelined Batch Streaming (No Stop-and-Wait):
Node_0never waits forLogIndex #1001ACK before sending#1002; it streams commands continuously over kernel-bypassUDP MulticastorSolarflare ef_vi / RDMAtoNode_1andNode_2simultaneously. - Majority Quorum (
Floor(N/2) + 1 = 2 of 3): As soon as whichever faster follower (MIN(RTT_Node1, RTT_Node2)) appendsLogIndex #1001to its pinned NVMe/Optane/DRAM ring and returns anAppendPositionACK (~6 uswire RTT!),CommitIndexadvances! - Tail-Jitter Masking: Notice the magic of a
3-nodeor5-nodequorum: ifNode_1experiences a transient400-microsecondSSD wear-leveling hiccup,Node_0 + Node_2(2/3majority!) keep committing uninterrupted at12 microseconds!
| Cluster Topology | Quorum Size (Q) | Max Simultaneous Node Failures | Split-Brain Safe? | Typical p99 Latency |
|---|---|---|---|---|
2-Node Active/Passive (Naive) | Needs external witness! | 0 if sync (2/2), or split-brain risk! | No (Classic split-brain trap!) | 12 us (Fragile) |
3-Node In-Hall Raft (Standard) | 2 of 3 nodes | 1 complete node crash | Yes (Mathematically disjoint) | 14 – 22 us |
5-Node In-Hall Raft (Critical) | 3 of 5 nodes | 2 simultaneous node crashes | Yes (Allows rolling maintenance + 1 crash!) | 18 – 28 us |
5-Node + Async Remote DR (SCI) | 3/5 Local + Async WAN | 2 Local OR 2-hr Regional DR | Yes (TermId + STONITH) | 18 – 28 us |
Consensus Cluster Sizing & Quorum Physics Comparison Matrix#
Comparing 2-node Master/Slave (dangerous!), 3-node Raft, 5-node Raft, and Cross-Region Geo-Paxos for matching engines.
- Tail-Latency Shielding of 3-Node Raft Over Single Synchronous Pair:
- Single Sync Follower Commit Time = Tnet + Tfollower1 (100% exposed to Follower1 p99.9 jitter!)
- 3-Node Raft Quorum Commit Time = Tnet + MIN( Tfollower1, Tfollower_2 )!
- If P(Single Follower Stall > 100us) = 0.001 (0.1%), then P(Both Stall Simultaneously) = (0.001)^2 = 0.000001 (1 in 1,000,000)!
Preventing Split-Brain Dual Leaders: Hardware Leases & Monotonic LeadershipTermId#
What prevents the deadliest distributed-systems bug—where Node_0 suffers a network partition, Node_1 + Node_2 elect Node_1 as new Leader (Term = 8), while partitioned Node_0 (Term = 7) accidentally releases a stale fill?
- Strict Output Gating After Quorum Commit:
Node_0never transmits an outboundExecutionReport (35=8)or ITCH trade print untilCommitIndex >= Command_LogIndexis acknowledged by a2/3majority! - Monotonic
LeadershipTermIdFencing on Gateways: Every outbound packet from the matching cluster carries(LeadershipTermId, LogSeqNum). The moment FIX/OUCH gateways and Multicast A/B feed handlers observeTerm = 8fromNode_1, they cryptographically/logically fence off and drop any late packet carryingTerm < 8!
Determinism Killers That Corrupt Follower Order Books Silently#
One non-deterministic line of code inside a matching engine causes Follower #2 order book to diverge silently from the Leader until failover day explodes! Enforce continuous Rolling State Checksum Verification (XOR / xxHash64 of every symbol book included in periodic Raft log heartbeats) and ban:
- Calling
clock_gettimeorstd::chrono::system_clock::nowinside matching logic (pass leader timestamp inside the sequenced command header!). - Iterating over unordered hash maps (
std::unordered_map/ JavaHashMapwith randomized hash seeds!). - Non-IEEE-754 extended
x87floating-point rounding (use 64-bit scaled fixed-point integers exclusively!).
Compare hot Raft failover against cold replay in deterministic event-log replay standby boot and model unhedged failover risk in our Terminal Margin VaR & ES Calculator.
Frequently asked questions
Why is a 5-node Raft cluster preferred over a 3-node cluster for Tier-1 national exchanges?
In a 3-node cluster (F = 1), taking 1 node offline for a BIOS/firmware patch leaves ZERO fault tolerance—if a second node crashes during maintenance, the market halts! A 5-node cluster (Quorum = 3, F = 2) can have 1 node undergoing maintenance and STILL survive an unexpected hardware crash with zero downtime!
Why does iterating over std::unordered_map or Java HashMap break Raft state-machine determinism?
Modern C++ and Java hash tables randomize per-process hash seeds (to prevent hash-flooding DoS) or depend on memory pointer addresses! If a Mass Cancel or Auction Uncross iterates over a hash map in different order on Node 1 vs Node 2, tie-breaking and outbound ExecID sequence numbers diverge permanently!
How fast do clients reconnect or switch when Raft elects Follower #1 as the new Leader in 35 milliseconds?
Clients NEVER reconnect to the matching engine directly! Clients stay connected via persistent TCP/FIX/OUCH sessions to stateless Front-End Order Gateways, which subscribe to the internal Aeron/Raft cluster bus and seamlessly route the next microsecond packet to the newly elected Term-8 Leader!
How does Raft avoid running out of disk/memory as the replicated command log grows to 500 million messages intraday?
Periodic off-thread Snapshot + Log Compaction: a background worker serializes a point-in-time binary snapshot of the deterministic order book at LogIndex = 100,000,000 (via copy-on-write fork or double-buffered ring), allowing older log segments to be archived to WORM storage.
Publication record
- Published
- 13 November 2026
- Updated
- 14 November 2026
- Author
- Marcus Thorne, CQF
- Review
- Review not recorded
- Automated fact check
- Passed in the stored pipeline record
- Source URLs
- 3 recorded
AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.
Academic & Institutional Attribution
Cite This Research & Embed Live Data Card
Standard academic & institutional research paper citation format.
Marcus Thorne, CQF. (2026, November 14). Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/raft-paxos-consensus-replicated-state-machines-in-matching-clustersRelated Reg Sci Disaster Recovery Outage Rca & Institutional Briefings
Explore Reg Sci Disaster Recovery Outage Rca Hub (20) →Reg Sci Disaster Recovery Outage Rca
SEC Regulation SCI, Active-Active Exchange Failover & Outage Post-Mortem Engineering GuideReg Sci Disaster Recovery Outage Rca
Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book ReconstructionReg Sci Disaster Recovery Outage Rca
Multicast A/B Market-Data Feed Arbitration: Air-Gapped Switch Fabrics & Microburst RecoveryReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem Anatomy: 32-Bit Sequence Wraparounds & Timestamp Epoch OverflowsReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: Opening-Auction Reference-Price Errors & LULD Halt CascadesReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: WAN BGP Route Leaks, Extranet Partitions & Asymmetric HaltsReg Sci Disaster Recovery Outage Rca
CTA & UTP SIP Market-Data Processor Failover: NBBO Staleness, Reg NMS Rule 611 & Direct FeedsReg Sci Disaster Recovery Outage Rca
SEC Regulation SCI Rule 1002: Incident Triage Trees, Immediate SEC Notification & De Minimis Safe HarborRegulatory Compliance
OKXICE moves ahead with tokenized stocks as blue chips push backBroker Fees and Calculators
NYSE Affiliate to Launch 24/7 Tokenized Stock Trading This MonthMarket Insights
Options traders bracing for more tech downside, says Cboe's Mandy Xu - CNBCExecutive Moves
Ripple Prime brokerage head to step down at year-end - Tech in Asia
Terminal Ledger Desk Data, Benchmarks & Contributor Records
CC BY 4.0 Open Data
Open Institutional Datasets (CSV / JSON) →
Download desk-compiled quantitative series & schemas.
2026 Reference Matrix
2026 Institutional Benchmarks & Thresholds →
Statutory capital, clearing, margin & rate conventions.
Editorial Records
Authors & Review Records →
View named contributors where those records are available.
Complete Index
Master Research & Dossier Archive →
Browse all published briefings by topic pillar & date.
