# Raft & Paxos Replicated State Machines in Matching Engines

> Low-latency consensus engineering for exchange matching clusters: Aeron Raft, 3-node vs 5-node quorums, leader leases, and split-brain fences.

Canonical: https://terminalledgerdesk.com/reports/raft-paxos-consensus-replicated-state-machines-in-matching-clusters
Last updated: 2026-11-14T19:50:30.771Z

**Publication:** Terminal Ledger Desk
**Category:** Reg Sci Disaster Recovery Outage Rca
**Published:** 2026-11-13T08:35:25.915Z
**Author:** Marcus Thorne, CQF (Certificate in Quantitative Finance (CQF), Low-Latency FIX & Matching Engine Architect)
**Reviewer:** Not recorded

## Answer-first summary

When a matching-engine blade suffers an uncorrectable ECC memory fault mid-session, LMAX Disruptor + Aeron Raft replicated state machines elect a bit-exact follower in milliseconds with zero lost fills.

## Key takeaways

- Never deploy a 2-node cluster without a third quorum witness—a 3-node or 5-node Raft cluster guarantees mathematical split-brain immunity and masks single-follower tail jitter!
- Replicate compact 64-byte sequenced input commands rather than bulky order-book mutations; deterministic single-threaded followers derive identical CLOB states automatically.
- Hold all outbound ExecutionReports (35=8) and market-data prints until 2/3 majority CommitIndex advances, stamping LeadershipTermId to fence stale partitioned leaders.
- Embed continuous rolling xxHash64 order-book checksums inside Raft heartbeats to catch non-deterministic hash-map or clock bugs immediately in UAT/production.

## Related financial entities

- **High-Availability Trading Failover Architecture** (HA-CLUSTER) — product: Clustered server configuration designed to guarantee uninterrupted execution processing even during sudden hardware failure, power loss, or network partitions.
- **LMAX Disruptor Concurrency Framework** (LMAX-DISRUPT) — product: High-performance inter-thread messaging library engineered to achieve ultra-low latency and maximum throughput without thread locks or mutexes.
- **Unified Single-Account Matching Engine** (UAME) — product: Electronic exchange architecture combining spot, perpetual futures, margin lending, and copy-trading sub-ledgers under a single real-time risk engine.

## Analysis

Traditional enterprise databases replicate state by locking rows, writing WAL disk pages, and shipping `2-Phase Commit` transactions (`5 to 25 milliseconds` per trade)—1,000x too slow for a central limit order book matching `300,000` orders per second! Modern exchanges (`Nasdaq`, `LMAX`, `MEMX`, `Euronext Optiq`, institutional FX/crypto venues) achieve **`RPO = 0` (zero lost fills)** at **`8 to 25 microseconds` replication latency** by pairing **Deterministic Single-Threaded State Machines** ([LMAX Disruptor Framework](/engines/lmax-disruptor-framework)) with kernel-bypass **Raft / Multi-Paxos Log Consensus (`Aeron Cluster`)**.

## Why Replicating Input Commands Beats Replicating Output Book Mutations

Connecting the mechanical-sympathy architecture of our [SEC Reg SCI & Outage Post-Mortem Hub](/reports/sec-regulation-sci-exchange-disaster-recovery-outage-postmortem-hub), suppose a single incoming `Marketable Limit Sweep` matches against **42 resting price levels**, modifying 85 order structs and generating 85 `ExecutionReport` messages:
- **State-Mutation Shipping (Slow)**: Serializing all 85 mutated order nodes + 42 price levels across the network consumes `~18 KB` and stalls the matching loop.
- **Deterministic Replicated State Machine (`RSM` — Fast)**: Ship ONLY the **`64-byte` sequenced inbound command** (`{LogIndex, LeaderTerm, HardwareTimestampNs, OrderPayload}`) across the 3-node Raft cluster **before** emitting outbound `ExecutionReports`! Because all 3 nodes (`1 Leader + 2 Hot Followers`) execute the **exact same single-threaded C++/Java matching code** on the **exact same sequenced byte stream** with zero wall-clock `gettimeofday`, `rand`, or thread races, all 3 in-memory order books stay **100% bit-for-bit identical** at all times!

![Three-node kernel-bypass Aeron Raft matching engine cluster showing Sequenced Command Multicast, Fastest-Follower 2/3 Quorum Commit in 12us, and Term-Fenced Gateway Egress](/api/figure?type=flow&title=3-Node%20Kernel-Bypass%20Raft%20Replicated%20Matching%20State%20Machine&subtitle=Leader%20Multicast%20Append%20-%3E%20Fastest%201-of-2%20Follower%20ACK%20%2812us%29%20-%3E%20Term-Fenced%2035%3D8%20Egress&steps=1.%20Leader%20Sequencer%3A%3AStamp%20Term%3D7%2C%20Index%3DN%2C%20Clock_ns%7C2.%20Kernel-Bypass%20Log%3A%3AMulticast%2064B%20cmd%20to%202%20Followers%7C3.%202%2F3%20Quorum%20Commit%3A%3AAdvance%20on%20MIN%28Follower_1%2C%20Follower_2%29%7C4.%20Term-Fenced%20Out%3A%3AExecute%20deterministic%20CLOB%20-%3E%20Emit%2035%3D8&metrics=QUORUM%20LATENCY%3A%3A~11%E2%80%9318%20us%3A%3ASame-hall%20Solarflare%20%2F%20RDMA%7CDATA%20LOSS%20%28RPO%29%3A%3A0%20Trades!%3A%3AStrict%20pre-egress%20quorum%20commit%7CLEADER%20ELECTION%3A%3A20%E2%80%9380%20ms%3A%3ATuned%20colocation%20heartbeat%20lease%7CBOOK%20DRIFT%20GUARD%3A%3AxxHash64%3A%3AContinuous%20per-10k-msg%20book%20hash)

## Pipelined Kernel-Bypass Raft (`Aeron Cluster` / `RDMA`): 12-Microsecond Quorum Math

How does a 3-node Raft cluster (`Node_0 Leader`, `Node_1 Follower`, `Node_2 Follower`) inside one colocation hall ([HA Trading Failover Architecture](/engines/ha-trading-failover-architecture)) commit an order in **`~12 microseconds`**?
1. **Pipelined Batch Streaming (No Stop-and-Wait)**: `Node_0` never waits for `LogIndex #1001` ACK before sending `#1002`; it streams commands continuously over kernel-bypass `UDP Multicast` or `Solarflare ef_vi / RDMA` to `Node_1` and `Node_2` simultaneously.
2. **Majority Quorum (`Floor(N/2) + 1 = 2 of 3`)**: As soon as **whichever faster follower** (`MIN(RTT_Node1, RTT_Node2)`) appends `LogIndex #1001` to its pinned NVMe/Optane/DRAM ring and returns an `AppendPosition` ACK (`~6 us` wire RTT!), `CommitIndex` advances!
3. **Tail-Jitter Masking**: Notice the magic of a `3-node` or `5-node` quorum: if `Node_1` experiences a transient `400-microsecond` SSD wear-leveling hiccup, `Node_0 + Node_2` (`2/3` majority!) keep committing uninterrupted at `12 microseconds`!

| Cluster Topology | Quorum Size (`Q`) | Max Simultaneous Node Failures | Split-Brain Safe? | Typical p99 Latency |
| :--- | :--- | :--- | :--- | :--- |
| **2-Node Active/Passive (`Naive`)** | Needs external witness! | `0` if sync (`2/2`), or split-brain risk! | **No (Classic split-brain trap!)** | `12 us` (Fragile) |
| **3-Node In-Hall Raft (`Standard`)** | **`2 of 3` nodes** | **`1` complete node crash** | **Yes** (Mathematically disjoint) | **`14 – 22 us`** |
| **5-Node In-Hall Raft (`Critical`)** | **`3 of 5` nodes** | **`2` simultaneous node crashes** | **Yes** (Allows rolling maintenance + 1 crash!) | **`18 – 28 us`** |
| **5-Node + Async Remote DR (`SCI`)** | `3/5` Local + Async WAN | `2` Local OR [2-hr Regional DR](/reports/sec-regulation-sci-rule-1001-two-hour-rto-geographic-backup-mandate) | **Yes** (`TermId` + STONITH) | **`18 – 28 us`** |

## Consensus Cluster Sizing & Quorum Physics Comparison Matrix

Comparing 2-node Master/Slave (dangerous!), 3-node Raft, 5-node Raft, and Cross-Region Geo-Paxos for matching engines.

```formula
Quorum_Majority(N) = FLOOR( N / 2 ) + 1 [Requires Odd N in {3, 5} so Failure_Tolerance F = (N - 1) / 2]

Tail-Latency Shielding of 3-Node Raft Over Single Synchronous Pair:
 Single Sync Follower Commit Time = T_net + T_follower_1 (100% exposed to Follower_1 p99.9 jitter!)
 3-Node Raft Quorum Commit Time = T_net + MIN( T_follower_1, T_follower_2 )!
 If P(Single Follower Stall > 100us) = 0.001 (0.1%), then P(Both Stall Simultaneously) = (0.001)^2 = 0.000001 (1 in 1,000,000)!
```

## Preventing Split-Brain Dual Leaders: Hardware Leases & Monotonic `LeadershipTermId`

What prevents the deadliest distributed-systems bug—where `Node_0` suffers a network partition, `Node_1 + Node_2` elect `Node_1` as new Leader (`Term = 8`), while partitioned `Node_0` (`Term = 7`) accidentally releases a stale fill?
- **Strict Output Gating After Quorum Commit**: `Node_0` **never** transmits an outbound `ExecutionReport (35=8)` or ITCH trade print until `CommitIndex >= Command_LogIndex` is acknowledged by a `2/3` majority!
- **Monotonic `LeadershipTermId` Fencing on Gateways**: Every outbound packet from the matching cluster carries `(LeadershipTermId, LogSeqNum)`. The moment FIX/OUCH gateways and [Multicast A/B feed handlers](/reports/multicast-a-b-market-data-feed-arbitration-switch-fault-isolation) observe `Term = 8` from `Node_1`, they **cryptographically/logically fence off and drop any late packet carrying `Term < 8`**!

![p99 commit latency and fault tolerance across 1-Node Unreplicated (unsafe), 3-Node In-Hall Kernel-Bypass Raft, 5-Node Metro Campus Raft, and Traditional Disk SQL 2-Phase Commit](/api/figure?type=bars&title=p99%20Order%20Commit%20Latency%20vs%20Fault%20Tolerance%20by%20Architecture&subtitle=Why%203-node%20%2F%205-node%20in-hall%20kernel-bypass%20Raft%20dominates%20legacy%20SQL%20replication&steps=3-Node%20In-Hall%20Bypass%20Raft%20%282%2F3%29%3A%3A16%20us%20p99%20%28Survives%201%20dead%20blade%2C%20RPO%3D0%29%7C5-Node%20In-Hall%20Bypass%20Raft%20%283%2F5%29%3A%3A21%20us%20p99%20%28Survives%202%20dead%20blades%2C%20RPO%3D0%29%7C3-Node%20Cross-Campus%20NJ%20Metro%20%2835km%29%3A%3A380%20us%20p99%20%28Survives%20full%20building%20loss!%29%7CLegacy%20Relational%20DB%202-Phase%20Commit%3A%3A14%2C500%20us%20p99%20%28900x%20slower%20%2B%20lock%20stalls%29&metrics=2-NODE%20CLUSTER%3F%3A%3AFORBIDDEN!%3A%3A2%2F2%20quorum%20%3D%20worse%20availability!%7CWALL-CLOCK%20CALLS%3A%3A0%20in%20Engine%3A%3ATimestamp%20injected%20in%20log%20header%7CHASHMAP%20ITERATE%3A%3ADeterministic%3A%3AIntrusive%20arrays%2Ftrees%20only%7CSNAPSHOT%20CADENCE%3A%3AOff-Thread%3A%3AFork%20%2F%20async%20ring%20compaction)

## Determinism Killers That Corrupt Follower Order Books Silently

One non-deterministic line of code inside a matching engine causes Follower #2 order book to diverge silently from the Leader until failover day explodes! Enforce continuous **Rolling State Checksum Verification** (`XOR / xxHash64` of every symbol book included in periodic Raft log heartbeats) and ban:
- Calling `clock_gettime` or `std::chrono::system_clock::now` inside matching logic (**pass leader timestamp inside the sequenced command header**!).
- Iterating over unordered hash maps (`std::unordered_map` / Java `HashMap` with randomized hash seeds!).
- Non-IEEE-754 extended `x87` floating-point rounding (**use 64-bit scaled fixed-point integers exclusively**!).

Compare hot Raft failover against cold replay in [deterministic event-log replay standby boot](/reports/deterministic-event-log-replay-cold-vs-warm-standby-orderbook-boot) and model unhedged failover risk in our [Terminal Margin VaR & ES Calculator](/calculators/terminal-margin-var-es-calculator).

## Frequently asked questions

### Why is a 5-node Raft cluster preferred over a 3-node cluster for Tier-1 national exchanges?

In a 3-node cluster (F = 1), taking 1 node offline for a BIOS/firmware patch leaves ZERO fault tolerance—if a second node crashes during maintenance, the market halts! A 5-node cluster (Quorum = 3, F = 2) can have 1 node undergoing maintenance and STILL survive an unexpected hardware crash with zero downtime!

### Why does iterating over std::unordered_map or Java HashMap break Raft state-machine determinism?

Modern C++ and Java hash tables randomize per-process hash seeds (to prevent hash-flooding DoS) or depend on memory pointer addresses! If a Mass Cancel or Auction Uncross iterates over a hash map in different order on Node 1 vs Node 2, tie-breaking and outbound ExecID sequence numbers diverge permanently!

### How fast do clients reconnect or switch when Raft elects Follower #1 as the new Leader in 35 milliseconds?

Clients NEVER reconnect to the matching engine directly! Clients stay connected via persistent TCP/FIX/OUCH sessions to stateless Front-End Order Gateways, which subscribe to the internal Aeron/Raft cluster bus and seamlessly route the next microsecond packet to the newly elected Term-8 Leader!

### How does Raft avoid running out of disk/memory as the replicated command log grows to 500 million messages intraday?

Periodic off-thread Snapshot + Log Compaction: a background worker serializes a point-in-time binary snapshot of the deterministic order book at LogIndex = 100,000,000 (via copy-on-write fork or double-buffered ring), allowing older log segments to be archived to WORM storage.


## Source URLs

- https://raft.github.io/raft.pdf
- https://github.com/real-logic/aeron
- https://www.sec.gov/files/rules/final/2014/34-73639.pdf

## Citation

- APA: Marcus Thorne, CQF. (2026). Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/raft-paxos-consensus-replicated-state-machines-in-matching-clusters
- Canonical article: [Raft & Paxos Replicated State Machines in Matching Engines: Zero-Loss Sub-Millisecond Failover](https://terminalledgerdesk.com/reports/raft-paxos-consensus-replicated-state-machines-in-matching-clusters)
- Site research index: [Terminal Ledger Desk](https://terminalledgerdesk.com/llms-full.txt)
