Skip to content
Terminal Ledger Desk
Directory

Reg Sci Disaster Recovery Outage Rca

Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book Reconstruction

Replaying 1.2 billion morning ITCH/order events from sequence 1 after a 14:30 ET crash takes 12+ minutes even at 1.5M msgs/sec—proving why periodic CoW snapshots plus short delta-log tail replay are essential.

By Marcus Thorne, CQFReview not recorded · Updated
Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book Reconstruction — Terminal Ledger Desk 2026 quantitative & supervisory data card
Terminal Ledger Desk (SEC Regulation SCI, Active-Active Exchange Failover & Outage Post-Mortems): Pointer-free flat-slab CoW snapshots, <7s afternoon boot, and poison-pill tombstoning.Source: Terminal Ledger Desk Institutional Research DeskLicense: CC-BY-4.0

Event sourcing is the architectural bedrock of every modern electronic exchange: every inbound order, cancel, auction uncross, and administrative halt is persisted as an immutable, sequentially numbered binary record (Seq = 1 .. N). Yet how a standby node consumes that event log determines whether a crashed partition recovers in 35 milliseconds (Hot), 1.8 seconds (Snapshot + Delta Warm), or 18 agonizing minutes (Cold Seq-1 Replay) during an afternoon volatility spike!

The Afternoon "Replay Cliff" of Naive Cold Sequence-1 Recovery#

As framed in our SEC Reg SCI & Outage Post-Mortem Hub, suppose a matching engine partition handling SPY, QQQ, and NVDA processes an average of 150,000 events/sec (2.7 billion events by 15:00 ET). Even if a stripped single-threaded C++ replay loop executes at a blistering 2,500,000 events/second (400 ns per event from NVMe RAID-0):

  • At 09:35 ET (45M events): Cold replay from Seq = 1 finishes in 18 seconds—seemingly acceptable in naive morning tests!
  • At 15:30 ET on a Fed day (3.2 Billion events): 3,200,000,000 / 2,500,000 = 1,280 seconds (21.3 minutes!)—plus another 2+ minutes draining the backlog accumulated while replaying! Many notorious exchange outages lasted hours precisely because restarting crashed nodes from Seq = 1 kept timing out health checks in a crash-restart loop!
Zero-pause POSIX fork Copy-on-Write slab checkpointing every 60 seconds paired with fast mmap load + short tail-log replay and Null-Sink output suppression
Zero-pause POSIX fork Copy-on-Write slab checkpointing every 60 seconds paired with fast mmap load + short tail-log replay and Null-Sink output suppressionVerified Exhibit

Copy-on-Write (fork) Binary Snapshots + Short Tail-Delta Replay#

Under our HA Trading Failover Architecture and LMAX Disruptor Framework, production engines bound worst-case disk boot time below 2.0 seconds at any hour of the day by combining Zero-Pause Copy-on-Write (CoW) Checkpoints with Tail Delta Replay:

  1. Pre-Allocated Contiguous Slab Arena: All Order and PriceLevel structs live inside a flat, pointer-free (index-addressed uint32_t slot_id) 16 GB memory slab—zero scattered C++ heap pointers requiring slow pointer-swizzling on load!
  2. Every 5 Minutes (fork CoW Snapshot): The engine calls POSIX fork (~120 us page-table clone) or switches a double-buffered ring; the child process writes the flat 16 GB slab straight to NVMe (6.5 GB/s sequential write = 2.4s in background) stamped Snapshot_Seq = 2,850,000,000, with zero stop-the-world pause on the live matching parent!
  3. Fast Recovery (mmap + 5-Min Tail): A cold-booting replacement node mmaps Snapshot_2.85B (2.2s read) and replays only the last < 5 minutes (~45M events = 18 seconds max, or 1.5s for 1-minute snapshots)!
Quantitative Distribution · Live Hot-Path Cost
Live Hot-Path Cost15:30 ET Recovery Time
01Hot Raft Follower (3/5-Node)~14 us quorum02Warm CoW Slab Snapshot + Ta…~0 ns (Async Co…03Cold Binary Log (Seq = 1)~0 ns04Out-of-Region DR (Chicago)~0 ns (Async WA…
Standby ArchitectureLive Hot-Path Cost15:30 ET Recovery TimeSurvives Poison-Pill Crash Bug?Hardware Cost
Hot Raft Follower (3/5-Node)~14 us quorum20 – 80 msNo (All replicas execute poison msg!)3x – 5x primary blades
Warm CoW Slab Snapshot + Tail~0 ns (Async CoW)2 – 12 secondsYes! (Admin tombstones bad Seq + replays 60s)1x standby + NVMe
Cold Binary Log (Seq = 1)~0 ns15 – 30 minutes late dayYes, but breaks Kubernetes/watchdog timeoutsStorage only
Out-of-Region DR (Chicago)~0 ns (Async WAN)<= 2 Hours (Reg SCI)Yes (Fresh book + Call Auction reopen)Full secondary campus

Standby Order-Book Recovery Architecture Comparison Matrix#

Comparing memory footprint, live-path jitter, afternoon 15:30 ET recovery SLA, and bug blast-radius immunity across four standby models.

Formula or model from the articleNot independently verified
T_recovery_total(t_crash) = ( Size_Slab_GB / B_nvme_read_GBps ) + ( R_msg_per_sec_live * (t_crash - t_last_snapshot) ) / R_replay_engine
  • Worked Afternoon Crash Recovery Comparison at t_crash = 15:30 ET (3.2B Cumulative Msgs, Live Rate = 200k/s, Replay Speed = 2.5M/s):
  • Without Snapshots (t_last = 09:30): 0s + (3,200,000,000 / 2,500,000) = 1,280.0 seconds (21 min 20 sec!)
  • With 60s Flat-Slab Snapshot (12 GB @ 6.5 GB/s): (12 / 6.5) + (200,000 * 60 / 2,500,000) = 1.85s + 4.80s = 6.65 seconds!

Why You Still Need Cold/Warm Replay Even With a 3-Node Hot Raft Cluster!#

Wait: if a 3-node or 5-node Hot Raft Cluster already fails over in 35 milliseconds, why build fast Snapshot + Event-Log Replay at all? Three life-saving reasons:

  1. Poison-Pill ("Deterministic Crash") Bug Recovery: Suppose a malformed exotic order (Seq #900,441,209) triggers a null-pointer dereference in the matching code. Because Raft is deterministic, that exact poison packet crashes ALL 3 hot Raft nodes simultaneously in 20 microseconds! Recovering requires patching/skipping #900,441,209 in the admin tool and booting clean from the Seq #900,000,000 snapshot in seconds instead of 25 minutes!
  2. Spinning Up a Replacement Raft Voter: When Node_2 hardware dies at 14:00 ET, a fresh spare blade must catch up 2.5B events quickly to restore 3/3 cluster health.
  3. Exact Forensic Incident Reproduction: Replaying the production snapshot + tail log inside staging (chaos engineering & staging fault injection) reproduces nanosecond bugs identically.
Comparison of 15:30 ET recovery duration (3.2 Billion intraday events) across Naive Seq-1 JSON/SQL Replay, Binary Seq-1 Replay, 5-Min CoW Snapshot + Tail, and Hot Raft Follower
Comparison of 15:30 ET recovery duration (3.2 Billion intraday events) across Naive Seq-1 JSON/SQL Replay, Binary Seq-1 Replay, 5-Min CoW Snapshot + Tail, and Hot Raft FollowerVerified Exhibit

Suppressing Duplicate Side Effects (Output Fence) During Replay Mode#

Critical engineering invariant: while a recovering node replays historical events Seq_Snapshot + 1 .. Seq_Last_Committed, its Outbound Gateway Emitter MUST be wired to a Null Sink (Replay_Silent_Mode = True) until Current_Seq == Seq_Last_Committed—otherwise replaying 45 million events blasts 45 million duplicate ExecutionReport (35=8) packets onto customer FIX API sockets! Size operational downtime exposure in our Terminal Margin VaR & ES Calculator.

Frequently asked questions

What is a "Poison-Pill" packet in a replicated matching engine, and how has it brought down major exchanges?

A Poison Pill is a syntactically valid inbound frame hitting an unhandled edge case (e.g., divide-by-zero on a 0-tick spread or negative price in an energy contract) that triggers SIGSEGV/panic. Because every Raft replica executes the exact same code on that sequenced log entry, all replicas crash together unless operators can tombstone that single SeqNum and fast-boot from a recent snapshot!

Does calling Linux fork on a 32 GB matching-engine process freeze trading while copying 32 GB of RAM?

No—Linux fork uses hardware MMU Copy-on-Write (CoW), copying ONLY page-table descriptors initially (~100 microseconds, or even faster with hugepages/double-buffered arenas); physical 4KB/2MB pages are duplicated incrementally only when the live parent mutates active price levels.

How do you guarantee a power crash mid-write does not leave a half-written corrupt Snapshot file?

Write every new checkpoint to a temporary filename (snapshot.2850000000.tmp), append a trailing SHA-256/CRC32C footer, call fsync on both file and directory, and atomically rename to snapshot.latest—retaining at least the last 3 verified snapshots on disk.

Should clientMoldUDP64 / ITCH market-data book builders also use Snapshot + Delta recovery?

Yes! Both Nasdaq MoldUDP64 (SoupBinTCP Glimpse) and CME MDP 3.0 Market Recovery Feeds (Channel 310+ Snapshot feeds) broadcast periodic full book snapshots stamped with LastMsgSeqNumProcessed so late-joining or crashed client feed handlers rebuild books in <2 seconds instead of rewinding to midnight.

Publication record

Published
13 November 2026
Updated
14 November 2026
Author
Marcus Thorne, CQF
Review
Review not recorded
Automated fact check
Passed in the stored pipeline record
Source URLs
3 recorded

AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.

Academic & Institutional Attribution

Cite This Research & Embed Live Data Card

Open Attribution License (CC BY 4.0)

Standard academic & institutional research paper citation format.

Marcus Thorne, CQF. (2026, November 14). Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book Reconstruction. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/deterministic-event-log-replay-cold-vs-warm-standby-orderbook-boot
Explore Reg Sci Disaster Recovery Outage Rca Hub (20) →

Terminal Ledger Desk Data, Benchmarks & Contributor Records

Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby | Terminal Ledger Desk