Reg Sci Disaster Recovery Outage Rca
Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book Reconstruction
Replaying 1.2 billion morning ITCH/order events from sequence 1 after a 14:30 ET crash takes 12+ minutes even at 1.5M msgs/sec—proving why periodic CoW snapshots plus short delta-log tail replay are essential.

Event sourcing is the architectural bedrock of every modern electronic exchange: every inbound order, cancel, auction uncross, and administrative halt is persisted as an immutable, sequentially numbered binary record (Seq = 1 .. N). Yet how a standby node consumes that event log determines whether a crashed partition recovers in 35 milliseconds (Hot), 1.8 seconds (Snapshot + Delta Warm), or 18 agonizing minutes (Cold Seq-1 Replay) during an afternoon volatility spike!
The Afternoon "Replay Cliff" of Naive Cold Sequence-1 Recovery#
As framed in our SEC Reg SCI & Outage Post-Mortem Hub, suppose a matching engine partition handling SPY, QQQ, and NVDA processes an average of 150,000 events/sec (2.7 billion events by 15:00 ET). Even if a stripped single-threaded C++ replay loop executes at a blistering 2,500,000 events/second (400 ns per event from NVMe RAID-0):
- At
09:35 ET(45Mevents): Cold replay fromSeq = 1finishes in18 seconds—seemingly acceptable in naive morning tests! - At
15:30 ETon a Fed day (3.2 Billionevents):3,200,000,000 / 2,500,000 = 1,280 seconds(21.3 minutes!)—plus another2+ minutesdraining the backlog accumulated while replaying! Many notorious exchange outages lasted hours precisely because restarting crashed nodes fromSeq = 1kept timing out health checks in a crash-restart loop!
Copy-on-Write (fork) Binary Snapshots + Short Tail-Delta Replay#
Under our HA Trading Failover Architecture and LMAX Disruptor Framework, production engines bound worst-case disk boot time below 2.0 seconds at any hour of the day by combining Zero-Pause Copy-on-Write (CoW) Checkpoints with Tail Delta Replay:
- Pre-Allocated Contiguous Slab Arena: All
OrderandPriceLevelstructs live inside a flat, pointer-free (index-addresseduint32_t slot_id)16 GBmemory slab—zero scattered C++ heap pointers requiring slow pointer-swizzling on load! - Every 5 Minutes (
forkCoW Snapshot): The engine calls POSIXfork(~120 uspage-table clone) or switches a double-buffered ring; the child process writes the flat16 GBslab straight to NVMe (6.5 GB/ssequential write =2.4sin background) stampedSnapshot_Seq = 2,850,000,000, with zero stop-the-world pause on the live matching parent! - Fast Recovery (
mmap+ 5-Min Tail): A cold-booting replacement nodemmapsSnapshot_2.85B(2.2sread) and replays only the last< 5 minutes(~45Mevents =18 secondsmax, or1.5sfor 1-minute snapshots)!
| Standby Architecture | Live Hot-Path Cost | 15:30 ET Recovery Time | Survives Poison-Pill Crash Bug? | Hardware Cost |
|---|---|---|---|---|
Hot Raft Follower (3/5-Node) | ~14 us quorum | 20 – 80 ms | No (All replicas execute poison msg!) | 3x – 5x primary blades |
| Warm CoW Slab Snapshot + Tail | ~0 ns (Async CoW) | 2 – 12 seconds | Yes! (Admin tombstones bad Seq + replays 60s) | 1x standby + NVMe |
Cold Binary Log (Seq = 1) | ~0 ns | 15 – 30 minutes late day | Yes, but breaks Kubernetes/watchdog timeouts | Storage only |
Out-of-Region DR (Chicago) | ~0 ns (Async WAN) | <= 2 Hours (Reg SCI) | Yes (Fresh book + Call Auction reopen) | Full secondary campus |
Standby Order-Book Recovery Architecture Comparison Matrix#
Comparing memory footprint, live-path jitter, afternoon 15:30 ET recovery SLA, and bug blast-radius immunity across four standby models.
- Worked Afternoon Crash Recovery Comparison at t_crash = 15:30 ET (3.2B Cumulative Msgs, Live Rate = 200k/s, Replay Speed = 2.5M/s):
- Without Snapshots (t_last = 09:30): 0s + (3,200,000,000 / 2,500,000) = 1,280.0 seconds (21 min 20 sec!)
- With 60s Flat-Slab Snapshot (12 GB @ 6.5 GB/s): (12 / 6.5) + (200,000 * 60 / 2,500,000) = 1.85s + 4.80s = 6.65 seconds!
Why You Still Need Cold/Warm Replay Even With a 3-Node Hot Raft Cluster!#
Wait: if a 3-node or 5-node Hot Raft Cluster already fails over in 35 milliseconds, why build fast Snapshot + Event-Log Replay at all? Three life-saving reasons:
- Poison-Pill ("Deterministic Crash") Bug Recovery: Suppose a malformed exotic order (
Seq #900,441,209) triggers a null-pointer dereference in the matching code. Because Raft is deterministic, that exact poison packet crashes ALL 3 hot Raft nodes simultaneously in 20 microseconds! Recovering requires patching/skipping#900,441,209in the admin tool and booting clean from theSeq #900,000,000snapshot in seconds instead of 25 minutes! - Spinning Up a Replacement Raft Voter: When
Node_2hardware dies at14:00 ET, a fresh spare blade must catch up2.5Bevents quickly to restore3/3cluster health. - Exact Forensic Incident Reproduction: Replaying the production snapshot + tail log inside staging (chaos engineering & staging fault injection) reproduces nanosecond bugs identically.
Suppressing Duplicate Side Effects (Output Fence) During Replay Mode#
Critical engineering invariant: while a recovering node replays historical events Seq_Snapshot + 1 .. Seq_Last_Committed, its Outbound Gateway Emitter MUST be wired to a Null Sink (Replay_Silent_Mode = True) until Current_Seq == Seq_Last_Committed—otherwise replaying 45 million events blasts 45 million duplicate ExecutionReport (35=8) packets onto customer FIX API sockets! Size operational downtime exposure in our Terminal Margin VaR & ES Calculator.
Frequently asked questions
What is a "Poison-Pill" packet in a replicated matching engine, and how has it brought down major exchanges?
A Poison Pill is a syntactically valid inbound frame hitting an unhandled edge case (e.g., divide-by-zero on a 0-tick spread or negative price in an energy contract) that triggers SIGSEGV/panic. Because every Raft replica executes the exact same code on that sequenced log entry, all replicas crash together unless operators can tombstone that single SeqNum and fast-boot from a recent snapshot!
Does calling Linux fork on a 32 GB matching-engine process freeze trading while copying 32 GB of RAM?
No—Linux fork uses hardware MMU Copy-on-Write (CoW), copying ONLY page-table descriptors initially (~100 microseconds, or even faster with hugepages/double-buffered arenas); physical 4KB/2MB pages are duplicated incrementally only when the live parent mutates active price levels.
How do you guarantee a power crash mid-write does not leave a half-written corrupt Snapshot file?
Write every new checkpoint to a temporary filename (snapshot.2850000000.tmp), append a trailing SHA-256/CRC32C footer, call fsync on both file and directory, and atomically rename to snapshot.latest—retaining at least the last 3 verified snapshots on disk.
Should clientMoldUDP64 / ITCH market-data book builders also use Snapshot + Delta recovery?
Yes! Both Nasdaq MoldUDP64 (SoupBinTCP Glimpse) and CME MDP 3.0 Market Recovery Feeds (Channel 310+ Snapshot feeds) broadcast periodic full book snapshots stamped with LastMsgSeqNumProcessed so late-joining or crashed client feed handlers rebuild books in <2 seconds instead of rewinding to midnight.
Publication record
- Published
- 13 November 2026
- Updated
- 14 November 2026
- Author
- Marcus Thorne, CQF
- Review
- Review not recorded
- Automated fact check
- Passed in the stored pipeline record
- Source URLs
- 3 recorded
AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.
Academic & Institutional Attribution
Cite This Research & Embed Live Data Card
Standard academic & institutional research paper citation format.
Marcus Thorne, CQF. (2026, November 14). Deterministic Event-Log Replay: Cold vs Warm vs Hot Standby Order-Book Reconstruction. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/deterministic-event-log-replay-cold-vs-warm-standby-orderbook-bootRelated Reg Sci Disaster Recovery Outage Rca & Institutional Briefings
Explore Reg Sci Disaster Recovery Outage Rca Hub (20) →Reg Sci Disaster Recovery Outage Rca
SEC Regulation SCI, Active-Active Exchange Failover & Outage Post-Mortem Engineering GuideReg Sci Disaster Recovery Outage Rca
Multicast A/B Market-Data Feed Arbitration: Air-Gapped Switch Fabrics & Microburst RecoveryReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem Anatomy: 32-Bit Sequence Wraparounds & Timestamp Epoch OverflowsReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: Opening-Auction Reference-Price Errors & LULD Halt CascadesReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: WAN BGP Route Leaks, Extranet Partitions & Asymmetric HaltsReg Sci Disaster Recovery Outage Rca
CTA & UTP SIP Market-Data Processor Failover: NBBO Staleness, Reg NMS Rule 611 & Direct FeedsReg Sci Disaster Recovery Outage Rca
SEC Regulation SCI Rule 1002: Incident Triage Trees, Immediate SEC Notification & De Minimis Safe HarborReg Sci Disaster Recovery Outage Rca
Clearly Erroneous Executions (CEE) & Trade Bust Bands: Post-Outage Unwind & Hedge Risk MathBroker Fees and Calculators
NYSE Affiliate to Launch 24/7 Tokenized Stock Trading This MonthMarket Insights
Options traders bracing for more tech downside, says Cboe's Mandy Xu - CNBCExecutive Moves
Ripple Prime brokerage head to step down at year-end - Tech in AsiaMarket Insights
Avalanche Won Over the NYSE. Now It Has to ‘Capture Value’ for AVAX
Terminal Ledger Desk Data, Benchmarks & Contributor Records
CC BY 4.0 Open Data
Open Institutional Datasets (CSV / JSON) →
Download desk-compiled quantitative series & schemas.
2026 Reference Matrix
2026 Institutional Benchmarks & Thresholds →
Statutory capital, clearing, margin & rate conventions.
Editorial Records
Authors & Review Records →
View named contributors where those records are available.
Complete Index
Master Research & Dossier Archive →
Browse all published briefings by topic pillar & date.
