Reg Sci Disaster Recovery Outage Rca
Multicast A/B Market-Data Feed Arbitration: Air-Gapped Switch Fabrics & Microburst Recovery
UDP multicast market data has no TCP retransmission safety net; institutional colocated desks subscribe simultaneously to physically air-gapped Feed A and Feed B switches, arbitrating packet-by-packet by sequence number.

High-speed exchange order books (CME MDP 3.0, Nasdaq TotalView-ITCH, NYSE Integrated, Cboe PITCH, Eurex EOBI) broadcast exclusively over unreliable Layer-3/4 UDP Multicast because maintaining 2,500 separate stateful TCP retransmission windows would add catastrophic head-of-line stalls inside the matching hall. How do trading firms achieve 99.9999% lossless book continuity over lossy UDP? Through Physically Air-Gapped Dual Multicast Fabrics (Feed A + Feed B) paired with lock-free First-Arrival Sequence Arbitration!
Exchange Dual-Plane Physical Isolation: Zero Shared Switches, Optics, or RP Routers#
Within CME Aurora DC3 and Equinix NY4/Carteret (detailed in our SEC Reg SCI & Outage Post-Mortem Hub), exchanges emit two logically identical copies of every market-data UDP datagram stamped with the exact same MsgSeqNum:
- Feed A (
224.0.31.x/ Optical Cross-Connect #1): Traverses Exchange Publisher A -> Spine/Leaf Switch Fabric A (e.g. Aristaplane) -> Client NIC Port 0. - Feed B (
224.0.32.x/ Optical Cross-Connect #2): Traverses Exchange Publisher B -> Completely separate Spine/Leaf Switch Fabric B (often even a different switch ASIC vendor or firmware train to prevent simultaneous software bugs!) -> Client NIC Port 1.
Fatal Client Cabling Sin: Never terminate Exchange Cross-Connect A and Cross-Connect B into the same single Top-of-Rack (ToR) switch in your colocation cabinet! Doing so collapses 200 million dollars of exchange dual-fabric redundancy into a single $18,000 switch power supply / ASIC reboot!
Zero-Branch C++ / FPGA Packet-by-Packet Expected_Seq Arbitrator#
Inside a Solarflare `ef_vi` C++ feed handler or inline FPGA, Feed A is not "Primary" and Feed B is not "Backup"—whichever physical wire delivers SeqNum == Expected_Seq first by even 40 nanoseconds wins on a packet-by-packet basis! Look at the deterministic state machine:
Packet_Seq < Expected_Seq: Already processed via the faster sibling leg45 nsago -> IncrementDup_Drop_Counterand discard in4 ns!Packet_Seq == Expected_Seq: Immediate Winner! DecodeSBE / ITCHbook update immediately, incrementExpected_Seq++, and drain any consecutive contiguous packets (Expected_Seq, Expected_Seq+1...) sitting in the small out-of-order reorder ring!Packet_Seq > Expected_Seq(Sequence Gap on Leg 1):Feed AdroppedSeq #804during a switch microburst! Place#805into a tiny64-slotlock-free Reorder Ring and start a250-microsecondGap Timer—99.98%of the time,Feed Bdelivers missing#804within2 to 15 microseconds, seamlessly bridging the gap without ever touching TCP Retransmit / Snapshot recovery!
| Exchange Multicast Feed | Transport Protocol | Dual-Feed Redundancy | Small-Gap Retransmit | Large-Gap Snapshot Channel |
|---|---|---|---|---|
CME Globex (MDP 3.0) | Binary SBE over UDP | Incremental Feed A + Feed B | TCP Historical Replay (<= 2,000 msgs) | Continuous UDP Market Recovery (W) + TCP |
Nasdaq (TotalView-ITCH 5.0) | MoldUDP64 | Dual multicast groups (A & B) | Unicast UDP MoldUDP64 Retransmit | ITCH Glimpse (SoupBinTCP point-in-time) |
Eurex T7 (EOBI / EMDI) | Binary SBE/ETI UDP | Service A + Service B (Distinct IPs) | Packet retransmission service | Cyclical Snapshot Multicast Stream |
NYSE Pillar (Integrated) | XDP Binary over UDP | Channel A + Channel B | XDP TCP Retransmission Request | XDP Refresh / Snapshot Request |
Multicast Feed Loss & Recovery Escalation Ladder Matrix#
Four-tier recovery escalation from nanosecond A/B merge to UDP Retransmit and Full Book Snapshot rebuild.
- Why Physical Independence Dominates Reliability:
- If Pdrop(SwitchA) = 10^-5 and Pdrop(SwitchB) = 10^-5:
- Air-Gapped Dual Switches + Dual NICs (Pcommon ~= 0): Pgap = 10^-5 * 10^-5 = 10^-10 (< 1 packet per year!)
- Both Feeds Plugged Into 1 Client ToR Switch (Pcommon = 10^-4): Pgap = 10^-4 (1,000,000x worse reliability!)
What If BOTH Feed A AND Feed B Drop the Same Microburst? (Freeze Quote Guard!)#
Suppose a massive 08:30:00.000 ET US CPI microburst (4.5 million packets/sec) overflows your own server PCIe / NIC RX ring (because a Linux housekeeping interrupt stole CPU Core 4 for 300 microseconds), dropping Seq #9001..#9018 on both Port 0 and Port 1 simultaneously!
- Mandatory Immediate Safety Action: The exact microsecond
Gap_Timerexpires (e.g. > 250 usunresolved gap on both A & B), mark that symbol/channel bookBOOK_STALE_DIRTY = TRUEinside shared memory! Your pre-trade collar (dynamic NBBO price collars) and quoting engine must instantly freeze aggressive entry and pull resting touch quotes—trading off a book missing18cancel/add events means hallucinating phantom top-of-book liquidity! - Recovery: Request missing
18packets from the venue UDP/TCP Retransmission Server (MoldUDP64 Retransmit/CME TCP Replay), or if gap> 1,000msgs, rebuild from the continuous Snapshot Recovery Channel (deterministic snapshot + delta replay).
Eliminating Client-Side RX Drops: isolcpus, Deep Switch Buffers & ef_vi Rings#
Ninety-five percent of "exchange multicast drops" blamed by junior engineers actually occurred inside the client own under-buffered cut-through switch (12 MB shallow buffer overflowing during fan-in!) or unpinned Linux kernel thread! Dedicate deep-buffer switches (Arista 7280R for non-HFT aggregation, or direct 1:1 Layer-1 optical tap/Arista 7130 to 4096-descriptor Solarflare ef_vi rings on isolcpus,nohz_full cores).
Quantify adverse-selection risk during feed gaps using our Terminal Margin VaR & ES Calculator and Algorithmic Kelly Allocation Calculator.
Frequently asked questions
Apart from reliability, how much latency does A/B packet-by-packet arbitration actually shave off market data?
Empirical colocation captures in Aurora DC3 and Carteret NY11 show that neither Feed A nor Feed B wins 100% of packets: transient switch queue occupancy causes A and B to trade leads by 30 to 900 nanoseconds throughout the day! Taking MIN(Arrival_A, Arrival_B) shaves significant p99 tail jitter compared to listening to Feed A alone.
What happens if an algorithm keeps quoting while missing just ONE "Order Delete" packet on Nasdaq ITCH?
That deleted 5,000-share bid stays stuck as a "Ghost Order" at the top of your local book forever—making your local Best Bid higher than the real market, causing your local book to look artificially locked/crossed whenever real offers move down, and tricking your strategy into buying at terrible prices!
Why do exchanges strictly throttle their UDP/TCP Market-Data Retransmission servers?
During a major network hiccup affecting 50 members simultaneously, if 500 client processes blast unlimited unicast retransmission requests at once, the retransmission server NIC saturates! Venues cap requests per minute—so client arbiters should share one local multicast/IPC retransmit proxy per rack rather than letting 20 processes request independently.
How do you synchronize a live Incremental UDP stream with an asynchronous Glimpse / Snapshot rebuild?
Buffer incoming live A/B incremental UDP packets in memory (say Seq 500,100..500,900) while downloading the Snapshot; read the Snapshot header LastMsgSeqNumProcessed (say 500,450), discard buffered incrementals <= 500,450, apply 500,451..500,900 in exact order, and unlock BOOK_STALE = FALSE!
Publication record
- Published
- 13 November 2026
- Updated
- 15 November 2026
- Author
- Marcus Thorne, CQF
- Review
- Review not recorded
- Automated fact check
- Passed in the stored pipeline record
- Source URLs
- 3 recorded
AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.
Academic & Institutional Attribution
Cite This Research & Embed Live Data Card
Standard academic & institutional research paper citation format.
Marcus Thorne, CQF. (2026, November 15). Multicast A/B Market-Data Feed Arbitration: Air-Gapped Switch Fabrics & Microburst Recovery. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/multicast-a-b-market-data-feed-arbitration-switch-fault-isolationRelated Reg Sci Disaster Recovery Outage Rca & Institutional Briefings
Explore Reg Sci Disaster Recovery Outage Rca Hub (20) →Reg Sci Disaster Recovery Outage Rca
SEC Regulation SCI, Active-Active Exchange Failover & Outage Post-Mortem Engineering GuideReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem Anatomy: 32-Bit Sequence Wraparounds & Timestamp Epoch OverflowsReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: Opening-Auction Reference-Price Errors & LULD Halt CascadesReg Sci Disaster Recovery Outage Rca
Exchange Outage Post-Mortem: WAN BGP Route Leaks, Extranet Partitions & Asymmetric HaltsReg Sci Disaster Recovery Outage Rca
CTA & UTP SIP Market-Data Processor Failover: NBBO Staleness, Reg NMS Rule 611 & Direct FeedsReg Sci Disaster Recovery Outage Rca
SEC Regulation SCI Rule 1002: Incident Triage Trees, Immediate SEC Notification & De Minimis Safe HarborReg Sci Disaster Recovery Outage Rca
Clearly Erroneous Executions (CEE) & Trade Bust Bands: Post-Outage Unwind & Hedge Risk MathReg Sci Disaster Recovery Outage Rca
Capacity Stress Testing & Microburst Telemetry Under SEC Regulation SCI Rule 1001(a)Market Insights
Options traders bracing for more tech downside, says Cboe's Mandy Xu - CNBCExecutive Moves
Ripple Prime brokerage head to step down at year-end - Tech in AsiaMarket Insights
Avalanche Won Over the NYSE. Now It Has to ‘Capture Value’ for AVAXMarket Insights
Nasdaq Jumps After CoreWeave Earnings
Terminal Ledger Desk Data, Benchmarks & Contributor Records
CC BY 4.0 Open Data
Open Institutional Datasets (CSV / JSON) →
Download desk-compiled quantitative series & schemas.
2026 Reference Matrix
2026 Institutional Benchmarks & Thresholds →
Statutory capital, clearing, margin & rate conventions.
Editorial Records
Authors & Review Records →
View named contributors where those records are available.
Complete Index
Master Research & Dossier Archive →
Browse all published briefings by topic pillar & date.
