Skip to content
Terminal Ledger Desk
Directory

Matching Engine Clob Colocation Ptp

Solarflare Onload / ef_vi vs DPDK Kernel Bypass: NIC RX/TX Ring & PIO Tuning

Standard Linux socket syscalls burn 8 to 15 microseconds in interrupt and sk_buff overhead; AMD/Solarflare ef_vi Programmed I/O (PIO) and Onload user-space stacks trim host RX/TX latency down to 290 nanoseconds.

By Marcus Thorne, CQFReview not recorded
Solarflare Onload / ef_vi vs DPDK Kernel Bypass: NIC RX/TX Ring & PIO Tuning — Terminal Ledger Desk 2026 quantitative & supervisory data card
Terminal Ledger Desk (Deterministic Matching Engines, NY4/LD4/TY3 Colocation & PTPv2 Clock Sync): Solarflare Onload, ef_vi PIO/CTPIO, TX path warming, and DPDK poll-mode ring architecture.Source: Terminal Ledger Desk Institutional Research DeskLicense: CC-BY-4.0

When an ITCH 5.0 or CME MDP 3.0 UDP frame hits a standard Linux server NIC, the hardware raises a MSI-X interrupt, wakes the kernel NAPI softirq handler, allocates an sk_buff, traverses iptables/netfilter, copies payload bytes across the ring-0/ring-3 Spectre/KPTI boundary, and finally unblocks epoll_wait—burning 8 to 15 microseconds before your strategy sees byte zero! Inside Equinix NY4 and CME Aurora, institutional C++/Rust gateways eliminate the OS kernel from the packet datapath using three distinct architectures: OpenOnload (LD_PRELOAD POSIX TCP/UDP), ef_vi (EtherFabric Virtual Interface Layer-2 direct rings), and DPDK Poll-Mode Drivers (PMD).

OpenOnload vs ef_vi vs DPDK: Choosing the Right Kernel-Bypass Layer#

Not every kernel-bypass framework solves the same trading problem—picking DPDK for single-session exchange TCP order entry often backfires:

  • AMD/Solarflare OpenOnload (X2522 / X3522 NICs): Intercepts standard POSIX socket, recv, and send via LD_PRELOAD, running a battle-tested RFC-compliant TCP/IP state machine entirely inside user space mapped directly to NIC hardware queues (VI). Zero application code rewrite; ~1.05 us half-RTT; preserves Linux ARP/routing control plane.
  • Solarflare ef_vi + Custom User-Space TCP (TCPDirect / Custom): Exposes raw Layer-2 DMA RX descriptor rings and Programmed I/O (PIO) TX buffers directly to C++. Ideal for raw UDP multicast (ITCH/MDP 3.0) and single-connection binary order ports (OUCH 5.0, iLink 3 SBE), hitting 280–340 ns TX+RX!
  • DPDK (mlx5 Mellanox/NVIDIA ConnectX-6/7 or Intel E810): Detaches the entire NIC port (or bifurcated queue in mlx5) into a vfio-pci Poll-Mode Driver capable of saturating 100GbE line rate (148.8 Mpps), yet requires bundling an external user-space TCP stack (F-Stack, mTCP, or custom) for TCP exchange sessions.
Solarflare ef_vi RX HugePage DMA descriptor ring and Write-Combining PIO / CTPIO direct SRAM outbound order transmission path
Solarflare ef_vi RX HugePage DMA descriptor ring and Write-Combining PIO / CTPIO direct SRAM outbound order transmission pathVerified Exhibit

Programmed I/O (PIO) vs Bus-Master DMA on Outbound Order Bursts#

How a NIC fetches your 64-byte outbound OUCH EnterOrder packet makes a 110-nanosecond difference! Under standard DMA TX, your CPU writes a ring descriptor and rings a PCIe doorbell; the NIC must then issue a PCIe Memory Read Request TLP back across the root complex to pull payload bytes (~160 ns PCIe round-trip penalty).

With Solarflare Programmed I/O (ef_pio) (and NVIDIA BlueFlame / WQE Inline), the CPU copies the pre-templated order frame directly across PCIe Write-Combining (WC) BAR memory into on-NIC SRAM, triggering instant MAC transmission on the final 64-byte flush without any PCIe read-back! Because market makers send orders rarely relative to market-data ticks, keeping WC buffers and TCP paths warm via dummy TX Warm (MSG_WARM / onload_zc_send dry-run) prevents instruction-cache and TLB decay—detailed in our Deterministic Matching Engines & Colocation Guide.

Quantitative Distribution · Typical Host Tick-to-Trade
Typical Host Tick-to-Trade
01Linux Kernel (epoll)11.4 us p50 / 4…02Solarflare OpenOnload1.05 us p50 / 1…03DPDK (mlx5 / ice PMD)0.41 us p50 / 0…04Solarflare ef_vi / TCPDirec…0.29 us p50 / 0…
Bypass ArchitectureAPI AbstractionTCP/IP Stack SupportTX Hardware MechanismTypical Host Tick-to-Trade
Linux Kernel (epoll)POSIX socketNative Kernel sk_buffRing-0 DMA Descriptor11.4 us p50 / 48.2 us p99
Solarflare OpenOnloadZero-code POSIX (LD_PRELOAD)Full User-Space RFC TCP/UDPPIO + MSG_WARM dry-run1.05 us p50 / 1.62 us p99
DPDK (mlx5 / ice PMD)rte_eth_rx_burst Layer-2External (F-Stack) or CustomWQE Inline / Doorbell DMA0.41 us p50 / 0.58 us p99
Solarflare ef_vi / TCPDirect (Aurora)Raw Layer-2 / zft_* Zero-CopyUltra-lean single-peer TCPOn-NIC PIO / CTPIO SRAM0.29 us p50 / 0.39 us p99

Kernel-Bypass NIC Stack Comparison Matrix (10GbE / 25GbE Colocation)#

Measured wire-to-application RX + TX half-round-trip latency on isolated bare-metal cores (isolcpus, C-states disabled).

Formula or model from the articleNot independently verified
T_tx_dma = T_desc_write + T_pcie_doorbell + T_pcie_read_req_rtt + T_mac_serialize
  • Ttxctpio = Twcbarfirst64B + Tcutthrough_mac [No PCIe Read-Back RTT!]
  • Where (128-byte OUCH / iLink 3 Order over PCIe Gen4 x8 @ 25GbE):
  • Tpciedoorbell = 55 ns; Tpciereadreqrtt = 155 ns; Tmacserialize = 41.0 ns
  • Twcbarfirst64B = 62 ns; Tcutthrough_mac = 38.0 ns
  • Worked Outbound Order Silicon Savings:
  • Ttxdma = 12 + 55 + 155 + 41 = 263 nanoseconds
  • Ttxctpio = 62 + 38 = 100 nanoseconds (saves 163 ns on every outbound order!).

Cut-Through RX Streaming (ctpio / X3522) & Ring Sizing Tradeoffs#

Sizing ef_vi or DPDK RX descriptor rings involves a sharp L3-cache tradeoff: setting rx_ring_size = 4096 absorbs massive CPI multicast bursts without hardware rx_nodesc_drop_cnt drops, yet cycling through 4,096 x 2,048-byte packet buffers (8.0 MiB) evicts strategy state from L2/L3 cache! Instead, low-latency feed handlers run compact 512 or 1,024 descriptor rings backed by 2MB HugePages while polling tightly on a dedicated isolcpus-pinned core that drains every datagram in < 90 ns.

On latest AMD Alveo X3522 hybrid NICs, CTPIO (Cut-Through PIO) begins streaming outbound Ethernet frames onto the optical transceiver before the final PCIe cache line even finishes arriving from the CPU—bridging the gap toward pure FPGA cut-through pipelines.

Host networking stack p99 latency across Linux epoll socket, io_uring, DPDK mlx5, Solarflare OpenOnload, and Solarflare ef_vi CTPIO
Host networking stack p99 latency across Linux epoll socket, iouring, DPDK mlx5, Solarflare OpenOnload, and Solarflare efvi CTPIOVerified Exhibit

Hardware Drop Counters & Compliance Tap Visibility#

Remember one operational caveat when switching from Linux sockets or Onload to raw ef_vi or vfio-pci DPDK: host-based tcpdump / libpcap can no longer see bypassed kernel traffic! Under SEC and MiFID II audit rules, always capture compliance PCAPs off an external Layer-1 passive optical tap or Arista 7130 switch rather than host software hooks.

Monitor ethtool -S (rx_crc_errors, rx_fifo_errors, port_rx_nodesc_drop) continuously via FIX API health telemetry, size slippage buffers in the Terminal Margin VaR & ES Calculator, and browse more NIC engineering guides in the Matching Engine, Colocation & PTP Topic Hub.

Frequently asked questions

When should a trading team use OpenOnload instead of writing directly to raw ef_vi?

Use OpenOnload when accelerating multi-session OMS gateways, QuickFIX/C++ routers, or vendor binaries where rewriting full TCP retransmission, ARP, and VLAN handling from scratch is impractical; reserve raw ef_vi/TCPDirect for dedicated ultra-low-latency market-data and sniper loops.

What is TX Path Warming (onload_zc_send MSG_WARM) and why does rare order flow need it?

If an HFT strategy processes 500,000 market-data ticks per second but fires an order only once every 3 seconds, the outbound TCP/SBE code path goes cold in L1i cache, uTLB, and branch predictors—adding 400–900 ns on the real order unless exercised every 500 us with a no-wire dry run.

Why does tcpdump miss packets when DPDK or ef_vi kernel bypass is active?

Because bypassed NIC queues DMA frames straight into user-space HugePages without ever allocating a Linux kernel sk_buff, the kernel packet tap hook (AF_PACKET) never sees them. Colocated cages tap optical splitters into dedicated hardware capture appliances instead.

How does NVIDIA/Mellanox bifurcated driver mode differ from Intel vfio-pci in DPDK?

Intel DPDK via vfio-pci unbinds the entire physical port from Linux (hiding eth0 from SSH/ARP), whereas Mellanox mlx5 uses a bifurcated kernel/user architecture that steers specific UDP/TCP flow rules to user-space DPDK queues while leaving ARP, PTP, and SSH on the kernel.

Publication record

Published
14 November 2026
Updated
14 November 2026
Author
Marcus Thorne, CQF
Review
Review not recorded
Automated fact check
Passed in the stored pipeline record
Source URLs
3 recorded

AI assistance recorded: anthropic/claude-opus-4.6. This field does not establish human review or source verification.

Academic & Institutional Attribution

Cite This Research & Embed Live Data Card

Open Attribution License (CC BY 4.0)

Standard academic & institutional research paper citation format.

Marcus Thorne, CQF. (2026, November 14). Solarflare Onload / ef_vi vs DPDK Kernel Bypass: NIC RX/TX Ring & PIO Tuning. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/solarflare-onload-ef-vi-vs-dpdk-kernel-bypass-nic-rx-tx-ring-tuning
Explore Matching Engine Clob Colocation Ptp Hub (20) →

Terminal Ledger Desk Data, Benchmarks & Contributor Records

Solarflare Onload & ef_vi vs DPDK Kernel Bypass Tuning | Terminal Ledger Desk