# Solarflare Onload & ef_vi vs DPDK Kernel Bypass Tuning

> Benchmark AMD/Solarflare OpenOnload, ef_vi Programmed I/O (PIO) TX warming, and DPDK Poll-Mode Drivers for sub-microsecond exchange order gateways.

Canonical: https://terminalledgerdesk.com/reports/solarflare-onload-ef-vi-vs-dpdk-kernel-bypass-nic-rx-tx-ring-tuning
Last updated: 2026-11-14T20:05:59.558Z

**Publication:** Terminal Ledger Desk
**Category:** Matching Engine Clob Colocation Ptp
**Published:** 2026-11-14T06:40:25.915Z
**Author:** Marcus Thorne, CQF (Certificate in Quantitative Finance (CQF), Low-Latency FIX & Matching Engine Architect)
**Reviewer:** Not recorded

## Answer-first summary

Standard Linux socket syscalls burn 8 to 15 microseconds in interrupt and sk_buff overhead; AMD/Solarflare ef_vi Programmed I/O (PIO) and Onload user-space stacks trim host RX/TX latency down to 290 nanoseconds.

## Key takeaways

- Solarflare ef_vi with CTPIO and NVIDIA ConnectX-7 Inline WQE slash host RX+TX p99 latency to 390–580 ns—up to 123x faster than standard Linux epoll sockets.
- Programmed I/O (PIO / CTPIO) pushes outbound order bytes directly across PCIe Write-Combining BARs into NIC SRAM, eliminating a 155-ns PCIe DMA read-back round-trip.
- Executing periodic dry-run TX Warming every 250–500 us keeps outbound TCP/order serialization code and branch predictors hot in L1 instruction cache.
- Keep RX descriptor rings compact (512–1,024 slots on 2MB HugePages) so packet buffer rotation never evicts active order-book cache lines from L3.

## Related financial entities

- **CME Aurora Financial Data Center Standard** (CME-AURORA) — product: Dedicated CME Group hosting facility in Aurora, Illinois, housing the Globex matching engine and offering direct low-latency colocation to clearing members.
- **Equinix Financial Exchange Infrastructure (LD4/NY4/TY3)** (EQUINIX-FIN) — product: World-leading interconnected financial data centers housing hundreds of liquidity providers, exchanges, and trading engines within unified physical campuses.
- **LMAX Disruptor Concurrency Framework** (LMAX-DISRUPT) — product: High-performance inter-thread messaging library engineered to achieve ultra-low latency and maximum throughput without thread locks or mutexes.

## Analysis

When an ITCH 5.0 or CME MDP 3.0 UDP frame hits a standard Linux server NIC, the hardware raises a MSI-X interrupt, wakes the kernel `NAPI` softirq handler, allocates an `sk_buff`, traverses `iptables/netfilter`, copies payload bytes across the ring-0/ring-3 Spectre/KPTI boundary, and finally unblocks `epoll_wait`—burning **`8 to 15 microseconds`** before your strategy sees byte zero! Inside [Equinix NY4 and CME Aurora](/engines/cme-aurora-colocation-standard), institutional C++/Rust gateways eliminate the OS kernel from the packet datapath using three distinct architectures: **OpenOnload (`LD_PRELOAD` POSIX TCP/UDP)**, **`ef_vi` (`EtherFabric Virtual Interface` Layer-2 direct rings)**, and **DPDK Poll-Mode Drivers (`PMD`)**.

## OpenOnload vs `ef_vi` vs DPDK: Choosing the Right Kernel-Bypass Layer

Not every kernel-bypass framework solves the same trading problem—picking DPDK for single-session exchange TCP order entry often backfires:

- **AMD/Solarflare OpenOnload (`X2522` / `X3522` NICs)**: Intercepts standard POSIX `socket`, `recv`, and `send` via `LD_PRELOAD`, running a battle-tested RFC-compliant TCP/IP state machine **entirely inside user space** mapped directly to NIC hardware queues (`VI`). Zero application code rewrite; `~1.05 us` half-RTT; preserves Linux ARP/routing control plane.
- **Solarflare `ef_vi` + Custom User-Space TCP (`TCPDirect` / Custom)**: Exposes raw Layer-2 DMA RX descriptor rings and **Programmed I/O (`PIO`)** TX buffers directly to C++. Ideal for raw UDP multicast (`ITCH`/`MDP 3.0`) and single-connection binary order ports (`OUCH 5.0`, `iLink 3 SBE`), hitting **`280–340 ns` TX+RX**!
- **DPDK (`mlx5` Mellanox/NVIDIA ConnectX-6/7 or Intel E810)**: Detaches the entire NIC port (or bifurcated queue in `mlx5`) into a `vfio-pci` Poll-Mode Driver capable of saturating `100GbE` line rate (`148.8 Mpps`), yet requires bundling an external user-space TCP stack (`F-Stack`, `mTCP`, or custom) for TCP exchange sessions.

![Solarflare ef_vi RX HugePage DMA descriptor ring and Write-Combining PIO / CTPIO direct SRAM outbound order transmission path](/api/figure?type=flow&title=Kernel-Bypass%20ef_vi%20RX%20Ring%20%26%20TX%20PIO%20%2F%20CTPIO%20Pipeline&subtitle=Zero-IRQ%20user-space%20RX%20polling%20%2B%20Write-Combining%20PCIe%20BAR%20push%20without%20DMA%20read-back&steps=1.%20Multicast%20RX%3A%3ANIC%20DMAs%20UDP%20into%20HugePage%20L3%20%28DDIO%29%7C2.%20Spin%20Poll%20VI%3A%3Aef_eventq_poll%28%29%20reads%20event%20in%2018%20ns%7C3.%20Strategy%20Calc%3A%3AUpdate%20O%281%29%20book%20%2B%20pre-warmed%20TX%20path%7C4.%20CTPIO%20Push%3A%3AWC%20BAR%20write%20streams%20straight%20to%20SerDes&metrics=EF_VI%20%2B%20CTPIO%3A%3A285%20ns%3A%3AHost%20RX%20%2B%20TX%20wire-to-wire%7CPIO%20vs%20DMA%20GAIN%3A%3A-115%20ns%3A%3AEliminates%20PCIe%20read%20TLP%7CKERNEL%20IRQS%3A%3A0%20%2F%20sec%3A%3A100%25%20user-space%20poll%20loop%7CRX%20RING%20FOOTPRINT%3A%3A512%20Slots%3A%3A1%20MiB%20stays%20L2%2FL3%20resident)

## Programmed I/O (`PIO`) vs Bus-Master DMA on Outbound Order Bursts

How a NIC fetches your 64-byte outbound `OUCH EnterOrder` packet makes a `110-nanosecond` difference! Under standard **DMA TX**, your CPU writes a ring descriptor and rings a PCIe doorbell; the NIC must then issue a **PCIe Memory Read Request TLP** back across the root complex to pull payload bytes (`~160 ns` PCIe round-trip penalty).

With Solarflare **Programmed I/O (`ef_pio`)** (and NVIDIA `BlueFlame` / `WQE Inline`), the CPU copies the pre-templated order frame **directly across PCIe Write-Combining (`WC`) BAR memory into on-NIC SRAM**, triggering instant MAC transmission on the final 64-byte flush without any PCIe read-back! Because market makers send orders *rarely* relative to market-data ticks, keeping WC buffers and TCP paths warm via **dummy `TX Warm` (`MSG_WARM` / `onload_zc_send` dry-run)** prevents instruction-cache and TLB decay—detailed in our [Deterministic Matching Engines & Colocation Guide](/reports/deterministic-matching-engine-clob-colocation-ptp-clock-sync-guide).

| Bypass Architecture | API Abstraction | TCP/IP Stack Support | TX Hardware Mechanism | Typical Host Tick-to-Trade |
| :--- | :--- | :--- | :--- | :--- |
| **Linux Kernel (`epoll`)** | POSIX `socket` | Native Kernel `sk_buff` | Ring-0 DMA Descriptor | `11.4 us` p50 / `48.2 us` p99 |
| **Solarflare `OpenOnload`** | Zero-code POSIX (`LD_PRELOAD`) | Full User-Space RFC TCP/UDP | `PIO` + `MSG_WARM` dry-run | `1.05 us` p50 / `1.62 us` p99 |
| **DPDK (`mlx5` / `ice` PMD)** | `rte_eth_rx_burst` Layer-2 | External (`F-Stack`) or Custom | `WQE Inline` / Doorbell DMA | `0.41 us` p50 / `0.58 us` p99 |
| **Solarflare `ef_vi` / `TCPDirect`** ([Aurora](/engines/cme-aurora-colocation-standard)) | Raw Layer-2 / `zft_*` Zero-Copy | Ultra-lean single-peer TCP | **On-NIC `PIO` / `CTPIO` SRAM** | **`0.29 us` p50 / `0.39 us` p99** |

## Kernel-Bypass NIC Stack Comparison Matrix (10GbE / 25GbE Colocation)

Measured wire-to-application RX + TX half-round-trip latency on isolated bare-metal cores (`isolcpus`, C-states disabled).

```formula
T_tx_dma = T_desc_write + T_pcie_doorbell + T_pcie_read_req_rtt + T_mac_serialize
T_tx_ctpio = T_wc_bar_first_64B + T_cut_through_mac [No PCIe Read-Back RTT!]

Where (128-byte OUCH / iLink 3 Order over PCIe Gen4 x8 @ 25GbE):
 T_pcie_doorbell = 55 ns; T_pcie_read_req_rtt = 155 ns; T_mac_serialize = 41.0 ns
 T_wc_bar_first_64B = 62 ns; T_cut_through_mac = 38.0 ns

Worked Outbound Order Silicon Savings:
 T_tx_dma = 12 + 55 + 155 + 41 = 263 nanoseconds
 T_tx_ctpio = 62 + 38 = 100 nanoseconds (saves 163 ns on every outbound order!).
```

## Cut-Through RX Streaming (`ctpio` / `X3522`) & Ring Sizing Tradeoffs

Sizing `ef_vi` or DPDK RX descriptor rings involves a sharp L3-cache tradeoff: setting `rx_ring_size = 4096` absorbs massive CPI multicast bursts without hardware `rx_nodesc_drop_cnt` drops, yet cycling through `4,096 x 2,048-byte` packet buffers (`8.0 MiB`) evicts strategy state from L2/L3 cache! Instead, low-latency feed handlers run compact **`512` or `1,024` descriptor rings** backed by `2MB` HugePages while polling tightly on a dedicated [isolcpus-pinned core](/reports/linux-isolcpus-nohz-full-rcu-nocbs-c-state-bios-jitter-elimination) that drains every datagram in `< 90 ns`.

On latest `AMD Alveo X3522` hybrid NICs, **`CTPIO` (`Cut-Through PIO`)** begins streaming outbound Ethernet frames onto the optical transceiver *before* the final PCIe cache line even finishes arriving from the CPU—bridging the gap toward [pure FPGA cut-through pipelines](/reports/fpga-tick-to-trade-cut-through-mac-pcs-transceiver-pipeline-budgets).

![Host networking stack p99 latency across Linux epoll socket, io_uring, DPDK mlx5, Solarflare OpenOnload, and Solarflare ef_vi CTPIO](/api/figure?type=bars&title=Host%20Network%20Stack%20p99%20Latency%20%28RX%20%2B%20TX%20Half-RTT%29&subtitle=25GbE%20loop-back%20tick-to-order%20benchmark%20on%20isolated%20Xeon%20core%20%28nanoseconds%29&steps=Solarflare%20X3522%20ef_vi%20CTPIO%3A%3A285%20ns%20p50%20%20%20390%20ns%20p99%7CNVIDIA%20ConnectX-7%20DPDK%20Inline%3A%3A410%20ns%20p50%20%20%20580%20ns%20p99%7CSolarflare%20OpenOnload%20%2B%20Warm%3A%3A1%2C050%20ns%20p50%20%20%201%2C620%20ns%20p99%7CLinux%20Kernel%20epoll_wait%20TCP%3A%3A11%2C400%20ns%20p50%20%20%2048%2C200%20ns%20p99&metrics=P99%20SPEEDUP%3A%3A123.6x%3A%3Aef_vi%20CTPIO%20vs%20Linux%20epoll%7CTX%20WARMING%3A%3AEvery%20500us%3A%3AKeeps%20I-cache%20%26%20BTB%20hot%7CBIFURCATED%20DRV%3A%3Amlx5%20%2F%20Onload%3A%3AKeeps%20SSH%2FARP%20on%20OS%20stack%7CPCAP%20METHOD%3A%3AExternal%20L1%3A%3AZero%20host%20capture%20overhead)

## Hardware Drop Counters & Compliance Tap Visibility

Remember one operational caveat when switching from Linux sockets or Onload to raw `ef_vi` or `vfio-pci` DPDK: **host-based `tcpdump` / `libpcap` can no longer see bypassed kernel traffic**! Under [SEC](/standards/sec) and MiFID II audit rules, always capture compliance PCAPs off an external [Layer-1 passive optical tap or Arista 7130 switch](/reports/layer-1-matrix-switches-metamako-arista-7130-pcap-tap-nanoseconds) rather than host software hooks.

Monitor `ethtool -S` (`rx_crc_errors`, `rx_fifo_errors`, `port_rx_nodesc_drop`) continuously via [FIX API](/software/fix-api) health telemetry, size slippage buffers in the [Terminal Margin VaR & ES Calculator](/calculators/terminal-margin-var-es-calculator), and browse more NIC engineering guides in the [Matching Engine, Colocation & PTP Topic Hub](/topics/matching-engine-clob-colocation-ptp).

## Frequently asked questions

### When should a trading team use OpenOnload instead of writing directly to raw ef_vi?

Use OpenOnload when accelerating multi-session OMS gateways, QuickFIX/C++ routers, or vendor binaries where rewriting full TCP retransmission, ARP, and VLAN handling from scratch is impractical; reserve raw ef_vi/TCPDirect for dedicated ultra-low-latency market-data and sniper loops.

### What is TX Path Warming (onload_zc_send MSG_WARM) and why does rare order flow need it?

If an HFT strategy processes 500,000 market-data ticks per second but fires an order only once every 3 seconds, the outbound TCP/SBE code path goes cold in L1i cache, uTLB, and branch predictors—adding 400–900 ns on the real order unless exercised every 500 us with a no-wire dry run.

### Why does tcpdump miss packets when DPDK or ef_vi kernel bypass is active?

Because bypassed NIC queues DMA frames straight into user-space HugePages without ever allocating a Linux kernel sk_buff, the kernel packet tap hook (AF_PACKET) never sees them. Colocated cages tap optical splitters into dedicated hardware capture appliances instead.

### How does NVIDIA/Mellanox bifurcated driver mode differ from Intel vfio-pci in DPDK?

Intel DPDK via vfio-pci unbinds the entire physical port from Linux (hiding eth0 from SSH/ARP), whereas Mellanox mlx5 uses a bifurcated kernel/user architecture that steers specific UDP/TCP flow rules to user-space DPDK queues while leaving ARP, PTP, and SSH on the kernel.


## Source URLs

- https://docs.amd.com/v/u/en-US/SF-114063-CD-ef_vi_User_Guide
- https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html
- https://github.com/Xilinx-CNS/onload

## Citation

- APA: Marcus Thorne, CQF. (2026). Solarflare Onload / ef_vi vs DPDK Kernel Bypass: NIC RX/TX Ring & PIO Tuning. Terminal Ledger Desk. https://terminalledgerdesk.com/reports/solarflare-onload-ef-vi-vs-dpdk-kernel-bypass-nic-rx-tx-ring-tuning
- Canonical article: [Solarflare Onload / ef_vi vs DPDK Kernel Bypass: NIC RX/TX Ring & PIO Tuning](https://terminalledgerdesk.com/reports/solarflare-onload-ef-vi-vs-dpdk-kernel-bypass-nic-rx-tx-ring-tuning)
- Site research index: [Terminal Ledger Desk](https://terminalledgerdesk.com/llms-full.txt)
