PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond UDP: RDMA as the Missing Link for Gigasample Software-Defined Radio

Carsten Andrich, Sebastian Giehl, Michael Schubert

arXiv:2609.05623v1eess.SP

TL;DR

High-rate SDRs still use UDP packetization and host reassembly, which strain the host’s packet-processing time and memory bandwidth. The paper proposes RDMA sample transport through registered host memory, with the SDR fetching transmit data as needed. The architecture keeps the CPU out of both sample data paths and has each sample byte cross the memory bus once.

  • Problem

    At gigasample rates, packet processing and sample reassembly place competing demands on the host, while users must supply transport and operating-systems expertise.

    Source: §I · p. 1, D. The Per-Packet Budget · p. 2, E. Copy Amplification · p. 3

  • Method

    The proposed RoCEv2 architecture has the SDR write received samples to registered host memory and read transmit samples from that memory when needed.

    Source: §I · p. 1, §VI · p. 6

  • Results

    The architecture keeps the CPU out of both data paths and reduces sample-byte memory-bus crossings to one; the X440’s 1.02 µs per-packet budget falls within the socket path’s processing cost.

    Source: D. The Per-Packet Budget · p. 2, §VI · p. 6

  • Takeaways & Limitations

    The transmit-side fetch removes the hard real-time pacing problem imposed on users by pushing samples toward an unobservable converter buffer.

    Source: §VI · p. 6

    The Linux software RoCE implementation reintroduces host CPU cost and may not reach gigasample rates.

    Source: C. RoCEv2 in Practice · p. 4

Abstract

from arXiv · show

Software-defined radio (SDR) front ends have gained an order of magnitude in sample rate within a single product generation, while the transport that carries their output has not changed: samples are packed into Ethernet datagrams and reassembled by the host processor. At gigasample rates this fails on two counts at once, exhausting the time the host has per packet and the memory bandwidth it has for copying, and no amount of tuning or kernel bypass recovers both. The burden has instead been passed to the user, who must supply operating system expertise that has nothing to do with radio. We show that remote direct memory access removes the problem rather than relocating it. The SDR places received samples directly into memory the host has registered, and fetches samples for transmission from that memory when its converters require them. The processor leaves the data path in both directions, each sample byte crosses the memory bus once, and the transmit path loses the hard real-time pacing problem that has always accompanied it. What remains for the user application is the signal processing itself, which should be all an SDR requires, even at gigasample rates.

I. INTRODUCTION

SDR sample rates have risen sharply while Ethernet transport still relies on UDP packet handling and host reassembly, creating packet-processing and memory-bandwidth limits at high rates. The paper proposes an RDMA/RoCEv2 architecture that moves sample transport out of the CPU data path; comparative measurements remain future work.

I. INTRODUCTION

  • A USRP X440 delivers 2 GSa/s, or 8 GB/s, while the X310 delivered 200 MSa/s per channel, or 0.8 GB/s, and both use the unchanged UDP-and-host-reassembly transport.The paper says the Xilinx Versal RF series supports up to 16 GSa/s, or 64 GB/s from a single channel.Source: §I · p. 1
  • At 8 GB/s the host has roughly one microsecond per packet, while reassembly makes each sample byte cross the memory bus five times.Two X440 channels generate 80 GB/s of memory traffic before application processing, nearly a contemporary desktop host’s bandwidth.Source: §I · p. 1
  • The transport burden requires users to manage buffers, packet handling, and flow control, demanding operating-systems expertise rather than radio expertise.The paper notes that researchers have resorted to periodic burst capture, and UHD offers an interface that hands transport back to the user.Source: §I · p. 1
  • RDMA over RoCEv2 lets the SDR write received samples into registered host memory and read transmit samples from it, leaving the CPU outside both data paths.RoCEv2 provides RDMA over ordinary Ethernet, with hardware support on 100+ GbE-class interfaces and a Linux software implementation at lower rates.Source: §I · p. 1
  • The proposed architecture configures connection parameters out of band without a device-side RDMA connection manager and has the SDR fetch transmit samples when needed.It also aggregates multiple links into one contiguous stream and adds 32 bytes of transport header to packets already carrying IP and UDP headers.Source: §I · pp. 1–2
  • The article presents the architecture and its rationale; comparative measurements are still ongoing.Source: §I · p. 2

II. HOW SDRS STREAM SAMPLES TODAY

Current Ethernet SDR transport packetizes continuous I/Q streams and leaves delivery into application memory to the host operating system and application. High-rate streaming consequently entails host tuning and user-managed buffers, with burst capture an imperfect fallback.

A. The Prevailing Model

  • The packetize–reassemble model segments continuous I/Q samples into Ethernet packets, while framing standards specify payloads but not how samples reach application memory.VITA 49, DIFI, CHDR, and eCPRI define sample encoding or payload framing; the host OS and application handle the remaining memory-delivery step.Source: A. The Prevailing Model · p. 2

B. The Tuning Tax

  • Continuous loss-free gigasample streaming requires pinned interrupt and thread placement, enlarged NIC rings and socket buffers, isolated receive cores, and sometimes a real-time kernel.Burst sampling reduces sustained host demand but does not reduce the per-packet cost, and cannot serve applications requiring zero measurement dead time.Source: B. The Tuning Tax · p. 2

C. Who Owns the Buffer?

  • Applications own buffers that must use hugepages, be locked in physical memory, reside on the NIC’s NUMA node, and match the link bandwidth–delay product.The API accepts buffers that violate these requirements; losses can emerge later, although the driver knows the rate and host topology that users generally do not.Source: C. Who Owns the Buffer? · p. 2

D. The Per-Packet Budget

The packet-rate budget and the memory-bandwidth cost of reassembly are distinct constraints on high-rate SDR streaming. The cited examples show that X440 and newer front ends can leave little packet-processing time and consume host memory bandwidth approaching or exceeding system peaks.

D. The Per-Packet Budget

  • At 1.02 µs per packet, the X440 falls within the socket path’s one-to-two-microsecond processing cost, leaving nothing for the application; later converters miss the budget by an order of magnitude or more.The Versal RF front end produces 64 GB/s from one channel, five times the capacity of a 100 GbE link.Source: D. The Per-Packet Budget · p. 2

E. Copy Amplification

  • Reassembly makes each sample byte cross the memory bus five times, and this cost scales with sample rate rather than packet rate.Table II distinguishes where the DMA write lands: a driver-owned buffer for socket and bypass paths, versus the application’s registered buffer for RDMA.Source: E. Copy Amplification · pp. 2–3, Table II · p. 3
  • Two X440 channels imply 80 GB/s of memory traffic before processing against roughly 100 GB/s host bandwidth; one Versal RF channel implies 320 GB/s against roughly 307 GB/s.These are worst-case traffic estimates assuming every crossing reaches DRAM; the stated DRAM capacities are theoretical peaks.Source: E. Copy Amplification · p. 3, Table II · p. 3

F. Kernel Bypass Is Not Enough

Kernel bypass removes per-packet system calls and interrupts, but it leaves packet reassembly and most memory traffic to the application. It also leaves transmit timing and multi-link aggregation unaddressed.

F. Kernel Bypass Is Not Enough

  • Kernel bypass removes the per-packet system call and interrupt, but the application still assembles packets into a contiguous sample block.The per-packet budget ceases to bind, while three of the original five memory crossings per sample byte remain.Source: F. Kernel Bypass Is Not Enough · p. 3
  • Bypass increases application responsibility for memory pools, queue configuration, and polling threads, while DPDK removes the interface from the operating-system network stack.The user expertise burden therefore increases rather than decreases.Source: F. Kernel Bypass Is Not Enough · p. 3
  • The second copy can in principle be avoided with scatter-gather reception through recvmmsg(), which UHD does not implement; the first copy remains.The first copy remains regardless.Source: F. Kernel Bypass Is Not Enough · p. 3
  • A reassembly buffer small enough to stay in last-level cache can make some otherwise-failing configurations feasible.Direct cache injection may reduce traffic further, but is outside the manuscript's scope.Source: F. Kernel Bypass Is Not Enough · p. 3
  • Neither DPDK nor AF_XDP addresses transmit pacing or aggregation across multiple links.The application must decide when to send without visibility into the transmit chain's consumption.Source: F. Kernel Bypass Is Not Enough · p. 3

G. Last Resort: Bypass the Abstraction

UHD remote streaming lets samples reach a destination application, but leaves that application responsible for packet-level reception and its associated flow-control and configuration burdens. This fallback reflects the limits of packetize–reassemble transport: SDR relevance is defined by the host application, so the full stream must travel.

G. Last Resort: Bypass the Abstraction

  • Remote streaming sends samples to a destination application's managed socket while retaining the Rx streamer as a proxy for stream commands.It is available for reception only.Source: G. Last Resort: Bypass the Abstraction · p. 3
  • The destination must handle CHDR parsing and flow control to enable it; automatic MTU detection is unavailable, and an incorrect MTU causes fragmentation and data loss.Overruns are reported to the controlling UHD session rather than the receiving destination.Source: G. Last Resort: Bypass the Abstraction · p. 3
  • The authors describe remote streaming as the remaining option after the packetize–reassemble model reached its limit.Within that model, they say there is no other way to give applications the control they need at these rates.Source: G. Last Resort: Bypass the Abstraction · p. 3
  • Transmit-side remote streaming would impose hard real-time pacing: samples must reach the converter fast enough to avoid underflow and slowly enough to avoid overflow.The described burden includes flow-control messages, precise sub-millisecond scheduling, and packet pacing.Source: G. Last Resort: Bypass the Abstraction · p. 3

A. Precedent, and How It Differs

  • Unlike systems that reduce data through front-end selection, integration, or filtering, SDR relevance is defined by the host application.The transport must therefore carry the full sample stream, whether or not an application permits reduction ahead of transfer.Source: A. Precedent, and How It Differs · pp. 3–4
  • The authors are not aware of a public-record application of RDMA to SDR sample streaming.Related radio astronomy and CERN readout systems use RDMA but typically transmit data reduced before it reaches the network.Source: A. Precedent, and How It Differs · pp. 3–4

B. What RDMA Changes

RDMA moves packet handling and sample transfers out of the host CPU and into network interfaces, enabling samples to move directly between the SDR and registered host memory. RoCEv2 carries RDMA over Ethernet, with software RoCE available where hardware offload is absent but potentially insufficient for gigasample rates.

B. What RDMA Changes

  • RDMA moves data between the network interface and registered host memory by DMA, with packetization and reassembly performed in hardware rather than by the CPU.There is no per-packet system call, interrupt, or reassembly loop; samples are written where the application reads them.Source: B. What RDMA Changes · p. 4
  • The SDR writes received samples into registered host memory and reads transmit samples from it; the host CPU need not run a thread or call for either transfer.The host defines where samples live and when streaming starts, while the SDR determines when each transfer occurs.Source: B. What RDMA Changes · p. 4
  • Packetization and reassembly remain, but move into the network interface's dedicated logic, which performs them at line rate.Source: B. What RDMA Changes · p. 4

C. RoCEv2 in Practice

  • RoCEv2 traffic shares the interface, subnet, and switching fabric with other traffic, while DPDK poll-mode drivers remove the interface from the network stack unless hardware supports bifurcation.Source: C. RoCEv2 in Practice · p. 4
  • RoCEv2 offload is standard on 100 GbE-class interfaces; Linux software RoCE (rxe) works over any Ethernet device but reintroduces CPU cost and may not reach gigasample rates.The authors say software RoCE is likely sufficient at 10, 25, and 40 GbE.Source: C. RoCEv2 in Practice · p. 4

IV. AN RDMA-NATIVE STREAMING ARCHITECTURE

The host reserves and registers sample memory, then configures the SDR out of band with the region and stream schedule. During reception, the SDR sends packets that the host NIC reassembles into a contiguous, header-free sample stream in registered memory.

A. Setup and the Control Plane

  • Before streaming, the host reserves and registers memory, selects a region or subset, and configures the SDR with where to write received or read transmitted samples.Source: A. Setup and the Control Plane · p. 4
  • The SDR's network logic computes packet addresses and supports one-time, burst, or circular-buffer use, as well as continuous streaming through queued consecutive bursts.Source: A. Setup and the Control Plane · p. 4
  • Configuration occurs out of band, carries no samples, and avoids a full RDMA connection manager by writing protocol parameters directly into the SDR network logic.The host sets stream start and end by wall-clock time or event; after that, the network logic carries out the stream on its own.Source: A. Setup and the Control Plane · p. 4

B. Receive: The SDR Sends

  • For reception, the host NIC reassembles incoming packets by DMA into registered memory, where consecutive payloads form a contiguous sample stream without headers or host-memory copies.Source: B. Receive: The SDR Sends · pp. 4–5
  • The host can optionally receive and consume immediate-data notifications for every packet or a subset, while timestamps and stream state travel on a separate, low-rate path.Source: B. Receive: The SDR Sends · p. 5

C. Transmit: The SDR Reads

The SDR initiates transmit reads according to its converter-buffer needs, replacing host-driven pacing with requests whose timing adapts to buffer fill and network delay. This removes the application’s flow-control, scheduling, and packet-pacing tasks.

C. Transmit: The SDR Reads

  • The network logic requests samples from registered host memory when needed, and the host NIC DMA-reads, packetizes, and returns them for reassembly into the converter buffer.The host CPU does not participate in the transfer.Source: C. Transmit: The SDR Reads · p. 5
  • The network logic continuously adapts request rate and rhythm to converter-buffer fill, so no flow-control protocol is needed.The party that knows the buffer fill level decides when to transfer.Source: C. Transmit: The SDR Reads · p. 5
  • Measuring request round-trip time lets the network logic adjust subsequent requests to changing network conditions without intervention.Lead time becomes a measured quantity rather than a configured guess.Source: C. Transmit: The SDR Reads · p. 5
  • The application no longer needs flow-control messages, precise sub-millisecond scheduling, or transmitted-packet pacing.Source: C. Transmit: The SDR Reads · p. 5

D. Tracking the SDR, and Handling Loss

The section describes how the host tracks sample progress and handles loss, then extends the architecture across multiple links and streams. Address-based placement permits out-of-order arrivals while preserving a contiguous stream without copies.

D. Tracking the SDR, and Handling Loss

  • A synchronized host–SDR clock and configured sample rate let the host derive the SDR’s current position in registered memory.The host’s safety margin from the SDR contributes latency to real-time applications.Source: D. Tracking the SDR, and Handling Loss · p. 5
  • Distinguishable placeholders reveal progress as samples overwrite them and localize gaps without packet-header inspection or interrupt load.A placeholder remaining where samples were expected identifies a gap.Source: D. Tracking the SDR, and Handling Loss · p. 5
  • Loss handling depends on transport reliability: reliable transport can resend samples or substitute zeros, while unreliable transport fills receive gaps with zeros and may rerequest or substitute transmit samples.Zero substitution avoids retaining sent packets until acknowledgment and keeps the stream aligned in time under hard deadlines.Source: D. Tracking the SDR, and Handling Loss · p. 5

E. Scaling Beyond a Single Link

  • When a channel exceeds one link’s rate, address-carrying packets can be distributed across links and placed correctly regardless of arrival order, scaling bandwidth linearly without reintroducing copies.The network logic distributes packets of one sample stream across the available links.Source: E. Scaling Beyond a Single Link · p. 5
  • Link-to-NIC mappings can be many-to-many, and packet headers can route independent sample streams into separately assigned memory regions.Memory regions may be distributed across NICs, storage devices, or hosts for load distribution.Source: E. Scaling Beyond a Single Link · pp. 5–6

F. What the Application Is Left With

The application receives a sample-filled memory region rather than managing the transport, while the driver handles registered-memory setup. The outlook describes transport substitutions and extensions to address-free placement, peripheral memory, and in-network sample reduction.

F. What the Application Is Left With

  • The driver reserves and registers memory and settles hugepage backing, page locking, and NUMA placement before handing the region to the application.The driver can make these choices using knowledge of sample rate, link, and host topology.Source: F. What the Application Is Left With · p. 6
  • The application works with a memory region of samples rather than packets, and need not size rings, tune interrupts, pace transfers, or implement flow control.Source: F. What the Application Is Left With · p. 6
  • The NIC, kernel code, or SDR network logic performs transport work, leaving application authors to focus on signal processing.The article says correctly implemented streaming at multi-gigasample rates is within reach of a few lines of Python.Source: F. What the Application Is Left With · p. 6

V. OUTLOOK

  • iWARP and InfiniBand can substitute for RoCEv2 if they DMA packetized samples to or from preregistered memory, place consecutive payloads contiguously without headers, and avoid host-memory copies.Source: §V · p. 6
  • Address-free placement can use packet chronology or sequence numbers instead of carrying each payload’s destination address.A scatter-gather NIC can separate headers from payloads and place adjacent payload segments sequentially without a copy.Source: §V · p. 6
  • Registered memory may reside on an accelerator or nonvolatile storage device, allowing direct NIC-to-peripheral transfer over PCIe peer-to-peer.In this arrangement, the bandwidth accounting no longer involves host memory.Source: §V · p. 6
  • Filtering, decimation, integration, buffering, or selective transmission in the network logic can reduce the amount of stream transferred when the application permits sample reduction.These operations change the carried amount, not the transport architecture.Source: §V · p. 6

VI. CONCLUSION

The conclusion argues that the existing transport exceeds host packet-processing and memory-bandwidth limits, while the proposed architecture moves sample transfer into hardware-managed registered memory. It identifies transmit pacing as a central burden, but notes that comparative measurement remains future work.

VI. CONCLUSION

  • At gigasample rates, the host cannot process packets within their time budget or reassemble tens of gigabytes per second without consuming memory bandwidth needed by the application.The conclusion says neither limit is a configuration problem and kernel bypass addresses only packet processing.Source: §VI · p. 6
  • The SDR writes received samples into host-registered memory and reads transmit samples when its converter needs them, with each sample byte crossing the memory bus once.The host defines the memory location and when streaming begins, then leaves the data path.Source: §VI · p. 6
  • Having the SDR request transmit samples removes the host’s hard real-time buffer-pacing problem and its associated flow-control, scheduling, and pacing work.The conclusion calls the transmit path the more consequential of the two.Source: §VI · p. 6
  • The architecture is implemented as a proof of concept, while comparison with the transports across the front ends remains ongoing work.Source: §VI · pp. 6–7
Loading 2609.05623v1…