PaperlayerView exact revision on arXiv

Source-linked AI summary

LatencyLab: A DPDK-Based P4 Pipeline Latency Measurement Framework for FPGA SmartNICs

Pavani Kuppili, Zhaoyang Han, Yicheng Qian, Suranga Handagala, Michael Zink, Miriam Leeser, Robert Ricci

arXiv:2609.39978v1cs.DC

TL;DR

Open FPGA P4 toolflows lack pipeline-boundary timestamping, leaving the latency added by P4 programs rarely measured. LatencyLab uses differential packet timing on an FPGA SmartNIC, and its measurements show tight, reproducible distributions that agree with independent hardware-clock and vendor-model checks.

  • Problem

    Open FPGA P4 datapaths lack pipeline-boundary timestamps and synchronized clocks, so the latency a P4 program adds is rarely measured.

    Source: §I · p. 1

  • Method

    LatencyLab measures the arrival-time difference between probe copies traversing matched P4 and bypass paths, using a DPDK-based receiver and null-bitstream calibration.

    Source: §I · p. 1, B. Why Open-NIC-Shell · p. 2, §VI · p. 7

  • Results

    For each reported program or FiveTuple latency mode, the 99th-percentile latency is at most 22 ns above the median, and single-clock program session medians reproduce to 1–2 ns across bring-ups.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The measurements show LatencyLab can characterize FPGA P4 latency without datapath timing support, with results corroborated by independent checks.

    Source: §VI · p. 7

    Characterizing queueing at high offered load, probing FiveTuple’s state ladder across many resets, and porting to other boards remain future work.

    Source: §VI · p. 7

Abstract

from arXiv · show

P4-programmable FPGA SmartNICs place packet processing directly on the wire, but open FPGA P4 toolflows do not expose timestamping at the pipeline boundary, so the latency a P4 program adds on the target FPGA is rarely measured. This paper presents LatencyLab, a DPDK-based measurement framework for FPGA P4 pipeline latency that needs neither PHC/PTP support on the datapath nor clock synchronization. The FPGA's two ports share a network segment, so the switch multicasts a copy of each probe packet to both: one copy passes through the VitisNetP4 pipeline, the other through a matched bypass path. A kernel-bypass DPDK receiver busy-polls both ports and timestamps every packet with the CPU timestamp counter (TSC) as it is retrieved from the NIC's receive circular buffer. The arrival-time difference of the two copies isolates the pipeline latency after calibration against a null bitstream carrying the same traffic; transmit time cancel in the subtraction. We evaluate four VitisNetP4 programs on an AMD Alveo U280, probing each with a 20,000-packet trace measured ten times per session over five independent sessions, all TSC-timestamped and reflected for hardware timestamping. The measured latency distributions are tight and reproducible: 99% of packets fall within 20 ns of the median, session medians repeating within 1 to 2 ns (FiveTuple 107/137 ns, Forward 149 ns, RemoveHeader 177 ns, Checksum 364 ns at 250 MHz). Two independent checks agree with the framework: a kernel-free reflector returns every probe pair to a ConnectX-5 NIC whose adapter clock reproduces the measured distributions within a few nanoseconds, quantile by quantile, and every measured packet falls 18 to 22 clock cycles below the vendor's worst-case latency bound.

I. INTRODUCTION

Open FPGA P4 datapaths lack the timing support needed to measure pipeline latency, while host-side timestamps can be obscured by PCIe, kernel, and quantization effects. LatencyLab measures latency differentially and reports reproducible distributions for four programs, with independent checks against hardware timestamps and the vendor model.

I. INTRODUCTION

  • Open FPGA P4 datapaths lack pipeline-boundary timestamps and synchronized clocks, while host timing can be dominated by PCIe, kernel, and quantization effects.Prior work used custom timestamping overlays, separate stamper FPGAs, or specialized CPU–NIC integration.Source: §I · p. 1
  • LatencyLab estimates P4 pipeline latency from the arrival-time difference between multicast probe copies sent through the P4 path and a matched bypass path.A null bitstream measures and removes the constant path asymmetry; shared transmit time and network path cancel in the subtraction.Source: §I · pp. 1–2
  • Independent NIC hardware timestamps and the vendor worst-case model both agree with LatencyLab’s estimates.The hardware-clock check reflects probe pairs without sending them through a kernel network stack; verification is needed once, not for each deployment.Source: §I · p. 2
  • Four VitisNetP4 program latency distributions were reproducible to 1–2 ns across independent hardware bring-ups.The paper also reports two reset-dependent FiveTuple modes approximately 31 ns apart, consistent with its separately clocked CAM.Source: §I · p. 2
  • FiveTuple showed two observed session-level latency modes approximately 31 ns apart, consistent with its separately clocked CAM.The paper presents this as a case study made visible by LatencyLab’s packet-level resolution.Source: §I · p. 2

II. BACKGROUND

The background describes the platform and testbed used to measure FPGA P4 pipeline latency. The measurements ran on bare-metal servers connected through a datacenter switch fabric, with an Alveo U280 under test.

A. Testbed

  • The testbed connects two bare-metal servers through a top-of-rack switch, with the receiver hosting an AMD Alveo U280 and two 100 GbE switch links.The sender replays probe traffic from a ConnectX-5 NIC; the receiver also has a ConnectX-5 NIC.Source: A. Testbed · p. 2
  • The receiver used a dual-socket Intel Xeon Gold 6226R host running Ubuntu 22.04 LTS, with DPDK polling cores pinned to one processor socket.The setup used Open-NIC-Shell, open-nic-dpdk, DPDK 20.11, VitisNetP4 2023.2, and Vivado 2023.2.Source: A. Testbed · p. 2

B. Why Open-NIC-Shell

Open-NIC-Shell supplies a two-path FPGA NIC architecture and an open DPDK polling path used by LatencyLab. The evaluation compares four VitisNetP4 programs with distinct packet-processing behavior.

B. Why Open-NIC-Shell

  • Open-NIC-Shell’s QDMA subsystem has an open DPDK poll-mode driver, enabling packet timestamps at NIC retrieval without kernel involvement.The paper selects the shell as an open-source deployment path for AMD FPGAs and shared research testbeds.Source: B. Why Open-NIC-Shell · p. 2
  • The VitisNetP4 and bypass receive paths share the user-logic clock, QDMA subsystem, and PCIe link, differing architecturally only by the P4 pipeline.This makes common path components cancel in a differential comparison.Source: B. Why Open-NIC-Shell · p. 2

C. P4 Programs Under Test

  • FiveTuple performs a CAM lookup on the extracted 5-tuple; its CAM clock is 300 MHz alongside the 250 MHz packet-pipeline clock.That separate clock is noted as relevant to the later latency case study.Source: C. P4 Programs Under Test · p. 2
  • Forward parses common Ethernet and transport headers and forwards on an IPv4 destination match; RemoveHeader strips VLAN, while Checksum also computes and matches the IPv4 checksum.Checksum is the deepest of the four pipelines; a null bitstream with bypass plugins on both ports provides the calibration reference.Source: C. P4 Programs Under Test · p. 2

D. What Host-Path Timing Shows: PCIe and Kernel Dominate

Host-visible loopback timing operates at microsecond scale, where added host crossings inflate both median and spread and leave the roughly 0.1–0.4 µs pipeline contribution unresolved.

D. What Host-Path Timing Shows: PCIe and Kernel Dominate

  • The preliminary setup replays packets from a cNIC to FPGA ports on the same host, then closes the loop through either hardware reflection or a software VLAN bridge.The hardware-reflection path includes two PCIe crossings; the software-reflection path adds another PCIe traversal and kernel network stack, for four crossings total.Source: D. What Host-Path Timing Shows: PCIe… · p. 2
  • Both loopback RTT distributions operate at microsecond scale, and the approximately 0.1–0.4 µs pipeline contribution is unresolved in either.Adding host crossings visibly inflates both the median and spread.Source: Figure 3 · p. 3, D. What Host-Path Timing Shows: PCIe… · p. 3

III. THE LATENCYLAB FRAMEWORK

LatencyLab estimates FPGA P4 pipeline latency by subtracting paired arrival times from matched pipeline and bypass paths, using DPDK TSC timestamps and null-bitstream calibration. Its probes are spaced to measure isolated transit latency rather than load-dependent queueing.

A. Overview

  • LatencyLab timestamps paired copies of each probe at both FPGA ports and subtracts their arrival times, removing shared transmit and network delay while retaining pipeline latency and port asymmetry.A null bitstream measures the remaining asymmetry for subtraction rather than assuming it is negligible.Source: §III · p. 3, A. Overview · p. 3

III. THE LATENCYLAB FRAMEWORK

  • The framework combines differential measurement with a DPDK receiver that timestamps packets using the CPU timestamp counter, while null-bitstream calibration removes the constant path asymmetry.These address the need to cancel shared path components and timestamp before kernel effects can distort packets.Source: §III · p. 3

B. Sender and Probe Traffic

  • Probes are sent 2 ms apart so each is isolated through the switch, pipeline, DMA engines, and poller, avoiding queueing delay that depends on load.The slow rate also lets the poller handle each copy in its own DMA burst and timestamp it immediately.Source: B. Sender and Probe Traffic · pp. 3–4

C. Kernel-Free Receiver

The receiver timestamps packets immediately after dequeue with synchronized host TSCs; separate reflection and null-bitstream procedures verify measurements and calibrate path asymmetry. The experiment repeats runs across independent hardware instantiations.

C. Kernel-Free Receiver

  • Each port is busy-polled on a dedicated core, with rte_rdtsc_precise() called immediately after packet dequeue and timestamps taken on synchronized invariant TSCs.TSC cycles are converted using DPDK’s reported frequency; any fixed intercore offset is included in calibrated asymmetry.Source: C. Kernel-Free Receiver · p. 4

D. Reflect Framework for Verification

  • A DPDK reflection path sends tagged probe copies to a ConnectX-5 for hardware receive timestamping, while fixed per-path TX queues keep queue differences as constant offsets.This reflector is for verification only; deployed LatencyLab does not require a hardware-timestamping NIC.Source: D. Reflect Framework for Verification · p. 4

E. Null Calibration

  • Null-bitstream calibration measures and removes port-and-queue asymmetry, with offsets of 14–28 ns and session-to-session spread under 7 ns.The same probe trace is replayed with both paths set to passthrough.Source: E. Null Calibration · p. 4

IV. MEASUREMENT MODEL

  • Each program is measured with 20,000 probes over ten runs per session and five independent sessions, with both paths observed concurrently.The protocol is repeated with the null bitstream, and every probe is TSC-timestamped and reflected for hardware timestamping.Source: §IV · p. 4

A. Differential Estimator

The estimator subtracts paired TSC arrival times to remove shared delay, then uses null-bitstream measurements to remove constant path asymmetry. It reports the distribution of calibrated per-probe estimates across probes and sessions.

A. Differential Estimator

  • The paired arrival-time difference retains P4 pipeline latency, constant path asymmetry, and measurement noise after shared transmit and path delays cancel.The two timestamps are taken for copies of the same probe.Source: A. Differential Estimator · pp. 4–5
  • The null-bitstream differential measures the constant asymmetry, and subtracting its median from the program differential leaves pipeline latency plus noise.The median of ten repeated runs limits the influence of atypical polling, DMA, or PCIe delays.Source: A. Differential Estimator · p. 5
  • Latency is reported as the distribution of per-probe estimates over all probes and sessions.The estimate aggregates repeated-run differentials after null calibration.Source: A. Differential Estimator · p. 5

B. Verifying the Framework

The framework is evaluated in two independent ways, with results presented in Sec. V-A.

1) Hardware-clock check:

LatencyLab’s distributions agree with an independent hardware-clock check, and all retained estimates remain below the vendor’s worst-case bounds. The results are narrow and reproducible, with an exception for the first probe after an idle gap.

1) Hardware-clock check:

2) Vendor-model check:

  • 53, 55, 66, and 109 cycles at 250 MHz are the worst-case bounds for FiveTuple, Forward, RemoveHeader, and Checksum, respectively.After excluding the first probe following an idle gap, all retained estimates remain below their corresponding bounds, with p99 margins of 12–18 cycles.Source: 2) Vendor-model check · p. 5, A. Latency of the Four Programs · p. 6

A. Latency of the Four Programs

  • p5–p99 spans are 31–36 ns, p99 is within 22 ns of the median, and session medians repeat within 1–2 ns across bring-ups.Fig. 4 and Table I compare LatencyLab estimates, hardware-clock measurements, and vendor bounds; after excluding startup probes, all retained estimates stay below their bounds.Source: A. Latency of the Four Programs · pp. 5–6, Figure 4 · p. 5, Table I · p. 6

B. Special Case: FiveTuple’s Two Latency States

FiveTuple exhibits two session-level latency modes, with a roughly 31 ns step between them. The observed pattern is consistent with its separately clocked CAM and is fixed at FPGA instantiation.

  • Four sessions sit at 105–107 ns, while session 1 sits at 137 ns, a step of about 31 ns with no intermediate values.The hardware-clock reflector saw the same split, and the state is chosen at instantiation and remains until the next flash and reset.Source: B. Special Case: FiveTuple’s Two… · p. 6, Figure 6 · p. 6
  • FiveTuple’s CAM runs at 300 MHz alongside the 250 MHz packet-pipeline clock, and the table is applied from two call sites.The proposed clock-domain crossing mechanism predicts a bring-up-specific latency that remains constant within a session and is absent from the three single-clock programs.Source: B. Special Case: FiveTuple’s Two… · p. 6
  • Packet-level differentials and session-structured measurements resolve the two stable per-reset latencies observed in the measured sessions, a distinction relevant to latency budgeting.Source: B. Special Case: FiveTuple’s Two… · pp. 6–7

VI. CONCLUSION

LatencyLab measures FPGA SmartNIC P4 pipeline latency at packet-level resolution without datapath PHC support, cross-device PTP synchronization, or kernel timestamping. On the U280, four programs show tight, reproducible estimates, with two independent checks agreeing with the framework.

  • A paired-port arrival-time differential cancels shared path components, while a null bitstream measures and removes the constant port-and-queue offset.The DPDK receiver timestamps both ports with one invariant clock at the NIC’s receive circular buffer.Source: §VI · p. 7
  • For four U280 programs, p99 latency is at most 22 ns above the median, and single-clock-program session medians repeat within 1–2 ns across bring-ups.A NIC hardware clock reproduces the distributions, and the vendor’s worst-case model bounds every estimate with similar headroom across all four programs.Source: §VI · p. 7
  • Future work includes characterizing queueing onset at high offered load, probing FiveTuple’s state ladder across more resets, and porting to other Open-NIC-Shell-class boards.Source: §VI · p. 7
Loading 2609.39978v1…