Source-linked AI summary
High Speed Network Processing for Scientific Computing in the Open Cloud Testbed
Miriam Leeser, Michael Zink, Suranga Handagala, Carlos Ruben Dell'Aquila, Verena Martinez Outschoorn, Rafael Coelho Lopes de Sa
TL;DR
Scientific workflows need high-bandwidth data movement and network visibility that connects traffic with scientific context. The paper implements FPGA-based SmartNIC functions in OCT, including SciTags packet monitoring, and reports identical processing results with imperceptible added delay and line-rate handling. The experiments demonstrate preliminary feasibility on AMD U280 FPGAs.
Problem
Data-intensive science faces growing wide-area bandwidth demands, while network visibility, optimization, and orchestration must support scientific workflows.
Source: §I · p. 1Method
The paper develops FPGA-based data-plane functions for packet parsing, SciTags marking and monitoring, telemetry, traffic steering, and related network services.
Source: §I · p. 1, §III · p. 3Results
FPGA processing produced results identical to bypass processing, with imperceptible delay at 40Gb/s and 100Gb/s and fewer retransmissions at high data rates.
Source: §I · p. 2, §III · p. 6Takeaways & Limitations
The work establishes a foundation for coordinating FPGA data-plane and control-plane capabilities for scientific traffic, without requiring host-CPU intervention for every packet.
Source: §IV · p. 6The reported experiments were conducted on AMD U280 FPGAs; extension to AMD V80s is planned.
Source: §IV · p. 6
Abstract
from arXiv · showhide
This paper presents an FPGA-based SmartNIC architecture designed to address critical network challenges in data-intensive scientific computing. As scientific workflows increasingly demand high-bandwidth data movement and real-time monitoring across wide-area networks, the Open Cloud Testbed (OCT) provides an ideal platform for investigating in-network processing solutions at line rate. Our approach focuses on implementing network visibility, optimization, and orchestration capabilities through FPGA acceleration. The proposed system leverages P4-programmable data-plane functions to support packet parsing, marking, telemetry extraction, traffic steering, and machine learning inference, all without requiring host CPU intervention or modifications to application servers. As a concrete example, our framework integrates the SciTags standard, which enables associating network traffic with scientific context through packet marking. We implement a high-performance network bridge that counts packets by their SciTag values, as identified in the IPv6 Flow Label. Experimental evaluation on AMD U280 FPGAs in OCT demonstrates the feasibility of this approach, showing imperceptible processing latency and line-rate packet handling, while accurately monitoring scientific data flows.
I. INTRODUCTION
Data-intensive science faces rising wide-area bandwidth demands, while visibility, optimization, and orchestration must work together to support scientific workflows. The paper explores FPGA-based network processing in OCT, including SciTags monitoring that operates at line rate with minimal added latency.
I. INTRODUCTION
- Data-intensive science and scientific AI workflows are driving demand for sustained wide-area bandwidth and low-latency, high-bandwidth data movement.Major computing sites are preparing for sustained traffic in the 400 Gb/s region and beyond.Source: §I · p. 1
- Visibility supplies labels and telemetry for optimization, which informs orchestration policies for scientific workflows.These capabilities address needs ranging from identifying traffic owners and purposes to managing shared bandwidth and requesting reserved paths.Source: §I · p. 1
- OCT FPGAs at the data-transfer edge can inspect, classify, mark, pace, and steer traffic at line rate without server changes or direct control of shared routers.The paper positions this edge processing as a way to investigate in-network processing on OCT’s network-connected FPGAs.Source: §I · p. 1
- The proposed platform brings together SciTags monitoring, telemetry, pacing, congestion management, QoS, dynamic IP/VLAN handling, and compact AI/ML inference.The project targets firmware-centric AMD/Xilinx SmartNICs using P4-based packet parsing, marking, telemetry, local steering, and ML control hooks.Source: §I · p. 1
- FPGA processing produced results identical to processor-only processing, with minimal extra latency and line-rate handling in preliminary experiments.The experiments used SciTags-compatible IPv6 packet markings on FPGAs directly connected to the network.Source: §I · p. 2
A. SciTags & SENSE
SciTags provide standardized scientific context for network traffic, while SENSE illustrates how workflow intent can be translated into end-to-end network services. FPGA-based endpoints are described as potential enforcement and telemetry points for these capabilities.
A. SciTags & SENSE
- SciTags associate traffic with standardized community and activity identifiers so network measurements can be correlated with experiments, production campaigns, or analysis activities.The tags are conveyed through packet marking to address the gap left by aggregate counters and sampled flow records.Source: A. SciTags & SENSE · p. 2
- FPGA SmartNICs can parse tags, timestamp packets, maintain per-class counters, and generate streaming summaries with limited host-CPU involvement.Summarized measurements can be exported to software services for correlation, anomaly detection, and longer-timescale diagnosis.Source: A. SciTags & SENSE · p. 2
- SENSE provides an intent-based model for end-to-end services, and FPGA-based endpoints or edge devices could enforce reservations and return telemetry.The passage also describes work connecting Rucio with SENSE to translate transfer priorities into bandwidth allocation.Source: A. SciTags & SENSE · p. 2
B. P4 on FPGAs
The paper uses OpenNIC as its FPGA-based SmartNIC foundation because it is optimized for AMD FPGAs.
B. P4 on FPGAs
- FPGA programmability allows developers to customize packet paths and create high-performance computing units.Source: B. P4 on FPGAs · p. 2
- OpenNIC and Corundum provide open-source FPGA NIC hardware designs and corresponding Linux drivers.Both designs connect Ethernet through MAC and Physical Interface Layers and connect to the host CPU through DMA.Source: B. P4 on FPGAs · p. 2
- The project employs OpenNIC because it is optimized for AMD FPGAs.The passage also notes an OpenNIC-based SmartNIC framework tailored for deep neural networks.Source: B. P4 on FPGAs · p. 2
C. OCT
OCT provides researchers with network-connected AMD FPGAs and has expanded its hardware and workflows to support varied FPGA projects. Its current resources connect to regional and national research networks.
C. OCT
- OCT provides bare-metal research servers and programmable FPGAs directly connected to the network.The testbed’s goal is to provide bare-metal servers to the research community.Source: C. OCT · p. 2
- OCT expanded from eight U280 accelerators to 24 and introduced a unified workflow for Vitis and custom FPGA flows.An automated golden-shell reset leaves the next user with a clean FPGA and reduces manual intervention between experiments.Source: C. OCT · p. 2
- OCT added Versal-based VCK5000 and V70 accelerators with AI engines for workloads such as object detection, image classification, and video analytics.Source: C. OCT · p. 3
- OCT currently provides 34 AMD FPGAs: 24 U280, four VCK5000, four V70, and two V80 accelerators.The testbed connects to US ATLAS Northeast Tier 2 and ESNet at 400 Gb/s, and to the NSF FABRIC research infrastructure.Source: C. OCT · p. 3
D. V80s in OCT
The V80 combines high-bandwidth memory and 800 Gb/s of aggregate network interfaces with hardened SoC functions that leave more FPGA fabric available for user logic.
D. V80s in OCT
- The V80 provides 32 GB of HBM2e memory with 820 GB/s peak bandwidth and four 200G QSFP56 ports totaling 800 Gb/s.The interfaces are described as supporting direct line-rate transmission of large data streams.Source: D. V80s in OCT · p. 3
- Compared with the U280’s UltraScale+ architecture, Versal Adaptive SoC provides hardened IP blocks that reduce LUT and routing use while improving performance and power efficiency.Examples include cryptographic accelerators and IEEE 1588 PTP engines.Source: D. V80s in OCT · p. 3
- Integrating those hardened resources leaves more FPGA fabric available for user logic.Source: D. V80s in OCT · p. 3
E. Build Machines
OCT moved its FPGA build tools into CloudLab to simplify a development workflow previously split between build and host environments.
E. Build Machines
- Moving build tools from MOC/NERC to CloudLab puts build and host nodes in one environment.Source: E. Build Machines · p. 3
- The former split required separate user accounts and synchronization of FPGA bitstreams and host executables across network domains.Source: E. Build Machines · p. 3
- OCT’s bare-metal build machines provide over 2 TB of aggregate memory, 32 Ryzen 5950X cores, and 128 EPYC 7C13 cores for building and emulating FPGA projects.CloudLab profiles let users provision virtual machines with specified CPU cores and memory.Source: E. Build Machines · p. 3
III. APPLICATIONS
The OCT application uses an FPGA bridge to count IPv6 packets by SciTag across two networks. Tests report matching packet counts, imperceptible added delay at 40 and 100 Gb/s, and fewer retransmissions at high rates.
III. APPLICATIONS
- The FPGA bridge counts IPv6 packets exchanged between two networks according to their SciTag value.The implementation and evaluation took place at MGHPCC, which houses the ATLAS NET2 cluster and OCT.Source: §III · p. 3
- A central FPGA node bridges external nodes on separate VLANs, counts packets by SciTag, and supports a regular-NIC throughput test as a reference.Source: §III · p. 3
- The evaluation uses OCT’s 40 Gb/s, 1500-byte-MTU NICs and 100 Gb/s, 9000-byte-MTU NICs to measure throughput degradation.Source: §III · p. 5
- Throughput testing compares regular-NIC reference traffic with traffic through the U280 bridge using the same NIC models and iperf3.The reference cases are 40 Gb/s and 100 Gb/s.Source: §III · p. 5
- The counter validation compares FPGA and host-application counts with packet numbers obtained by offline analysis of saved packets.Source: §III · pp. 5–6
- FPGA-processed packets produced exactly the same counter results as packets bypassing the FPGA; at 40 and 100 Gb/s, processing added imperceptible delay and the FPGA design had fewer retransmissions at high rates.Source: Table III · p. 6, Table II · p. 6, §III · p. 6
- The authors plan to move all processing to V80s and connect a 400 Gb/s interface; they expect the FPGA to handle that traffic at line rate and preprocess data before the processor.Source: §III · p. 6
IV. CONCLUSIONS
The work demonstrates an FPGA data plane for SciTags insertion and monitoring plus programmable VLAN manipulation, while the corresponding policy-driven control plane remains a next-stage development.
IV. CONCLUSIONS
- FPGA firmware supports high-throughput SciTags insertion and monitoring, programmable VLAN manipulation, and packet classification and modification for data-intensive physics applications.DSCP marking and processing are under development to extend traffic prioritization and quality-of-service policies.Source: §IV · p. 6
- The planned control plane will replace static global SciTags, VLAN, and DSCP configurations with configurable policies, initially scoped to IPv6 prefixes or subnets.It will map scientific activities to data-plane actions and associate SENSE-provisioned services with VLANs.Source: §IV · p. 6
- A programmatic interface will expose policy capabilities and firmware telemetry, including per-activity packet and byte counters, to monitoring systems and network orchestrators.The separation keeps FPGA packet processing deterministic while policies are created and retired dynamically in software.Source: §IV · p. 6
- Experiments used AMD U280 FPGAs in OCT, with planned extension to AMD V80s, which offer higher-speed networking.The project demonstrates OCT's flexibility for scientific applications and in-network processing.Source: §IV · p. 6