PaperlayerView exact revision on arXiv

Source-linked AI summary

PCM: Programmable Congestion Management for Terabit RDMA NICs

Mikhail Khalilov, Timo Schneider, Tommaso Bonato, Siyuan Shen, Marcin Chrapek, Yehor Kozyr, Alberto Merino, Rip Sohan, Surendra Anubolu, Mohan Kalkunte, Ahmad Ghalayini, Abdul Kabbani, Torsten Hoefler

arXiv:2610.13130v1cs.NI

TL;DR

Workload-specific congestion policies are difficult to reuse because programmable RDMA engines bind code to platform APIs, and no common interface combines portability with efficient execution across vendors and transports. PCM separates event-driven policy handlers from platform integration, and spans eleven policies and four datapaths, with 1.56 Tbit/s goodput in UCCL and near-per-ACK processing capacity on 400G and 800G NICs.

  • Problem

    Programmable RDMA engines tie policy code to particular architectures, transports, or vendors, and no common interface combines source portability with efficient execution across them.

    Source: §1 · p. 1

  • Method

    PCM separates portable, event-driven policy handlers from platform integration through control objects, signals, triggers, and optimized backends.

    Source: §1 · pp. 1–2

  • Results

    PCM spans eleven policies and four datapaths, sustains 1.56 Tbit/s goodput in UCCL, and achieves near-per-ACK processing capacity on vendor 400G and 800G NICs.

    Source: §1 · p. 2, §8 · p. 12

  • Takeaways & Limitations

    PCM's design informed ongoing Ultra Ethernet standardization toward a portable congestion-management interface.

    Source: §8 · p. 12

    Exact per-packet cache updates require coherent event delivery and sufficient compute resources, as in synchronous simulator validation.

    Source: §5.4 · p. 6

Abstract

from arXiv · show

Datacenter congestion control (CC) and load balancing (LB) evolve rapidly as workload demands change, but deployment depends on RDMA hardware and transport interfaces. Emerging programmable RDMA engines tie policy code to vendor APIs. Operators must therefore port each policy for every platform they deploy. We address this with PCM, a programming model that separates portable congestion management from platform integration on datacenter NICs. Developers write event-driven handlers that they can debug in simulation and reuse on real hardware. PCM exposes flow telemetry as signals and uses triggers to support per-event reactions and coarser responses to accumulated feedback. Vendors provide optimized backends that map signals and output controls to native NIC mechanisms with defined feedback and control semantics. We demonstrate PCM's expressivity by implementing eleven well-known datacenter CC and LB algorithms. We deploy PCM across four distinct datapaths: an Ultra Ethernet simulator, CPU-based UCCL, and on-NIC offloads on NVIDIA ConnectX-8 and Microsoft 1P. On CX-8, PCM-Swift achieves processing capacity equivalent to one reaction per three 4 KiB packets at 800 Gbit/s, using one-fifth of the NIC's hardware threads. In UCCL, it matches native Swift's 1.56 Tbit/s goodput with 64 KiB chunk-level control. PCM's design informed ongoing Ultra Ethernet work toward a portable congestion management interface.

1 Introduction

Workload-specific congestion control and load balancing evolve faster than ASIC redesign cycles, while programmable RDMA engines still tie policy code to vendor or transport APIs. PCM separates policy development from platform integration through portable handlers and optimized backends.

1 Introduction

  • Workload-specific policies evolve faster than RDMA transport ASICs can be taped out, tested, and deployed; programmable engines still bind code to particular architectures, transports, or vendors.The paper notes that silicon deployment takes multiple years and cannot keep pace with changing AI models, topologies, or congestion signals.Source: §1 · p. 1
  • PCM separates policy development from platform integration using portable control objects and optimized platform backends.Handlers retain their source across supported targets, while deployment parameters and behavioral validation remain target-specific.Source: §1 · pp. 1–2
  • Native accessors, parallel per-flow execution, and accumulated feedback let portable handlers approach bare-metal performance while supporting fine or coarser reactions.Hardware-aware signal accumulation decouples handler execution from event arrival rates.Source: §1 · p. 2
  • PCM derives expressivity, portability, and performance requirements from workload needs and existing mechanisms, and defines hardware-friendly abstractions for per-event and coarser reactions.Source: §1 · p. 2
  • Eleven policies span UET-htsim, UCCL, NVIDIA CX-8, and Microsoft 1P, with full congestion-control offload.PCM implements eleven datacenter CC and LB algorithms and shell generators for four platforms.Source: §1 · p. 2
  • PCM's design informed ongoing standardization work toward programmable congestion management for Ultra Ethernet.Source: §1 · p. 2

2 Motivation

Three use cases motivate reusable congestion policies: incast-aware tuning, mixed load-balancing requirements, and destination-specific congestion control. The examples show how policy choices affect flow completion time and delivery behavior.

2 Motivation

  • 3× maximum FCT speedup came from smaller incast-tuned initial windows, which avoided 98.6% of baseline packet drops.The comparison used MP-RDMA in UET-htsim across ten configurations and ten endpoint mappings per configuration.Source: §2 · p. 2, Figure 2 · p. 2
  • REPS+ECMP improved maximum FCT by 1.51–2.61× for main traffic and 1.48–1.80× for background traffic versus cluster-wide ECMP.REPS controlled 115 of 128 flows, while ECMP served 13 background flows to avoid multipath-induced reordering.Source: §2 · p. 2, Figure 2 · p. 2
  • Per-flow policies can match congestion control to destination: Swift for low-latency intradatacenter traffic and BBR for delay-tolerant inter-datacenter traffic.Source: §2 · p. 2
  • Table 1 distinguishes mechanisms by whether policies run within or independently of the datapath and by their required interfaces and execution support.Source: Table 1 · p. 3

3 Related Work

Existing congestion-management mechanisms bind policy code to hardware, transport interfaces, or host runtimes. PCM addresses these dependencies with portable CC/LB source and native execution, while retaining platform-specific mappings.

3 Related Work

  • PCM targets source portability across substrates, defined signal and control semantics, and native execution on embedded NIC cores.This extends policy reuse from simulation to NIC deployment.Source: §3 · p. 3
  • Hardware and transport-specific systems expose native events, state, and controls, but policy changes can require synthesis or remain tied to vendor interfaces.Examples include FPGA and P4-like designs, DOCA PCC callbacks, and Falcon's asynchronous transport engine.Source: §3 · p. 3
  • Language and runtime dependencies also limit reuse: eBPF, Presto, and UCCL policies rely on substrate-specific callbacks, datapaths, classes, or APIs.Presto's and UCCL's C++ policies cannot compile with the DOCA PCC C11 compiler.Source: §3 · p. 3
  • CCP preserves source across host datapaths but routes policy updates through a separate user-space process, adding delays incompatible with fresh per-packet decisions.On the evaluated Xeon Gold 6140, a NewReno cycle takes 10 µs, versus a 20 ns 4 KiB packet arrival interval at 1.6 Tbit/s.Source: §3 · pp. 3–4
  • PCM's shell accumulates events, evaluates triggers, snapshots state, executes handlers, and commits outputs while mapping named controls to native mechanisms.Source: Figure 3 · p. 4

4 Design Requirements

PCM's design requirements follow from diverse workload needs and platform differences in interfaces, feedback, and execution capacity. The model must express varied policies, support source reuse, and execute efficiently without assuming every platform can provide identical behavior.

4 Design Requirements

  • Expressivity requires support for sender-side congestion control, receiver-side credit control, and stateful load balancing.Source: §4 · p. 4
  • Portability requires separating handler logic from native APIs, state layouts, measurement units, and arithmetic, while validating each deployment's behavior.Backends must reject handlers when a platform cannot provide a required signal or enforce a required control.Source: §4 · p. 4
  • Performance requires low overhead and scalable execution across independent flows, using native feedback granularity where possible and coarser reactions when validated.Source: §4 · p. 4
  • The paper examines these requirements through algorithm coverage, policy reuse across transports, and measurements of execution cost, goodput, and scaling.Source: §4 · p. 4

5 PCM Programming Model

PCM expresses congestion-management policies as portable, stateful per-flow handlers, while platform shells translate their signals and controls into native mechanisms. Its abstractions support synchronous and asynchronous execution, but accumulated feedback can limit the fidelity of some per-packet behaviors.

5 PCM Programming Model

  • PCM handlers are stateful functions over per-flow state, independent of hardware details, and may combine non-overlapping controls from multiple handlers.Matching flow events to handler contexts remains outside PCM and depends on the transport, steering engine, and addressing.Source: §5 · p. 4

5.1 Unified Datapath Abstractions

  • Signals expose telemetry through SUM, MIN, MAX, or LAST accumulators, while triggers evaluate accumulated values against thresholds to invoke handlers.RAW, DELTA, and MAGNITUDE comparisons support reactions ranging from per-packet events to less frequent threshold-driven responses.Source: §5.1 · pp. 4–5

5.2 Handler Compilation

  • A handler package combines C source and a specification that a platform backend compiles into a shell mapping events to signals and controls to native mechanisms.Compilers can fold accesses into native loads and stores; shells cannot restore unavailable capabilities, and the compiler checks interface support before deployment.Source: §5.2 · p. 5

5.3 Handler Execution Modes

  • Synchronous handlers execute on the flow-event path, whereas asynchronous handlers run in a separate control plane; relaxed snapshots preserve per-signal order but may mix feedback epochs.The proposed bounded-skew approach uses sequence numbers to discard over-skew snapshots; the evaluated asynchronous CPU shell implements relaxed and zero-skew snapshots.Source: §5.3 · pp. 5–6

5.4 Real-World Handler Examples

  • Examples span ACK/NACK-based sender congestion control, EQDS receiver credit pacing, and REPS path caching, whose exact per-packet updates require coherent event delivery and sufficient compute.REPS uses a LAST path signal, which cannot preserve all ACK-to-path associations in a coalesced batch.Source: §5.4 · p. 6

6 Implementation

PCM is implemented through platform-specific shells for CPU-based UET-htsim and UCCL, ConnectX-8, and Microsoft 1P. The integrations use different execution paths and native controls tailored to their targets.

6 Implementation

  • The platform integrations comprise approximately 600 lines for htsim, 1000 for UCCL, 2500 for Microsoft 1P, and 2000 for CX-8.The htsim and UCCL implementations are in C++; the Microsoft 1P and CX-8 integrations use C/Python.Source: §6 · pp. 6–7

6.1 CPU Reference Shell

  • The CPU reference shell supports UET-htsim and UCCL, with synchronous and asynchronous storage and transport integrations.UET-htsim maps simulator events to shell inputs; UCCL connects ACK, NACK, and RTT feedback to synchronous or asynchronous congestion-control paths.Source: §6.1 · p. 7

6.2 ConnectX DOCA PCC Shell

  • The ConnectX-8 shell offloads handlers to DPA cores through DOCA PCC callbacks, with per-flow state and native QP-rate control.DOCA PCC provides 16 DPA cores with 16 hardware threads each; because its datapath control is rate-only, window-based algorithms use a Swift pacing adaptation to approximate rate from CWND and EWMA(RTT).Source: §6.2 · p. 7

6.3 Microsoft 1P NIC Shell

  • The Microsoft 1P shell runs asynchronous congestion-control callbacks on eight hardware threads co-located with its RDMA engine and maps controls to native 1P APIs.The shell dispatches events to per-flow state.Source: §6.3 · pp. 7–8

7 Evaluation

PCM is evaluated across simulation, CPU-based UCCL, and NIC offloads, with matched baselines and targeted tests of abstraction cost, reuse, scaling, and reaction capacity. Results show native-level practical goodput in UCCL, high fairness in a live ConnectX-8 incast, and performance shaped by reaction timing, thread placement, and flow-state footprint.

7.1 Experimental Setup

  • Evaluation validates all eleven handlers in UET-htsim, tests sender congestion control on three live platforms, and uses synthetic traces for LB and receiver-side control.LB and receiver-CC live tests are unavailable because the CX-8 and M-1P NICs expose no public hooks for them.Source: §7.1 · p. 8, Table 3 · p. 8

7.2 Cost of Abstraction

  • UCCL reaches the same 1.56 Tbit/s practical goodput as native Swift, while PCM adds 1.5× isolated CC input/output cost.Overall TX+RX costs average up to 1.06× and 1.20× native execution for synchronous and asynchronous PCM, respectively.Source: §7.2 · p. 9

7.3 PCM-Swift Case Study

  • Under the tested incast, Q32.32 and Q42.22 complete flows within 0.3% of reference maximum FCT, while Q16.16 oscillates and causes 1,030 retransmissions.Fairness remains within 0.001 of reference for the two 64-bit variants; at scheduling delay N = 32, median CWND fairness stays at or above 0.99, but falls to 0.818 at N = 256.Source: §7.3 · pp. 9–10, Figure 6 · p. 9
  • In five live 3:1 incast phases, PCM-Swift maintained mean Jain fairness indices of at least 0.99 for both pacer rate and Write goodput.The optimized handler sustained one invocation per four ACKs with one flow per 100 Gbit/s plane.Source: §7.3 · p. 10, Figure 7 · p. 10

7.4 PCM Offloading Scalability

  • With 64 scattered DPA threads and 4,096 flows, PCM-Swift reaches 13.5 Mcalls/s, compared with 11.6 Mcalls/s under compact mapping.At 131,072 flows, both placements sustain nearly 9 Mcalls/s; throughput declines as the working set grows from 0.875 to 1.75 MiB.Source: Figure 9 · p. 11, §7.4 · p. 12

7.5 Performance Across Handlers

  • ConnectX’s 48 DPA threads sustain 9.5–15.1 million CC calls/s and 18.6–48.8 million LB calls/s; for Swift, capacity corresponds to one input event per two to three ACKs.Source: §7.5 · p. 12

7.6 Lessons Learned

  • Handler capacity depends on reaction timing, thread placement, and flow-state footprint, so PCM deployment requires workload-specific validation and worker provisioning.The authors recommend validating numerical behavior and reaction timing in simulation before deployment.Source: §7.6 · p. 12

8 Conclusion

PCM spans eleven congestion-control and load-balancing policies across four datapaths, with reported high-throughput control and ongoing influence on Ultra Ethernet standardization.

8 Conclusion

  • PCM spans eleven policies and four datapaths, from simulation and software transports to NIC offload.The conclusion presents this breadth as making portable congestion management practical for high-performance RDMA.Source: §8 · p. 12
  • PCM policies sustain per-chunk control at 1.56 Tbit/s goodput in UCCL and achieve near-per-ACK processing capacity on vendor 400G and 800G NICs.Source: §8 · p. 12
  • PCM's design informs ongoing Ultra Ethernet standardization toward a portable congestion management interface.Source: §8 · p. 12

A Behavioral Validation of PCM-Swift in UCCL

Under synthetic RTT injection, PCM-Swift's CWND adjustment range closely tracks native UCCL Swift across synchronous and asynchronous modes. Relaxed and locked snapshot modes show small differences in their nonzero adjustments.

A Behavioral Validation of PCM-Swift in UCCL

  • A synthetic 500 µs RTT is injected every 25th CC update to model transient network congestion.In asynchronous runs, the control-plane worker runs on a separate core; the asynchronous CPU shell supports relaxed and locked snapshot modes.Source: A Behavioral Validation of PCM-Swift… · p. 16
  • 1.2% is the difference between median nonzero adjustments in relaxed and locked snapshot modes; their 1st and 99th percentiles differ by less than 1%.The datapath observes an unchanged window in 10.9% of relaxed-mode samples and 10.7% of locked-mode samples; with the worker on the datapath thread’s SMT sibling, median adjustments differ by 0.7%.Source: A Behavioral Validation of PCM-Swift… · p. 16
  • Across all modes, PCM-Swift’s CWND adjustment range closely tracks native UCCL Swift under synthetic RTT injection.Figure 11 compares native UCCL Swift, the synchronous CPU shell, and two asynchronous snapshot modes; each curve uses 400,000 datapath observations from one run.Source: Figure 11 · p. 16, A Behavioral Validation of PCM-Swift… · p. 16

B Specification for the MP-RDMA Handler

The MP-RDMA handler specification is identified as Listing 4, an ACK-handling specification for Listing 1.

B Specification for the MP-RDMA Handler

Loading 2610.13130v1…