PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Network Support for RDMA

Radhika Mittal, Alexander Shpiner, Aurojit Panda, Eitan Zahavi, Arvind Krishnamurthy, Sylvia Ratnasamy, Scott Shenker

arXiv:1806.08159v1cs.NI

TL;DR

PFC causes management and performance problems, while current RoCE NICs depend on a lossless fabric for good performance. The paper proposes IRN, which improves loss recovery and adds bounded end-to-end flow control, and reports better performance without PFC plus feasible implementation overheads.

  • Problem

    Current RoCE NICs require a lossless fabric for good performance, but PFC can cause management complexity, head-of-the-line blocking, congestion spreading, and deadlocks.

  • Method

    IRN modifies RoCE NICs with selective loss recovery and BDP-FC, which bounds in-flight packets by the network bandwidth-delay product.

  • Results

    IRN performs better than current RoCE NICs and does not require PFC to achieve high performance; its implementation is within feasibility limits with small chip-area and NIC-memory requirements.

  • Takeaways & Limitations

    IRN offers a RoCE NIC design that eliminates the need for PFC while improving performance, and its functionality can be implemented on commercial hardware.

  • Takeaways & Limitations

    Reported implementation results are not optimal ASIC results because HLS-based FPGA synthesis can be up to 2× less optimal than Verilog and FPGAs are generally less optimal than ASICs.

Abstract

from arXiv · show

The advent of RoCE (RDMA over Converged Ethernet) has led to a significant increase in the use of RDMA in datacenter networks. To achieve good performance, RoCE requires a lossless network which is in turn achieved by enabling Priority Flow Control (PFC) within the network. However, PFC brings with it a host of problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks. Rather than seek to fix these issues, we instead ask: is PFC fundamentally required to support RDMA over Ethernet? We show that the need for PFC is an artifact of current RoCE NIC designs rather than a fundamental requirement. We propose an improved RoCE NIC (IRN) design that makes a few simple changes to the RoCE NIC for better handling of packet losses. We show that IRN (without PFC) outperforms RoCE (with PFC) by 6-83% for typical network scenarios. Thus not only does IRN eliminate the need for PFC, it improves performance in the process! We further show that the changes that IRN introduces can be implemented with modest overheads of about 3-10% to NIC resources. Based on our results, we argue that research and industry should rethink the current trajectory of network support for RDMA.

1 Introduction

RoCE made RDMA practical in Ethernet datacenters but depends on PFC-enabled lossless networking, which introduces serious network-management and performance problems. The paper proposes IRN, showing that modest NIC changes can remove PFC while improving performance.

  • RoCE and PFC: RoCE NICs achieve good performance only over lossless networks, prompting operators to use Ethernet Priority Flow Control.RoCE combines direct host-memory access with basic transport functionality, while PFC minimizes packet loss.
  • RoCE and PFC: PFC can cause head-of-the-line blocking, congestion spreading, management complexity, and occasional deadlocks.
  • Research question: The paper asks whether RoCE NICs can be redesigned so RDMA performs well without a lossless network fabric.
  • IRN approach: IRN adds more efficient loss recovery and basic end-to-end flow control to current RoCE NICs.The evaluation uses extensive simulations and examines protocol extensions and NIC hardware-resource overheads.
  • Results: IRN can eliminate PFC and often performs better without it, while adding about 3-10% NIC resource overhead with no deterioration in message rates.
  • Relation to iWARP: IRN combines the simpler implementation proximity of RoCE with the loss-handling philosophy associated with iWARP.The paper characterizes iWARP as architecturally right but unnecessarily complex because it implements the full TCP/IP stack on the NIC.

2 Background

The background explains why RoCE inherited loss-insensitive transport from lossless RDMA networks and why PFC became necessary in practice. It also contrasts IRN’s architecture with iWARP and reports limited empirical evidence favoring RoCE’s current NIC implementation.

  • RDMA transport background: Infiniband RDMA uses credit-based flow control and go-back-N recovery because packet drops are rare in lossless HPC clusters.Out-of-order packets are discarded, and the sender retransmits packets after the last acknowledged packet.
  • RoCE transport: RoCE adopted Infiniband’s go-back-N transport design and used PFC to make Ethernet networks lossless.
  • Priority Flow Control: PFC pauses upstream senders when queues exceed thresholds, but its flow-agnostic reaction can produce unfairness, head-of-line blocking, pause spreading, and deadlocks.Congestion-control mechanisms such as DCQCN and Timely reduce sending rates but do not remove the need for PFC.
  • iWARP versus RoCE: iWARP implements the full TCP stack in hardware and translates TCP byte streams into RDMA segments, making its NIC design complex.
  • iWARP versus RoCE: 3× higher latency and 4× lower throughput were measured for iWARP than RoCE on 64-byte batched RDMA Writes using a single queue-pair.The comparison used similar NIC specifications and 40Gbps link speeds.
  • Evidence limitation: The measured price and performance differences are anecdotal because factors beyond transport complexity could explain them.
  • Paper’s contribution: The paper argues that endhost NIC loss recovery can outperform a lossless network without requiring iWARP’s full TCP-stack implementation.IRN is presented as closer to RoCE in performance and complexity while avoiding PFC.

3 IRN Design

IRN modifies RoCE in two main ways: selective loss recovery and BDP-based end-to-end flow control. These changes reduce redundant retransmissions and unnecessary queuing while limiting the NIC state needed to track losses.

  • Design goal: IRN seeks to eliminate PFC through minimal changes to the RoCE NIC rather than a more sophisticated performance-maximizing design.
  • Loss recovery: IRN improves RoCE’s go-back-N recovery with selective retransmission that preserves out-of-order packets and retransmits only lost packets.Its receiver sends NACKs carrying cumulative and selective acknowledgment information.
  • Loss recovery: IRN uses adaptive timeout behavior because selective acknowledgments are useful with multiple in-flight packets, whereas short messages rely on timeouts.High timeouts can increase short-message tail latency, while low timeouts can cause spurious retransmissions.
  • BDP-FC: BDP-FC caps each flow’s outstanding packets at the network bandwidth-delay product, computed from the longest path and packet MTU.A sender transmits a new packet only when its in-flight count is below this cap.
  • BDP-FC: BDP-FC improves performance by reducing unnecessary network queuing and reduces NIC loss-tracking state by bounding out-of-order arrivals.
  • Architectural scope: IRN separates loss recovery from TCP-style congestion control and operates directly on RDMA segments rather than TCP byte streams.This avoids the translation layers required by iWARP’s TCP-based design.

4 Evaluating IRN’s Transport Logic

Extensive simulations evaluate IRN against current RoCE across workloads, congestion-control schemes, and performance metrics. IRN without PFC consistently outperforms RoCE with PFC, while efficient loss recovery is especially important to these gains.

  • Experimental settings: IRN is evaluated using a commercial-vendor RoCE simulator modeling Mellanox ConnectX4 NICs and RDMA queue pairs.Flows use Poisson arrivals, random destinations, and a realistic heavy-tailed size distribution dominated by small messages and large-flow bytes.
  • Baseline comparison: IRN without PFC is compared against RoCE with PFC as the default evaluation baseline.
  • Basic results: 2.8-3.7×: IRN outperforms RoCE across three metrics because BDP-FC reduces queuing and IRN avoids PFC’s congestion spreading.
  • Basic results: 1.5-2×: enabling PFC with IRN increases each metric, despite 8.5% packet drops without PFC.PFC causes head-of-the-line blocking and congestion spreading, whereas efficient recovery confines packet-loss effects to the affected flow.
  • Basic results: 1.5-3×: disabling PFC degrades RoCE because go-back-N recovery creates redundant retransmissions and wastes time and bandwidth.
  • Explicit congestion control: With Timely or DCQCN, IRN remains up to 1.5-2.2× better than RoCE; PFC changes IRN performance by less than 1% improvement or about 3.4% degradation.For RoCE, enabling PFC still improves performance by 1.35× to 3.5× under explicit congestion control.

5 Implementation Considerations

IRN requires endpoint changes to support acknowledgements, selective out-of-order delivery, and correct RDMA completion semantics without switch modifications. Its implementation avoids large NIC packet buffers by placing out-of-order data directly in application memory, but must address compatibility and ordering challenges.

  • IRN extends RDMA packet formats within IP and UDP headers, so implementation changes affect endpoints without requiring switch changes.
  • IRN uses per-packet acknowledgements for BDP-FC and loss recovery, adding read acknowledgements because requesters otherwise do not acknowledge Read responses.The design reuses an available reliable-connected-QP opcode and adds responder timeouts for Read operations.
  • IRN instead DMAs out-of-order packets to their final application-memory addresses and tracks arrivals with bitmaps, reducing storage from 1KB per packet to a few bits.The approach extends prior partial out-of-order support to all RDMA operations.
  • Supporting out-of-order packets requires carrying first-packet metadata more broadly, explicit WQE sequence numbers, and tracking last-packet completion information until earlier packets arrive.IRN adds headers such as RETH or sequence information where needed and uses a 2-bitmap to defer MSN, WQE expiration, and CQE actions safely.
  • Applications that poll the last packet of a Write for completion are incompatible with IRN’s out-of-order placement, although IRN preserves Write completion guarantees specified by RDMA.The paper identifies this polling pattern as violating the RDMA specification and describes officially supported alternatives.

6 Evaluating Implementation Overheads

IRN’s implementation overheads are evaluated through memory analysis, FPGA synthesis, and end-to-end simulations. The results indicate modest resource demands and feasibility, with modeled retransmission and header costs.

  • Memory Requirements: IRN adds only 3-10% of current NIC cache for thousands of QPs and tens of thousands of WQEs on 100Gbps links.The added state includes transport-logic metadata and responder-side Read timeout state.
  • Validation: Correctness was validated by matching synthesized-module output traces against corresponding simulator traces.The RTL was also exported as IP blocks using Vivado HLS.
  • Synthesis Results: Each IRN packet-processing module uses under 1% of FPGA flip-flops and 2% of LUTs, totaling 1.35% and 4%, respectively.Supporting 100Gbps links raises total usage to 2.66% of flip-flops and 9.5% of LUTs on the same device.
  • Synthesis Results: The synthesized modules add at most 16.5ns latency, while receiveData sustains 45.45Mpps, equivalent to 372Gbps for MTU-sized packets.The receiveData throughput also exceeds the observed 39.5Mpps maximum of the evaluated Mellanox RoCE NIC.
  • Caveat: FPGA results provide feasibility intuition rather than optimal ASIC estimates because HLS and FPGA implementations are less optimal than direct Verilog and ASICs.The paper expects IRN to be integrated into an ASIC alongside the existing RoCE implementation.
  • End-to-End Effects: End-to-end simulations model a worst-case 2µs retransmission delay and additional packet headers, including up to 16 bytes for Write packets.Send packets receive 6 extra bytes, while Read responses require no additional headers.
  • Feasibility: IRN remains within feasibility limits, and two commercial NIC vendors confirmed that its design can be implemented on hardware NICs.Mellanox was implementing a version of IRN for a subsequent release.

7 Discussion and Related Work

The discussion frames IRN as either an incremental RoCE redesign or a simpler, faster realization of iWARP’s loss-handling philosophy. It also outlines a backward-compatible deployment path and relates IRN to credit-based flow control.

  • Backwards Compatibility: IRN can be deployed through dual RoCE/IRN NIC modes, negotiating IRN between endpoints and falling back to RoCE when unsupported.Operators can continue using PFC during incremental deployment because IRN performs with or without PFC.
  • Overhead Comparison: Figure 12 compares IRN with worst-case overheads against IRN without overheads and RoCE in the default scenario.The comparison isolates the effect of implementation overheads relative to the two reference configurations.
  • Flow Control: Credit-based flow control shares PFC’s head-of-line blocking, congestion spreading, and potential deadlocks.The paper therefore applies its observations about PFC to credit-based flow control.

A Evaluating IRN Across Varying Experimental Scenarios

The appendix evaluates IRN across scenarios using average slowdown, average flow-completion time, and tail flow-completion time. It reports both IRN’s absolute metrics and ratios against PFC-enabled baselines.

  • Metrics: The evaluation reports average slowdown, average FCT, and tail FCT for each scenario.For each metric, it provides IRN’s absolute value and ratios comparing IRN against IRN + PFC and other PFC-enabled configurations.

A.1 Varying link utilization levels.

IRN’s advantage without PFC is robust across utilization, bandwidth, and topology scales, although higher bandwidth narrows the performance gap. Increasing utilization strengthens IRN’s relative advantage as PFC congestion effects grow.

  • Varying Link Utilization Levels: 30% to 90% link utilization is used to test robustness across increasing network load.The corresponding results are presented in Table 3.
  • Varying Link Utilization Levels: As utilization rises, IRN without PFC increasingly outperforms both IRN + PFC and RoCE + PFC.The paper attributes this trend to stronger congestion spreading drawbacks from PFC at higher utilization.
  • Varying Link Bandwidth: Bandwidth variation from 10Gbps to 100Gbps narrows the performance gap between IRN without PFC and the two PFC-enabled cases.At higher bandwidth, the relative cost of the round trip needed to react to packet drops without PFC increases.
  • Varying Topology Scale: Topology scaling from 6-port switches with 54 servers to larger configurations preserves roughly similar performance trends.The larger evaluated configurations use 8-port switches with 128 servers and 10-port switches with 250 servers.

A.4 Varying workload.

IRN’s key performance trends remain consistent across workload patterns and individual flow sizes. The evaluation therefore reports robustness beyond the default aggregated workload.

  • A.4 Varying workload.: The alternative workload uses medium-to-large flows with a uniform size distribution, representing RDMA for storage or background tasks.The default workload instead mixes short and large messages with a heavy-tailed distribution.
  • A.4 Varying workload.: IRN’s key trends hold for the alternative workload pattern.Table 6 reports the corresponding results.
  • A.4 Varying workload.: Individual flow sizes in the default workload show no significant deviation from the trends in aggregated metrics.

A.5 Varying buffer size.

IRN’s performance advantages are robust to buffer-size changes, while the relative effects of PFC and IRN vary with available buffering. Parameter variations also produce only small differences from the default results.

  • A.5 Varying buffer size.: As buffer size decreases from 480KB toward 60KB, PFC’s drawbacks increase because of more pauses and greater congestion spreading.
  • A.5 Varying buffer size.: As buffer size increases, the performance difference between IRN with and without PFC decreases because fewer PFC frames and packet drops occur.
  • A.5 Varying buffer size.: As buffer size increases, IRN’s benefit over RoCE+PFC increases because BDP-FC relatively reduces queuing delay more.The passage expects similar behavior in shared-buffer switches.
  • A.5 Varying buffer size.: Changing IRN’s RTOhigh and RTOlow parameters produces very small differences from default results, indicating robustness to parameter settings.The evaluation varies RTOhigh to 2× and 4× the ideal and uses higher N values for RTOlow.

B Additional Implementation Details

IRN preserves RDMA completion semantics while adapting receive-queue handling for shared receive queues. Its implementation accounts for Send/Receive, Write with Immediate, and existing end-to-end credit behavior.

  • B Additional Implementation Details: IRN guarantees that completion of a later Send, Write with Immediate, or Atomic operation implies completion of previous Writes.This follows the RDMA specification’s responder-buffer conditions.
  • B Additional Implementation Details: For Send/Receive, IRN assigns receive sequence numbers when receive WQEs are dequeued from the shared receive queue.When sequence number 4 arrives after losses, four WQEs are dequeued and assigned numbers 1 through 4.
  • B Additional Implementation Details: For Write with Immediate, IRN expires the appropriate Receive WQE only after all packets of the message have been received.The first outstanding WQE is selected using in-order message-completion tracking.
  • B Additional Implementation Details: Current RoCE uses piggybacked receive credits, while probes and RNR NACKs handle cases where the responder lacks credits.Out-of-sequence probes without credits should be dropped rather than processed against the wrong memory address.

B.4 NACKs due to Other Errors

IRN distinguishes out-of-sequence NACKs from NACKs caused by other errors. It avoids treating the former as errors while retaining go-back-N recovery for the latter.

  • B.4 NACKs due to Other Errors: IRN treats out-of-sequence NACKs as normal behavior rather than requester errors.
  • B.4 NACKs due to Other Errors: For NACKs caused by other errors, such as receiver-not-ready conditions, IRN performs go-back-N recovery.
  • B.4 NACKs due to Other Errors: If an out-of-sequence packet would trigger an error NACK, IRN discards it at the responder without processing it or sending a NACK.
  • B.4 NACKs due to Other Errors: For Send with Invalidate, IRN can enforce a fence so earlier Writes complete before the invalidating operation executes.
Loading 1806.08159v1…