PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan Tallent, Kevin Barker

arXiv:1903.04611v1cs.ARcs.DCcs.NIcs.PF

TL;DR

Modern multi-GPU applications need stronger interconnect performance, yet the behavior and application impact of contemporary GPU interconnects remain insufficiently understood. The paper evaluates six interconnect technologies across six platforms and finds topology-dependent NUMA effects, motivating topology-aware performance modeling and tuning.

  • Problem

    The characteristics of modern GPU interconnects and their impact on real-world multi-GPU application performance remain insufficiently understood, limiting performance tuning and delivery.

  • Method

    The paper evaluates six GPU interconnects across six GPU servers and HPC platforms using Tartan microbenchmarks for Peer-to-Peer and Collective communication.

  • Results

    The evaluation identifies four new intra-node GPU communication NUMA effects associated with NVLink topology, connectivity, routing, and PCIe chipset design.

  • Takeaways & Limitations

    Choosing GPU combinations and interconnects according to topology can affect communication efficiency and overall multi-GPU application performance.

  • Takeaways & Limitations

    Scale-out applications may be constrained by communication interfaces such as MPI, whose CPU-oriented abstractions can hinder use of faster GPU interconnects.

Abstract

from arXiv · show

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

1 INTRODUCTION

Multi-GPU computing is increasingly necessary for demanding workloads, but the characteristics and application-level impact of modern GPU interconnects remain insufficiently understood. The paper addresses this gap through systematic characterization of these interconnects and their effects on multi-GPU execution.

  • Motivation: Demanding workloads in machine learning, big data, and planet-scale simulations increasingly require scalable multi-GPU computing.Large systems have demonstrated substantial GPU counts for neural-network training and climate simulation.
  • Motivation: Multi-GPU systems scale within nodes through platforms such as DGX-1 and DGX-2, and across nodes through systems such as Summit and Sierra.DGX-1 and DGX-2 integrate 8 and 16 GPUs, respectively, while Summit and Sierra support GPU-integrated scale-out.
  • Research gap: GPU-oriented interconnects changed multi-GPU communication beyond shared PCIe, but their characteristics and real-world performance impact remain unknown.This uncertainty limits efforts to use these interconnects for performance tuning and delivery.
  • Approach: The paper characterizes PCIe, NVLink-V1, NVLink-V2, NV-SLI, NVSwitch, and GPUDirect across latency, bandwidth, topology, routing, efficiency, and NUMA effects.Measurements cover Peer-to-Peer and Collective communication patterns.

2 MODERN GPU INTERCONNECT

The paper surveys the architectures, topologies, and communication mechanisms of PCIe, NVLink generations, NV-SLI, NVSwitch, and GPUDirect across evaluated platforms. These technologies differ substantially in connectivity, routing, bandwidth, and support for intra- or inter-node communication.

  • Technologies and platforms: The evaluation covers PCIe, NVLink-V1, NVLink-V2, NV-SLI, NVSwitch, and GPUDirect-enabled InfiniBand across six GPU servers and HPC platforms.The platforms map different technologies to P100/V100 DGX-1, RTX2080 SLI, DGX-2, SummitDev, and Summit.
  • PCIe: PCIe connects GPUs and CPUs through tree-based topologies, with DGX-1 using PCIe switches and QPI and DGX-2 using two switch levels.PCIe is slower than CPU-DRAM interconnects and can bottleneck GPU acceleration and GPU P2P communication.
  • NVLink-V1: NVLink-V1 supports direct CPU-GPU and GPU-GPU P2P reads and writes through bidirectional links, with communication efficiency correlated to packet size.The paper reports NVLink-V1 as twice as efficient as PCIe overall.
  • NVLink topologies: P100-DGX-1 uses a Hypercube Mesh whose stronger within-plane connectivity yields UMA locally and NUMA across planes, with explicit routing for non-direct links.SummitDev similarly partitions four GPUs into two fully connected subnetworks joined through an X-Bus.
  • NVLink-V2: NVLink-V2 increases link slots and per-link bandwidth, while V100-DGX-1 uses a dual-bandwidth Backbone Ring and Summit connects two three-GPU subnetworks through a 64 GB/s X-Bus.The Summit topology has no direct GPU connections between its two subnetworks.
  • NV-SLI and NVSwitch: NV-SLI links two Turing GPUs through one x8 NVLink-V2 link, while NVSwitch provides non-blocking crossbar connectivity for DGX-2's 16 GPUs.Each GPU can connect to six NVSwitches, delivering 300 GB/s through a single switch hop within a baseboard.
  • GPUDirect: GPUDirect-RDMA allows third-party PCIe devices such as InfiniBand adapters to access GPU device memory directly without CPU assistance.The paper focuses on the relationship between InfiniBand and GPU memory rather than reviewing InfiniBand itself.

3 GPU INTERCONNECT MICROBENCHMARKING

The paper uses Tartan microbenchmarks to evaluate GPU interconnect communication across Peer-to-Peer and Collective patterns, emphasizing topology, latency, bandwidth, efficiency, and NUMA behavior. The reported figures expose distinct communication patterns for PCIe and NVLink.

  • Benchmarking approach: Tartan microbenchmarks evaluate intra-node and inter-node GPU interconnects under Peer-to-Peer and Collective communication patterns.The measurements include topology, latency, bandwidth, message-size efficiency, and NUMA effects.
  • Unidirectional bandwidth: Figure 6 compares DGX-1 PCIe and NVLink unidirectional bandwidth using GPU-pair matrices, revealing PCIe anti-locality and NVLink neighbor and remote-node patterns.NVLink-V2's additional NUMA types are clearer in bidirectional bandwidth measurements.
  • Latency: Figure 5 compares P100- and V100-DGX-1 pairwise latency, distinguishing local, neighboring, and remote communication across PCIe and NVLink.The figure uses block colors to expose communication disparities and NUMA structure.

3.1 Intra-Node P2P Communication

Intra-node P2P performance varies with interconnect topology, connectivity, routing, and chipset design. NVLink exhibits stronger topology-related NUMA effects, while NVSwitch provides largely uniform remote access.

  • Latency and bandwidth: PCIe latency is roughly uniform across switch, local-socket, and QPI-bridge GPU pairs, indicating no latency NUMA effect.PCIe bandwidth nevertheless decreases for GPUs sharing the same PCIe switch, including four GPUs sharing a Level-2 switch in DGX-2.
  • Latency and bandwidth: NVLink latency is around 9µs for directly connected GPUs, but manual routing increases latency by about 2x on P100-DGX-1 and 3x on V100-DGX-1.NVLink-V2 also shows higher latency than NVLink-V1, potentially because of a deeper pipeline or lower operating frequency.
  • NUMA effects: NVLink-V1 and NVLink-V2 show NUMA among neighboring and remote nodes across both latency and bandwidth.NVLink-V2 additionally differentiates neighboring GPUs connected by one versus two links, and remote GPUs according to routing choice.
  • NUMA effects: NVLink bidirectional bandwidth makes its three NUMA effects more obvious than unidirectional bandwidth.For NVLink-V2, routes traversing different numbers of dual-bandwidth links can produce different bandwidth while latency remains similar.
  • NV-SLI: NV-SLI has symmetric two-GPU bandwidth with no NUMA effect, while bidirectional bandwidth doubles relative to unidirectional bandwidth.NV-SLI latency is about 8µs for opposite-GPU access, compared with about 13µs through PCIe and 5µs for local access.
  • NVSwitch: NVSwitch provides consistent remote bandwidth and nearly homogeneous latency, including across accesses requiring two switch hops.The results confirm an all-to-all fully connected organization in DGX-2.
  • Efficiency on message size: At message sizes of at least 4MB, NVLink-V1 and NVLink-V2 approach bandwidth saturation; DGX-2 NVSwitch one-hop and two-hop bandwidth remain aligned.For messages larger than 64KB, PCIe remote access can deliver higher bidirectional bandwidth than PCIe neighbors because of anti-locality.

3.2 Intra-Node Collective Communication

Collective communication depends on topology-aware orchestration, synchronization, and message-size-sensitive performance. Across intra-node systems, NCCL behavior reflects both network structure and implementation choices.

  • Communication model: NCCL constructs ring networks that pipeline partitioned data to provide near-optimal bandwidth for standard collective operations across network topologies.NCCL-V1 supports intra-node PCIe/QPI, whereas NCCL-V2 supports NVLink, PCIe, NVSwitch, NV-SLI, InfiniBand, and IP networks.
  • DGX-1: Collective latency increases almost linearly with the number of participating GPUs, while odd-GPU curve disalignment likely reflects NCCL design rather than NVLink-V2 P2P NUMA.The latency behavior is similar across NCCL versions and the two DGX-1 platforms.
  • DGX-1: PCIe collective bandwidth decreases as more GPUs participate, whereas NVLink bandwidth increases because additional connected links contribute capacity.NVLink-V2 reaches about 1.6x NVLink-V1 bandwidth with 4 GPUs and about 2x with 8 GPUs under a 1GB payload.
  • DGX-1: For 8 GPUs, PCIe collective bandwidth saturates around 16MB, while NVLink saturates around 256MB across the five collective patterns.The five patterns follow similar message-size trends despite different saturation points.
  • NV-SLI: On the two-GPU SLI system, NV-SLI collective latency is around 18µs and bandwidth is generally similar to PCIe, except reduce-scatter performs worse on both.Reduce and all-reduce have lower PCIe latency than the other collective operations and even lower latency than NV-SLI.
  • NVSwitch: With 16 GPUs on DGX-2, NVSwitch aligns collective bandwidth curves across message sizes, unlike PCIe and NVLink where intermediate-size divergences remain.Because NVSwitch is UMA, the authors attribute the remaining PCIe and NVLink disalignment to NUMA effects in those networks.

3.3 Inter-Node P2P Communication

Inter-node P2P evaluation compares five transfer configurations on SummitDev and Summit, including GPUDirect-RDMA and host-memory staging. GPUDirect-RDMA performs differently across the two generations.

  • Platforms and configurations: The evaluation measures inter-node P2P latency and bandwidth on SummitDev with four P100 GPUs per node and Summit with six V100 GPUs per node.Both systems use NVLink and support GPUDirect, with SummitDev using NVLink-V1 and Summit using NVLink-V2.
  • Platforms and configurations: The five configurations compare direct GPUDirect-RDMA with pinned or unpinned host-memory staging, each with GPUDirect enabled or disabled.The measurements span message sizes from 4B to 1GB.
  • SummitDev: On SummitDev, GPUDirect-RDMA performs worst from 4KB to 64KB for latency and from 4KB to 256KB for bandwidth, but becomes advantageous from 4MB onward.Its bandwidth degrades sharply after 64MB, suggesting that splitting large transfers into 64MB multiples may be preferable.
  • Summit: On Summit, GPUDirect-RDMA provides the lowest latency and highest bandwidth among the five configurations.Its advantage is especially clear for latency at message sizes ≤1MB and bandwidth at sizes ≥16KB.

3.4 Inter-Node Collective Communication

Inter-node collective communication is evaluated across two to eight participant nodes using one GPU per node. Latency is mostly stable with scale, while bandwidth shows topology- and operation-dependent effects but reaches substantial saturation levels.

  • Evaluation setup: Inter-node collective measurements vary participant nodes from 2 to 8, using one GPU per node, 4B for startup latency, and 1GB for sustainable bandwidth.Bandwidth-versus-message-size measurements are also reported for SummitDev and Summit.
  • Latency: Collective latency remains flat as node count increases for most operations, with all-reduce showing the main divergence on SummitDev and a smaller divergence on Summit.The authors suggest a joint effect of NCCL implementation and GPUDirect interconnect technology.
  • Bandwidth: SummitDev shows strong NUMA effects for reduce-scatter and all-gather at 3 and 5 nodes, while Summit shows them at 3, 5, 6, and 7 nodes.Overall bandwidth remains unchanged despite these effects, unlike the decreasing PCIe and increasing NVLink patterns reported for inter-node P2P.
  • Bandwidth: Collective bandwidth converges and saturates around 32/64MB, reaching nearly 16/32 GB/s on SummitDev and Summit, respectively.The reported values indicate substantially stronger GPU inter-node communication performance on Summit than SummitDev.

4 GPU INTERCONNECT BENCHMARKING

The evaluation measures how modern GPU interconnects affect intra-node and inter-node multi-GPU applications. NVLink often improves communication efficiency without substantially improving whole applications, whereas GPUDirect-RDMA provides clearer scale-out benefits.

  • Evaluation setup: The Tartan Benchmark Suite evaluates intra-node scale-up with and without NVLink on P100-DGX-1 and V100-DGX-1, and scale-out with five memory-transfer scenarios on SummitDev and Summit.The scale-up study uses seven applications; the scale-out study binds each MPI rank to one GPU.
  • Intra-node scale-up: NVLink’s communication improvements do not generally translate into substantial whole-application speedup, except for CSM and GMM.Most evaluated applications use CPU-master/GPU-slave execution, with communication primarily between CPUs and GPUs.
  • Intra-node scale-up: NCCL may avoid D2D transfers over NVLink for small messages, instead using H2D and D2H transfers to exploit available interconnect bandwidth.This behavior was observed around BroadcastKernelSmall() and under similar conditions for PLN and TRK.
  • Programming-model implications: Faster interconnects are difficult to exploit when communication is mediated by CPU-centric models or abstract interfaces such as MPI.The paper identifies GPU-initiated communication and NCCL integration into MPI as potential ways to improve performance delivery.
  • Inter-node scale-out: GPUDirect improves performance for inter-node communication, while pinned memory is especially beneficial when combined with GPUDirect.GPUDirect-RDMA is especially helpful for BRQ and MAM on SummitDev and overall on Summit.
  • Inter-node scale-out: Summit’s stronger fundamental communication bandwidth reduces relative speedups compared with SummitDev, despite no degradation in communication efficiency.The authors suggest PinnedMem-GPUDirect for SummitDev and GPUDirect-RDMA for Summit.

5 DISCUSSION

The discussion emphasizes that modern GPU interconnects are not transparently interchangeable. NUMA effects, heterogeneous networks, and communication-efficiency constraints complicate performance optimization and motivate new programming models.

  • NUMA effect: PCIe, NVLink-V1, and NVLink-V2 show strong NUMA effects on tested platforms, while NVSwitch and NV-SLI show UMA.The reported NUMA causes include topology, position, connectivity, routing, sharing, and chipset design.
  • Heterogeneity: Heterogeneous interconnect networks may operate exclusively, concurrently, or cooperatively, requiring runtime choices about selection and integration.The paper notes that NVLink has a strong bandwidth advantage over PCIe rather than a latency advantage.
  • Research directions: The evaluation motivates adaptive, portable, and tractable multi-GPU programming models that address interconnect complexity and improve communication use.It also motivates performance models for GPU task allocation, scheduling, and migration in shared environments.
  • Research directions: New communication patterns and libraries should match underlying interconnects, including Summit’s dual-subnetwork NVLink topology and its data-exchange constraints.The discussion specifically highlights data reuse and X-bus bandwidth when distributing data across the two islands.

6 RELATED WORK

Related work covers intra-node GPU memory and communication systems alongside MPI-based multi-node GPU computing. Prior studies address programmability, GPU-aware communication, overlap, and MPI-constrained algorithm design.

  • Intra-node GPU Computing: Prior intra-node work investigates NUMA optimization, hybrid-memory-cube memory networks, GPU-aware MPI, and automatic multi-GPU partitioning.These efforts target communication guidance, memory-management programmability, standard-MPI GPU communication, and partition automation.
  • Multi-node GPU Computing: Prior multi-node work integrates CUDA movement into MPI, overlaps computation with communication, analyzes communication patterns, and matches GPU algorithms to MPI constraints.The cited approaches span software and hardware designs for GPU clusters.

7 CONCLUSION

The paper characterizes six modern GPU interconnect types across six GPU servers and HPC platforms using intra-node and inter-node communication benchmarks. It reports four new intra-node NUMA effects and derives optimization guidance for multi-GPU systems.

  • Conclusion: The study evaluates PCIe, NVLink-V1, NVLink-V2, NV-SLI, NVSwitch, and InfiniBand with GPUDirect-RDMA using the Tartan Benchmark Suite.The evaluation covers six GPU servers and HPC platforms and both Peer-to-Peer and Collective communication patterns.
  • Conclusion: The evaluation identifies four new NUMA effects for intra-node GPU communication and proposes observations for practical optimization guidelines.The stated aim is to support more mature multi-GPU programming, execution, and performance models.
Loading 1903.04611v1…