PaperlayerView exact revision on arXiv

Source-linked AI summary

GPU-Initiated Communication: Dissecting Down to the Bone

Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat

arXiv:2610.01380v1cs.DCcs.NIcs.PF

TL;DR

GPU-initiated communication serves fine-grained, latency-critical workloads, but library comparisons conflate hardware-mechanism and surrounding software costs. This paper isolates GPU and CPU-proxy paths with minimal transports and production-library measurements, finding that software choices, queueing, and traffic shape performance and resource costs.

  • Problem

    Library comparisons conflate hardware-mechanism costs with software choices, despite GPU-initiated communication serving fine-grained, latency-critical workloads such as MoE.

    Source: §1 · p. 1

  • Method

    The paper implements minimal GPU-submission and CPU-proxy transports to isolate mechanisms, then measures them alongside production libraries on four NVIDIA platforms.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Results

    Software choices shape performance: a minimal GPU path issues an 8 B write in 0.7 µs and completes in 4.0 µs, while a tuned proxy has lower median round trip (5.9 vs. 6.9 µs).

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Benchmarkers should match completion scope and queue count, report GPU and host operating states, and reserve queues for latency-sensitive traffic.

    Source: §6 · p. 12

    Raw-memory polling must handle ordering because persistent kernels lack the implicit synchronization of a kernel-launch boundary.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

GPU-initiated communication lets GPU threads post RDMA operations directly to the NIC. It underpins NVSHMEM, NCCL GIN, and DeepEP, which serve the fine-grained, latency-critical communication of Mixture-of-Experts (MoE) models, yet its performance characteristics and optimizations remain scarcely documented beyond source code, and library comparisons fail to separate the costs of the hardware mechanism from those of the library around it. This paper dissects GPU-initiated communication at the GPU-NIC boundary. We first detail the GPU-side network path: queue placement, work-request construction, doorbell ordering, and completion semantics. We then introduce mini-gda and mini-proxy, minimal transports for the GPU and CPU-proxy submission paths, and measure them alongside NVSHMEM IBGDA, NCCL GIN, DeepEP, UCCL-EP, MSCCL++, and fabric-lib on NVIDIA H100, H200, B200, and GB200 platforms. A minimal GPU path issues an operation in 0.7 $\mu$s and completes in 4.0 $\mu$s; libraries add up to 4.6 $\mu$s of issue time through queue management, memory ordering, and completion scope, and issue time scales with the SM clock. A tuned CPU proxy matches or beats the GPU path at idle, at the cost of a dedicated core whose operating state sets its latency and throughput. On either path, sharing a queue with bulk traffic raises latency by one to three orders of magnitude. Reaching the 260 M msg/s ceiling of our InfiniBand platform requires doorbell batching and queue parallelism, and both have resource costs: communication code can reduce GPU block residency even when unused, and all-to-all traffic loses 59% of its NIC message rate at about 3,000 active connections. The submission path alone therefore does not predict communication performance. Our experiment code and results are available at https://github.com/ParCoreLab/Dissecting-GPU-Communication-Experiments.

1 Introduction

GPU-initiated communication moves RDMA control onto the GPU, serving fine-grained, latency-critical workloads such as MoE, but existing library comparisons conflate mechanism and software costs. The paper isolates GPU and CPU-proxy submission paths and finds that both performance and resource costs depend on software choices, queueing, and traffic.

1 Introduction

  • Library-level comparisons conflate hardware-mechanism costs with queue counts, batching, memory fences, completion scope, and API overhead.Prior work reports proxies lower latency for a single unloaded operation and GPU paths higher rates for concurrent small messages [13].Source: §1 · p. 1
  • The paper isolates mechanisms with mini-gda and mini-proxy, then evaluates production libraries across NVIDIA H100, H200, B200, and GB200 platforms.The evaluation measures single-operation cost, proxy trade-offs, message rate, and transport resource costs.Source: §1 · pp. 1–2
  • A minimal GPU path issues an 8 B write in 0.7 µs and completes in 4.0 µs; libraries add up to 4.6 µs of issue time.Ordering scope changes issue time 3.7× across safe configurations, and all-QP completion doubles put+completion latency from 1 to 16 queues.Source: §1 · p. 2
  • A tuned proxy completes within 0.07 µs of the minimal GPU path and has lower median round trip (5.9 vs. 6.9 µs), at the cost of a dedicated core.The proxy's operating state determines its results; shared queues degrade under load, while reserving a queue recovers one to two orders of magnitude.Source: §1 · p. 2
  • One submitting thread posts 1.8 M msg/s; cooperative publication reaches 21.6 M msg/s on one queue, and independent queues reach 260 M msg/s.A host thread ringing doorbells for GPU-built WQEs reaches 257–258 M msg/s, so GPU submission is not required for the peak.Source: §1 · p. 2
  • Communication code can reduce useful kernel throughput by up to 37% even when dormant, while all-to-all traffic loses 59% of NIC message rate near 3,000 active connections.A send-only NIC maintains about 242 M msg/s through 32,768 active queues; Dynamically Connected transport does not remove the all-to-all decline.Source: §1 · p. 2

2 Background

The background introduces the GPU-direct networking path, RDMA queue organization, and distinctions among who decides, constructs, and submits communication work.

2 Background

  • GPUDirect technologies progressively remove the host from networking: RDMA exposes GPU memory to the NIC, Async shifts synchronization onto the device, and kernel-initiated successors move submission into the kernel.IBGDA and GDAKI [13] let GPU threads build queue elements, ring doorbells over PCIe, and poll completions.Source: §2 · p. 2
  • GDRCopy provides low-latency CPU access to GPU memory for proxy paths, while direct verbs create queues, doorbell pages, and memory registrations.Transport libraries place these resources in GPU memory.Source: §2 · p. 2
  • RDMA uses Queue Pairs containing circular Send and Receive Queues of Work Queue Elements; completed signaled WQEs produce CQEs in completion queues.Unsignaled WQEs produce no CQE of their own.Source: §2 · pp. 2–3
  • The GPU-submitted path uses GPU-resident queues and in-kernel WQE construction; in the CPU-proxy path, the host performs steps (1)–(3) and (6), while the GPU enqueues a descriptor.The figure contrasts where work is performed across the GPU and host paths.Source: Figure 1 · p. 3
  • Communication control and submission are distinct: GPU-initiated work can be GPU-submitted or proxy-submitted, and WQEs may be constructed by either the CPU or GPU.Table 1 summarizes the queue, posting, and completion choices of the evaluated libraries.Source: §2 · p. 3

3 The GPU–NIC Boundary

A network transfer requires NIC-reachable queues, work-request construction, ordered doorbell updates, and completion polling; libraries implement these steps differently.

3.1 Memory geography

  • GPU-submitted transports place RDMA queues in GPU-accessible memory for SM updates and NIC DMA, but doorbells require mapped UAR pages; NVSHMEM can forward doorbells through a CPU proxy when mapping is disallowed.Mapping a third-party PCIe BAR into GPU address space is treated as a potential security hazard.Source: §3.1 · p. 3

3.2 Work requests, submission, and doorbell semantics

  • An mlx5 send WQE packs 16-byte segments into 64-byte blocks; DC adds a destination AV, while inline payload trades a NIC DMA read for a larger WQE.Submission updates dbrec before an ordered 64-bit UAR store; batching can raise throughput but may delay operations, and WQE and payload stores must be visible before announcement.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 The completion path

  • Seeing a CQE or flag does not by itself guarantee that the SM can observe the preceding payload; raw-memory polling must handle this ordering for correctness.The text identifies persistent kernels as vulnerable because they lack the implicit synchronization of a kernel-launch boundary.Source: §3.3 · p. 4

3.4 Transport choice

  • RC binds each local QP to one remote QP, simplifying destination specification but increasing per-peer state; DC reduces initiator QPs while adding an address-vector segment to each send.NIC state and locality can affect message rate, and changing a DCI destination may require additional NIC work.Source: §3.4 · p. 4

4 Mechanism Evaluation

The evaluation separates GPU–NIC mechanism costs from library and proxy overhead, and examines how queue sharing, batching, concurrency, traffic role, connection reuse, and payload shape affect latency, throughput, kernel resource use, and connection scaling. It shows that performance depends on more than submission cost or idle latency: communication code can consume GPU resources before sending, while useful-work throughput and NIC message rate vary with these factors.

4.1 The cost of one operation

  • With GPU-scope ordering, issue takes 0.70 µs versus 2.62 µs with system-scope ordering on P-IB; all-QP quiet doubles completion latency from 1 to 16 QPs, unlike per-QP quiet.Removing fences reaches 0.19 µs issue and 3.46 µs through completion, but the authors do not establish that this configuration is safe; issue time also follows SM clock on GPU and proxy paths.Source: §4.1 · pp. 5–6, Table 4 · p. 6
  • A minimal GPU path issues an 8 B write in 0.7 µs and completes in 4.0 µs; libraries add up to 4.6 µs of issue time, while a tuned proxy matches its completion time and has lower median round-trip latency.The tuned proxy requires a dedicated CPU core, whose operating state affects its latency, tail, and rate.Source: Table 3 · p. 5, §4.1 · p. 7

4.2 The proxy design space

  • At 64 background CTAs, shared queues drive proxy latency into tens of milliseconds, while a private GDAKI context keeps p50 at 12–16 µs with 73–79 M msg/s carried.A shared GDAKI context instead answers in 362 µs while carrying 2.6 M msg/s; reserving queues recovers much of the interference loss.Source: §4.2 · pp. 7–8, Figure 6 · p. 7
  • Raising mini-proxy’s batch limit from 1 to 16 increases message rate 5.5×, and on P-GB200 it reaches 140 M msg/s, 90% of IBGDA’s 156 M msg/s.On P-IB, proxies remain an order of magnitude below NVSHMEM IBGDA’s 250 M msg/s; at larger payloads, paths converge in goodput, though completion latency remains different.Source: §4.2 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

4.3 Message rate and resource costs

  • Issuing 16 WQEs per doorbell raises one thread’s rate 2.6×, from 1.79 to 4.70 M msg/s; cooperative warp sharing raises a single QP to 30.7 M msg/s with two or more warps.More threads or larger batches do not raise the single-QP ceiling in the tested setup; publication order can constrain library rates, and adding a warp synchronization roughly triples GDAKI’s rate at 32 threads per context.Source: §4.3.1 · p. 9
  • 250–260 M msg/s requires parallel queues and batching: mini-gda and NVSHMEM reach this ceiling with 16 shared QPs and 4,096 threads, while mini-gda also reaches it with 64 batched single-thread QPs.Increasing NVSHMEM from 2 to 16 QPs raises rate about 5× at one CTA per QP; queues also lengthen PE-wide quiet and add active NIC connections.Source: §4.3.1 · p. 9
  • In the compute caller, communication code leaves mini-gda, mini-proxy, and separately compiled NVSHMEM near full throughput, while inlined NVSHMEM and GDAKI lose 11–15%.The comparison includes code compiled but not executed, as well as send-and-compute and send-wait-compute variants.Source: §4.3.2 · p. 9
  • Communication code can reduce useful-work throughput before a message is sent, including a 27% cost from dormant mini-gda in a streaming caller.Waiting for each 8 B message adds 2–5% loss with mini-gda or GDAKI and 19–27% with NVSHMEM or mini-proxy; separately compiled NVSHMEM's waiting loss rises from 7% to 20% and 64% at 16, 64, and 528 RC QPs per PE.Source: §4.3.2 · p. 10
  • DeepEP V1 dispatch finishes issuing about 25 µs into a 0.49 ms operation, so reducing issue cost alone addresses only a small part of execution.The remaining time is receive-side waiting and copying; Combine also finishes issuing early, then waits at grid synchronization.Source: §4.3.2 · p. 10
  • Connection reuse flattens simultaneous-traffic throughput to 139–143 M msg/s across 1,024–4,096 QPs, while larger payloads hide much of the message-rate loss.At 4,096 QPs, simultaneous traffic retains 55% of send-only goodput with 128 B writes and 98% from 256 B onward; changing connection on every write also hurts send-only traffic at large sets.Source: §4.3.3 · p. 10, Figure 10 · p. 11
  • Send-only NICs sustain about 242 M msg/s through 32,768 active QPs, while NICs that also receive fall from 152 to 74 M msg/s.Receive-only traffic declines at roughly ten times larger active sets; connection counts that are inexpensive for one traffic role may be costly for another.Source: §4.3.3 · p. 10, Figure 10 · p. 11
  • All-to-all loses 59% of its two-peer rate by roughly 3,000 active connections, and DC retains only 23% at 2,976 DCI–peer pairs.At 2,976 pairs, RC with the same 256-write bursts retains 56%; DC costs 0.45 µs above RC through completion, but changing destination on every WQE cuts NVSHMEM DC message rate 60×.Source: §4.3.3 · p. 10, Figure 10 · p. 11

5 Related Work

Prior work studies GPU communication at the library, application, and RDMA-engineering levels. This paper complements those studies with mechanism-level measurements and examines the libraries that consume these mechanisms, including paths beyond NVIDIA GPUs and ConnectX NICs.

GPU communication studies and HPC work

  • Mechanism-level measurements complement studies of GPU communication stacks, application strategies, and GPU–NIC mechanisms by independently varying WQE construction, ordering, queue sharing, and completion.The work also adopts the initiation-versus-submission taxonomy established by The Landscape of GPU-Centric Communication.Source: §5 · p. 11

RDMA performance engineering

  • RDMA engineering work identifies NIC cache pressure from large QP counts, while this paper varies traffic roles, connection reuse, and payload to examine active-connection costs on ConnectX-7.Related work also reports differing inbound and outbound scaling and bidirectional degradation.Source: §5 · p. 11

Expert-parallelism libraries

  • DeepEP, pplx-kernels, and newer ecosystem libraries consume the mechanisms dissected here, while host-thread designs such as fabric-lib and UCCL-EP [31] target portability.The paper positions its measurements as explaining differences in published library performance.Source: §5 · p. 11

Beyond NVIDIA GPUs and ConnectX NICs

  • GPU-submitted communication also appears on AMD, Intel, and AWS EFA paths, and HPE Slingshot supports GPU-triggered commands prepared by the host.The measured mechanisms apply across these paths, although their costs differ.Source: §5 · p. 11

6 Conclusion

The paper isolates GPU-initiated communication costs from library overhead using minimal GPU and proxy transports across four NVIDIA platforms. Its measurements connect queue, ordering, and completion choices to latency, message rate, and GPU and NIC resource costs.

6 Conclusion

  • The mechanism itself is cheap, while library choices around queue management, ordering, and completion determine latency, message rate, and required GPU and NIC resources.The study uses mini-gda and mini-proxy to separate mechanism costs from library costs across four NVIDIA platforms.Source: §6 · p. 12
  • Benchmark GPU and proxy paths at reported processor states, provision queues for isolation and capacity, and budget NIC connections by traffic role, reuse, and payload.Compile and measure communication in its final caller because dormant communication code can reduce useful throughput.Source: §6 · p. 12
  • The released mini-gda, mini-proxy, and benchmark suite provide an explainable reference point and performance oracle for transport users and developers.The code and results are available as open source.Source: §6 · p. 12

A Methodology Details

The appendix reports measurement checks and conditions for the methodology, SM-clock sensitivity, and completion-scope experiments.

A Methodology Details

  • Adding a 50 µs responder delay shifts round trips by 49.6–50.3 µs; the reported latency runs use one P-IB pair, with warm-up operations discarded.The delay check also covers P-GB200, and the appendix distinguishes the pair used for loaded-RTT curves from the pair used for other measurements.Source: A Methodology Details · p. 14

B.1 SM-clock sensitivity

  • The P-RoCE clock sweep measures six SM clocks from 502–1,845 MHz, and Table 6 fits latency as T(f) = C/f + B, where C is an effective clock-sensitive coefficient.The sweep uses three shuffled passes of 5,000 operations per cell.Source: §B.1 · p. 14, Table 6 · p. 14

B.2 Completion scope

  • Table 7 compares NVSHMEM completion scopes using the same internal put, while Table 8 reports GB200 put-plus-completion with automatic DCI sizing selected by DCI requested = 0.The completion-scope experiment measures after warm-up at a measured 1,965 MHz; Table 8 uses a 2,062 MHz SM clock.Source: Table 7 · p. 14, Table 8 · p. 14, §B.2 · p. 14

C.1 Operating state

The appendix reports worker operating-state conditions, proxy measurement scope, and an MSCCL++ change that raises measured proxy capacity.

C.1 Operating state

  • Cold workers follow 45 s idle at 2.5 GHz, while warm workers follow 20 s sustained load at 4.0 GHz; CPU-only preconditioning reproduces warm results.GPU-submitted controls change little after CPU-only preconditioning.Source: §C.1 · p. 14
  • The proxy latency table measures cold and warm runs in one session on one pair, while the capacity table reports selected P-IB proxy capacity..Source: Table 9 · p. 14, Table 10 · p. 14
  • Passing memory handles by const reference removes shared-pointer refcount contention, raising P-IB capacity from 2.5 to 7.5 M msg/s over one to eight services.The patched build retains one signaled post per message and is used throughout the paper.Source: §C.1 · p. 14

C.2 Queue isolation

Table 11 reports loaded 8 B round-trip p50 with 64 background CTAs, including shared and private initiator rates for mini-proxy on P-IB and P-GB200.

C.2 Queue isolation

  • 47.5/44.9 M msg/s on P-IB and 61.8/62.9 M msg/s on P-GB200 are mini-proxy’s shared/private initiator rates.The configuration is T4/R32/B16.Source: Table 11 · p. 14
  • The loaded measurement is 8 B RTT p50 with 64 background CTAs.Paced rows compare reserved queues at approximately 30 M msg/s.Source: Table 11 · p. 14
  • GDAKI uses eight threads per CTA and a fixed 65-context pool, with the shared probe using one background CTA’s context.UCCL-EP uses eight workers.Source: Table 11 · p. 14

C.3 Payload size

The section compares single-operation put-plus-completion across payload sizes on P-IB and reports selected pointer-write goodput on P-GB200; the latter measurements do not locate a crossover.

C.3 Payload size

  • Figure 11 plots single-operation put-plus-completion against payload size on P-IB for the paths in Figure 8.NVSHMEM uses all-QP quiet.Source: Figure 11 · p. 15
  • Table 12 reports selected P-GB200 pointer-write goodput using four CTAs at 4 KiB and one CTA at 64 KiB.Configurations include mini-proxy T4/R32/B16, IBGDA with 16 RC QPs, GIN Proxy with four requested workers and one context per CTA, and GDAKI with 32 threads and one context per CTA.Source: Table 12 · p. 15
  • Coarse payload sizes and changing geometry do not locate a crossover in the P-GB200 pointer-write comparison.The table’s selected measurements therefore do not establish a crossover point.Source: Table 12 · p. 15

D.1 Queue publication

On P-GB200, longer warm-up cuts GDAKI’s aggregated rate, while synchronizing after each put preserves it; Table 13 also reports controls for P-IB and P-GB200.

D.1 Queue publication

  • 67 to 17 M msg/s: lengthening warm-up from 100 to 1,000 iterations cuts GDAKI’s aggregated rate on P-GB200.Synchronizing after each put keeps the rate at 82 M msg/s in both cases; the passage attributes the sensitivity to lane drift.Source: §D.1 · p. 15
  • 106–110 M msg/s: synchronizing after each put raises the P-IB 32-thread GDAKI case.The table’s P-IB GDAKI setup uses 16 contexts, one per CTA.Source: Table 13 · p. 15
  • Table 13’s P-GB200 GDAKI control uses 16 contexts of 32 threads, aggregates every 16 puts, and measures 10,000 iterations.The setup disables NCCL’s RAS monitoring; NVSHMEM and mini-gda use 256 threads per CTA and one QP per CTA, with mini-gda publishing 32 WQEs per doorbell.Source: Table 13 · p. 15

D.2 Enclosing-kernel resources

Dormant NVSHMEM code can sharply reduce a compute caller’s retained throughput, while the section also reports kernel-timing, waiting-loss, and DeepEP configuration details.

D.2 Enclosing-kernel resources

  • Kernel times are medians of three paired passes of 15 launches.The inlined build sets NVSHMEM_ENABLE_ALL_DEVICE_INLINING=ON.Source: §D.2 · p. 15
  • Table 14 reports waiting loss relative to sending without waiting, using 16 individual 8 B messages per block.The setup is a P-IB compute caller with eight live values (sm_90, -O3).Source: Table 14 · p. 15
  • 15% versus 100%: with 160 live values, a compute caller retains this throughput when NVSHMEM is compiled separately versus inlined.At 224 live values, it retains 8% versus 25%, respectively, with dormant NVSHMEM code and the earlier instrumented builds.Source: §D.2 · p. 15
  • DeepEP V1’s combine finishes issuing after 24 µs, and its median warp spends 0.93 ms at grid synchronization before reduction.The configuration uses NVSHMEM 3.4.5, two four-GPU P-IB nodes, 128 tokens per rank, hidden size 7,168, 288 experts, top-8 routing, and all-RDMA traffic across five runs.Source: §D.2 · p. 15

D.3 Active connections

The dense 128-PE sweep is fit with a plateau followed by a power-law decline; P-H100 controls specify active-connection normalization and selected payload-measurement settings.

D.3 Active connections

  • The dense 128-PE sweep is fit in log space to a plateau joined to a power-law decline.The fit uses least squares.Source: §D.3 · p. 15
  • GPU active-connection runs use CPU-forwarded doorbells and normalize to each configuration’s two-peer control.Host verbs normalize to their one-sender control.Source: Table 15 · p. 15
  • Selected payload controls use 4,096 active QPs per NIC, four peers, and 32 writes per connection visit.Goodput is the median sender rate per pass, then the median [min, max] over passes; the ratio divides the two medians.Source: Table 16 · p. 15

Related papers

Loading 2610.01380v1…