Source-linked AI summary
HOCCL: Offloading Collective Communication from GPU Cores to Accelerate Distributed Training
Yao Fei, Gongming Zhao, Hongli Xu, Jin Fang, Jiacheng Zhu, Shuo Xu, Kun Huang, Zhuolong Yu
TL;DR
Communication operations compete with computation for GPU SMs, limiting resources available for training. HOCCL moves communication data movement to DMA engines; it preserves near-state-of-the-art communication performance while freeing SM capacity and improving end-to-end throughput.
Problem
Communication operations contend with computation for SMs, reducing the SM capacity available to compute kernels and limiting training throughput.
Source: §1 · pp. 1–2Method
HOCCL is a zero-SM collective communication framework combining CPU–GPU synchronization, pipelined P2P transfers, and bandwidth-oriented collective scheduling.
Source: §1 · p. 2Results
HOCCL loses less than 3% versus NCCL while using no SMs for communication, frees nearly 10% of total GPU SMs, and improves end-to-end throughput by up to 5%.
Source: §1 · p. 2Takeaways & Limitations
HOCCL suggests communication design should consider preserving GPU resources for training, not only standalone communication speed.
Source: §7 · p. 13HOCCL’s extra NVLink forwarding traffic may contend with latency-sensitive TP, though interference is typically limited by the scale-up and scale-out bandwidth disparity.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Large language model training involves massive computation on GPU streaming multiprocessors (SMs), the primary compute units of GPUs. Since SMs host specialized accelerators such as Tensor Cores, their efficient utilization is critical to training efficiency. Unfortunately, existing collective communication systems compete with computation for SMs, as they consume SMs for communication-related data movement and synchronization operations. We observe that communication can, in principle, be driven by DMA engines, thereby eliminating SM involvement in communication. Based on this insight, we propose HOCCL, a zero-SM collective communication framework consisting of three components: a stream manager, a point-to-point (P2P) executor, and a collective scheduler. The stream manager preserves operator-level temporal ordering with other GPU kernels. The P2P executor enables zero-SM point-to-point communication, while the collective scheduler orchestrates P2P transfers to maximize bandwidth. Experiments show that HOCCL preserves near-peak communication performance, achieving within 3% of the state of the art on average, while eliminating communication occupancy on nearly 10% of total GPU SMs. By freeing SM resources for computation, HOCCL improves end-to-end training throughput by up to 5%.
1 Introduction
The introduction identifies communication’s competition with computation for SMs as a training bottleneck and motivates HOCCL, a zero-SM framework that offloads data movement to DMA engines. It reports near-state-of-the-art communication performance while freeing SM capacity and improving training throughput.
1 Introduction
- For GPT-6.7B, overlapped PP and DP together average 11.2 SMs, nearly 10% of an H800’s total, reducing SM capacity available to compute kernels.TP and EP peak at 16 and 32 SMs, respectively, but overlap little with computation and therefore do not contend for SM resources.Source: §1 · p. 1
- RNICs cannot access arbitrary user buffers, motivating registered staging, while dynamic tensor addresses force repeated registration that can cause nearly 10% end-to-end training slowdown.VMM-managed buffers may also be incompatible with RNIC registration, potentially incurring more than 10% additional fragmentation overhead.Source: §1 · pp. 1–2
- DMA offload is feasible because engines are lightly used during training, but CPU coordination, roughly 5× startup latency, and serialized-flow scheduling pose challenges.The measured startup latency leads to more than 20% communication performance loss; poor task ordering can cause deadlock, bandwidth imbalance, and long-tail latency.Source: §1 · p. 2
- Trace analysis identifies communication’s contention with computation for shared SMs as a factor that slows training and reduces throughput.Source: §1 · p. 2
- HOCCL combines CPU–GPU synchronization, pipelined P2P transfers that hide DMA startup latency, and topology- and flow-size-aware scheduling to maximize bandwidth.Its collective scheduler supports arbitrary user-defined collective communication patterns.Source: §1 · p. 2
- HOCCL loses less than 3% versus NCCL while using no SMs for communication; end-to-end training frees nearly 10% of total GPU SMs and improves throughput by up to 5%.The evaluation uses benchmarks and end-to-end Megatron-LM training on 32 NVIDIA H800 GPUs.Source: §1 · p. 2
2 Background and Motivation
Communication can occupy SMs while computation is active, and higher contention coincides with slower transformer-layer computation. Existing approaches use SMs to stage data or rely on RNIC-registered buffers, while DMA engines can move data without consuming SM resources.
2.1 Distributed Training Workloads
- PP and DP are background traffic that can overlap computation, whereas TP and EP are foreground traffic on the critical path with little room for overlap.PP overlaps transfers or gradient communication with other micro-batches; DP can prefetch the next layer’s parameters during current-layer computation.Source: §2.1 · pp. 2–3
- The contention rate measures the fraction of SMs used by a traffic class while compute kernels are active; Isum sums rates across traffic classes.The study focuses on background traffic and reports rates for representative models and parallelization strategies.Source: §2.1 · p. 3
- At Isum ≈10%, forward and backward transformer-layer computation slow down by 8.5% and 7.7%, respectively, as contention rises.The figure reports contention rates by traffic class and shows forward and backward compute time increasing with contention.Source: Figure 2 · p. 3, §2.1 · p. 3
2.2.1 SM-Based Communication Mechanism
- NCCL communication kernels use SMs to copy user data into an RDMA staging buffer and coordinate progress with the CPU proxy and NIC.After staging, GPU threads notify the proxy, which posts NIC work; completion information is then relayed back to the GPU.Source: §2.2.1 · p. 4
2.2.2 Registered-Buffer-Based Communication Mechanism
- VCCL [7] lets the RNIC read runtime-registered user buffers directly, but VMM layouts and recurring registration costs constrain this approach.Non-contiguous physical pages can prevent registration under advanced virtual memory management; repeated registration can reach millisecond scale and spike to tens of milliseconds.Source: §2.2.2 · p. 4
2.3 DMA Engines
- DMA engines move data without consuming SM resources, but NCCL’s copy-engine collectives are limited to intra-node transfers, unlike HOCCL’s cross-node focus.Modern DMA engines also support high-bandwidth inter-GPU transfers over NVLink.Source: §2.3 · p. 4
3 System Design
HOCCL combines a stream manager, point-to-point executor, and collective scheduler to provide zero-SM communication while preserving stream ordering and coordinating transfers across GPU communication domains.
3.1 Overview of HOCCL
- HOCCL pairs stream-ordering control, zero-SM point-to-point transfers, and collective scheduling to balance NVLink, RDMA, and DMA-engine use.The stream manager coordinates the CPU and GPU; the executor handles intra- and inter-node transfers; the scheduler organizes flows and chunks.Source: §3.1 · pp. 4–5
3.2 Stream Manager
- A CPU–GPU handshake preserves stream order, while shared flags and a per-chunk state machine coordinate RNIC and DMA progress.The handshake uses start and completion notifications; chunk flags encode idle, DMA copying, and RNIC-in-progress states.Source: §3.2 · pp. 5–6
3.3.1 Inter-node Communication
- Inter-node transfers stage data through a symmetric GPU’s buffers and pipeline DMA copies with RNIC transfers to amortize startup overhead.Multiple QPs keep chunks in flight, hiding CPU–GPU control latency and DMA startup overhead.Source: §3.3.1 · p. 6
3.3.2 Intra-node Communication
- Intra-node transfers use CPU-side IPC-handle exchange to resolve receiver addresses at runtime, enabling direct DMA between GPU buffers without SM involvement.CPU-side completion signaling lets the stream manager notify the receiving GPU stream when the transfer finishes.Source: §3.3.2 · pp. 6–7
3.4 Collective Scheduler
- The scheduler expands logical flows with forwarding traffic, decomposes the resulting matrix into contention-free RDMA stages, then interleaves fixed-size DMA chunks by flow size.Stages also fill available NVLink slack; chunk interleaving gives larger flows proportionate service while revisiting smaller flows regularly.Source: §3.4 · pp. 7–9
4 Limitations
HOCCL adds NVLink forwarding traffic and does not naturally support reduction-based operators such as AllReduce and ReduceScatter.
4 Limitations
- AllReduce and ReduceScatter fall outside HOCCL’s target scope because they require tightly coupled communication and reduction computation.The passage notes that high performance for these operators typically relies on SM-side execution.Source: §4 · p. 9
- Each RDMA transfer adds sender- and receiver-side NVLink traffic, which may contend with latency-sensitive foreground communication.The passage says this interference is typically limited because the forwarding rate is bounded by scale-out bandwidth.Source: §4 · p. 9
- A strict-priority switch suspends HOCCL background tasks while foreground TP traffic is active, preventing HOCCL from extending its critical path.Background tasks resume after foreground traffic completes.Source: §4 · p. 9
5 Implementation
HOCCL extends NCCL and adds a PyTorch-compatible backend designed for minimal integration effort. Figure 7 presents point-to-point bandwidth.
5 Implementation
- HOCCL extends the NCCL codebase with a runtime of more than 3K lines of C++.The core runtime implements HOCCL.Source: §5 · p. 9
- Figure 7 presents point-to-point bandwidth.Source: Figure 7 · p. 9
- A PyTorch-compatible backend matches the torch.distributed abstraction; switching between NCCL and HOCCL requires only changing an environment variable.The backend is a lightweight Python extension with approximately 1K lines of C++ code.Source: §5 · p. 9
6 Evaluation
The evaluation uses communication benchmarks and Megatron-LM workloads on H800 GPUs to assess bandwidth, CPU overhead, and training throughput. HOCCL preserves strong communication performance while improving end-to-end throughput and reducing compute time; its host-side work remains within measured CPU slack.
6.1 Experiment Setup
- Experiments use 32 H800 GPUs, NCCL and VCCL baselines, 2 MB–1 GB communication tests, and 6.7B/32B Megatron-LM workloads.HOCCL accelerates PP send/recv and DP AllGather; ReduceScatter falls back to NCCL.Source: §6.1 · pp. 9–10
6.2 Communication Benchmark
- At 1 GB, HOCCL reaches 192.2 GB/s intra-node, exceeding NCCL-8SM by 23%, and 48.8 GB/s inter-node, near the effective 50 GB/s NIC bandwidth.HOCCL lags on small messages, where DMA startup and pipeline setup are harder to amortize.Source: §6.2 · p. 10
- At 1 GB, HOCCL reaches 48.2 GB/s for 16-GPU cross-node AllGather, comparable to NCCL-4SM at 47.9 GB/s and NCCL-8SM at 48.3 GB/s.Source: §6.2 · p. 10, Figure 8 · p. 11
- At 1 GB, HOCCL reaches 57.7 GB/s for 16-GPU All-to-All, matching NCCL-4SM at 57.5 GB/s, while NCCL-8SM is 17% faster at 67.7 GB/s.Source: §6.2 · p. 11, Figure 8 · p. 11
- HOCCL's CPU launch cost is 9–12 µs intra-node and grows from 11.8 µs to 624.6 µs inter-node; registration latency can spike to 20–30 ms under load.The inter-node growth reflects host submission of more DMA/RDMA work as transfers are split into chunks.Source: §6.2 · p. 11, Figure 9 · p. 11
6.3 End-to-End Training
- 2.4% (6.7B) and 5.1% (32B): HOCCL improves throughput over NCCL, reaching 214.8 and 491.7 TFLOP/s, respectively.Source: §6.3 · pp. 11–12, Figure 11 · p. 12
- At mbs=2, HOCCL reduces forward compute time from 13.12 ms to 12.41 ms (5.4%) and backward compute time from 22.94 ms to 21.08 ms (8.1%).Source: §6.3 · p. 12, Figure 12 · p. 12
- NCCL and HOCCL maintain positive CPU slack, typically 20–50 ms, while VCCL reaches zero slack, indicating CPU overhead can stall GPU execution.The evaluation defines CPU slack as GPU start time minus CPU launch end time.Source: §6.3 · p. 12, Figure 10 · p. 12
7 Conclusion
HOCCL demonstrates that collective communication progress can move from GPU SMs to dedicated hardware, reducing interference between communication and computation. The conclusion argues that communication libraries should be assessed by their coexistence with end-to-end training, not only standalone throughput.
7 Conclusion
- Distributed training communication is often optimized for bandwidth while its consumption of GPU compute resources is overlooked.Source: §7 · p. 13
- HOCCL moves communication progress off GPU SMs and onto dedicated hardware, reducing interference between communication and computation.Source: §7 · p. 13
- Communication libraries should be evaluated by how effectively they coexist with end-to-end training, as well as by raw communication throughput.Source: §7 · p. 13