PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in Charge

Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, Mohamed Wahib, Didem Unat

arXiv:2610.15878v1cs.DC

TL;DR

Host-side control of kernels, communication, and synchronization remains in multi-GPU execution, including when data transfers go directly between devices. The paper proposes a CPU-free model using persistent kernels, specialized thread blocks, GPU-initiated communication, and device-side synchronization, and evaluates it on iterative solvers. On 8 NVIDIA A100 GPUs, it reports improved stencil performance and CG speedups over CPU-controlled counterparts.

  • Problem

    Although GPU-to-GPU data transfers can bypass the CPU, the CPU still directs multi-GPU execution and synchronization.

    Source: §1 · p. 1

  • Method

    The model combines persistent kernels, thread-block specialization, GPU-initiated communication, and device-side synchronization to move control flow to devices.

    Source: §1 · p. 1

  • Results

    On 8 NVIDIA A100 GPUs, stencil performance improves by 18.8% to 96.2%, and Standard CG and Pipelined CG achieve 1.54x and 1.63x speedups on 18 sparse matrices versus CPU-controlled counterparts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model demonstrates performance improvements in iterative solvers, especially in communication latency-bounded scenarios.

    Source: §7 · p. 10

    Cooperative Groups barriers require launches without more thread blocks than can run concurrently, and successive tiling for larger domains can be less efficient than hardware scheduling.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

This paper proposes a fully autonomous execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch. In a typical multi-GPU application, the host serves as the orchestrator of execution by directly launching kernels, issuing communication calls, and acting as a synchronizer for devices. We argue that this orchestration, or control flow path, causes undue overhead and can be delegated entirely to devices to improve performance in applications that require communication among peers. For the proposed CPU-free execution model, we leverage existing techniques such as persistent kernels, thread block specialization, device-side barriers, and device-initiated communication routines to write fully autonomous multi-GPU code and achieve significantly reduced communication overheads. We demonstrate our proposed model on two broadly used iterative solvers, 2D/3D Jacobi stencil and Conjugate Gradient(CG). Compared to the CPU-controlled baselines, the CPU-free model can improve 3D stencil communication latency by 58.8% and provide a 1.63x speedup for CG on 8 NVIDIA A100 GPUs. The project code is available at https://github.com/ParCoreLab/CPU-Free-model.

1 INTRODUCTION

Multi-GPU communication can bottleneck scaling, while host-side control remains in the execution path even when GPUs transfer data directly. The paper combines device-side execution and synchronization techniques, evaluates them on iterative solvers, and reports performance gains over CPU-controlled baselines.

1 INTRODUCTION

  • CPU orchestration persists even with direct GPU-to-GPU transfers: the host enqueues kernels and communication, manages overlap, and synchronizes devices.Earlier GPU interconnect advances allowed devices to bypass intermediate CPU buffers for data movement.Source: §1 · p. 1
  • The CPU-free model combines persistent kernels, thread block specialization, GPU-initiated communication, and device-side synchronization to move execution control to devices.The paper presents these components as a systematic way to implement CPU-free applications, although the individual techniques had been studied in other contexts.Source: §1 · pp. 1–2
  • The model is demonstrated on 2D/3D Jacobi stencils and Conjugate Gradient, whose iterations require communication and computation.The paper identifies communication-bound settings as particularly relevant to the proposed model.Source: §1 · p. 1
  • Stencil kernels improve performance by 18.8% to 96.2%, while Standard CG and Pipelined CG achieve 1.54x and 1.63x speedups on 18 sparse matrices using 8 NVIDIA A100 GPUs.The comparisons are against CPU-controlled counterparts.Source: §1 · p. 2

2 BACKGROUND & MOTIVATION

Even after direct GPU-to-GPU data movement became possible, the CPU continued to direct multi-GPU execution and synchronization. The section contrasts this host-controlled path with CPU-free stencil execution and reports lower communication overhead and latency.

2 BACKGROUND & MOTIVATION

  • For iterative solvers, the CPU launches a kernel each iteration to enforce timestep ordering; the paper calls these implicitly synchronized kernels discrete kernels.The CPU-controlled stencil example also uses host-managed neighbor synchronization and stream synchronization before advancing iterations.Source: §2 · p. 2
  • In CPU-controlled multi-GPU execution, the host issues communication calls, enqueues computation and communication on separate streams, and synchronizes devices.The host remains involved even when the underlying data path directly connects GPUs.Source: §2 · p. 2
  • At 8 GPUs, CPU-free 2D Jacobi communication overheads are up to 5.9x lower across domain sizes than in the CPU-controlled baseline.The baseline explicitly overlaps communication and computation using host-side CUDA events and memory-copy calls; the CPU-free version uses GPU-initiated calls.Source: §2 · p. 3
  • The baseline's communication takes over 96% of execution time, with only 19% overlapped; the CPU-free version reclaims 89.7% of full execution time by reducing communication latency 17.5 times.Figure 1c compares communication/computation overlap and execution times.Source: §2 · p. 3

3.1 Overview

The CPU-free execution model moves both control and data movement from the host to GPUs. It combines persistent execution, specialized thread blocks, GPU-initiated transfers, and device-side synchronization.

3.1 Overview

  • The CPU-free model removes the CPU from the data and control paths using persistent kernels, thread-block specialization, GPU-initiated transfers, and device-side synchronization.Figure 2 presents these concepts as the model's basis.Source: §3.1 · p. 3, Figure 2 · p. 3

3.1.1 Persistent Kernels.

  • A persistent kernel moves the solver's time loop onto the GPU, avoiding a return to the CPU each iteration for synchronization.Traditional iterative kernels are enqueued separately for each timestep and rely on CPU-provided implicit synchronization.Source: §3.1.1 · p. 3

3.1.2 Thread Block Specialization.

  • Thread-block specialization assigns communication to some blocks and bulk computation to the remaining blocks, enabling communication/computation overlap within one kernel.The design uses specialization to reduce communication latency.Source: §3.1.2 · p. 3

3.1.3 GPU Initiated Data Movement.

  • GPU-initiated transfers move data directly between peer GPUs without host involvement, using NVSHMEM for device-side data movement.The transfers serve both data exchange and synchronization between neighboring GPUs.Source: §3.1.3 · p. 3

3.1.4 Device-Side Synchronization.

  • CUDA Cooperative Groups device-wide barriers synchronize communication and computation blocks at each iteration's end; NVSHMEM signaling handles synchronization between peer GPUs.This removes the need for CPU-orchestrated kernel launches solely to synchronize threads within a kernel.Source: §3.1.4 · p. 3

3.2 Benefits of CPU-free Execution

CPU-free execution combines communication and computation inside a persistent kernel, with specialized blocks handling stencil boundaries and interior work. The stated benefits include reduced host overhead, lower communication overhead, improved overlap, and retaining on-chip data across iterations.

3.2 Benefits of CPU-free Execution

  • For the 2D 5-point stencil, communication blocks handle boundary computation and halo updates while computation blocks handle the inner domain.The stencil domain is split into boundary and inner regions.Source: Figure 3 · p. 4
  • Fusing communication and computation in one kernel reduces kernel-launch overhead and eliminates frequent GPU-CPU synchronization overheads.Traditional implementations use separate streams and host synchronization to overlap the phases.Source: §3.2 · pp. 3–4
  • Device-side communication calls can be issued as soon as data is ready, increasing asynchrony and reducing communication overhead.Device-side signaling also reduces cross-GPU synchronization latency.Source: §3.2 · p. 4
  • Persistent kernels can maintain communication/computation overlap even when a smaller per-GPU domain leaves the GPU partly idle.The passage contrasts this with traditional overlap, which requires a domain size that saturates the device.Source: §3.2 · p. 4
  • Keeping the time loop in one kernel lets shared memory and registers retain intermediate data across iterations instead of being wiped at each timestep.The paper reports observing caching benefits in stencil methods.Source: §3.2 · p. 4

3.3 Use-Case 1: Stencil Methods

Jacobi stencils require data exchange and synchronization at each iteration, making them a natural use case for CPU-free execution. The design overlaps boundary communication and computation within persistent GPU kernels.

3.3 Use-Case 1: Stencil Methods

  • Each GPU’s domain is split into inner and boundary regions, with neighboring GPUs’ halo values required at every timestep.The inner region can be computed while communication with neighbors proceeds.Source: §3.3 · p. 4
  • Persistent kernels move the timestep loop onto the GPU, where specialized thread blocks compute boundaries, exchange halos, and synchronize before the next iteration.Boundary blocks wait for neighboring halos, compute boundary values, write them to neighbors’ buffers, and signal availability; all blocks synchronize at each loop end.Source: §3.3 · p. 4
  • A two-kernel, separate-stream alternative is more modular but adds a synchronization point and showed no significant performance change versus the single-stream design.The separate persistent kernels independently manage boundary and inner-domain processing.Source: §3.3 · p. 4
  • The communication scheme also extends to PERKS by applying it to the single-GPU stencil implementation with minimal changes.The paper notes that iterative methods can cache intermediate results across timesteps instead of committing them to global memory each iteration.Source: §3.3 · pp. 4–5

3.4 Use-Case 2: Conjugate Gradient

The paper adapts CPU-free execution to Standard and Pipelined CG, whose communication includes neighbor-vector access for SpMV and global reductions for dot products. Pipelined CG uses auxiliary vectors to enable overlap between dot-product reductions and SpMV.

3.4 Use-Case 2: Conjugate Gradient

  • The solver applies to linear systems Ax = b where A is symmetric positive definite.The paper implements the CPU-free model for both Standard CG and Pipelined CG.Source: §3.4 · p. 5
  • SpMV needs neighboring GPUs’ vector entries, while dot products need a global reduction and synchronization; Saxpy operates locally without communication.Prior work cited in the paper reports that dot-product reduction and synchronization can bottleneck scaling across nodes.Source: §3.4 · p. 5
  • Pipelined CG adds three Saxpys but enables dot-product reductions to overlap SpMV and combines the reductions into one, removing a global synchronization step.Standard CG has few overlap opportunities because its operations have strict dependencies.Source: §3.4 · p. 5
  • Persistent kernels fuse Pipelined CG’s up to nine per-iteration operations into one kernel and move the time loop onto the GPU.The operations are six Saxpys, two dot products, and one SpMV.Source: §3.4 · p. 5
  • One thread block handles global dot-product reductions while the remaining blocks perform SpMV, overlapping communication and computation when possible.When overlap is unavailable, the communication block joins the compute blocks.Source: §3.4 · p. 5
  • GPU-side reduction and device-grid and multi-GPU barriers replace CPU-involved synchronization in the Pipelined CG kernel.The implementation uses a GPU-initiated reduction for local dot contributions, then synchronizes within the device and across devices.Source: §3.4 · p. 5

4 IMPLEMENTATION

The implementation combines CUDA Cooperative Groups and NVSHMEM to run synchronization and communication within GPU kernels. For stencil methods, it builds on NVIDIA’s multi-GPU samples and integrates specialized thread blocks and PERKS.

4 IMPLEMENTATION

  • CUDA Cooperative Groups and NVSHMEM provide the APIs for device-side synchronization and communication in the implementation.The authors note that the approach could also apply to AMD GPUs through ROCm Cooperative Groups and ROC_SHMEM.Source: §4 · p. 5

4.1 Stencil Methods

  • The stencil implementation uses NVIDIA’s multi-GPU programming-model samples as its baseline for direct communication-overhead comparisons.The repository’s samples focus on communication optimization, not necessarily on the most optimized computational kernels.Source: §4.1 · pp. 5–6

4.1.1 Synchronization and Halo Exchange.

  • Neighboring GPUs exchange halo data through NVSHMEM signaling, while grid.sync() synchronizes thread blocks within each device.Boundary blocks wait for iteration-specific flags; the combined put-and-signal operation moves halo data and updates the corresponding flag.Source: §4.1.1 · p. 6

4.1.2 Thread Block

  • The implementation allocates thread blocks between inner and boundary work, with the boundary-block count adjusted to domain size.Proportional allocation is important for smaller domains, where boundary computation and communication can otherwise constrain overlap.Source: §4.1.2 · p. 6

4.1.3 PERKS Integration.

  • PERKS is restricted to the inner domain and treated mostly as a black box, with minimal changes to synchronize with communication each iteration.Boundary layers remain immutable halos for PERKS; its exclusion of read-only halos from caching lets it read boundary-stream writes from global memory each iteration.Source: §4.1.3 · p. 6

4.2 Conjugate Gradient

The CG implementation partitions vectors across GPUs, launches persistent kernels, and uses NVSHMEM for communication and synchronization. Its cooperative-launch requirements constrain occupancy, and software tiling can introduce inefficiencies on large domains.

4.2 Conjugate Gradient

  • Vectors are divided across GPUs and allocated as equally sized NVSHMEM symmetric objects; redundant rows handle partitions that do not divide evenly.Each GPU keeps a copy of the matrix and processes a logical chunk.Source: §4.2 · p. 6

4.2.1 Synchronization.

  • Persistent CG kernels use cooperative launch, with grid.sync() for within-device synchronization and NVSHMEM barriers across devices.The CPU-controlled baselines use the host-side stream-based barrier equivalent.Source: §4.2 · p. 6, §4.2.1 · p. 6

4.2.2 Communication.

  • SpMV fetches neighboring vector entries with a device-initiated NVSHMEM call, while CPU-free dot-product reductions use the block-level reduction API.Warp- and thread-level reduction variants performed worse than the block-level version; the CPU-controlled baseline uses a host-side stream-based reduction.Source: §4.2.2 · p. 6

4.3 Limitations

  • Cooperative Groups’ device-wide barriers require cooperative launches without block oversubscription, limiting occupancy to blocks that can run concurrently.For larger domains, successive tiling is assigned by the programmer or software, and the implementation’s tiling can be less efficient than hardware scheduling.Source: §4.3 · p. 6

5 EVALUATION

The evaluation compares CPU-free and CPU-controlled Jacobi and CG implementations on 8 NVIDIA A100 GPUs. CPU-free performance gains vary with domain and matrix characteristics, with communication-bound and overlap-friendly cases showing particular benefits.

5 EVALUATION

  • The evaluation uses 8 A100 GPUs connected through NVLink, repeats each experiment five times, and reports the minimum runtime.Small, medium, and large domains are categorized by whether they under-saturate, utilize, or over-saturate the device.Source: §5 · p. 7, §5.1.1 · p. 7

5.1.1 Experiment Setup and Code Variants.

  • Jacobi baselines include host-controlled copy and overlap variants, GPU-initiated P2P with host synchronization, and NVSHMEM with CPU-launched kernels.The CPU-free variant uses device-side APIs; the 3D evaluation adapts existing 2D baselines to a 3D7pt stencil partitioned across z.Source: §5.1.1 · p. 7

5.1.2 Scaling

  • At 8 GPUs, weak-scaling speedups over Baseline NVSHMEM were 41.6% for small domains and 48.2% for medium domains.Against Baseline Copy Overlap, the corresponding speedups were 96.2% and 95.7%.Source: §5.1.2 · p. 7
  • For 3D Jacobi, CPU-free performance stays largely flat under strong scaling while CPU-controlled baselines degrade as GPU count increases.For the largest domain, the PERKS kernel with CPU-free communication achieved an 18.8% speedup over the baseline.Source: §5.1.2 · p. 8
  • CPU-free execution is not always optimal; its strongest fit is when CPU-induced latency is substantial and communication can overlap with computation.The paper reports that CPU-controlled implementations serialize communication and computation in small-to-medium domains.Source: §5.1.2 · p. 8

5.2.1 Experiment Setup and Code Variants.

  • CG tests use selected SuiteSparse SPD matrices and compare CPU-free Standard and Pipelined variants against corresponding CPU-controlled baselines.The CG variants run for 5,000 iterations without preconditioning, and numerical stability is outside the study’s scope.Source: §5.2.1 · p. 8

5.2.2 Scaling Study.

  • 1.63x and 1.54x geometric mean speedups over CPU-controlled Pipelined and Standard CG, respectively, were achieved across the evaluated matrices.CPU-free versions outperform CPU-controlled baselines on almost all matrices, though speedup varies by matrix.Source: §5.2.2 · p. 8
  • Irregular matrices with load imbalance show comparatively subpar speedups, while low-sparsity matrices can offer more communication and overlap opportunity.For several irregular matrices, some GPUs spend longer on SpMV, constraining all versions by single-GPU performance.Source: §5.2.2 · p. 9

6 RELATED WORK

Related work addresses reduced CPU involvement through persistent kernels, GPU-initiated communication, and execution models combining these approaches. The cited work spans GPU workloads, networking, and runtime systems.

6 RELATED WORK

  • Prior work describes persistent kernels as improving load balancing and kernel fusion, proposes WhippleTree uberkernels for graphics workloads, and applies persistent kernels to GPU key-value stores to reduce overheads.Belviranli et al. use a persistent kernel for a task-based execution model; Chen et al. present a task-scheduling framework using persistent and discrete kernels.Source: §6 · pp. 9–10
  • GPU shared memory and registers have also been used to avoid global-memory accesses and speed up single-GPU stencil and CG baselines [44].Source: §6 · p. 9
  • GPU networking work progressed from early device network stacks with performance and correctness issues to GPU-triggered network operations and GPU-NIC memory-model work.Some approaches used CPU helper threads.Source: §6 · p. 9
  • NVSHMEM provides fine-grained GPU-to-GPU movement from CUDA kernels, and version 2.6.0 enabled kernel-initiated communication to bypass the CPU proxy thread.Source: §6 · pp. 9–10

7 CONCLUSION

The paper presents a federated, CPU-free multi-GPU model for stencil and Conjugate Gradient solvers, using persistent kernels, thread-block specialization, device-initiated communication, and synchronization. Its experiments report improved performance, especially when communication latency is a bottleneck; broader applications and guidance on when the model is advantageous remain future work.

7 CONCLUSION

  • The model combines persistent kernels, thread-block specialization, device-initiated communication, and synchronization for multi-GPU stencil and CG solvers.It removes host-side communication and synchronization routines from the control path and uses persistent kernels to access on-chip memory across iterations.Source: §7 · p. 10
  • Experiments on 8 NVIDIA A100 GPUs showed improved performance, especially in communication latency-bounded scenarios.Source: §7 · p. 10
  • The authors plan to apply the model to more applications, specify when it is advantageous, and explore automating ports from CPU-controlled applications.Source: §7 · p. 10
Loading 2610.15878v1…