PaperlayerView exact revision on arXiv

Source-linked AI summary

A HIP-Compatible Accelerator Backend for Fourier-Bessel Particle-in-Cell Simulations on CPU/DCU Heterogeneous Clusters

Jingliang Fan, Ruiqing He, Yang Wan, Jiandong Shang, Hengliang Guo, Qiang Chen

arXiv:2609.06680v1physics.comp-phcs.DC

TL;DR

FBPIC’s Numba CUDA backend limits direct deployment on HIP-compatible DCUs. This paper replaces that execution path with explicit C/C++ kernels integrated through CuPy RawKernel, preserving the Python workflow. The backend reports speedups on V100 and scaling results across multiple DCUs.

  • Problem

    FBPIC’s Numba CUDA GPU path cannot be directly reused as a robust, efficient backend for HIP/DCU systems.

    Source: §1 · p. 1

  • Method

    The backend replaces performance-critical Numba GPU kernels with explicit C/C++ kernels integrated through CuPy RawKernel while preserving the Python workflow and device-resident data flow.

    Source: §1 · p. 1

  • Results

    On V100, speedups were 1.32×, 1.44×, and 1.54× over original FBPIC; four DCUs achieved 1.88× strong-scaling speedup and 2.72× aggregate throughput at approximately 68% weak-scaling efficiency.

    Source: §1 · p. 2, §7 · p. 15

  • Takeaways & Limitations

    The backend provides a practical reference strategy for porting Python scientific applications whose performance-critical components depend on platform-specific JIT frameworks.

    Source: §7 · p. 15

    At larger process counts, inter-node communication and synchronization limit scaling as local computation becomes less able to amortize communication costs.

    Source: §6.5 · p. 14

Abstract

from arXiv · show

FBPIC (Fourier-Bessel particle-in-cell) is a high-performance simulation code for relativistic plasma and accelerator physics. Its original accelerator backend relies on Numba CUDA, which limits its direct deployment on accelerators using the HIP (Heterogeneous-Compute Interface for Portability) programming environment, such as DCU (Deep Computing Unit) accelerators. In this work, we develop an accelerator backend compatible with HIP that enables FBPIC to run efficiently on DCU platforms while preserving its Python user interface and high level simulation workflow. For the evaluated LWFA (laser-wakefield acceleration) workloads, the proposed backend achieves 1.32-1.54x speedups over the original FBPIC implementation on an NVIDIA V100 GPU and enables efficient execution on the DCU platform. We also summarize the key lessons learned from porting FBPIC to the DCU platform. Multi-DCU experiments achieve a 1.88x strong-scaling speedup on four accelerators and a 2.72x increase in aggregate throughput at approximately 68\% weak-scaling efficiency, with communication analysis identifying inter-node communication and synchronization as the main scalability limitations. Beyond FBPIC, the proposed approach provides a practical reference for porting and optimizing other scientific computing applications developed with Python on heterogeneous accelerator platforms.

1. Introduction

FBPIC’s Numba CUDA backend limits its direct use on HIP/DCU platforms, motivating a portable accelerator path that preserves its Python workflow. The paper introduces that path and reports CUDA and multi-DCU performance results.

1. Introduction

  • FBPIC’s Fourier-Bessel spectral formulation retains essential 3D effects for nearly axisymmetric systems with fewer degrees of freedom than full 3D Cartesian PIC.The formulation also avoids numerical dispersion associated with conventional finite-difference time-domain PIC solvers.Source: §1 · p. 1
  • FBPIC’s Python/Numba CUDA design supports productive NVIDIA GPU execution but cannot be directly reused as a robust, efficient backend for DCU/HIP systems.Numba’s ROCm target has been officially unmaintained since version 0.54.0.Source: §1 · p. 1
  • The backend replaces performance-critical Numba GPU kernels with explicit C/C++ kernels integrated through CuPy RawKernel, while retaining device-resident data and the existing MPI decomposition.The high-level Python workflow remains unchanged, and the backend adds no host-device data movement in the time-stepping loop.Source: §1 · pp. 1–2
  • 1.32–1.54× speedups over original FBPIC were achieved on an NVIDIA V100 for evaluated workloads; four DCUs reached 1.88× strong-scaling speedup and 2.72× aggregate throughput at approximately 68% weak-scaling efficiency.The reported multi-DCU profiling identifies inter-node communication and synchronization as primary limitations at larger process counts.Source: §1 · p. 2

2. Current state of the art

The section situates FBPIC among accelerator-oriented PIC codes and identifies a portability gap for Python-JIT implementations. It presents CuPy RawKernel as a route toward a HIP-compatible backend while preserving FBPIC’s usability.

2. Current state of the art

  • FBPIC’s Fourier-Bessel quasi-cylindrical solver is suited to near-axisymmetric wakefield problems and offers lower cost than full 3D Cartesian PIC for such geometries.The spectral approach also addresses numerical dispersion associated with finite-difference field solvers.Source: §2 · p. 2
  • Performance-portable PIC codes include WarpX, PIConGPU, Smilei, and HiPACE++, with implementations targeting multiple architectures or GPU clusters.WarpX targets NVIDIA, AMD, and Intel GPUs; PIConGPU adopted cupla over Alpaka, while HiPACE++ reports near-optimal strong scaling from 1 to 512 GPUs.Source: §2 · p. 2
  • FBPIC combines a high-level Python workflow with a spectral quasi-cylindrical algorithm, but its Numba CUDA GPU backend limits portability to AMD/DCU-class accelerators.The ROCm target was moved to unmaintained status in Numba 0.54.0.Source: §2 · p. 2
  • A minimal-change numba.hip experiment reported interoperability and functionality limits and lower-than-expected performance, while CuPy RawKernel offers a Python integration route for CUDA and ROCm kernels.The paper positions its work as addressing the gap between preserving FBPIC usability and running efficiently across GPU platforms.Source: §2 · pp. 2–3

3. Parallel implementation of FBPIC on DCU accelerators

The implementation preserves FBPIC’s longitudinal MPI decomposition while replacing its intra-device execution layer. Explicit CUDA/HIP-compatible kernels and a CuPy interface retain the Python workflow and device data model.

3.1. Computational structure and porting requirements of FBPIC

  • 1D MPI decomposition exchanges guard cells and migrating particles between neighboring subdomains, while GPU threads process particles and grid elements within each subdomain.The Python Simulation object schedules the PIC cycle and owns field, particle, diagnostic, and boundary-communication components.Source: §3.1 · p. 3
  • The PIC cycle gathers fields, pushes particles, deposits charge and current, advances spectral fields, and exchanges boundary data across subdomains.Gathering uses irregular reads, deposition requires conflict resolution for shared grid updates, and the spectral solver combines Fourier and Hankel transforms.Source: §3.1 · p. 3
  • Boundary handling transfers guard-cell values, accumulated sources, and migrating particles; local packing, unpacking, and rearrangement remain accelerator-intensive operations.Communication volume depends on guard-region width and migrating particles rather than the full local particle population.Source: §3.1 · p. 3

3.2. Kernel execution layer for DCU platforms

  • Because Numba’s ROCm target is unmaintained, the original GPU path cannot serve as a stable, performance-controllable execution path for DCU; the kernels are rewritten in explicit C/C++ using common CUDA/HIP syntax.The rewrite removes dependence on Python JIT translation and aligns the computational core with native CUDA and HIP programming models.Source: §3.2 · p. 3
  • CuPy RawKernel connects the rewritten kernels to Python and uses CUDA or HIP-backed compilation interfaces while retaining the same Python-side invocation mechanism.A valid cached binary skips compilation on later use.Source: §3.2 · pp. 3–4
  • The three-layer architecture preserves the Python application interface, dispatches kernels through a backend layer, and keeps particle and field state in CuPy arrays.Shared kernel source is used across CUDA and HIP where practical, with architecture-specific variants allowed when needed for correctness or performance.Source: §3.2 · p. 4

4. Lessons from the DCU port

Functional portability alone does not fully exploit the DCU, so the port examines kernel reimplementation, occupancy, thread-block tuning, and particle locality. It combines those changes with kernel-specific tuning and alternative particle organization.

4.1. Portable kernel reimplementation and compiler optimization

  • The rewritten C/C++ kernels use CuPy’s runtime compilation path, reducing first-use compilation latency and host-side launch overhead while retaining a consistent Python interface across CUDA and HIP.The custom-kernel path reuses cached binaries, and the device arrays remain resident through the timestep loop.Source: §4.1 · pp. 4–5, Figure 1 · p. 5

4.2.1. Register pressure

  • Selective loop unrolling balances instruction-level parallelism against register pressure, occupancy, and spilling in particle kernels.The implementation unrolls short fixed-trip-count loops when register allocation stays below an occupancy-relevant threshold; other loops remain iterative.Source: §4.2.1 · pp. 5–6

4.2.2. Thread-block configuration optimization

  • Thread-block tuning varies configurations by kernel structure because block size and shape affect wavefront utilization, occupancy, memory coalescing, and atomic contention.Figure 2 compares normalized candidate runtimes and identifies selected configurations; experiments warmed candidates before measurement and changed only launch geometry.Source: §4.2.2 · p. 6, Figure 2 · p. 7
  • 24.30% less time per simulation step (149.171 ms to 112.918 ms), or 1.32× speedup, followed kernel-specific launch tuning against the original configurations.The evaluation used the complete LWFA time-stepping loop.Source: §4.2.2 · p. 6

4.3. Particle locality

  • The port examines particle organization because spatial grouping can improve field-gather locality, while periodically sorting particles incurs attribute-array movement.The proposed alternative avoids physically reordering all particle attributes.Source: §4.3 · p. 6

4.3.1. Current deposition

  • Deposition is harder to parallelize than particle pushing or field gathering because particles can update overlapping grid locations, creating contention and data-race risks.Cell-based processing locally aggregates contributions before global writes, though overlapping neighboring-cell stencils still require atomics.Source: §4.3.1 · pp. 6–7

4.3.2. Particle sorting and binning

  • Binning assigns particles to spatial bins using counts, a prefix scan, and an auxiliary index array, leaving particle attributes in their original arrays.This organizes particle indices rather than physically permuting all particle attributes.Source: §4.3.2 · pp. 7–8
  • The binning/counting implementation has a total timestep cost comparable to physical reordering because histogram, prefix, index-generation, and indirect-access costs offset reduced array movement.The result shows that reducing sorting complexity alone does not necessarily reduce end-to-end PIC runtime.Source: §4.3.2 · p. 8

5. Verification of numerical consistency

Verification combines an analytical linear-wakefield benchmark with a higher-intensity LWFA test of cross-backend consistency. The tests assess agreement with analytical fields and preservation of coupled particle-field evolution across implementations.

5.1. Linear wakefield benchmark

  • All implementations passed the linear benchmark limits: Ez errors were 7.408%, 7.409%, and 7.410%, and Er errors 6.508%, 6.509%, and 6.507%.For Original FBPIC, Ported CUDA, and Ported DCU, respectively; the limits were 8% for Ez and 11% for Er.Source: §5.1 · p. 8
  • Ported CUDA's relative L2 differences were 5.85×10−6 for Ez and 1.04×10−4 for Er; Ported DCU's were 7.98×10−6 and 1.25×10−4.These differences were substantially smaller than those between the numerical and analytical solutions.Source: §5.1 · p. 8

5.2. LWFA benchmark

  • At iteration 1750, Ported CUDA and Ported DCU preserve the original wakefield extent and the positions of the density cavity and boundary accumulation.The field morphology, low-density cavity, and high-density accumulation were consistent across all three implementations.Source: §5.2 · p. 9, Figure 5 · p. 10
  • Ex ϵ2 values were 5.19 × 10−4 for Ported CUDA and 4.86 × 10−4 for Ported DCU, with no backend-dependent phase displacement apparent at the plotted resolution.The laser-pulse region and its internal oscillation pattern were consistent across implementations.Source: §5.2 · p. 9, Figure 6 · p. 10

6. Performance results

The ported backend improves V100 performance and runs on the target DCU platform, with the best multi-DCU results at four accelerators. Communication and synchronization costs limit further scaling, especially across nodes.

6.1. Experimental platform

  • The evaluation compares Original FBPIC and the ported CUDA backend on an NVIDIA V100 and tests DCU execution on Sugon 8000 Hygon BW1000 accelerators using DTK/HIP.The LWFA benchmark exercises field evolution, particle pushing, interpolation, and current deposition.Source: §6.1 · pp. 10–11, §6.2 · p. 11

6.2. Single accelerator performance

  • The ported CUDA backend achieves 1.32×, 1.44×, and 1.54× speedups over Original FBPIC on the NVIDIA V100 across the three LWFA problem sizes.The speedup increases as problem size decreases, where overhead from the original Numba-based kernel path accounts for a larger fraction of each timestep.Source: §6.2 · p. 11, Figure 7 · p. 11

6.3. Strong scaling performance on DCUs; 6.4. Weak scaling performance on DCUs

  • Four DCUs deliver a 1.88× strong-scaling speedup; under weak scaling, four DCUs reach 2.72× normalized aggregate throughput at approximately 68% efficiency, while eight add little throughput.At eight DCUs, weak-scaling throughput reaches only 2.84×, an increase of about 4.4% over four DCUs.Source: §6.3 · p. 12, §6.4 · pp. 12–13

6.3. Strong scaling performance on DCUs

  • At eight DCUs, strong-scaling time rises to 187.7 ms and efficiency falls to 11.8%, as less local work amortizes boundary exchange and synchronization.The eight-rank run also spans two nodes, adding inter-node communication and synchronization overhead; four DCUs give the minimum execution time for this workload.Source: §6.3 · p. 12

6.4. Weak scaling performance on DCUs

  • At eight ranks, weak-scaling time reaches 125.3 ms and efficiency falls to 35.5%, while normalized aggregate throughput increases only from 2.72× to 2.84×.The additional four accelerators improve aggregate throughput by only about 4.4%.Source: §6.4 · pp. 12–13

6.5. Communication analysis of Multi-DCU scalability

  • Under strong scaling, MPI time rises from 30.8 s to 94.2 s per rank as its application-time share increases from 20.2% to 53.4%; per-interface communication volume remains about 53 GB.With less local computation but similar per-interface communication, the computation-to-communication ratio declines; mpiP time also includes synchronization and waiting.Source: §6.5 · p. 13, Figure 10 · p. 14

7. Conclusion

The HIP-compatible backend reproduces the original FBPIC numerical behavior while improving performance on the evaluated NVIDIA V100 and DCU workloads. Multi-DCU results show scaling gains through four accelerators, while communication and synchronization limit larger-scale gains.

7. Conclusion

  • The ported implementations reproduced the original FBPIC numerical behavior, with relative L2 differences no greater than 5.19×10^-4 in reported field comparisons.No backend-dependent phase shift or structural distortion was observed.Source: §7 · p. 14
  • On an NVIDIA V100 GPU, the ported backend achieved speedups of 1.32×, 1.44×, and 1.54× across the three LWFA problem sizes.The results show that increased portability did not compromise execution efficiency on the evaluated CUDA platform.Source: §7 · pp. 14–15
  • On DCU, kernel tuning reduced average simulation-step time from 149.171 ms to 112.918 ms, a 24.30% reduction and 1.32× speedup.Four DCUs achieved a 1.88× strong-scaling speedup.Source: §7 · p. 15
  • Under weak scaling, four DCUs delivered 2.72× the single-DCU aggregate throughput at approximately 68% efficiency, while scaling to eight DCUs produced limited or negative gains.mpiP profiles attribute the degradation primarily to inter-node communication, synchronization, and a less favorable ratio of local computation to interface communication cost.Source: §7 · p. 15
  • The approach is presented as a reference for porting Python scientific applications, with future work targeting communication overlap, device-aware MPI exchange, and workload-aware kernel configuration.The implementation and tuning experiments were conducted on NVIDIA V100 and DCU accelerators.Source: §7 · p. 15
Loading 2609.06680v1…