Source-linked AI summary
Exploiting multi-scale parallelism for large scale numerical modelling of laser wakefield accelerators
Ricardo A. Fonseca, Jorge Vieira, Frederico Fiúza, Asher Davidson, Frank S. Tsung, Warren B. Mori, Luís O. Silva
TL;DR
Large-scale 3D LWFA simulations are computationally demanding, motivating efficient use of high-performance computing resources. The paper adapts OSIRIS to exploit HPC parallelism, and reports scaling beyond 10^6 cores with sustained performance above 2 PFlop/s.
Problem
Method
The paper develops HPC parallelism for OSIRIS, including shared-memory processing and vector-accelerated PIC simulations, alongside multidimensional load balancing.
Source: §4 · p. 7, §6 · p. 10, §8 · p. 14Results
The code scaled beyond 10^6 cores and sustained more than 2 PFlop/s in floating-point performance.
Source: §8 · p. 14Takeaways & Limitations
The demonstrated parallelism supports large-scale numerical modeling of laser wakefield accelerators.
Source: §8 · p. 14The multidimensional load-balance method assumes separable computational load, and boundary motion offers little optimization room at very high core counts.
Source: §4 · p. 7
Abstract
from arXiv · showhide
A new generation of laser wakefield accelerators, supported by the extreme accelerating fields generated in the interaction of PW-Class lasers and underdense targets, promises the production of high quality electron beams in short distances for multiple applications. Achieving this goal will rely heavily on numerical modeling for further understanding of the underlying physics and identification of optimal regimes, but large scale modeling of these scenarios is computationally heavy and requires efficient use of state-of-the-art Petascale supercomputing systems. We discuss the main difficulties involved in running these simulations and the new developments implemented in the OSIRIS framework to address these issues, ranging from multi-dimensional dynamic load balancing and hybrid distributed / shared memory parallelism to the vectorization of the PIC algorithm. We present the results of the OASCR Joule Metric program on the issue of large scale modeling of LWFA, demonstrating speedups of over 1 order of magnitude on the same hardware. Finally, scalability to over $\sim 10^6$ cores, and sustained performance over $\sim 2$ PFlops is demonstrated, opening the way for large scale modeling of laser wakefield accelerator scenarios.
1. Introduction
Laser wakefield acceleration can generate high-energy particles over short distances, but understanding and optimizing it requires accurate modeling of complex plasma dynamics. Full 3D simulations are computationally demanding, motivating code adaptations for high-performance computing systems.
1. Introduction
- LWFA combines a nonlinear plasma wake with relativistic particle motion, requiring full electromagnetic kinetic models that track individual particles.The wakefields are electromagnetic, and electron trajectories can cross while reaching highly relativistic energies.Source: §1 · p. 2
- Realistic LWFA modeling must follow an intense, approximately 50 fs laser over roughly 1–100 cm while tracking wake evolution and accelerating beams.Full 3D simulations are computationally heavy and require high-end HPC systems.Source: §1 · p. 2
- Numerical modeling has predicted low-energy-spread electron bunches later demonstrated experimentally, with energies now reaching approximately 2 GeV.The cited sequence is [3, 4] for predictions, for experimental demonstrations, and for the reported energy.Source: §1 · p. 2
- The paper examines how to exploit multi-scale parallelism in OSIRIS to run large-scale LWFA simulations efficiently on high-end HPC systems.It addresses code scaling to 10^5–10^6 cores and hardware acceleration features.Source: §1 · p. 2
2. Numerical modeling of LWFA
The section describes an explicit electromagnetic particle-in-cell approach for modeling LWFA and estimates the scale of a representative 10 GeV stage. It also outlines the hierarchy of parallelism available in modern HPC systems.
2. Numerical modeling of LWFA
- The explicit relativistic PIC loop deposits particle current on a grid, advances electromagnetic fields with an FDTD solver, then interpolates fields to update particle motion.Grid spacing resolves the smallest relevant physical scale, typically the laser wavelength, and the timestep follows the Courant condition.Source: §2 · p. 3
- A modeled 10 GeV LWFA stage is estimated to require about 10^9 cells, 10^7 timesteps, 10^10 particles, and 10^19 calculations.The estimate assumes a roughly 0.5 m stage, plasma density of 10^17 cm^-3, and eight particles per cell.Source: §2 · p. 3
- Modern HPC systems commonly combine distributed-memory nodes, shared-memory CPU cores, and SIMD vector units, all of which codes must use efficiently.Some systems also include node-level accelerators such as GPGPUs or Xeon Phi cards, which the paper does not cover in detail.Source: §2 · pp. 3–4
3. Scaling to ∼105 −106 cores
At high core counts, LWFA particle accumulation creates severe load imbalance under standard spatial domain decomposition. The section describes shared-memory particle processing as a way to balance work within nodes, while noting its overhead and measured scaling limits.
3. Scaling to ∼105 −106 cores
- At approximately 10^5 cores, localized particle accumulation behind the laser made an LWFA run about nine times slower than a warm-plasma benchmark on the same partition.The measured maximum-to-average particle load ratio was 8.68 and the imbalance persisted through most of the simulation.Source: §3 · pp. 4–5, Figure 1 · p. 5
3.1. Shared memory parallelism
- Shared-memory parallelism assigns a larger simulation region to each process and evenly distributes its particles across threads, balancing work within each node.Global imbalance can remain, but larger shared regions reduce the likelihood of severe imbalance.Source: §3.1 · p. 5
- To avoid thread conflicts during current deposition, each thread deposits to its own current-grid copy, followed by a parallel reduction into one grid.Zeroing and reducing multiple copies add overhead, so scaling to more cores requires balancing that cost against the parallel work.Source: §3.1 · pp. 5–6
- On Jugene, performance degradation was negligible through four threads per MPI process; on Jaguar, it stayed below 10% through six threads but reached 27% at 12 threads.The 12-thread drop was attributed to additional calculations and memory-affinity issues.Source: §3.1 · p. 6
4. Multi-dimensional Dynamic Load Balance
The section presents multidimensional dynamic load balancing as an approach to redistributing computational load across nodes. Its LWFA example improved average imbalance but did not improve runtime because the benefit did not offset repartitioning overhead.
4. Multi-dimensional Dynamic Load Balance
- The proposed multidimensional method assumes computational load is separable into independent directional functions, an assumption that may not hold for scenarios such as spherical plasma.Only one-dimensional partitioning has a single best solution; improving balance in one direction can worsen it in others.Source: §4 · pp. 6–7
- At very high core counts, node boundaries have little room to move because each node approaches the minimum cell limit, reducing optimization opportunities.The section also notes that domain redistribution can carry significant communication overhead.Source: §4 · pp. 6–7
- A multidimensional partition improved average imbalance by 30% in the LWFA example but did not improve performance because reshaping overhead outweighed the gain.Partitions were concentrated near the laser propagation axis where particles accumulate, with wider partitions toward the simulation-box edge.Source: §4 · p. 7, Figure 3 · p. 7
5. Using vector (SIMD) units
Explicit SIMD implementation targets the particle push and current deposition, which account for about 90% of execution time. The approach preserves existing particle data structures while reorganizing data for vector processing and adapting the PIC algorithm to vector operations.
5. Using vector (SIMD) units
- About 90% of execution time is spent on the particle push and electric current deposition, making them the main targets for explicit SIMD optimization.The authors note that rethinking the algorithm and implementing vector code explicitly is needed for optimal performance.Source: §5 · pp. 7–8
5.1. Data structures for vector code
- A 4×3 transpose uses SIMD shuffles to convert interleaved particle positions into separate x, y, and z vectors, costing about 3 operations per particle versus about 350 for a linear push.The conversion supports vector processing while retaining the existing serial data layout.Source: §5.1 · p. 8
- Storing particle positions relative to the local mesh vertex, normalized to cell size, with the vertex index supports accurate, stable single-precision simulations and uniform noise across the grid..Source: §5.1 · p. 8
- Field interpolation gathers values for particles at varying grid locations, while current deposition adds a dummy component so four cell values can be accumulated with vector operations.The dummy component adds a small memory overhead without affecting the global code structure.Source: §5.1 · pp. 8–9
5.2. Vector version of the PIC algorithm
- The vector PIC algorithm processes nV particles through interpolation, motion updates, trajectory creation, and storage; current accumulation needs special handling to avoid conflicts when particles share a cell.Trajectory splitting is mostly serial, and current contributions are accumulated by looping through virtual trajectories using vector operations.Source: §5.2 · p. 9
6. OASCR Joule Metric
OASCR Joule Metric tests on Jaguar evaluated OSIRIS performance improvements for large-scale LWFA simulations. The same-hardware tests showed substantial gains, and full-system runs measured both PIC throughput and floating-point performance.
6. OASCR Joule Metric
- In repeated SIMD (SSE) tests on the same parallel partition and unchanged hardware, peak code performance improved to 179.9 G particles/s and all LWFA simulations achieved ∼7× speedups.The simulations ran on ∼55 thousand cores; Table 1 reports baseline and improved code versions.Source: §6 · p. 10, Table 1 · p. 10
- Baseline LWFA performance was 9–20 times lower because of load imbalance.The test set included 200 TW and 1 PW laser scenarios, as well as a warm-plasma test.Source: §6 · p. 10
- On the full Jaguar system, warm-plasma performance was about 0.73 × 10^12 particles/s, while peak FP performance reached 0.74 PFlops, or 42% of HPL.Table 2 summarizes performance on about 220,000 cores.Source: Table 2 · p. 11, §6 · p. 11
- With only four times the cores, the code hyper-scaled in both scenarios, achieving 9.48–16.8× speedups; quadratic interpolation used ∼2.7× more operations than linear interpolation but ran only ∼1.8× slower.The shared-memory algorithm improved load balance, while vector optimizations improved floating-point efficiency.Source: §6 · p. 11
- At full-system scale, LWFA run 1 reached 7.09 × 10^10 particles/s and a 16.8× speedup, reducing its estimated runtime from 2 days 18 hours to under 4 hours.The reported core push time was 3.12 µs; the speedup over improved results at 55,000 cores was 2.4×.Source: §6 · pp. 11–12
7. Performance on Tier-0 Systems
Tier-0 tests evaluate OSIRIS performance on a workstation, Sequoia, and BlueWaters using warm-plasma workloads. The tests demonstrate petascale performance and scaling to Sequoia’s full system.
Workstation baseline
- 134.4 million particle pushes per second per CPU were achieved for linear interpolation on an 8-core workstation, with approximately 50% floating-point efficiency.The test used the CPUs’ AVX vector unit in single precision.Source: §7 · p. 12, Table 3 · p. 13
Sequoia
- Over 96% weak-scaling efficiency and 75% strong-scaling efficiency were measured on Sequoia, with the final strong-scaling run taking 44.2 s versus approximately 3.5 hours at 4096 cores.The strong-scaling efficiency drop was attributed mainly to load imbalance and a small problem size per core.Source: §7 · p. 13
BlueWaters
- 2.2 PFlop/s, or 31% of peak theoretical CPU-partition performance, was sustained on BlueWaters across 772,480 cores.The test used approximately 10^13 particles.Source: §7 · p. 13
8. Overview
The paper develops OSIRIS techniques for exploiting parallelism in large-scale LWFA modeling. The resulting tools scale to over 10^6 cores and sustain more than 2 PFlop/s, while shared-memory parallelism addresses high-core-count load imbalance.
Overview
- Over 10^6 cores and 2 PFlop/s were demonstrated while modeling over 10^10 cells and 10^13 particles.The reported performance exceeded 2 × 10^12 particle pushes per second using quadratic interpolation.Source: §8 · p. 14
- Shared-memory parallelism provided perfect load balance within each node and significant speedup, whereas dynamic boundary adjustment did not improve LWFA performance.The latter was limited by small per-core simulation volumes and load-redistribution overhead.Source: §8 · p. 14
- The tools are presented as opening avenues between theoretical computational studies and laboratory experiments through full-scale, accurate, quantitative simulations.The paper notes that current HPC systems provide the necessary computing power, but require efficient use.Source: §8 · pp. 14–15