Source-linked AI summary
PKDGRAV3: Beyond Trillion Particle Cosmological Simulations for the Next Era of Galaxy Surveys
Douglas Potter, Joachim Stadel, Romain Teyssier
TL;DR
Large galaxy surveys require accurate nonlinear-gravity predictions from simulations with enormous volumes, mass resolution, and particle counts. The paper develops PKDGRAV3 by combining FMM, adaptive individual time-steps, GPU acceleration, and low-memory execution. It evolves 2 trillion particles to z = 0 in less than 80 hours and benchmarks 8 trillion particles on Titan.
Problem
Future galaxy surveys require better-than-1% nonlinear matter predictions, large volumes, high mass resolution, and fast repeated simulations across cosmological models.
Method
PKDGRAV3 combines the Fast Multipole Method with adaptive individual particle time-steps, GPU acceleration, and memory optimization for large N-body simulations.
Results
The code evolves 2 trillion particles from z = 49 to z = 0 in less than 80h on 4000+ GPU nodes and benchmarks 8 trillion particles on 18000 Titan nodes.
Takeaways & Limitations
PKDGRAV3 establishes the feasibility of extreme N-body simulations for preparing and analyzing large galaxy surveys.
Takeaways & Limitations
All benchmarks use fixed particle mass and the same run parameters as the 2×10^12-particle production simulation rather than relaxed accuracy settings.
Abstract
from arXiv · showhide
We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18000 nodes and a peak performance of 10 Pflops.
1 Overview of the Problem
Future galaxy surveys require highly accurate nonlinear-gravity predictions, motivating simulations with enormous particle counts, large volumes, and fast time-to-solution. The paper reports a 2 trillion-particle LCDM simulation evolved from z = 49 to z = 0 in less than 80 hours on 4000+ GPU-accelerated nodes.
- Upcoming LSST, Euclid, and WFIRST surveys aim to constrain cosmological parameters below the percent level using weak lensing and galaxy clustering.
- N-body simulations provide theoretical predictions for nonlinear matter dynamics by evolving macro-particles under mutual gravitational attraction.
- The simulation’s full-sky matter distribution contains nonlinear, filamentary structure on small scales.
- 2 trillion particles are required to achieve the needed mass resolution for modeling small dark matter haloes within a roughly 2 Gpc/h volume.
- The paper reports a 2 trillion-particle LCDM simulation evolved from z = 49 to z = 0 in less than 80h on 4000+ GPU-accelerated nodes.
2 Current State of the Art
Cosmological simulations must evolve fixed-duration universes with vast particle counts while exploiting increasingly capable supercomputers. Earlier tree and hybrid methods reached trillion-scale simulations, while FMM offers O(N) scaling and PKDGRAV3 targets 8 trillion particles with low memory use.
- Cosmological simulations must cover 13.7 Gyr while fitting the largest possible particle count into available computer memory.
- Earlier efforts included 0.3 trillion particles with GADGET-3, 1 trillion with GreeM, 3.7×10^12 benchmarks with HACC, and 1 trillion with 2HOT.
- The simulated density field spans underdense voids to halo centers with densities five orders of magnitude above the mean.
- Barnes-Hut and related tree or P3M codes scale as O(N ln N), whereas FMM can provide O(N) scaling for tree-based gravity solvers.
- PKDGRAV3 combines a high-performing FMM implementation with memory optimization, reaching 10 Pflops and 8 trillion particles on 18000 Titan nodes.
3 Algorithmic Improvements
PKDGRAV3 improves large cosmological simulations through FMM gravity calculations, adaptive multi-stepping with dual trees, GPU-suitable interaction evaluation, and memory-conscious execution. These choices target accurate forces while reducing unnecessary work and enabling extreme particle counts.
- 3.1 Fast Multipole Method: PKDGRAV3 creates PP, PC, CC, and CP2 interaction lists whose evaluations produce particle forces, with opening-criterion calculations costing about 2% of total computing.
- 3.1 Fast Multipole Method: FMM uses tree-based multipole expansions and an asymmetric opening criterion to control force-error magnitude and spatial correlations.
- 3.2 Multiple Time Stepping with Dual Trees: Individual particle time-steps are restricted to 2^-l multiples of a base step, assigning particles to hierarchical rungs according to dynamical activity.
- 3.2 Multiple Time Stepping with Dual Trees: Dynamically active regions automatically receive more time-steps, while only sink cells containing particles on the relevant or higher rungs are walked.
- 3.2 Multiple Time Stepping with Dual Trees: Dual trees reduce repeated full-tree costs when active particles fall below about 5%, although dual trees require at least two rungs below the fixed rung.
- GPU acceleration evaluates interaction and multipole work, while CPU execution handles geometrically complex tree walking and branch-divergent operations.
- 3.4 Memory: PKDGRAV3 minimizes memory usage through direct I/O, memory-balanced domain decomposition, reduced tree storage, and careful node-level partitioning.
4 Performance Results
PKDGRAV3 achieves efficient trillion-particle cosmological simulation through adaptive multi-stepping, GPU acceleration, and scalable O(N) performance. Benchmarks validate runtime predictions, strong scaling, and feasibility of simulations up to 8 trillion particles.
- Multi-stepping: A factor-of-4 to 5 speed-up comes from multi-stepping, which produced approximately 5000 ± 10% effective time steps.Particles choose their own time-step rung, while the average step takes about 50 seconds excluding on-the-fly analysis.
- GPU acceleration: GPU acceleration reduced a complete step from 6507s to 1629s, a 4.0x improvement in time-to-solution.A single force evaluation improved from 1138s on CPU-only execution to 119.5s with the GPU.
- Weak scaling: 3.8 million particles per second per node was achieved by PKDGRAV3 for a 1 trillion-particle weak-scaling comparison, versus 1.7 for HACC and 1.2 for 2HOT.The comparison used the same simulation size of 1.0 × 10^12 particles.
- Strong scaling: A 100-fold node increase incurred a 70% penalty, yielding a projected runtime slightly longer than one hour.The baseline projected simulation time was approximately 67 hours.
- Raw performance: 10 Pflops were measured on Titan, compared with its 17.59-Pflops LINPACK performance.The reported measurement used mixed-precision floating-point code.
5 Implications
PKDGRAV3’s reduced time-to-solution changes cosmological simulations from explanatory tools into required components of future survey-data analysis, while continued speed gains are expected to further reshape workflows.
- The observed performance gains require continued algorithmic refactoring for new hardware, with co-design offering another route to improvement.
- Future simulations will be needed to extract fundamental physical parameters from survey data and must become part of data-analysis pipelines.
- PKDGRAV3’s time-to-solution is expected to reach 8 hours or less within the decade as computational speed continues to increase.
- Raw particle data will not be stored and post-processed; instead, data analysis will occur during the simulation.
A Computational Challenges
The production run encountered a slow node and a rare GPU completion failure, but controlled work submission mitigated the latter and the combined problems had little effect on total runtime.
- One substandard node caused the entire simulation to take twice as long because other nodes waited for its work to finish.The problem disappeared after restarting without that node; its exact cause was unknown but likely a rogue process.
- A rare GPU problem caused accepted work to never complete after GPU workload crossed a threshold during the simulation.Sending work in a more controlled fashion eliminated or greatly reduced the problem, with slightly decreased performance.
- The two failures had very little impact on total runtime, which ended almost exactly at the predicted duration.