PaperlayerView exact revision on arXiv

Source-linked AI summary

AbacusSummit: A Massive Set of High-Accuracy, High-Resolution $N$-Body Simulations

Nina A. Maksimova, Lehman H. Garrison, Daniel J. Eisenstein, Boryana Hadzhiyska, Sownak Bose, Thomas P. Satterthwaite

arXiv:2110.11398v1astro-ph.CO

TL;DR

Large-scale structure surveys need high-precision mock data for sub-percent comparisons, systematic-error control, and pipeline testing. This paper releases AbacusSummit, a large suite of high-accuracy N-body simulations spanning cosmologies, realizations, and data products. The suite reaches 70M particles/sec per node early and 45M particles/sec per node at z = 0.1, while totaling nearly 60 trillion particles.

  • Problem

    Forthcoming large-scale structure surveys require high-precision mock data for sub-percent comparisons, systematic-error control, and cosmological-parameter pipeline testing.

  • Method

    The authors construct and publicly release AbacusSummit, using the Abacus N-body code to produce simulations across Planck2018 ΛCDM, secondary cosmologies, derivative grids, and covariance boxes.

  • Results

    70M particles/sec per node at early times and 45M particles/sec per node at z = 0.1, with nearly 60 trillion particles across the suite.

  • Takeaways & Limitations

    AbacusSummit provides a large, diverse, publicly available simulation and data-product resource for next-generation large-scale structure cosmology.

  • Takeaways & Limitations

    Small-scale correlation-function convergence is limited by particle mass resolution as well as force softening, with the mass-resolution dependence scaling as a^-1/2.

Abstract

from arXiv · show

We present the public data release of the AbacusSummit cosmological $N$-body simulation suite, produced with the $\texttt{Abacus}$ $N$-body code on the Summit supercomputer of the Oak Ridge Leadership Computing Facility. $\texttt{Abacus}$ achieves $\mathcal{O}\left(10^{-5}\right)$ median fractional force error at superlative speeds, calculating 70M particle updates per second per node at early times, and 45M particle updates per second per node at late times. The simulation suite totals roughly 60 trillion particles, the core of which is a set of 139 simulations with particle mass $2\times10^{9}\,h^{-1}\mathrm{M}_\odot$ in box size $2\,h^{-1}\mathrm{Gpc}$. The suite spans 97 cosmological models, including Planck 2018, previous flagship simulation cosmologies, and a linear derivative and cosmic emulator grid. A sub-suite of 1883 boxes of size $500\,h^{-1}\mathrm{Mpc}$ is available for covariance estimation. AbacusSummit data products span 33 epochs from $z=8$ to $0.1$ and include lightcones, full particle snapshots, halo catalogs, and particle subsets sampled consistently across redshift. AbacusSummit is the largest high-accuracy cosmological $N$-body data set produced to date.

1. INTRODUCTION

The paper introduces AbacusSummit as a public, high-accuracy N-body simulation suite designed for forthcoming large-scale structure surveys. It combines diverse cosmologies, large particle counts, and publicly released data products for cosmological applications.

  • Motivation: Forthcoming surveys require high-precision mock data to support sub-percent observation comparisons, systematic-error control, and pipeline testing.Approximate methods can generate sample data, but accurate cosmological N-body simulations remain a central tool for forecasts.
  • Contribution: AbacusSummit releases a massive suite of high-accuracy N-body simulations produced with the Abacus code on Summit.The release is intended for the next generation of large-scale structure surveys.
  • Implementation: Abacus achieves 10^-5 fractional force accuracy through a novel Poisson-equation solver and GPU-accelerated software engineering.The code’s force accuracy and performance are central technical features of the suite.
  • Suite design: Nearly 60 trillion particles span 97 cosmological models, including large core boxes, resolution-study resimulations, and 1883 smaller boxes for statistical-error investigations.Most core boxes contain 330 billion particles with mass 2 × 10^9 h^-1 M⊙.
  • Paper scope: The paper documents suite specifications, production methods, performance, and publicly released data products for unrestricted use.The article is part of a series covering Abacus, the CompaSO halo finder, and merger trees.

2. AbacusSummit: SUITE SPECIFICATIONS

AbacusSummit is a broad public simulation suite designed for high-accuracy large-volume studies, cosmological parameter exploration, covariance estimation, and code or resolution comparisons. It combines 97 cosmologies, nearly 60 trillion particles, extensive redshift coverage, and diverse halo and particle products.

  • Scientific scope: The suite targets high-accuracy Planck2018 ΛCDM simulations, secondary cosmologies, covariance estimation, and comparisons across codes, resolutions, and parameters.Its design choices in cosmological parameters, realizations, and data products were selected to support these applications.
  • Simulation scale: Nearly 60 trillion particles comprise the suite, including 25 base Planck2018 boxes of side 2 h^-1 Gpc with particle mass 2 × 10^9 h^-1 M⊙.Additional simulations provide higher and lower mass resolution for resolution studies and prototyping.
  • Cosmological coverage: 97 cosmological models span Planck2018 ΛCDM, matched flagship cosmologies, massive-neutrino cases, and an 8-dimensional derivative and emulator grid.The derivative grid uses matched positive and negative parameter steps around the fiducial c000 cosmology.
  • Simulation scale: 1883 periodic 500 h^-1 Mpc boxes support statistical-error studies and covariance-matrix estimation.These boxes use the base mass resolution.
  • Data products: Each simulation provides particle subsamples, halo catalogs, merger trees, light cones, density estimates, and projected light-cone maps.CompaSO L1 halo catalogs with comprehensive statistics are available at primary and secondary redshifts.
  • Code parameters: A typical base-resolution simulation requires roughly 1100 timesteps to reach z = 0.1, with timesteps decreasing rapidly near z = 15 and approaching 0.001H^-1_0 at late times.The late-time behavior follows the use of proper force softening.

3. ABACUS

Abacus decomposes gravitational forces into independently computed near- and far-field terms, combining direct GPU summation with Taylor-expanded far-field approximations. Its slab-pipeline and rotating domain decomposition support large simulations while imposing memory and light-cone boundary constraints.

  • Force solver: Abacus splits each particle’s gravitational force into near-field and far-field contributions computed with separate techniques.The near-field radius R determines the division, balancing accuracy and computational cost.
  • Force solver: p = 8 Taylor expansions with R = 2 provide excellent far-field force accuracy while maintaining high performance.The derivatives tensor is precomputed from grid geometry and reused at every timestep.
  • Force solver: Direct O(N^2) summation computes near-field forces on GPUs using a compact spline softening kernel.The regular geometry exposes parallelism and efficient memory access, communication, and computation patterns.
  • Timestep pipeline: Each timestep alternates between singlestep force integration and convolution, which produces new Taylor coefficients for the next singlestep.Singlestep also moves particles, updates cells, and computes new multipoles.
  • Pipeline and parallelization: The slab pipeline overlaps CPU far-field work, GPU near-field work, particle integration, and data movement while processing only a rolling subset of slabs.This design avoids holding the whole simulation state in memory and naturally supports distributed execution.
  • Pipeline and parallelization: The rotating 1D domain decomposition sends slab data asynchronously between neighboring nodes, overlapping communication with computation.Each node’s assigned simulation slice changes cyclically rather than remaining fixed.
  • Constraints: Light cones are accurate only to approximately 1.2 h−1 Mpc from transverse edges because periodic extrapolation is not performed there.The estimate is conservative because particles typically move less than one cell width per timestep.

4.1. Overview

Abacus achieves high particle-update rates on Summit by combining a force solver that overlaps CPU and GPU computation with hardware-specific optimizations. Performance is about 70M particles/sec per node at early times and 45M particles/sec per node at z = 0.1.

  • Performance: 70M particles/sec per node is achieved at early times, falling to 45M particles/sec per node at z = 0.1.After z ∼1, GPU near-force computation dominates and performance declines.
  • Performance: The force solver drives performance by overlapping substantial CPU and GPU work through the slab-pipeline structure.Hardware-specific optimizations further improve speed on Summit.

4.2. The Summit System

Summit provides a large, GPU-equipped platform for Abacus, whose performance is evaluated through timing and GPU-throughput measurements. The system-level analysis includes concurrent CPU/GPU work and a conservative comparison with theoretical GPU performance.

  • Hardware: Summit nodes combine two 22-core POWER9 processors with six NVIDIA Tesla V100 GPUs, 512 GB RAM, and 96 GB GPU-accessible high-bandwidth memory.Each node has a theoretical double-precision capability of 40 TF.
  • Timing overview: CPU work is approximately constant while GPU near-field force time increases with clustering, with both workloads running concurrently.Convolution precedes the CPU/GPU work at each timestep.
  • GPU performance: GPU performance is measured in billions of pairwise interactions per second, including host-device communication overhead.The theoretical maximum assumes 26 floating-point operations per interaction and all operations execute as FMA.

4.3. Performance

AbacusSummit’s performance depends on balancing CPU and GPU work as clustering evolves, with singlestep dominating runtime and near-field work becoming increasingly limiting at late times. The implementation achieves high throughput but also exposes I/O, memory, and host-side contention costs.

  • Throughput: 79.5 seconds yields 69.2 million particles per second at z = 90.5, whereas 122 seconds yields 45.1 million particles per second at z = 0.105.These timings are for a typical 2 h^-1 Gpc box executed on 60 compute nodes.
  • Runtime composition: Singlestep accounts for 85–90% of an Abacus timestep’s runtime, while convolution contributes 10–15%.Singlestep updates particles and computes forces; convolution operates on cell multipoles.
  • Workload balance: The optimal CPD is around 1701, or 67 particles per cell, because it balances near-field and far-field workloads while minimizing time-to-solution.GPU work grows as the simulation becomes more clustered, so no single setting balances every timestep.
  • Epoch dependence: Before z ∼1, CPU work masks the 39-second GPU near-field computation; by z = 0.1, near-field work reaches 97 seconds and dominates runtime.The late-time singlestep total is 109 seconds after final GPU forces complete.
  • GPU performance: Peak GPU performance reaches 2200G direct pairwise force calculations per second, about 56 TFlops and 60% of peak theoretical performance.Lower off-peak performance is attributed to early-time flop-to-byte ratios and sparse late-time kernels.
  • Bottlenecks: I/O-bound steps increase GPU performance by about 20%, indicating likely early-time host-side resource contention.Reduced memory pressure allows positions and accelerations to be filled and communicated more efficiently.
  • Overheads: Network transfer is a single burst taking about 5% of runtime and arriving before it is needed, while MPI spinning and memory-management operations add overhead.A typical base simulation requires about 1100 timesteps and 1800 node-hours to reach z = 0.1.

4.4. SIMD Multipoles and Taylors

Abacus accelerates CPU multipole and Taylor calculations with POWER9 VSX SIMD instructions and manually generated loop unrolling. The optimized kernels achieve substantial production throughput while preserving results to rounding error.

  • Vectorization: VSX SIMD intrinsics process two double-precision particle coordinates per 128-bit vector, with meta-code manually unrolling the multipole and Taylor loops.Eight-fold particle-loop unrolling is reported as optimal.
  • Multipoles: Production multipole processing reaches about 740 million particles per second, or about 20 particles per core.Artificial benchmarks were only a few to 10 percent faster per core.
  • Taylor kernels: Taylor kernels are slower per particle but more efficient in FLOPS because their per-particle operation count is several times higher than multipoles.The vectorized calculations agree with non-vectorized results to within rounding error.

4.5. Thread Scheduling and NUMA

Abacus schedules CPU, GPU, and I/O work around Summit’s NUMA structure to preserve locality and overlap computation. Its production configuration uses SMT2, pinned CPU threads, and NUMA-specific GPU queues.

  • NUMA locality: Threads process only the slab half belonging to their NUMA node, with core affinity preventing migration and cross-node contention.A custom OpenMP task scheduler dynamically schedules work within NUMA nodes.
  • Thread allocation: SMT2 provides 84 hardware threads, of which 70 serve OpenMP work, 12 interface with GPUs, and two handle memory freeing and I/O.SMT2 was selected because it produced the best performance among SMT1, SMT2, and SMT4.
  • GPU scheduling: GPU work units use one queue per NUMA node, allowing dynamic dispatch while maintaining NUMA locality.Pinned host memory supports host-GPU communication, and its initial pinning overhead is masked by convolution.

4.6. Production Pipeline

AbacusSummit used an automated Python pipeline to coordinate simulation preparation, execution, and post-processing across four OLCF systems. This automation prioritized efficient production and reduced human error.

  • 4.6. Production Pipeline: A Python pipeline generated simulation parameters, queued initial conditions, executed Abacus, queued post-processing, and transferred data across the production workflow.The workflow involved Summit, Rhea, OLCF HPSS, and NERSC.
  • 4.6. Production Pipeline: Over 150 simulation boxes made automation a priority for efficient production and error reduction.
  • 4.6. Production Pipeline: Initial-condition generation and post-processing were important components alongside the lengthy Abacus execution step.

4.7. Initial Conditions

AbacusSummit generated initial conditions with zeldovich-PLT in an out-of-core workflow on Rhea, enabling very large problems to be handled on a single node.

  • 4.7. Initial Conditions: zeldovich-PLT generated initial conditions out-of-core by buffering state on disk.
  • 4.7. Initial Conditions: The workflow used Rhea’s cluster and Alpine network file system rather than parallelizing initial-condition generation across distributed memory systems.
  • 4.7. Initial Conditions: The method can produce initial conditions for very large problems on a single node.

4.8. Post-processing

AbacusSummit post-processing converted raw simulation outputs into compressed, self-describing products while dynamically scheduling heterogeneous tasks. Typical jobs required a few to a dozen node-hours.

  • 4.8. Post-processing: Each simulation box was post-processed after its terminal redshift to compress data products and package them in self-describing ASDF files.
  • 4.8. Post-processing: Heterogeneous post-processing workloads were dynamically dispatched within batch allocations as nodes became available.
  • 4.8. Post-processing: Task return codes were recorded in status files that supported retrying failed jobs.
  • 4.8. Post-processing: A typical post-processing job took a few to a dozen node-hours, depending on whether full time slices were written.

5. COSMOLOGICAL OPPORTUNITIES WITH AbacusSummit

AbacusSummit supports high-precision large-scale-structure studies through extensive halo catalogs, specialized clustering statistics, phase-matched cosmological comparisons, and accuracy tests. The results also identify resolution and halo-definition boundaries that users must account for.

  • 5.1. Scope: AbacusSummit was designed to identify halos for high-precision large-scale-structure applications and clustering studies across galaxies, matter, and cosmological parameters.
  • 5.1. Scope: 56.0 billion, 68.4 billion, and 67.3 billion L1 halos were catalogued at z = 2, 1.1, and 0.2, respectively.For halos above 250 particles, the corresponding counts were 5.7, 9.1, and 10.7 billion.
  • 5.1. Scope: The modified statistic ∆ξ(r) averages the correlation function within radius r before subtracting ξ(r), retaining localized acoustic information while avoiding a zero crossing.The statistic is also insensitive to constant offsets in ξ and systematic errors on scales much larger than r.
  • 5.1. Scope: Halos above 10^14 h^-1 M⊙ at z = 1.1 had number density 1.6 × 10^-6 h^3 Mpc^-3 and bias exceeding 7, with scale-dependent clustering and a broader acoustic peak.These properties suggest care when interpreting high-mass cluster correlation functions.
  • 5.1. Scope: Phase-matched power-spectrum ratios had lower sample variance than 25 independent boxes until k = 1.0 h Mpc^-1 and were equivalent to more than 50 boxes at k = 0.1 h Mpc^-1.
  • 5.2. Accuracy: The force solver achieved typical relative force errors of 10^-5, while small-scale correlation functions between resolutions matched within 1% over approximately 35–90 h^-1 kpc across z = 0.2–1.4.The comparison used particle masses 2 × 10^9 h^-1 M⊙ and 3.5 × 10^8 h^-1 M⊙.

6. CONCLUSION

AbacusSummit is a uniquely large and diverse public N-body dataset designed for broad large-scale-structure science applications. Its extensive validated data products and persistent initial-lattice memory at low density characterize both its utility and a measurable numerical feature.

  • Roughly 60 trillion particles make AbacusSummit the largest N-body dataset produced to date.
  • The suite provides a large-volume, high-accuracy Planck2018 ΛCDM simulation set plus a grid of 96 other cosmologies.
  • Figure 10 compares Lagrangian-plane cutouts at z = 2.5, z = 1.1, and z = 0.1, showing initial-lattice memory persisting at low redshift in low-density regions.
  • Validated public products include particle subsamples, halo catalogs, merger trees, kernel density estimates, light cones, and projected light-cone density maps.
Loading 2110.11398v1…