Source-linked AI summary
Nyx: A Massively Parallel AMR Code for Computational Cosmology
Ann Almgren, John Bell, Mike Lijewski, Zarija Lukić, Ethan Van Andel
TL;DR
Nyx addresses the need for many large-scale cosmological simulations by modeling dark matter and gas dynamics with adaptive mesh refinement. The code is validated on cosmological tests and shows excellent weak scaling to almost 50,000 processors.
Problem
Cosmological simulations are needed to connect matter distributions with cosmological models, but nonlinear evolution and uncertain initial conditions require many simulations rather than one heroic run.
Method
Nyx is an N-body and gas-dynamics code that evolves dark-matter particles and gas using Eulerian block-structured adaptive mesh refinement and particle-mesh gravity.
Results
Nyx shows excellent weak scaling to almost 50,000 processors and is validated in pure dark-matter and adiabatic hydrodynamics simulations.
Takeaways & Limitations
Nyx provides a massively parallel framework for large-scale cosmological simulations with validated dark-matter and gas-dynamics capabilities.
Takeaways & Limitations
The Santa Barbara comparison does not use exactly the same initial conditions or starting redshift as the reference simulations, so exact agreement is not expected.
Abstract
from arXiv · showhide
We present a new N-body and gas dynamics code, called Nyx, for large-scale cosmological simulations. Nyx follows the temporal evolution of a system of discrete dark matter particles gravitationally coupled to an inviscid ideal fluid in an expanding universe. The gas is advanced in an Eulerian framework with block-structured adaptive mesh refinement (AMR); a particle-mesh (PM) scheme using the same grid hierarchy is used to solve for self-gravity and advance the particles. Computational results demonstrating the validation of Nyx on standard cosmological test problems, and the scaling behavior of Nyx to 50,000 cores, are presented.
1. Introduction
Nyx is introduced to address the scale, physics, and hardware demands of modern cosmological simulations. It combines dark-matter particles, gas dynamics, adaptive mesh refinement, and massively parallel computation.
- Scientific and computational motivation: A single simulation is insufficient for many cosmological questions, motivating ensembles with different cosmologies and approximations to governing physics.Simulations also help validate semi-analytic models and connect theory with observation.
- Scientific and computational motivation: Future surveys require simulations spanning roughly 1 Gpc while resolving scales near 10 kpc or smaller.Such simulations may require tens of billions of particles and multiple refinement levels.
- Scientific and computational motivation: Cosmological simulations must incorporate baryonic physics because gas effects become increasingly significant at smaller scales.Radiative cooling and heating remain uncertain, while galaxy formation on small, dense scales remains unresolved.
- Nyx design: Nyx models dark matter as Lagrangian particles gravitationally coupled to an inviscid ideal fluid representing baryonic matter.The fluid uses an Eulerian finite-volume framework with block-structured AMR, while the same mesh supports particle-mesh gravity and particle evolution.
- Nyx design: Nyx provides multiple time-subcycling strategies, including user-specified and optimal subcycling choices.These options allow the runtime time-stepping pattern to vary across refinement levels.
- Nyx design: Nyx targets efficient use of future hardware through massively parallel computation and support for multicore shared-memory nodes.The code is built on BoxLib, which supports hierarchical MPI and OpenMP programming and modular routines portable to GPUs.
2. Basic Equations
Nyx formulates expanding-universe gas dynamics and dark-matter evolution in comoving coordinates. Dark matter is represented by particles, while baryons and dark matter jointly determine the gravitational field.
- Expanding Universe: The expanding-universe formulation introduces the scale factor a into the gas conservation laws, with a = 1/(1 + z).The scale factor can also serve as a time variable in the formulation.
- Expanding Universe: Comoving baryonic density is related to proper density by ρb = a^3ρproper, while U denotes peculiar proper baryonic velocity.These variables define the comoving-frame gas state.
- Gas Dynamics: Nyx evolves internal and total energy together and synchronizes them at the end of each time step.The synchronization procedure depends on the magnitude of internal energy relative to total energy.
- Gas Dynamics: The gas equations include pressure, gravitational, and combined heating and cooling source terms.The equation of state uses p = (γ − 1)ρe with γ = 5/3, while ΛHC represents heating and cooling.
- Dark Matter: Dark matter evolves according to the collisionless Boltzmann equation, which Nyx approximates by Monte Carlo sampling with discrete particles.Particle i has comoving location xi and peculiar proper velocity ui, and its acceleration is evaluated from the gravitational field.
- Self-gravity: Dark matter and baryons jointly contribute to the gravitational field, which is obtained by solving for the gravitational potential from the comoving dark-matter density.The gravitational acceleration is g = −∇φ.
3. Code overview
Nyx advances cosmological gas and dark matter with comoving gas dynamics, particle-mesh gravity, and coordinated time integration. Its algorithm combines flux-based fluid updates, particle kick-drift-kick evolution, gravity solves, and source-term treatments.
- Gas dynamics: The gas state is advanced in comoving coordinates using fluxes and gravitational, internal-energy, and heating/cooling source terms.The formulation includes the scale factor a in the state, fluxes, and source terms.
- Gas dynamics: Nyx computes hydrodynamic fluxes with an unsplit Godunov method using characteristic tracing and full corner coupling.The formulation preserves second-order temporal accuracy with the expanding-universe scale factor included.
- Dark matter: Dark matter particles are advanced with a kick-drift-kick sequence using gravitational acceleration interpolated from cell centers to particle locations.The first kick advances velocities by Δt/2, the drift uses a time-centered velocity, and the final velocity update follows gravity at the new time.
- Self-gravity: The particle-mesh gravity pipeline deposits particle mass with cloud-in-cell, solves the potential using multigrid, and differentiates it to obtain gravitational acceleration.On a uniform mesh, the centered-difference and interpolation choices make a particle’s self-force identically zero.
- Time integration: The timestep is the smallest of constraints from dark-matter motion, gas hyperbolic evolution, and scale-factor growth.The scale factor is limited to changing by no more than 1% per coarsest-level step; non-zero source terms may impose additional constraints whose details are deferred.
- Single-level integration algorithm: Single-level integration advances the scale factor and gas, transports particles, recomputes particle density and gravity, then corrects source terms and particle velocities.Heating and cooling may use predictor-corrector source treatment or Strang splitting, with implementation details deferred for specific mechanisms.
4. AMR
Nyx uses a dynamically regridded, block-structured AMR hierarchy with configurable time-subcycling strategies. Its recursive multilevel advance coordinates fluid evolution, particle motion, gravity solves, synchronization, and conservation corrections across levels.
- AMR hierarchy: Nyx represents the solution on nested rectangular grids, refining spatially by a factor of two between levels and allowing dynamically changing finest levels.Particles are assigned to the finest grid containing their locations, with ghost-cell regions supporting particle motion between redistributions.
- AMR hierarchy: User-selected tagging criteria, including overdensity or arbitrary fluid and particle variables, trigger construction of finer grid patches during regridding.Existing fine-grid data is copied, while newly refined regions are interpolated from coarser grids using limited piecewise linear profiles.
- Subcycling options: Nyx supports no subcycling, standard subcycling, user-specified patterns, and optimal subcycling that adapts the pattern to level-specific timestep constraints.Standard subcycling keeps ∆t/∆x constant across levels, whereas optimal subcycling recomputes the pattern at specified coarse-time intervals.
- Subcycling options: More complex subcycling patterns may benefit large parallel computations, but their computational-efficiency trade-offs are deferred to future work.The deferred cases include parallelization over refinement levels rather than only over grids at one level.
- Multilevel algorithm: The timestep at each level is derived from its independently computed maximum timestep and the selected subcycling ratios, with n_ℓ defining fine-level steps per coarse step.A complete multilevel advance advances all levels to the same time; n_ℓ=1 for no subcycling and n_ℓ=r^ℓ for standard subcycling.
- Multilevel algorithm: The recursive Advance procedure updates gas and particles, performs gravity solves and source corrections, recursively advances finer levels, then synchronizes and refluxes hydrodynamic quantities.Explicit refluxing and averaging down maintain consistency and conservation between fine and coarse levels.
5. Software Design and Parallel Performance
Nyx is built on BoxLib's hybrid MPI/OpenMP framework, with distributed grids and particles supporting large parallel simulations. On NERSC Hopper, its weak-scaling study reached 49,152 processors with less than a 50% total-time increase under a 1000-fold processor increase.
- Software design: Nyx uses BoxLib's hierarchical MPI/OpenMP programming model for parallel block-structured AMR applications.BoxLib separates C++ infrastructure from Fortran90 numerical kernels and supports modular routines designed for portability, including to GPUs.
- Software design: Particles are distributed according to their finest-level grid location, so particle-grid interactions require no inter-node communication and scale linearly with particle count.Particle movement and velocity updates likewise require no communication between processors or nodes.
- I/O design: Nyx writes self-describing checkpoint and plotfile directories containing AMR hierarchy metadata and level-specific data for restart, analysis, and visualization.The format supports user-specified output intervals and automatic byte-order and precision translation when files are read on other machines.
- Parallel performance: The weak-scaling experiment placed one 128^3 grid on each six-core NUMA node and reported hybrid MPI/OpenMP results on NERSC Hopper.The replicated Santa Barbara problem was expanded by replicating its original 256^3 particles across domain directions.
- Parallel performance: 49,152 processors produced less than a 50% increase in total time relative to 48 processors, despite a 1000-fold increase in processor count.Timings separated hydrodynamics, multigrid setup and initialization, and multigrid V-cycles; each solve used seven V-cycles to reach 10^-12 tolerance.
6. Validation
Nyx is validated through idealized particle tests and cosmological dark-matter and gas simulations. Its results generally agree with established codes, while comparisons identify resolution and initial-condition boundaries.
- Validation tests: Nyx reproduces the two-particle orbit with deviations below 0.0002% of the radius and kinetic-energy differences below 0.0004%.The study isolates the particle-location and velocity update component in a resolution test.
- Validation tests: Analytical tests including MacLaurin spheroids and Zel’dovich pancakes validate particle dynamics and second-order gravity-solver convergence.These tests supplement the two-particle orbit study with known analytical solutions.
- Dark Matter Only Simulations: Nyx and Gadget-2 show a strong match in dark-matter correlation functions, with convergence toward Gadget-2 as resolution increases.The AMR run at effective 1024^3 resolution closely matches the uniform 1024^3 result.
- Dark Matter Only Simulations: Nyx agrees with Gadget-2 for halo mass functions, while common refinement strategies suppress the low-mass end when small halos form across the domain.The reported AMR behavior is consistent with criteria established in prior code comparisons.
- Dark Matter Only Simulations: Halo correlation functions converge quickly, and AMR and uniform 1024^3 runs show excellent agreement in resolved halo inner structure.The smallest-halo correlation function retains a low-resolution offset before converging.
- Santa Barbara Cluster Simulations: Nyx obtains Santa Barbara cluster properties of 1.15 × 10^15M⊙, 2.7 Mpc, and NFW concentration 7.1, consistent with reported comparison values.The comparison values are (1.1 ± 0.05) × 10^15M⊙, 2.7 ± 0.04 Mpc, and a rough concentration guideline of 7.5.
- Santa Barbara Cluster Simulations: Nyx agrees well with other AMR codes in cluster radial profiles, including central entropy flattening that contrasts with SPH results.The profiles include dark-matter density, gas density, pressure, and entropy.
7. Conclusions and Future Work
Nyx is presented as a large-scale cosmological N-body and gas-dynamics code designed for efficient parallel execution. The authors report validation and nearly 50,000-processor weak scaling, with future work targeting additional gas physics and cosmological models.
- Conclusions: Nyx is designed for large-scale cosmological simulations and efficient utilization of tens of thousands of processors.The paper presents it as an N-body and gas-dynamics code.
- Conclusions: Timings to almost 50,000 processors show excellent weak scaling behavior.The reported scaling result concerns the code's ability to maintain performance as processor count and workload increase.
- Conclusions: The paper validates Nyx in pure dark-matter runs and in dark matter with adiabatic hydrodynamics.These validation results cover both gravity-only and coupled gas-and-dark-matter simulations.
- Future Work: Future work will add source-term details and simulations with gas heating and cooling mechanisms for applications requiring increased fidelity.The authors identify these mechanisms as targets for future papers.
- Future Work: Planned extensions include dynamical dark energy and modifications of Einstein’s gravity beyond ΛCDM.The existing grid structure and multigrid Poisson solver are described as supporting extensions to nonlinear elliptic methods.