Source-linked AI summary
NekRS, a GPU-Accelerated Spectral Element Navier-Stokes Solver
Paul Fischer, Stefan Kerkemeier, Misun Min, Yu-Hsiang Lan, Malachi Phillips, Thilina Rathnayake, Elia Merzari, Ananias Tomboulides, Ali Karakus, Noel Chalmers, Tim Warburton
TL;DR
Large thermal-fluid simulations demand scalable methods for billions of degrees of freedom and long integrations, especially on GPU-based leadership systems. NekRS combines high-order spectral elements, GPU-oriented OCCA kernels, and Nek5000/libParanumal developments, demonstrating large-scale performance up to 27,648 V100 GPUs and 60B gridpoints. Its scope includes incompressible Navier–Stokes simulations, with conjugate heat transfer supported but not further discussed in the formulation.
Problem
Thermal-fluid simulations can involve billions of degrees of freedom and hundreds of thousands of timesteps, creating weeks- or months-long runtimes and motivating GPU-oriented HPC methods.
Method
NekRS is a C++/OCCA GPU-oriented solver combining high-order spectral elements, Nek5000 scaling and interfaces, and libParanumal high-performance kernels.
Results
NekRS demonstrates weak- and strong-scaling to 27,648 V100 GPUs for more than 175M spectral elements, corresponding to 60B gridpoints, while sustaining 80–90% of realizable bandwidth-limited peak.
Takeaways & Limitations
NekRS provides a scalable GPU-based Navier–Stokes solver that supports large reactor-geometry simulations and retains the standard Nek5000 interface and features.
Takeaways & Limitations
The formulation discussion omits the energy equation despite current support for conjugate heat transfer, and strong scaling depends on sufficient local points per GPU.
Abstract
from arXiv · showhide
The development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages scalable developments in the SEM code Nek5000 and in libParanumal, which is a library of high-performance kernels for high-order discretizations and PDE-based miniapps. Critical performance sections of the Navier-Stokes time advancement are addressed. Performance results on several platforms are presented, including scaling to 27,648 V100s on OLCF Summit, for calculations of up to 60B gridpoints.
1. Introduction
NekRS addresses the extreme scale and runtime demands of turbulent thermal-fluid simulations by providing a GPU-oriented, open-source solver that builds on established HPC software and supports existing Nek5000 applications.
- Billions of degrees of freedom and hundreds of thousands of timesteps can require weeks or months on leading supercomputers.Reducing these runtimes is a principal objective of high-performance computing.
- Thermal-fluid applications span reactor hydraulics, combustion, geophysical, vascular, astrophysical, and fundamental turbulence problems with challenges in scale resolution, multiphysics, and complex domains.Simulation is particularly important where experimental data are expensive or impossible to obtain.
- GPU-based exascale platforms motivate exploiting powerful node architectures for thermal-fluid analysis.NekRS is presented as a GPU-oriented open-source code emerging from two HPC software projects.
- NekRS combines scalable developments from Nek5000 with portable, optimized OCCA kernels and numerical components from libParanumal.libParanumal includes solvers, mesh handling, approximation, time stepping, gather-scatter, halo exchanges, and transport-physics miniapps.
- NekRS preserves the standard Nek5000 interface and features, including conjugate heat transfer, so existing application source code and data files can be used on GPUs.
2. Formulations
NekRS formulates incompressible Navier–Stokes and thermal transport with high-order spectral elements, time splitting, and iterative subsolvers designed for efficient matrix-free GPU execution.
- 2. Formulations: The formulation targets advection-dominated, potentially fully turbulent flows governed by incompressible Navier–Stokes and energy equations.High Reynolds and Peclet numbers require accurate discretizations to limit numerical dispersion and dissipation.
- 2.1. BDF Time Discretization: BDF time discretization forms an implicit Stokes substep, while prior quantities enter through standard BDFk/EXTk extrapolation or a semi-Lagrangian alternative.The coefficients βj and αj are kth-order BDF and extrapolation coefficients, respectively.
- 2.1. BDF Time Discretization: Semi-Lagrangian treatment evaluates prior velocity at characteristic foot points and avoids off-grid interpolation by solving a hyperbolic advective subproblem.The method can avoid the CFL constraint on timestep size.
- 2.2. Implicit Stokes Solve: The fractional-step Stokes solve decouples advection, viscous Helmholtz, and pressure-Poisson subproblems for physics-specific iterative treatment.Pressure is intrinsically the stiffest substep because it governs the fastest modes in the incompressible model.
- 2.3. Spectral Element Discretization: High-order spectral elements represent fields with tensor-product polynomials on globally unstructured curvilinear hexahedra.GLL points provide well-conditioned operators, pointwise quadrature, and a diagonal mass matrix.
- 2.3. Spectral Element Discretization: Matrix-free assembly forms vectors rather than matrices, and factored operators apply local stiffness actions while Q and QT handle communication.The spectral-element communication pattern has a unit-depth stencil for all N.
- 2.3. Spectral Element Discretization: Tensor-product differentiation uses efficient contractions, giving leading-order O(nN) work with favorable O(N) work-to-storage ratios.The full gradient combines tensor contractions, pointwise geometric transformations, and reusable geometric data.
- 2.3. Spectral Element Discretization: GLL quadrature supplies discrete inner products and a local diagonal mass matrix, while higher-order integration is required for the degree-3N advection integrand.The Jacobian and quadrature weights define elementwise integration.
3. Parallel GPU Development
NekRS combines GPU-resident SPMD execution, portable OCCA kernels, scalable partitioning, and communication strategies tailored to high-order spectral elements. Kernel and preconditioner refinements improve pressure-solve performance while exposing remaining bandwidth and pressure-step costs.
- Parallel GPU execution: NekRS runs one MPI rank per GPU with data resident on the device, copying to the host only when needed.This supports distributed-memory time advancement of the incompressible Navier–Stokes equations.
- Portable GPU kernels: OCCA abstracts OpenCL, CUDA, and HIP targets, while NekRS extends libParanumal kernels and preserves the standard Nek5000 interface.This combines performance portability with reuse of application-specific source code and data files.
- Scaling and partitioning: GPU strong scaling reaches 2–4 million points per MPI rank, compared with 2,000–4,000 points per rank for Nek5000 on Mira.NekRS routinely runs 175 million elements at N = 7, corresponding to n = 60B.
- Parallel communication: gslib selects pairwise, crystal-router, or all-reduce communication by trialing strategies for the given adjacency graph.The optimization was particularly important for distributed coarse-grid AMG, where it produced a reported 10× benefit.
- High-order kernels: The 2D advection thread structure avoids the 1,024-thread limit reached by 3D structures at Nq = 10 and reduces slow shared-memory loads nearly threefold.For Nq ≤10, runtime JIT compilation allows selecting the more performant 2D or 3D kernel.
- Preconditioning performance: A 4.85× pressure-solve speedup results from adaptive gather-scatter, FP32-based smoothing, and projection-based initial guesses.Despite this reduction, pressure-related kernels still account for about 25% of wall-clock time; characteristics account for about 18%.
4. Application Performance
NekRS demonstrates strong GPU performance and scalability across thermal-fluids applications, including comparisons with Nek5000, rod-bundle scaling to all of Summit, and results across GPU architectures.
- 4.1. Comparison of Summit and Mira: 4.8× faster: NekRS reduced spacer-grid time per step to 0.14 seconds on 98 Summit nodes versus 0.68 seconds for Nek5000 on Mira.The runs used comparable parallel efficiency and equivalent tolerances, timestep sizes, and other parameters.
- 4.2. Summit Scaling Performance: 2.5M points per GPU: rod-bundle cases exhibited excellent strong scaling to all Summit V100s at the 80% efficiency level.The paper identifies n/P, rather than processor count alone, as the leading indicator of parallel scalability.
- 4.2. Summit Scaling Performance: 18% efficiency loss: the Rod-1717 case maintained reasonable weak scaling over a 53-fold increase in processor count.The tests used n/P = 2.1M, beyond the strong-scale limit, so communication overhead substantially influenced the results.
- 4.2. Summit Scaling Performance: 54% efficiency: full-core weak scaling declined with only a 17-fold increase in processor count, with degradation correlated with higher neighbor counts.At P = 27648, Full-Core had a weak-scale time 20% higher than Rod-1717.
- 4.3. Performance on Other GPU Architectures: 1.5× faster: the NVIDIA A100 outperformed the V100 in preliminary single-GPU results, while MI100 and MI60 reached 85% and 60% of Summit V100 performance.The AMD results used OCCA’s HIP backend, but hand-tuning remained necessary for optimized performance on each device.
- 4.3. Performance on Other GPU Architectures: 30B grid points: the target configuration of 300,000 pebbles was described as within Summit’s current performance envelope.A 44,257-pebble example contained 4.5B grid points and used 1,788 V100s, with 84% of time spent in the pressure solve.
5. Conclusions
NekRS demonstrates GPU-based Navier–Stokes simulation at extreme scale, sustaining high bandwidth-limited performance on Summit. The solver maintains 80% parallel efficiency at practical local problem sizes and has preliminary results on newer GPU platforms.
- 27,648 V100 GPUs on Summit handled more than 175M spectral elements, equivalent to n = 60B gridpoints.The demonstrated reactor-geometry calculations included both weak- and strong-scaling studies.
- 80–90% of the realizable bandwidth-limited peak was sustained by NekRS.
- 80% parallel efficiency was realized for local problem sizes of n/P ≈2.5M, where P denotes the number of GPUs.
- Preliminary timing data were presented for NVIDIA A100 and AMD MI100 GPUs.