PaperlayerView exact revision on arXiv

Source-linked AI summary

A FFT-based finite-difference solver for massively-parallel direct numerical simulations of turbulent flows

Pedro Costa

arXiv:1802.10323v4physics.flu-dyn

TL;DR

The paper addresses the lack of a general massively parallel DNS implementation supporting varied homogeneous pressure boundary conditions. It presents CaNS, combining a second-order pressure-correction scheme with FFT-based Poisson solving and a 2D pencil-like decomposition. The implementation is validated against canonical-flow benchmarks, and its performance is reported as consistent with the performance of the parallelization library it uses.

  • Problem

    A general massively parallel DNS implementation supporting the range of homogeneous pressure boundary conditions available to second-order FFT-based Poisson solvers was lacking.

    Source: §1 · p. 3

  • Method

    CaNS uses a second-order finite-difference pressure-correction method with eigenfunction-expansion FFT-based Poisson solving and a 2D pencil-like decomposition.

    Source: §1 · p. 3, §5 · p. 14

  • Results

    The code was validated by comparing channel-flow profiles with DNS data, and its performance was reported as consistent with that of the 2DECOMP&FFT library.

    Source: §4.1.2 · p. 11, §4.2 · p. 14

  • Takeaways & Limitations

    The method provides a general implementation for different combinations of homogeneous pressure boundary conditions in massively parallel DNS.

    Source: §5 · p. 14

    The validation covered selected canonical laminar and turbulent flows; wall-bounded flows with inflow/outflow conditions are identified as configurations not considered.

    Source: §5 · p. 14

Abstract

from arXiv · show

We present an efficient solver for massively-parallel direct numerical simulations of incompressible turbulent flows. The method uses a second-order, finite-volume pressure-correction scheme, where the pressure Poisson equation is solved with the method of eigenfunction expansions. This approach allows for very efficient FFT-based solvers in problems with different combinations of homogeneous pressure boundary conditions. Our algorithm explores all combinations of pressure boundary conditions valid for such a solver, in a single, general framework. The method is implemented in a 2D pencil-like domain decomposition, which enables efficient massively-parallel simulations. The implementation was validated against different canonical flows, and its computational performance was examined. Excellent strong scaling performance up to $10^4$ cores is demonstrated for a domain with $10^9$ spatial degrees of freedom, corresponding to a very small wall-clock time/time step. The resulting tool, CaNS, has been made freely available and open-source.

1. Introduction

The pressure Poisson solve is a demanding part of DNS, and FFT-based direct solvers can be efficient but have been difficult to scale across multiple directions. This work presents CaNS, a massively parallel solver using a second-order finite-difference pressure Poisson method for a wider range of homogeneous pressure boundary conditions.

1. Introduction

  • Finite-difference methods can reproduce several lower-order flow-observable moments with spectral-method fidelity at less computational effort, while requiring more resolution and memory bandwidth per FLOP.They also offer greater versatility for boundary conditions and complex geometries, including immersed-boundary implementations.Source: §1 · p. 2
  • Second-order finite-difference pressure-correction methods have been used for diverse turbulent-flow applications, including rough-wall channels, porous media, particle-laden flows, and Rayleigh–Bénard convection.They are often used when dense immersed boundaries already reduce overall accuracy to second order.Source: §1 · p. 2
  • The pressure Poisson equation is often the most demanding Navier–Stokes solve, while FFT-based direct solvers can be O(10) times faster than multigrid and satisfy the solution to machine precision.The cited comparison concerns second-order finite-difference Poisson equations.Source: §1 · p. 2
  • FFT parallelization is challenging because it requires all points in one direction, complicating distributed-memory decomposition across multiple directions.This difficulty had restricted scaling to very high core counts.Source: §1 · p. 2
  • The work addresses the lack of a general massively parallel DNS implementation supporting the range of homogeneous pressure boundary conditions available to second-order FFT-based solvers.Earlier high-performing implementations used DFTs where at least two directions were periodic.Source: §1 · p. 3
  • CaNS combines a second-order finite-difference pressure-correction method with FFT-based Poisson solving to simulate a wide range of canonical flows.The numerical tool is freely available.Source: §1 · pp. 1, 3

2. Numerical Method

The numerical method advances incompressible-flow equations on a structured Cartesian grid using a pressure-correction scheme and three-step Runge–Kutta integration. Its Poisson solver transforms two directions according to homogeneous pressure boundary conditions, then solves tridiagonal systems in the third direction.

2. Numerical Method

  • The flow equations use a structured Cartesian grid and standard second-order finite differences with staggered grid points.The equations are coupled through pressure correction and integrated with a low-storage, three-step Runge–Kutta scheme.Source: §2 · pp. 3–4
  • The three-step Runge–Kutta temporal scheme is reported to yield overall second-order temporal and spatial accuracy for DNS of turbulent flows.Its coefficients are αk = {8/15, 5/12, 3/4}, βk = {0, −17/60, −5/12}, and γk = αk + βk.Source: §2 · p. 4
  • For prescribed velocity boundaries, the correction-pressure boundary condition must be homogeneous Neumann so the projection step does not alter the velocity condition.The pressure and velocity boundary conditions therefore cannot be specified independently.Source: §2 · p. 4

2.1. Poisson Equation

  • The Poisson solver uses boundary-condition-dependent transforms in x and y, then solves the resulting tridiagonal systems in z with Gauss elimination.The method requires constant grid spacing in x and y; z may be non-uniform.Source: §2.1 · pp. 4–5
  • The method reduces the seven-diagonal finite-difference Poisson system to tridiagonal systems by applying discrete transforms in two domain directions.The transforms and eigenvalues depend on the boundary conditions, and their inverses satisfy the corresponding homogeneous conditions.Source: §2.1 · p. 5
  • The solver applies forward transforms in x and y, solves NxNy tridiagonal pressure systems in O(NxNyNz) operations, then applies inverse transforms.The forward and inverse transform stages each have operation counts proportional to Nz(NxNy log Nx + NyNx log Ny).Source: §2.1 · pp. 5, 7

3. Implementation

The implementation distributes the domain in pencil-like subdomains and transposes data between alignments for the Poisson solver. It combines discrete transforms, tridiagonal solves, and parallel I/O with MPI-based libraries.

3. Implementation

  • The code uses a pencil-like domain decomposition with hybrid MPI-OpenMP parallelization; p pencils align in z, while the solver transposes data to x- and y-aligned layouts.The left-most layout is used throughout most algorithm steps.Source: §3 · p. 7, Figure 1 · p. 8
  • Two-cell-wide finite-difference stencils use halo cells and four pairwise exchanges per update, with negligible computational overhead.Source: §3 · p. 7
  • Keeping a 2D decomposition avoids the slab approach’s limit of Nx p ≤ Nz, but requires an extra all-to-all operation per discrete transform.Hybrid parallelization relaxes the slab restriction by a factor O(10) on the described hardware, but can remain constraining for large problems.Source: §3 · p. 7
  • FFTW3’s Guru interface handles the discrete transforms through a common syntax, while LAPACK DGTSV solves the resulting tridiagonal systems.For periodicity in z, the cyclic tridiagonal system is reduced to two tridiagonal problems.Source: §3 · pp. 7–8
  • The Poisson workflow transposes the right-hand side, transforms in x and y, solves tridiagonal systems, then applies backward transforms in y and x with further transpositions.The solve step uses Gauss elimination for NxNy systems.Source: §3 · p. 8
  • Parallel I/O is handled by 2DECOMP&FFT using MPI-I/O.Source: §3 · p. 8

4. Validation and Computational performance

The solver was validated against canonical wall-bounded and Taylor–Green flows, with reported agreement against reference DNS. Strong and weak scaling tests examine performance across large core counts and domain sizes.

4.1.1. Lid-driven cavity flow

  • The cavity centerline velocity profiles at Re = 1000 agree excellently with reference DNS data.The compared profiles are u(0, 0, z) and w(x, 0, 0).Source: §4.1.1 · p. 9, Figure 2 · p. 9

4.1.2. Pressure-driven turbulent channel and square duct flow

  • Channel statistics closely match Kim et al. at Reτ = 180, while the duct diagonal profile agrees with Gavrilakis and its secondary flow reaches about 2% of Ub.The channel visualization shows three-dimensional coherent structures; the duct comparison concerns the mean streamwise velocity along its diagonal.Source: §4.1.2 · pp. 10–11, Figure 3 · p. 10, Figure 4 · p. 11

4.1.3. Taylor-Green vortex

  • Mean viscous dissipation in the Taylor–Green vortex peaks at t = 9L/U and agrees well with reference data.The reported dissipation trend follows small-scale vortex generation and later decreases in the absence of external power input.Source: §4.1.3 · p. 12, Figure 6 · p. 13

4.2. Computational performance

  • 0.5 s/core/timestep was reached in strong scaling to about 10^4 cores on a 1024^3-cell domain.The authors report superlinear speedup at smaller core counts, likely due to cache effects; the timings are consistent with the performance of 2DECOMP&FFT.Source: §4.2 · pp. 12, 14, Figure 7 · p. 14
  • Weak scaling showed up to 17% deterioration from 256 to 4096 cores at about 2·10^6 grid cells per task.The authors infer that strong-scaling timings are likely to extend to much larger problem sizes.Source: §4.2 · p. 14, Figure 7 · p. 14

5. Conclusions and outlook

The paper presents a second-order FFT-based pressure-Poisson solver for massively parallel DNS, generalized to different combinations of homogeneous pressure boundary conditions. It reports scaling to about 10^4 cores for 10^9 spatial degrees of freedom, while identifying validation scope and possible extensions.

Conclusions and outlook

  • About 10^4 cores and 10^9 spatial degrees of freedom are reached with a very small wall-clock time.The authors state that these figures will probably scale for larger problem sizes.Source: §5 · p. 14
  • To the authors’ knowledge, this is the first general implementation supporting the homogeneous pressure-boundary combinations valid for eigenfunction expansions.Earlier parallel implementations were restricted to at least two periodic directions.Source: §5 · p. 14
  • The method was validated on distinct canonical laminar and turbulent flows, while wall-bounded inflow/outflow configurations remain possible extensions.The authors identify these configurations as obvious examples of cases that could also have been considered.Source: §5 · p. 14
  • For low-Reynolds-number flows or extremely high resolution, implicit diffusion integration can be advantageous, and the tool implements it for homogeneous velocity boundary conditions in x and y.The implicit diffusion treatment removes the time-step restriction due to viscous effects.Source: §5 · pp. 14–15
  • The open-source code is intended as a base solver on which methods for complex geometries or multiphase flows can be implemented.The paper connects such combined methodologies with physical insights into flows requiring massively parallel DNS.Source: §5 · p. 15
Loading 1802.10323v4…