Source-linked AI summary
A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence
Pablo D. Mininni, Duane L. Rosenberg, Raghu Reddy, Annick Pouquet
TL;DR
The paper presents a hybrid MPI-OpenMP implementation for scalable pseudospectral computations of turbulent flows on high-core-count systems. It reports good scalability to about 20000 processors and identifies thread/process configurations for performance optimization, while noting increased buffer-memory requirements at large resolutions.
Problem
Distributed-memory implementation of pseudospectral computations is hard, motivating hybrid parallelism for high-resolution turbulent-flow simulations.
Method
The authors present a new hybrid MPI-OpenMP implementation that extends a scalable MPI-parallelized pseudospectral code.
Results
Good scalability is achieved up to about 20000 processors on the Kraken system, with performance depending on the MPI-process and OpenMP-thread configuration.
Takeaways & Limitations
The reported configurations provide guidance for choosing MPI processes and OpenMP threads to optimize performance on different platforms.
Takeaways & Limitations
Increased buffer-memory requirements at large resolutions can prevent the code from fitting into memory and require careful consideration.
Abstract
from arXiv · showhide
A hybrid scheme that utilizes MPI for distributed memory parallelism and OpenMP for shared memory parallelism is presented. The work is motivated by the desire to achieve exceptionally high Reynolds numbers in pseudospectral computations of fluid turbulence on emerging petascale, high core-count, massively parallel processing systems. The hybrid implementation derives from and augments a well-tested scalable MPI-parallelized pseudospectral code. The hybrid paradigm leads to a new picture for the domain decomposition of the pseudospectral grids, which is helpful in understanding, among other things, the 3D transpose of the global data that is necessary for the parallel fast Fourier transforms that are the central component of the numerical discretizations. Details of the hybrid implementation are provided, and performance tests illustrate the utility of the method. It is shown that the hybrid scheme achieves near ideal scalability up to ~20000 compute cores with a maximum mean efficiency of 83%. Data are presented that demonstrate how to choose the optimal number of MPI processes and OpenMP threads in order to optimize code performance on two different platforms.
1 Introduction
Pseudospectral DNS resolves turbulent scales accurately but requires global spectral transforms that are difficult to distribute across many processors. The paper presents a hybrid MPI-OpenMP scheme designed for high-core-count systems and reports strong scalability.
- Motivation: DNS resolves all spatial and temporal scales, but exascale resolutions needed for accurate geophysical flows are not currently feasible.Petascale systems can accommodate 10^15 grid points, corresponding to Rv ∼10^7 and sufficient scale separation for studying complex turbulent flows.
- Numerical method: Pseudospectral methods efficiently use FFTs and accurately capture multiscale interactions with little or no numerical dissipation or dispersion.These properties support quantification of small-scale dissipative effects arising from nonlinear turbulent interactions.
- Parallelization challenge: Global spectral transforms make pseudospectral methods hard to implement in distributed memory environments.Domain decomposition enables local serial FFTs after transpositions, but transpositions require global communication.
- Parallelization challenge: MPI slab decompositions can parallelize multidimensional FFTs effectively, but processor counts are limited and larger MPI-only configurations are costly or difficult to tune.The main communication challenge is the all-to-all exchange required by transpositions; some non-Fourier transforms cannot use this technique.
- Contribution: The paper develops a hybrid MPI-OpenMP scheme by augmenting an MPI-scalable decomposition with loop-level directives and multithreaded FFTs.The implementation targets multicore architectures and portability without requiring detailed knowledge of network topology or cache.
- Contribution: The hybrid implementation is presented as the first published high-core-count hybrid model attempted in a pseudospectral CFD context and is reported to scale well.The paper introduces a new domain-decomposition picture for multicore nodes and evaluates OpenMP overhead, full-hybrid scalability, and performance.
2 The pseudo-spectral method and the underlying domain decom-
The method represents turbulent-flow fields in a Fourier basis and computes nonlinear terms in physical space before transforming them with FFTs. Its underlying slab decomposition enables local transforms but requires a global transpose through all-to-all communication, limiting processor scaling.
- The pseudospectral formulation represents each velocity component as a truncated Galerkin expansion in the Fourier basis.
- Nonlinear terms are evaluated in physical space, transformed to spectral space with FFTs, and projected onto a divergence-free velocity space.
- FFT-based multidimensional transforms are central to parallel pseudospectral computing and high-Reynolds-number turbulent hydrodynamics simulations.
- In the 1D decomposition, each processor owns an N × N × M slab, with M = N/NP, because data distribution occurs along one direction.
- Local slab FFTs are followed by an all-to-all global transpose, producing P × N × N slices so the remaining one-dimensional FFT can run locally.
- The slab scheme minimizes all-to-all operations when implemented properly but limits MPI processes to N and can lose linear scaling as communication dominates computation.
3 Implementation of the hybrid scheme
The hybrid implementation combines coarse-grain MPI slab decomposition with OpenMP threads and threaded FFTs within each MPI task. It addresses slab-only scaling limits while reducing MPI communication and buffer requirements.
- The hybrid scheme uses MPI for coarse-grain slab decomposition and OpenMP for inner-level parallelization within each MPI task.
- Each MPI task distributes slab work among threads through loop-level OpenMP regions, while multi-threaded FFTs accelerate computation within the slab.
- The outer multidimensional FFT and interprocess communication remain unchanged at the MPI level, while local transposition is parallelized within each slab.
- OpenMP parallelizes outer loops when slab planes are sufficient and inner loops otherwise, providing two within-node decomposition choices.
- Cache-blocking keeps inner-loop data small enough for cache, with tunable cache size supporting optimization across shared or separate thread caches.
- Reducing MPI processes lowers MPI calls, communicated data, and MPI buffer sizes, and supports MPI I/O when using tens of thousands of cores.
4 Scalability and performance
Performance tests compare MPI/OpenMP configurations on bluefire and kraken, showing platform-dependent thread scaling and strong scalability at high core counts. The results also identify configurations that improve efficiency as resolution and processor counts increase.
- MPI versus OpenMP: Fixed MPI-task counts: increasing MPI tasks from 1 to 6 produces scaling comparable to pure OpenMP parallelization.The hybrid configuration is most useful for large simulations, large processor counts, and many cores within the same socket.
- MPI/OpenMP configurations: Fixed processor counts: two threads outperform one for large MPI-task counts, but adding more threads generally provides no substantial additional speed-up.This pattern is observed more clearly for the N = 512 runs.
- Efficiency trends: N = 1024: efficiency improves with higher resolution when using two threads at a fixed processor count, and also with four threads when doubling MPI processes.The stated reference uses NP0 = N/2 and nthd = 1.
- High-core-count scalability: ∼20000 cores: good speedup is achieved on kraken, with mean efficiencies of 76%, 83%, 61%, and 71% across the reported runs.The maximum observed efficiency is slightly higher than the corresponding mean efficiency.
5 Discussion and conclusion
The hybrid MPI-OpenMP model combines two-level parallelization with a domain decomposition suited to hierarchical, multicore systems. Tests identify scalability limits, configuration choices, and memory constraints across Bluefire and Kraken.
- Method: The model augments slab-based pseudospectral decomposition with loop-level OpenMP parallelization, creating a two-level parallelization scheme.This changes how the computational domain is mapped across MPI processes and threads.
- Performance: About 20000 processors yielded good scalability on Kraken, while Bluefire showed thread-overhead limitations at small socket core counts.The overhead became clearer below a resolution threshold N and reduced scalability.
- Performance: For large enough problems, 12 threads per MPI process provided the best scalability, whereas 6 threads reduced performance time until each MPI process had sufficient workload.The preferred MPI/OpenMP configuration depended on problem size, resolution, thread count, and MPI-task count.
- Scope and implications: The approach can extend to non-Fourier spectral methods and non-unit aspect-ratio domains, while fewer MPI processes may reduce buffer-memory requirements.The paper also identifies increasing cores per socket as a setting where the hybrid approach may provide better performance.