Source-linked AI summary
waLBerla: A block-structured high-performance framework for multiphysics simulations
Martin Bauer, Sebastian Eibl, Christian Godenschwager, Nils Kohl, Michael Kuron, Christoph Rettinger, Florian Schornbaum, Christoph Schwarzmeier, Dominik Thönnes, Harald Köstler, Ulrich Rüde
TL;DR
waLBerla addresses the challenge of using current supercomputers efficiently across multiple levels of parallelism and heterogeneous hardware. It provides block-structured simulation building blocks, couples fluid and particle methods, and generates CPU or GPU code from high-level formulations. The framework supports applications across fluid, particle, and coupled simulations, with continuous integration testing for software and performance stability.
Problem
Efficiently programming current supercomputers requires exploiting multiple levels of parallelism, while heterogeneous architectures with accelerators complicate development.
Source: §1 · p. 1Method
waLBerla provides block-structured simulation components, supports parallel kernels and coupled fluid-particle methods, and generates CPU or GPU code from high-level descriptions.
Source: §1 · p. 2, §2 · p. 4, §4.1 · p. 11Results
The framework has supported applications ranging from finite-difference phase-field and finite-element earth-mantle simulations to fluid, particle, and coupled fluid-particle applications.
Source: §2 · p. 4, §9 · p. 30Takeaways & Limitations
Block-structured partitioning supports flexible treatment of complex geometries, while code generation permits integration of new physical models.
Source: §9 · pp. 30–31
Abstract
from arXiv · showhide
Programming current supercomputers efficiently is a challenging task. Multiple levels of parallelism on the core, on the compute node, and between nodes need to be exploited to make full use of the system. Heterogeneous hardware architectures with accelerators further complicate the development process. waLBerla addresses these challenges by providing the user with highly efficient building blocks for developing simulations on block-structured grids. The block-structured domain partitioning is flexible enough to handle complex geometries, while the structured grid within each block allows for highly efficient implementations of stencil-based algorithms. We present several example applications realized with waLBerla, ranging from lattice Boltzmann methods to rigid particle simulations. Most importantly, these methods can be coupled together, enabling multiphysics simulations. The framework uses meta-programming techniques to generate highly efficient code for CPUs and GPUs from a symbolic method formulation. To ensure software quality and performance portability, a continuous integration toolchain automatically runs an extensive test suite encompassing multiple compilers, hardware architectures, and software configurations.
1. Introduction
waLBerla is an open-source framework designed to exploit parallelism on supercomputers for a broad range of scientific simulations. It combines performance-oriented computing with flexible multiphysics capabilities and code generation.
1. Introduction
- waLBerla targets massively parallel HPC systems with fully distributed data structures, MPI or hybrid MPI/OpenMP parallelization, and high node-level performance across compilers and hardware.Each process holds only local and adjacent data, so its memory use does not depend on total simulation size.Source: §1 · p. 2
- waLBerla uses code generation to produce time-critical numerical kernels from high-level, domain-specific formulations, reducing duplicated code across hardware targets.The paper presents the approach as a way to address portability while developing efficient kernels.Source: §1 · p. 2
- The framework combines an explicit HPC focus with flexible adaptive mesh refinement and load-balancing routines.The introduction contrasts this focus with general frameworks such as AMROC, p4est, and Peano.Source: §1 · p. 2
- The framework combines lattice Boltzmann fluid simulations with rigid-particle methods, enabling large-scale coupled fluid-particle simulations in a monolithic code.The modules are interconnected, and the coupling is described as not compromising performance or scalability.Source: §1 · p. 2
- waLBerla has evolved from a 2007 prototype through two architectural overhauls, with newer features including rigid-body dynamics, adaptive mesh refinement, and code generation.The current iteration has been continuously developed since 2012; version 4.2 includes most of the listed features.Source: §1 · p. 2
2. Framework overview
waLBerla’s framework overview describes modular infrastructure for block partitioning, structured-grid kernels, and coupled numerical methods. Its components support parallel simulation development, while the framework has also served as a basis for applications beyond its original focus.
2. Framework overview
- The framework’s modular C++ implementation is supported by extensive testing, developer collaboration tools, a GUI, and Python interfaces for simulation pre- and postprocessing.The Python interface is described as providing efficient pre- and postprocessing workflows.Source: §2 · pp. 3–4
- The block forest manages domain partitioning independently of simulation data structures and supports static or dynamic refinement, load balancing, checkpointing, and inter-block data exchange.Checkpointing writes and reads simulation snapshots for stopping, restarting, or recovery after hardware failure.Source: §2 · p. 3
- waLBerla’s kernels operate on structured fields, with data structures and kernels designed for shared-memory parallelization, vectorization, and GPUs.Geometry and mesh-file utilities support complex domains and boundary setups.Source: §2 · pp. 3–4
- The framework includes a rigid particle dynamics module that uses the same domain partitioning concepts as LBM while operating on different data structures.Its particle algorithms and applications are presented alongside fluid-particle coupling strategies.Source: §2 · p. 4
- Code generation produces kernels for field and particle data structures and aims to separate method description from performance optimization.The paper frames code generation as a way to address tensions between kernel performance and maintainability.Source: §2 · p. 4
- waLBerla’s flexible, generic design has supported projects ranging from finite-difference phase-field simulations to finite-element earth-mantle dynamics.These projects are summarized as extensions built on the framework.Source: §2 · p. 4
3. Block-structured domain partitioning
waLBerla’s BlockForest partitions domains into blocks that can be refined and distributed, while layered communication and checkpointing support parallel execution and resilience.
3.1. Overview
- The octree-based BlockForest partitions a cuboidal domain into independently refinable blocks, maintains a 2:1 size ratio between neighbors, and assigns each block to exactly one process.Processes know their assigned blocks and neighbors, keeping partition-information memory per process independent of process count; blocks can also hold arbitrary simulation data.Source: §3.1 · pp. 4–5, Figure 2 · p. 5
3.2. Adaptive refinement and load balancing
- Adaptive repartitioning marks blocks for refinement or coarsening, load-balances a lightweight proxy, then updates the forest and redistributes simulation data.Application-specific callbacks can use criteria such as flow vorticity or particle count; balancing options include Morton and Hilbert curves and ParMETIS.Source: §3.2 · pp. 5–6
3.3. Distributed memory parallelization
- The MPI communication stack ranges from reusable low-level serialization and buffering to block-aware schemes that specify what data moves and how.Buffered communication uses PackInfos for block data and can handle refinement transfers; schemes can also expose asynchronous communication to overlap with computation.Source: §3.3 · pp. 6, 8
- The direct MPI-datatype approach avoids explicit serialization and copying, while buffered communication is easier for custom data and can combine different data in one message.The framework offers both approaches so users can choose for their application.Source: §3.3 · p. 8
3.4. Checkpointing and resilience
- Distributed in-memory checkpoints replicate process-group data, enabling recovery from single-node failures when group members span different nodes.After failure, surviving processes restore their subdomains and the failed partner’s data if needed, then redistribute data; the implementation scales beyond 260,000 processes.Source: §3.4 · p. 9
- Checkpoint creation scales independently of the number of processes involved.The in-memory scheme avoids disk writes, addressing checkpoint costs from memory traffic, disk I/O, and memory requirements.Source: §3.4 · pp. 9–10
4. Stencil codes on structured grids
waLBerla combines block-distributed structured grids with configurable fields and sweeps, GPU support, and geometry tools for setup and evaluation.
4.1. Structured grids
- StructuredBlockForest maps block-local and global coordinates for uniformly sized grids within blocks, while block-level refinement allows geometric flexibility.Blocks without overlap with a surface-mesh-defined domain are discarded; fields and ghost layers support synchronization and transfers between neighboring resolutions.Source: §4.1 · pp. 10–11
4.2. Fields and sweeps
- Sweeps separate numerical kernels from their data, allowing independent optimization and per-block selection, including CPU or GPU execution.Ghost-layer exchange before or after a sweep propagates updated data to neighboring blocks; the framework also supplies common and optimized lattice Boltzmann sweeps.Source: §4.2 · p. 11
- Field stores spatial coordinates and multiple per-cell values in a configurable AoS or SoA layout chosen to suit kernel performance.Accessors and iteration constructs hide the layout, support linear traversal, and enable OpenMP-parallel iteration; padding and alignment support SIMD operations.Source: §4.2 · pp. 11–12
4.3. GPU support
- GPU fields mirror CPU capabilities and layouts, allowing bulk transfers while computation and boundary treatment remain on the GPU across many time steps.CPU mirror fields support initialization and later post-processing and I/O, which need not be ported to the GPU; indexing strategies simplify CUDA thread-to-cell mapping.Source: §4.3 · p. 12
- GPU communication supports CUDA-aware MPI for direct transfer from GPU memory and adapts buffered communication for GPU buffers.Using the same memory layout for CPU and GPU fields permits fast transfers with a single memcpy call.Source: §4.3 · p. 12
4.4. Geometry handling
- Geometry setup supports primitives, image-derived boundaries, and surface meshes, using signed-distance calculations and an octree to accelerate mesh queries.Meshes guide block allocation and refinement as well as cell-level boundary conditions, whose parameters can be attached to mesh vertices or faces.Source: §4.1 · p. 10, §4.4 · p. 13
- For static geometry with few boundary cells, preprocessing can list boundary-cell indices so separate kernels handle each boundary condition.For evaluation and output, waLBerla can gather selected data or construct and coarsen local isosurfaces into a combined mesh instead of writing voxel data.Source: §4.4 · pp. 13–14
5. Numerical methods and applications
waLBerla’s applications span lattice Boltzmann and rigid-particle methods, with monolithic coupling for particulate-flow simulations.
5. Numerical methods and applications
- Lattice Boltzmann simulations are the method for which waLBerla was initially designed.The paper introduces these simulations before its rigid-particle and coupled-flow applications.Source: §5 · p. 14
- The rigid particle dynamics module contains methods for particle simulations and is presented with application examples.The section presents this module after lattice Boltzmann simulations.Source: §5 · p. 14
- waLBerla discusses monolithic coupling of fluid and particle methods for particulate-flow simulation.The coupled case follows the separate lattice Boltzmann and rigid-particle presentations.Source: §5 · p. 14
5.1. Lattice Boltzmann method
This section reviews waLBerla’s lattice Boltzmann method (LBM), its performance and scaling, and examples of fluid-flow applications. It describes the method’s PDF-based updates and supported collision models, alongside extensions for refinement and complex simulations.
5.1.1. Overview
- LBM represents fluid particles with PDFs on a regular DdQq grid, then advances them through collision and streaming steps using supported SRT, TRT, MRT, and cumulant methods.waLBerla also supports locally varied relaxation rates, multiple boundary conditions and force models, and relaxation-parameter adaptation for methods such as turbulence models and entropic stabilization.Source: §5.1.1 · pp. 14–15
- For costly uniform-grid cases, waLBerla implements a mass-conserving local-refinement method that uses fine grids only in selected regions and coarser grids elsewhere.Source: §5.1.1 · p. 15
5.1.2. Performance and scaling
- Generic stencil kernels scale across one node but perform poorly; loop unrolling, fused stream/collision, common-subexpression elimination, and SIMD produce the fastest version.The optimized version saturates available memory bandwidth at 8 of 16 cores.Source: §5.1.2 · pp. 15–16
- Perfect weak scaling reaches all 458 752 JUQUEEN cores with pure MPI or hybrid OpenMP+MPI; Table 1 also summarizes experiments on CPU and GPU systems.The Figure 5 experiments use a D3Q19 TRT collision model, with performance reported in MLUPS.Source: Figure 5 · p. 16, §5.1.2 · p. 16, Table 1 · p. 16
5.1.3. Example applications
- A coronary-artery simulation handles geometry covering only 0.3% of its bounding-box volume by applying load balancing to reduce workload peaks from partially covered blocks.Source: §5.1.3 · p. 17
- Vocal-fold flow simulations use static refinement in the narrow channel, later improved with adaptive refinement that keeps fine resolution in turbulent-vortex regions.Source: §5.1.3 · p. 17
- waLBerla’s rigid-particle module creates porous-bed sphere arrangements used to study laminar and turbulent flows and derive empirical models and interface conditions.Source: §5.1.3 · p. 17
- Complex-mesh boundaries and grid refinement support simulation of fluid flow through a chaotic crystal structure used as a diesel particulate filter.Source: §5.1.3 · p. 17
5.2. Rigid particle dynamics
waLBerla’s rigid particle dynamics module builds on block-based domain partitioning and supports parallel particle simulations, scalable execution, and applications from granular packings to carbon nanotubes.
5.2.1. Data structures
- BodyStorage assigns particles to blocks, while overlap ghosts are synchronized each step and serialization supports particle-aware dynamic load balancing.Infinite-extension bodies are held in a separate global structure.Source: §5.2.1 · p. 18
5.2.2. Algorithms
- The simulation loop detects contacts, resolves them with soft or hard contact models, integrates motion, and synchronizes distributed simulations.Broad-phase choices include linked lists or hierarchical hash grids; integration options include Euler and Velocity Verlet.Source: §5.2.2 · pp. 18–19
5.2.3. Performance and scaling
- Weak scaling remains perfect through 1 835 008 processes, while strong scaling is good down to 1280 particles per process.When simulations become too small, communication and internal data-structure maintenance can exceed the gains from adding cores.Source: Figure 7 · p. 19, §5.2.3 · p. 19
- Run-time load balancing cuts hopper-discharge time per step by approximately a factor of 8 while compensating workload-driven performance variation.The measurements use 8 million particles on 1296 processes.Source: Figure 9 · p. 20, §5.2.3 · p. 20
5.2.4. Applications
- Particle simulations study validated gravitational random dense packings and model carbon nanotubes under in-plane loading by assembling larger non-rigid particles from smaller rigid ones.The packing simulations extract structural properties of granular materials.Source: §5.2.4 · pp. 20–21
5.3. Fluid-particle coupling
waLBerla couples its LBM fluid solver and particle module within one framework, supporting resolved and unresolved particle-fluid simulations and applications at large scale.
5.3. Fluid-particle coupling
- Monolithic coupling gives both solvers direct access to shared data structures, avoiding potentially costly interfaces between separate frameworks.This coupling supports large-scale fluid-particle simulations.Source: §5.3 · p. 21
5.3.1. Geometrically fully resolved simulations
- For geometrically resolved simulations, particles map onto the Eulerian grid; momentum exchange updates particle motion, while no-slip boundaries represent their effect on fluid.Higher-order boundary conditions improve boundary accuracy, and adaptive refinement can maintain fine resolution around particles while coarsening other regions.Source: §5.3.1 · pp. 21–22, Figure 10 · p. 21
5.3.2. Simulated scenarios
- A riverbed-flow DNS used 16.2·10^9 fluid cells and 7.7·10^6 spherical particles, and the authors describe it as one of the largest such simulations carried out until then.The coupled simulations showed excellent scalability on SuperMUC Phase2.Source: §5.3.2 · p. 23
- Laterally displaced pillars create vertical flow that sorts particles by size, and simulations support investigation and optimization of these separation devices.The approach also supports particles represented by randomly generated polygonal meshes in sediment-bed simulations.Source: §5.3.2 · p. 23
5.3.3. Volumetric coupling method
- Unresolved coupling represents particles smaller than a fluid cell using empirical drag and lift models, reducing computational cost for systems impractical to resolve fully.The approach has also been used to study fine-sediment infiltration with a deposition model.Source: §5.3.3 · pp. 23–24
5.4. Coupling to non-hydrodynamic fields
waLBerla supports non-hydrodynamic fields through PDE solvers and couples electrostatics and solute transport with LBM hydrodynamics in electrokinetic simulations.
5.4. Coupling to non-hydrodynamic fields
- Available PDE solvers include Jacobi, Gauß-Seidel, SOR, conjugate gradient, a PFFT-based spectral solver, and multigrid combinations.The PDE examples include the Poisson equation for electrostatics.Source: §5.4 · p. 24
- Electrokinetic simulations solve solute diffusion by finite differences, electrostatics with the available PDE solvers, and hydrodynamics with LBM.The method is based on Capuani et al..Source: §5.4 · p. 24
- Other fields typically exert forces on the fluid, while the fluid couples back through solute advection.The method was extended to incorporate particles through the momentum-exchange method.Source: §5.4 · p. 24
6. Code generation
waLBerla uses code generation to address the cost of developing optimized kernels and adaptable particle software, while retaining readable generated code.
6. Code generation
- Code generation targets the cost of manually optimizing many kernel variants for different models and hardware, balancing performance with maintainability and portability.The framework generates time-critical kernels from a high-level domain-specific formulation.Source: §6 · p. 24
6.1. Pystencils
- Pystencils generates CPU and GPU stencil code from symbolic descriptions and has been used for LBM and alloy-solidification phase-field simulations.Its transformations operate on expression trees, with backends for C/C++, CUDA, OpenCL, and LLVM IR.Source: §6.1 · p. 25
6.2. MESA-PD
- MESA-PD separates data from algorithms and uses Jinja templates to generate particle data structures and dependent algorithms for each simulation’s properties.The generated code remains readable and is intended to reduce boilerplate while adapting algorithms and data structures to simulation needs.Source: §6.2 · pp. 25–26
7. Supporting infrastructure
waLBerla’s supporting infrastructure combines automated development checks with tools for inspecting internal simulation data and configuring, steering, and evaluating simulations through Python.
7.1. Development infrastructure
- CI runs compiler- and configuration-varied builds, correctness tests, static analysis, coverage checks, and reproducible performance tests.At the time described, 39 setups ran at each commit and 121 in extended nightly builds.Source: §7.1 · p. 26
7.2. Graphical user interface
- The GUI supports developer debugging by showing internal data in situ, including stepwise state inspection and breakpoints at arbitrary control-flow points.Its modular widgets show selected-block data and can be extended for custom data types.Source: §7.2 · pp. 27–28, Figure 12 · p. 27
7.3. Python interface
- The Python interface supports simulation setup, computational steering, result evaluation, and visualization, extending beyond the GUI’s developer-debugging role.Python configuration supports mathematical expressions and parameter dependencies; callbacks can analyze data in situ and store selected results rather than full VTK states.Source: §7.3 · p. 28
8. waLBerla extensions
Projects built on waLBerla illustrate its use beyond its core repository, including unstructured-grid multigrid, two-phase-flow applications, and phase-field simulations.
8.1. HyTeG
- HyTeG applies waLBerla concepts to unstructured triangle and tetrahedral meshes, building hierarchies of structured fields suited to geometric multigrid.It also reuses waLBerla’s CMake build system and core facilities including buffers, math operations, and logging.Source: §8.1 · p. 29
8.2. Free surface lattice Boltzmann method
- The free-surface LBM extension models two-phase scenarios where one fluid phase’s flow is treated as negligible, and has been used for food foaming and additive-manufacturing simulations.The additive-manufacturing application couples it with an advection-diffusion LBM to simulate an electron beam melting a powder bed.Source: §8.2 · p. 29
8.3. Phase-field methods
- An optimized phase-field implementation achieved an 80-fold single-node speedup over a basic C implementation, and code generation automates the optimizations across phase-field models.The approach uses an abstract formulation in terms of a continuous free energy functional; a 3D Ag-Al-Cu directional-solidification simulation ran on Hornet with 2420 × 2420 × 1474 cells.Source: §8.3 · p. 29, Figure 15 · p. 30
9. Conclusion
waLBerla has developed from a lattice-Boltzmann CFD framework into a modular HPC toolkit for varied and coupled applications. Its block-structured approach, code generation, and continuous integration support performance and software stability.
9. Conclusion
- Block-structured partitioning supports complex geometries and multiple levels of parallelism for fluid, particle, and coupled fluid-particle applications.The framework’s performance and scalability focus supports efficient use of HPC systems for detailed physical-process studies.Source: §9 · p. 30
- Continuous integration with extensive unit, integration, and performance tests is presented as ensuring feature and performance stability while encouraging developer changes..Source: §9 · p. 30
- Optimizing transformations turn high-level descriptions of compute-intensive work into efficient CPU or GPU code, supporting future HPC architectures and integration of new physical models.Source: §9 · pp. 30–31