Source-linked AI summary
LBFAST: A Lightweight Moment-Represented Lattice Boltzmann Solver for Multi-GPU Architectures
Marco Lauricella, Andrea Montessori, Giorgio Amati, Filippo Spiga, Adriano Tiribocchi, Massimo Bernaschi, Sauro Succi
TL;DR
Standard LBM stores all discrete populations, making large three-dimensional GPU simulations memory intensive. LBFAST instead reconstructs populations from reduced moments, and its reported memory comparison and validation demonstrate the compact formulation’s results.
Problem
Storing the full set of populations makes standard LBM memory intensive, especially for three-dimensional simulations on GPU architectures.
Source: §1 · p. 1Method
Results
At 512^3, double-precision main arrays use 20 GB versus 38 GB (D3Q19) and 54 GB (D3Q27), with validation reporting agreement with benchmark behavior.
Source: §3 · p. 5, §4 · pp. 6–8Takeaways & Limitations
Abstract
from arXiv · showhide
We present LBFAST, a GPU-oriented lattice Boltzmann solver based on a lightweight moment-represented formulation, in which post-collision populations are reconstructed on the fly from a reduced set of moments rather than stored explicitly. This approach significantly lowers the memory footprint, enabling large three-dimensional simulations within the constraints of modern accelerator architectures, where VRAM capacity and bandwidth are critical resources. The method is assessed through standard single- and two-component benchmarks demonstrating good accuracy and stability. Extensive scaling experiments on multi-GPU systems show near-ideal weak scaling up to 512 GPUs and sustained performance across different velocity sets. The combination of reduced memory usage, competitive throughput, and stable energy efficiency makes the proposed formulation a practical route for large-scale lattice Boltzmann simulations on current and emerging HPC platforms.
1. Introduction
LBFAST applies a lightweight, moment-represented formulation to GPU-accelerated lattice Boltzmann simulations, reconstructing populations on demand rather than storing them in full. It extends this approach to distributed multi-GPU simulations and an incompressible multiphase formulation.
1. Introduction
- The standard LBM’s full population storage makes three-dimensional GPU simulations memory intensive, a recognized practical limitation for large-scale simulations.The paper notes that GPU-based LBM is often constrained by memory bandwidth as well as memory footprint.Source: §1 · pp. 1–2
- Earlier moment-based work connected reduced storage to hydrodynamically relevant fields, while Hermite-based regularization projects non-equilibrium distributions onto a controlled subspace.For athermal models, recursive equilibrium Hermite coefficients allow higher-order coefficients to be reconstructed from lower-order moments.Source: §1 · pp. 1–2
- Moment representation stores density, momentum, and the non-equilibrium stress tensor; in D3Q19, this reduces stored variables from 19 populations to 10 moments per site.Populations are reconstructed on the fly when needed for collision and propagation.Source: §1 · p. 2
- LBFAST reconstructs post-collision populations from reduced hydrodynamic and kinetic variables, then streams them without storing a full population array in global device memory.The flow solver evolves pressure and velocity, while density and viscosity are reconstructed from the phase field.Source: §1 · p. 2
- Compared with Tiribocchi et al., LBFAST adds distributed-memory multi-GPU decomposition and replaces the weakly compressible color-gradient multiphase model with an incompressible velocity–pressure solver coupled to conservative Allen–Cahn phase evolution.The paper assesses the solver through numerical validation and strong- and weak-scaling measurements.Source: §1 · p. 2
2. Methods
The method couples an incompressible velocity–pressure flow solver to a phase field, using a lightweight Hermite-moment LBM with on-the-fly population reconstruction. Its formulation combines phase-dependent material properties, conservative interface evolution, forcing, and multiple velocity sets.
2. Methods
- The velocity–pressure solver evolves velocity and pressure, while density and viscosity are reconstructed from the phase field transported by a conservative Allen–Cahn equation.The phase field determines both material properties through linear blending laws.Source: §2 · p. 3
- The conservative Allen–Cahn equation evolves the phase field using diffusion and an interface-compression contribution that counterbalances numerical diffusion and preserves a sharp profile.The compression coefficient is λ = 4D/W, and the interface normal is computed from the phase-field gradient.Source: §2 · p. 3
- The flow solver indirectly integrates the continuity and momentum equations using a pulled LBM update that combines collision and streaming.The relaxation time sets kinematic viscosity, and a correction using prior-step Hermite coefficients supplies second-order temporal accuracy.Source: §2 · pp. 3–4
- The distribution functions are reconstructed as a second-order-truncated Hermite expansion, with equilibrium coefficients expressed using velocity and rescaled pressure and non-equilibrium content held in the second-order tensor.The projection filters non-hydrodynamic degrees of freedom.Source: §2 · pp. 3–4
- The total force includes capillary, buoyancy, viscosity-gradient, and non-equilibrium-stress contributions, with the discrete forcing term defined by the Guo scheme.The capillary-force coefficients are chosen to recover prescribed surface tension and interface thickness.Source: §2 · p. 4
- The formulation supports D3Q19, second-order D3Q27, and higher-order D3Q27h velocity sets, with the latter using recursive Hermite closure relations.The update is described as independent of the specific discrete velocity set.Source: §2 · p. 4
- Spatial derivatives for forcing and phase evolution use D3Q27-weighted lattice stencils, while the conservative Allen–Cahn equation uses a forward-time, centered-space discretization.The stated rationale is to maintain stability while reducing computational cost and memory use.Source: §2 · p. 4
3. Implementation
LBFAST stores a compact set of Hermite fields and reconstructs populations during a fused collide-and-stream kernel. Its GPU implementation combines tiled data access with asynchronous MPI communication across one-, two-, or three-dimensional domain decompositions.
3. Implementation
- The CUDA Fortran implementation uses OpenACC and MPI, with domains decomposed along one, two, or three directions and non-blocking communication overlapped with interior computation.MPI_Waitall completes the exchanges before boundary nodes are updated.Source: §3 · pp. 4–5
- Hermite fields are arranged in tiles mapped to CUDA thread blocks; coalesced accesses and shared-memory halo buffers handle the non-local streaming step.Populations are reconstructed on the fly and temporarily stored in shared memory to limit global-memory traffic.Source: §3 · p. 5
- A fused pull-form collide-and-stream kernel reconstructs populations locally and accumulates their contributions directly into output moments, avoiding global storage of full populations.The output moments serve as local accumulators, reducing memory footprint and increasing arithmetic intensity.Source: §3 · p. 5
- The implementation stores 10 second-order Hermite fields with double buffering, compared with 19 or 27 populations per site in the standard D3Q19 and D3Q27 schemes.The retained fields comprise one scalar, three velocity components, and six independent components of the symmetric second-order tensor.Source: §3 · p. 5
- For a 512^3 domain in double precision, main lattice arrays require 20 GB with LBFAST, versus 38 GB for D3Q19 and 54 GB for D3Q27 population schemes.These correspond to 47.4% and 63.0% lower memory occupation than D3Q19 and D3Q27, respectively.Source: §3 · p. 5
4. Validation and Benchmarking
The single- and two-component benchmarks assess LBFAST’s accuracy across vortex decay, layered flow with a viscosity contrast, and surface-tension estimation. The reported results show close agreement with analytical predictions, with a larger discrepancy at the diffuse interface in the layered-flow test.
Taylor–Green vortex
- For Taylor–Green decay, νfit = 0.04079 for D3Q27, 0.040794 for D3Q19, and 0.040791 for D3Q27h, with relative deviations of ∼1.3 × 10−3, ∼1.3 × 10−3, and ∼1.2 × 10−3, respectively.The test used a periodic 512^3 box at Re ≃80; the D3Q27 kinetic-energy curve closely follows the analytical decay, with minor late-time deviations.Source: §4 · pp. 5–6, Figure 1 · p. 6
Layered Poiseuille flow
- At the interface, the largest relative errors were 4.18 × 10−2 for D3Q19, 4.18 × 10−2 for standard D3Q27, and 3.97 × 10−2 for D3Q27h.The velocity profiles agree well overall; the interfacial deviation is expected because the analytical solution assumes a sharp interface while the model uses a diffuse one.Source: §4 · p. 7, Figure 2 · p. 7
Laplace test
- The Laplace-test pressure jump follows the expected ∆P = 2σ/R, and all three stencils give consistent surface-tension estimates within statistical uncertainty.The measured surface tension is σ = 2.796 × 10−2 ± 9.976 × 10−4 for D3Q19, σ = 2.796 × 10−2 ± 9.979 × 10−4 for standard D3Q27, and σ = 2.796 × 10−2 ± 9.979 × 10−4 for D3Q27h.Source: §4 · pp. 7–8
5. Performance Analysis
Performance was assessed on Leonardo using up to 512 GPUs. The section quantifies throughput with GLUPS and introduces parallel efficiency to evaluate scalability.
Experimental platform
- The largest reported calculations used 512 GPUs across 128 Leonardo compute nodes, each equipped with four NVIDIA A100 GPUs with 64 GB of HBM2 memory.The code was compiled with the NVIDIA HPC SDK and executed using CUDA-aware MPI.Source: §5 · p. 8
Performance metrics
- GLUPS measures total lattice updates per second, expressed in billions, as a solver-throughput metric.The section also introduces parallel efficiency to evaluate scalability.Source: §5 · p. 8
- The section compares configurations using throughput and parallel efficiency as its performance and scalability measures.The supplied passage introduces parallel efficiency but does not state its defining expression.Source: §5 · p. 8
N GLUPS(1), (13)
LBFAST extends the lightweight moment-represented strategy to distributed-memory multi-GPU systems without storing full population arrays. Its weak scaling remains close to ideal through 512 GPUs, while performance and energy use vary with velocity set and benchmark type.
Implementation and evaluation
- LBFAST stores reduced moment fields with flip–flop double buffering and never stores full population arrays in global device memory, targeting distributed-memory scaling up to 512 GPUs.The evaluation covers numerical validation, MPI communication, domain decompositions, and strong and weak scaling.Source: N GLUPS(1), (13) · pp. 8–9
Strong scaling
- Strong-scaling efficiency is good up to several hundred GPUs, then gradually saturates as communication costs rise with shrinking local subdomains.D3Q19 achieves the highest GLUPS, followed by D3Q27, while D3Q27h is more expensive; this ordering holds across both component counts and precisions.Source: N GLUPS(1), (13) · p. 9
Component-count comparison
- Two-component systems achieve lower GLUPS than one-component systems because phase-field evolution and interfacial forces add memory accesses and arithmetic, with a larger gap in double precision.The double-precision gap is more pronounced where memory-bandwidth limitations are more severe.Source: N GLUPS(1), (13) · p. 9
Weak scaling
- Weak scaling remains close to ideal through 512 GPUs; the 1D decomposition performs slightly better than 2D and 3D, whose communication burden grows with neighbor count.The decompositions exchange data with two, eight, and up to twenty-six neighbors, respectively.Source: N GLUPS(1), (13) · p. 9, Figure 3 · p. 10
Energy efficiency
- Energy per lattice update stays approximately constant as GPU count increases; D3Q19 is most energy-efficient, while D3Q27h has the highest energy cost per update.The comparison is for weak scaling of the single-component Taylor–Green benchmark using a 3D MPI decomposition.Source: N GLUPS(1), (13) · p. 10, Figure 4 · p. 11
Comparison with other solvers
- One-to-one performance comparisons with other multiphase solvers are not straightforward because hardware, programming models, and lattice Boltzmann formulations differ.The paper reports earlier single-GPU lightweight-versus-population results, but positions this work as addressing deployment in a distributed-memory multi-GPU setting.Source: N GLUPS(1), (13) · pp. 8, 10
6. Conclusions
LBFAST avoids storing full population arrays by reconstructing distributions from a reduced set of variables, reducing memory footprint while preserving hydrodynamic accuracy. Benchmarks and multi-GPU experiments report robust physical behavior, good scaling, and stable energy per lattice update.
6. Conclusions
- LBFAST reconstructs distributions on the fly from reduced variables instead of storing full population arrays, reducing memory footprint while preserving hydrodynamic accuracy.Source: §6 · p. 11
- Taylor–Green vortex decay and multicomponent Laplace benchmarks reproduce expected physical behavior across a wide parameter range without compromising stability.Source: §6 · p. 11
- LBFAST shows good strong- and weak-scaling on Leonardo to several hundred GPUs, with multidimensional domain layouts advantageous at large scales.The experiments also compare velocity sets, which remain competitive in scalability despite higher-order models requiring additional operations.Source: §6 · p. 11
- Higher-order velocity sets require additional operations yet remain competitive in scalability, trading computational cost against physical fidelity.Source: §6 · p. 11
- Energy per lattice update remains stable in weak scaling, with increasing parallelism introducing no additional energy overhead per unit of work.Source: §6 · p. 11
- The authors present LBFAST as a promising tool for large-scale GPU applications, including multicomponent and thermally fluctuating systems.Source: §6 · pp. 11–12