Source-linked AI summary
Large-Scale DNS of Gas-Solid Flow on Mole-8.5
Qingang Xiong, Guofeng Zhou, Bo Li, Ji Xua, Xiaojian Fang, Junwu Wang, Xianfeng He, Xiaowei Wang, Limin Wang, Wei Ge, Jinghai Li
TL;DR
The paper addresses the computational demands of resolving multiscale structures in gas-solid flow, where experimental and theoretical methods have limited capability. It implements DNS with coupled LBM, IMB, and DEM on a GPU cluster; the simulations show particle clustering and report about 20x speedup over one mainstream CPU core, with good scalability up to 600 GPUs.
Problem
Understanding multiscale gas-solid flow structures requires massive parallel computing, while experimental and theoretical methods have so far been limited in exploring their fundamental mechanisms.
Source: Introduction · pp. 1–2Method
The paper implements gas-solid DNS by coupling LBM and IMB for gas flow with DEM for particle motion on a GPU cluster.
Source: Conclusions and prospects · p. 10Results
Particle clustering is observed, and the implementation achieves about 20x speedup over one mainstream CPU core with good scalability up to 600 GPUs.
Source: Conclusions and prospects · p. 10Takeaways & Limitations
The authors present parallel LB-IMB-DEM DNS as a promising approach for exploring fundamental mechanisms in gas-solid flow.
Source: Introduction · p. 1, Conclusions and prospects · p. 10The simulated domain remains too small to contain large clusters, and the reported increase in slip velocity is consequently limited.
Source: §2 · p. 8
Abstract
from arXiv · showhide
Direct numerical simulation (DNS) for gas-solid flow is implemented on a multi-scale supercomputing system, Mole-8.5, featuring massive parallel GPU-CPU hybrid computing, for which the lattice Boltzmann method (LBM) is deployed together with the immersed moving boundary (IMB) method and discrete element method (DEM). A two-dimensional suspension with about 1,166,400 75-micron solid particles distributed in an area of 11.5cm x46cm, and a three-dimensional suspension with 129,024 solid particles in a domain of 0.384cm x1.512cm x0.384cm are fully-resolved below particle scale and distinct multi-scale heterogeneity are observed. Almost 20-fold speedup is achieved on one Nvidia C2050 GPU over one core of Intel E5520 CPU in double precision, and nearly ideal scalability is maintained when using up to 672 GPUs. The simulations demonstrate that LB-IMB-DEM modeling with parallel GPU computing may suggest a promising approach for exploring the fundamental mechanisms and constitutive laws of complex gas-solid flow, which are, so far, poorly understood in both experiments and theoretical studies.
Introduction
DNS can resolve the smallest continuum scales in complex flows, but multiphase simulations are computationally demanding. The section motivates GPU-parallel LBM-IMB as a way to study gas-solid flow structures at larger scales.
Introduction
- Multiphase DNS is computationally more extensive than single-phase turbulent-flow DNS, largely because enforcing no-slip conditions at phase interfaces is challenging.Conventional finite-difference and finite-volume approaches are described as cumbersome for this task.Source: Introduction · p. 1
- Particle methods adapt well to complex interfaces, but their high computational cost and relatively low accuracy limit application to larger systems.The cited examples include SPH and MaPPM methods used in gas-solid suspension simulations.Source: Introduction · pp. 1–2
- LBM-IMB had demonstrated efficient gas-solid DNS on a single CPU core, while Mole-8.5 offered a GPU cluster suited to scaling the approach.A prior study observed a transition from homogeneous to heterogeneous structures as the solid/gas density ratio increased from 2.5 to 1500, running each case within one week.Source: Introduction · p. 2
- Massive parallel computing is necessary to study multiscale gas-solid flow structures in two and three dimensions.The paper presents DNS as a tool for exploring mechanisms that experimental and theoretical methods have so far been limited in examining.Source: Introduction · pp. 1–2
Numerical scheme
The numerical scheme couples a lattice Boltzmann gas solver with immersed moving boundaries and DEM particle interactions. Its interface treatment enforces no-slip coupling, while particle forces and torque are accumulated over covered lattice sites.
Numerical scheme
- The LBGK model uses D2Q9 in two dimensions and D3Q19 in three dimensions, with immersed moving boundaries embedded as a body force to couple gas and solids.The method is used to realize the no-slip condition at the gas-solid interface.Source: Numerical scheme · p. 2
- The local solid area/volume fraction ε measures the portion of a lattice covered by solid particles.This fraction enters the immersed-boundary formulation.Source: Numerical scheme · p. 2
- Hydrodynamic forces and torque on a solid particle are summed over the n lattices it covers.The immersed-boundary treatment supplies the interphase force contributions.Source: Numerical scheme · p. 2
- Particle interactions use a frictionless DEM contact treatment with a stiffness coefficient and collision terms that account for energy dissipation.The stated collision mode resembles hard-disk or hard-sphere collisions and is readily carried out on GPUs.Source: Numerical scheme · p. 3
Parallel implementation
The implementation combines GPU kernels with CPU-GPU and inter-node communication on Mole-8.5. It uses lookup tables to reduce interface calculations and a hybrid memory and messaging design to handle distributed simulations.
Parallel implementation
- A particle neighbor list is rebuilt roughly every 100 time steps, reducing the cost of repeated neighbor searches.The approach uses extra memory to reduce computation on particle collisions.Source: Parallel implementation · p. 3
- Accurately computing γ on boundary lattices creates complex GPU control flow and requires topology updates at every step.The lookup-table method addresses this computational challenge, particularly for three-dimensional boundaries.Source: Parallel implementation · pp. 3–4
- A precomputed lookup table accelerates the interface parameter γ by indexing it from lattice-particle distance and orientation, trading some accuracy for speed.The values are stored in GPU constant memory to avoid repeated intensive calculations.Source: Parallel implementation · p. 4
- GPU memory and throughput limits make multi-GPU execution necessary for simulations requiring billions of lattices and millions of time steps.A Tesla C2050 holds only tens of millions of lattices and reaches about 200 million lattice updates per second in three-dimensional double precision.Source: Parallel implementation · p. 4
- The implementation combines CUDA, shared memory, and MPI, using shared memory for intra-node communication and MPI for inter-node transfers.GPUDirect is used to reduce communication time among GPUs on different nodes.Source: Parallel implementation · p. 4
1. Validation
The validation compares GPU and CPU implementations with Fluent for three-dimensional flow past a fixed sphere. It reports close agreement between single- and multi-GPU gas velocities and selects a lattice-resolution compromise for subsequent simulations.
1. Validation
- The validation case is three-dimensional gravity-driven flow past a fixed sphere, simulated on a single CPU, multiple GPUs and CPUs, and Fluent 6.2.The passage identifies Table 1 as containing physical parameters and Table 2 as containing simulation parameters and main results.Source: §1 · pp. 5–6
- The particle-diameter-to-lattice-spacing ratio d/l is set to 10 as a compromise between simulation accuracy and computational cost.This resolution is used in the simulations that follow.Source: §1 · p. 5
- The gas-velocity difference between multi-GPU and single-GPU implementations is on the order of 10^-13.The paper treats this agreement as evidence for the correctness of the parallel GPU program.Source: §1 · p. 5
2. Simulation results
The simulations achieved about 20x single-GPU speedup and nearly linear multi-GPU scaling while resolving gas-solid suspensions with multiscale clusters. The 2D results show strong intracluster drag and elevated slip velocity, while the 3D domain was too limited to develop large clusters.
2. Simulation results
- A single GPU achieved about 20x speedup over one core of an Intel E5520 CPU, with slightly higher speedup as domain size increased.Table 3 reports computation steps per second for a single GPU and a single CPU.Source: §2 · pp. 6–7
- Multi-GPU strong scalability remained almost linear below approximately 600 GPUs.The scalability results are shown in Figure 3.Source: §2 · pp. 6–7
- Gas velocity was significantly lower inside clusters than in dilute regions, implying much larger particle drag there and supporting assumptions in the EMMS model.The cited EMMS works are Li and Kwauk (1994), Li, et al. (1999), and Li, et al. (2005).Source: §2 · p. 7
- The time-averaged steady-state slip velocity was 0.017, significantly above the homogeneous-suspension prediction of Wen and Yu (1966) and quantitatively consistent with EMMS predictions.The slip velocity and drag-force evolution are shown in Figure 5.Source: §2 · pp. 7, 9
- Three-dimensional DNS gave qualitatively similar conclusions, but its size limitations left large clusters absent and made the slip-velocity increase less evident.The 3D particle and flow snapshots and slip-velocity evolution are shown in Figures 6 and 7.Source: §2 · pp. 7–10
- The authors believe these large-scale, fully resolved results can provide more meaningful information to higher-level simulation methods, including two-fluid and discrete-particle models, and eventually improve their accuracy.Source: §2 · pp. 6, 8
Conclusions and prospects
The parallel GPU implementation enables large gas–solid DNS, with multi-scale particle clustering observed in two- and three-dimensional suspensions. Its reported speedup and scalability support its use as a promising tool, while PCIe transfer and three-dimensional topology remain implementation challenges.
Conclusions and prospects
- The implementation couples parallel LBM with IMB for gas flow and DEM for solid-particle motion, combining CUDA, shared-memory, and MPI programming.Source: Conclusions and prospects · p. 10
- Multi-scale particle clustering was observed in simulations of 1,166,400-particle 2D and 129,024-particle 3D suspensions, supporting the EMMS model for heterogeneous multi-phase systems.Source: Conclusions and prospects · p. 10
- About 20x speedup over one mainstream CPU core and good scalability up to 600 GPUs were achieved.Source: Conclusions and prospects · p. 10
- PCIe transfer remains a limitation of the first implementation; asynchronous communication can overlap data transfer and computation, while complex 3D topology still requires substantial effort.Source: Conclusions and prospects · p. 10