Source-linked AI summary
Mixed-Precision SEM-Based CFD Simulations on GPUs: A Taylor-Green Vortex case
Yanxiang Chen, Manuel Münsch, Roman Iakymchuk
TL;DR
The paper asks how mixed precision can reduce CFD cost and energy without sacrificing accuracy when runtime is distributed beyond Krylov convergence. It evaluates hierarchical precision control in matrix-free SEM-based Taylor–Green vortex simulations, finding about 34% lower time- and energy-to-solution for selected fp32-bounded configurations while improving robustness over global fp32.
Problem
Mixed-precision CFD needs to reduce cost and energy while preserving accuracy, but fine-grained Krylov-focused methods have limited applicability when SEM simulation runtime is distributed across multiple components.
Method
The study evaluates hierarchical block-, kernel-, and step-level precision control in full matrix-free SEM Taylor–Green vortex simulations using Neko.
Results
About 34% reduction in both time-to-solution and energy-to-solution is achieved by the best fp32-based mixed-precision candidate for the high Reynolds number case, relative to the fp64 baseline.
Takeaways & Limitations
Mixed precision for matrix-free SEM-based CFD is best treated as a simulation-level control problem rather than solely as Krylov-solver optimization.
Takeaways & Limitations
The study does not fully decouple storage precision by simulation component because doing so would require substantial architectural restructuring in Neko.
Abstract
from arXiv · showhide
Mixed precision is a promising approach for reducing the computational cost and energy consumption of Computation Fluid Dynamics (CFD) simulations, but its effectiveness depends strongly on where precision is reduced within the full simulation pipeline. In this work, we study Taylor-Green vortex case using Neko, a matrix-free CFD solver based on the spectral element method (SEM). Profiling shows that the fluid time step is not dominated by Krylov convergence alone: the velocity and pressure solvers require only a small number of iterations per step, while a substantial fraction of runtime is spent in other SEM operators and solver components. Motivated by this structure, we propose a three-level hierarchical mixed-precision control model. Two groups of configurations are evaluated in environments bounded by 64-bit floating-point (fp64) and 32-bit floating-point (fp32) precision, respectively. The fp64-bounded group identifies accuracy sensitive components and shows that SEM-focused fp32 computation is a promising direction for future optimization. The fp32-bounded group provides the main practical benefit. For the high Reynolds number case studied, selected configurations reduce both time- and energy-to-solution by about 34% relative to the fp64 baseline, while improving robustness compared with global fp32. Targeted fp16 kernel overrides are also explored, showing potential for selected operations but increased sensitivity in gradient-based quantities such as enstrophy. Overall, these results indicate that mixed-precision for matrix-free SEM-based CFD should be treated as a simulation-level control problem rather than solely as a Krylov-solver optimization.
1 Introduction
The paper argues that fine-grained mixed-precision methods designed for Krylov solvers have limited end-to-end applicability in SEM-based CFD, motivating simulation-level precision control. In the TGV case, solvers perform few iterations and account for only part of the runtime.
- Mixed-precision CFD seeks lower memory traffic, execution time, and energy while retaining high-accuracy solutions.
- Existing mixed-precision Krylov methods mainly switch precision within matrix-based solver operations involving stored matrices or preconditioners.
- The proposed study extends prior velocity-only analyses to full SEM Taylor–Green vortex simulations with both velocity and pressure solves.
- 11,193 cumulative CG iterations versus 3,764 GMRES iterations make the TGV case CG-dominant, while velocity and pressure solves contribute about 19% and 7% of total simulation time.
- About 0.06 s average fluid-step time, with few iterations per step, limits the end-to-end savings available from optimizing inner Krylov operations alone.
2 Methodology
The methodology exposes mixed-precision choices hierarchically across fluid-solve blocks, hotspot kernels, and simulation time steps under a global runtime precision bound. It focuses experimentally on block-level control while exploring kernel overrides and future step-level adaptation.
- The global runtime precision bound rp constrains precision choices across block, kernel, and time-step levels.
- Fluid-solve blocks group SEM, velocity, and pressure components, with SEM covering advection, RHS assembly, boundary treatment, and projection/correction.
- Kernel-level rules can override block defaults for dominant solver kernels, enabling finer precision tuning.
- Precision can vary across time steps because full precision is needed early, whereas settled solvers may tolerate lower precision.
- The study emphasizes block-level precision control, with selected kernel-level experiments, rather than a fully implemented storage-precision separation.
3 Taylor-Green Vortex
The Taylor–Green Vortex benchmark models the temporal decay of a three-dimensional vortex and its transition from smooth to turbulent flow. Experiments vary Reynolds number, mesh polynomial order, and solver settings while tracking enstrophy evolution.
- Benchmark definition: The benchmark evolves a three-dimensional vortex in a periodic box with initial velocity fields specified analytically.The domain is [−Lπ, Lπ]3, and the initial w component is zero.
- Benchmark definition: The study considers Re = 360, Re = 1600, and Re = 10,000, defined using initial velocity, length scale L, and viscosity ν.The Reynolds number is based on the maximum initial velocity magnitude v0, largest-structure length L, and kinematic viscosity ν.
- Experimental cases: Re = 360 supports profiling, Re = 1600 is the main mixed-precision benchmark, and Re = 10,000 serves as a precision-stress test.For Re = 360 and Re = 1600, simulations use 32 × 32 × 32 elements with polynomial orders P = 5 to P = 8.
- Numerical setup: Time advancement uses adaptive stepping targeting CFL = 0.4, dealiasing, and fixed third-order integration; velocity uses CG-Jacobi and pressure uses GMRES-HSMG.The velocity and pressure absolute stopping tolerances are 10−6 and 10−4, respectively; the pressure projection space dimension is 20.
- Flow evolution: The initially smooth flow transitions to turbulence with small-scale structures and isotropic decay, while kinetic energy decays and enstrophy peaks at t/tc = 6.43447162.For Re = 360 and p = 7, the volume-averaged enstrophy peak is 2.03058781.
4 Experimental results
The experiments compare hierarchical mixed-precision configurations under fp64- and fp32-bounded environments for Taylor–Green vortex simulations. Group B provides the main performance gains, while selected higher-precision components improve robustness and accuracy depends on the diagnostic quantity.
- Experimental setup: Eight configurations compare global fp64 and fp32 baselines with block-level precision assignments for SEM, velocity, and pressure components.Group A uses rp=fp64, whereas Group B uses rp=fp32 with double-precision accumulation and selected local storage optimizations.
- Experimental setup: The study uses Re = 1600, polynomial degree P = 8, and a 32×32×32 mesh containing 32,768 elements.Devices are selected to keep approximately 7,000 to 10,000 elements per device.
- 4.1 Performance: Group A slightly increases runtime and energy relative to the fp64 baseline because fp32 compute paths do not overcome repeated type casting and fp64 storage overheads.Solver counts remain similar to the fp64 baseline.
- 4.1 Performance: Group B substantially reduces time- and energy-to-solution, despite a slight increase in pressure iterations, making fp32-based mixed precision the more practical performance direction.The comparison is reported for the Re = 1600, P = 8 case against the double-precision baseline configurations A-0/B-0.
- 4.2 Accuracy: For Re = 10,000 and P = 7, A-2 minimizes kinetic-energy maximum error at eE,max = 0.5462%, while A-4 and A-3 minimize enstrophy maximum and peak-time errors at 2.8763% and 0.0104%.The accuracy test is a precision-stress test rather than a DNS-quality reference because the mesh is substantially finer than required for DNS at this Reynolds number.
- 4.2 Accuracy: B-1 minimizes enstrophy maximum error at eΩ,max = 3.0603%, B-3 minimizes peak-time error at eΩ,peak = 0.1174%, and B-6 shows quantity-dependent fp16 sensitivity.B-6 achieves low eE,max but relatively larger enstrophy errors, so fp16 is better treated as a targeted kernel option.
5 Conclusion and Future Work
The study finds that mixed-precision control must cover the full SEM-based CFD time-stepping loop, not only Krylov solvers. Under an fp32 global bound, selected configurations deliver practical performance gains, while further work targets broader precision adaptation and GPU low-precision residency.
- Few Krylov iterations per time step distribute runtime across the full PN−PN fluid time-stepping loop.
- The model exposes precision control at the block, kernel, and step levels.
- Group A identifies accuracy-sensitive components, with SEM-focused fp32 compute promising future optimization once lower-precision GPU storage becomes resident.
- Approximately 34% lower time-to-solution and energy-to-solution are achieved by the best fp32-based mixed-precision candidate in the high-Reynolds-number case, with improved robustness versus global single precision.
- Targeted fp16 kernel overrides can benefit selected kernels, while future work addresses higher Reynolds numbers, step-based control, and reduced GPU conversion overhead.