Scientific multi-agent reinforcement learning for wall-models of turbulent flows

H. Jane Bae; Petros Koumoutsakos

Source-linked AI summary

Scientific multi-agent reinforcement learning for wall-models of turbulent flows

H. Jane Bae, Petros Koumoutsakos

arXiv:2106.11144v2Jun 21, 2021physics.flu-dyncs.LGphysics.comp-ph

TL;DR

Turbulent-flow simulations need wall models that avoid prohibitive near-wall resolution and limitations from calibrated assumptions or narrow training distributions. The paper introduces SciMARL, in which cooperating computational agents learn LES wall-closure policies from reward information. The resulting models generalize from channel-flow training to different Reynolds numbers and boundary-layer settings while reproducing key wall-flow behavior.

Problem
Near-wall turbulence remains difficult to model because high-resolution LES is costly, existing models use flow-specific empirical calibration, and supervised learning can struggle beyond training distributions.
Method
SciMARL treats LES computational elements as cooperating reinforcement-learning agents that learn wall-closure actions from scalar rewards tied to wall-shear-stress prediction.
Results
The models trained on turbulent channel flows at Reτ = 2000, 4200 and 8000 generalize to turbulent boundary layers and turbulent channel flow at extreme Reynolds numbers.
Takeaways & Limitations
SciMARL provides a limited-data route to wall-model discovery, with the LLWM reproducing wall-shear-stress and off-wall-velocity correlations similar to fully resolved flow.
Takeaways & Limitations
The method’s extrapolation to much larger (hm)+ values may be challenging and can require wall-normal grid refinement with Ny ∼Re[4–6].

Abstract

from arXiv · show

The predictive capabilities of turbulent flow simulations, critical for aerodynamic design and weather prediction, hinge on the choice of turbulence models. The abundance of data from experiments and simulations and the advent of machine learning have provided a boost to turbulence modeling efforts. However, simulations of turbulent flows remain hindered by the inability of heuristics and supervised learning to model the near-wall dynamics. We address this challenge by introducing scientific multi-agent reinforcement learning (SciMARL) for the discovery of wall models for large-eddy simulations (LES). In SciMARL, discretization points act also as cooperating agents that learn to supply the LES closure model. The agents self-learn using limited data and generalize to extreme Reynolds numbers and previously unseen geometries. The present simulations reduce by several orders of magnitude the computational cost over fully-resolved simulations while reproducing key flow quantities. We believe that SciMARL creates unprecedented capabilities for the simulation of turbulent flows.

Introduction

Near-wall turbulence modeling remains costly and difficult because existing approaches depend on calibrated assumptions, solver-sensitive constants, or training distributions that do not generalize broadly. SciMARL addresses these limitations by learning wall-model policies through cooperating agents and limited reward-based information.

Motivation: Wall-resolved LES still requires O(Re13/7) grid points, making near-wall simulations prohibitive despite being cheaper than DNS.Near-wall stress-producing eddies shrink linearly toward the wall, increasing grid requirements.
Limitations of prior models: Existing wall models commonly rely on RANS parametrizations with empirical coefficients calibrated for particular equilibrium flows.This calibration limits their intended behavior across separated flows, roughness, and transition.
Limitations of prior models: Dynamic wall models are robust to Reynolds-number and grid changes but remain sensitive to numerical methods, SGS models, and solver or geometry settings.The cited challenges arise from near-wall numerical errors and limited use beyond structured incompressible solvers.
Machine-learning gap: Supervised-learning wall and SGS models show promise but often struggle to generalize beyond the distributions represented in their training data.Their parameters are commonly learned from single-step prediction errors.
Proposed approach: SciMARL uses computational elements as cooperating reinforcement-learning agents whose actions compensate for closure and numerical errors.Agents optimize scalar rewards rather than requiring detailed spatiotemporal supervision.

Results

SciMARL distributes local agents across channel walls, where they observe local flow states, receive rewards, and collectively update a shared policy. The wall model targets correct wall shear stress while accounting for numerical discretization effects.

Agent organization: Agents are distributed along each wall and provide local states, rewards, and actions while maintaining a single domain-wide policy.The policy is updated through repeated agent interactions with the evolving flow environment.
Agent organization: The reinforcement-learning loop advances the system after each action, then returns a new state and scalar reward for the next decision.This supplies the temporal interaction structure used to infer the policy.
State formulation: SciMARL nondimensionalizes states using viscosity and modeled instantaneous friction velocity to support applicability across flow parameters.The modeled quantities can be obtained from the wall-model output without prior flow knowledge.
Objective: The wall model predicts wall shear stress so that friction velocity, mean velocity profiles, and turbulence intensities can be evaluated accurately.Wall shear stress is the stated target of the wall-model policy.

Velocity-based wall model

The velocity-based wall model uses local velocity information and its wall-normal variation to adjust modeled wall shear stress incrementally. Its training reflects improvement relative to the previous time step and uses log-region behavior.

State and action: The velocity-based model observes instantaneous velocity, its wall-normal derivative, and the sampling height as its state.The sampling point is located at wall-normal height hm.
State and action: Agents adjust modeled wall shear stress through a multiplicative action factor an(x, z) ∈[0.9, 1.1].The bounded adjustment avoids requiring the model to output the exact Reynolds-number-dependent stress directly.
Reward: The reward is incremental and proportional to improvement in predicted wall shear stress relative to the previous time step.Additional reward is used to stabilize agent behavior.

Log-law-based wall model

The log-law-based wall model represents the near-wall velocity through instantaneous local log-law coefficients rather than prescribed canonical coefficients. This state choice supports learning beyond the sampled wall-normal locations and toward broader Reynolds numbers and grid resolutions.

Physical basis: The second wall model uses the logarithmic-layer velocity profile as its physical basis.The log layer is described as present in flows with inner-outer scale separation.
Physical basis: The log-law formulation contains the von Karman constant κ and intercept constant B, whose exact values depend on flow configuration and wall roughness.The study uses values associated with a canonical smooth zero-pressure-gradient boundary layer.
State formulation: Its states are instantaneous local coefficients κm and Bm computed from velocity, velocity gradients, and wall-normal location.The model does not input the a priori known κ and B values from the log law.
Generalization: Because κm and Bm do not depend on y∗, the model can learn log-law behavior outside the trained wall-normal range.The passage identifies this as an advantage over the velocity-based model for extension to higher Reynolds numbers or coarser grids.

State-action map

The VWM and LLWM learn state-dependent wall-shear-stress adjustments that receive positive rewards when correcting deviations from log-law behavior. LLWM states cluster near the true log-law coefficients, while VWM extrapolation is limited beyond its training range.

VWM agents up- or down-shift wall shear based on whether the velocity–height state lies above or below the log-law profile.The model learns this behavior without prior knowledge of the log-law coefficients.
The state-action maps are conditioned on positive rewards for channel flows at Reτ = 2000, 4200, 8000.
VWM extrapolation to much larger sampling heights may be challenging because training covered only a limited range of (hm)+.The paper identifies wall-normal grid refinement as a way to keep sampling heights within the trained range.
LLWM agents similarly adjust wall shear according to whether local slope and intercept estimates under- or over-predict the log law.Most LLWM states cluster around the true values of 1/κ and B, supporting the intended control mechanism.

Testing: Turbulent channel flow

Channel-flow tests compare VWM and LLWM friction-velocity errors and mean velocity profiles across Reynolds numbers. LLWM remains accurate across sampling heights, whereas VWM requires the sampling point to stay within its trained range.

VWM produces high errors at Reτ = 2 × 10^4 and 5 × 10^4 when (hm)+ lies outside its trained range.
LLWM friction-velocity prediction error remains below 4% across the tested channel-flow cases.Its error increases with Reynolds number, likely because of stronger streamwise wall-normal-gradient variation and departure from trained sampling heights.
LLWM mean velocity profiles align with the log law regardless of (hm)+.
Refining the wall-normal grid to place the first off-wall point within the trained (hm)+ range significantly decreases VWM errors.The training range cited for VWM is 150 < (hm)+ < 1200.
LLWM results are comparable to the equilibrium wall model up to Reτ ≈ 10^5.The comparison uses the widely used EQWM with an empirical coefficient tuned for this flow configuration.

Testing: Spatially evolving turbulent boundary layer

The LLWM is tested on a zero-pressure-gradient, spatially evolving turbulent boundary layer over Reθ = 1000 to 7000. Its predicted skin-friction coefficient is comparable to empirical values despite training only on channel flow.

The boundary-layer simulation spans Reθ = 1000 to 7000, with Reθ defined using momentum thickness.
Figure 5 plots Cf against Reθ using LLWM symbols and an empirical Cf line.
LLWM predictions of the skin-friction coefficient Cf are comparable to empirical Cf values across the full simulation domain.
The model adapts to streamwise wall-shear-stress variation despite being trained only on a channel-flow simulation.

Distribution of wall-shear stress

The LLWM reproduces the approximately 0.3 wall-shear-stress/off-wall-velocity correlation observed in DNS, unlike the perfectly correlated EQWM.

Figure 6 compares instantaneous x–z snapshots of off-wall streamwise velocity fluctuations and wall-shear stress for EQWM and LLWM.
Figure 7 presents the cross-correlation coefficient between wall-shear stress and instantaneous off-wall streamwise velocity.
DNS observes a wall-shear-stress/off-wall-velocity correlation coefficient of 0.3, while deterministic EQWM produces perfect correlation.
The LLWM reduces the correlation to a maximum of approximately 0.3, matching the expected DNS correlation.

Potential of SciMARL wall models

SciMARL wall models match a tuned RANS-based equilibrium model while using limited in-situ learning data and state variables suited to varied simulations.

SciMARL wall models perform as well as the RANS-based EQWM tuned for the particular flow configuration.
For Reτ = 4200 channel flow, LLWM training uses O(10^3) CPU-hours and less than 1GB storage, versus O(10^7) CPU-hours and more than 100TB for DNS generation.
LLWM uses instantaneous wall-parallel velocity, its wall-normal gradient, and distance from the wall as states.
LLWM is described as extendable to complex geometries, different numerical methods, and different SGS models without prescribed tunable parameters.

Discussion

SciMARL discovers wall-model closures with limited data, trains on moderate-Reynolds-number channel flows, and generalizes to boundary layers and extreme Reynolds numbers.

Discussion: SciMARL fuses LES with cooperating multi-agent reinforcement learning, using recovery of the correct mean wall-shear stress as a reward.
Discussion: Training used turbulent channel flows at Reτ = 2000, 4200, and 8000, followed by tests on a turbulent boundary layer and extreme Reynolds numbers.
Discussion: The VWM captures mean velocity profiles across Reynolds numbers when its wall-normal sampling location lies within the training set.
Discussion: LLWM produces wall-shear-stress/off-wall-velocity correlations similar to fully resolved flow and unlike classical RANS models.
Discussion: The authors anticipate extending SciMARL to wall-bounded flows exhibiting a log law, including roughness, stratification, and compressibility.

Methods

SciMARL trains wall models through distributed multi-agent reinforcement learning, using flow simulations, replay-based policy updates, and wall-stress rewards. The models are evaluated across Reynolds numbers, geometries, and numerical configurations.

Reinforcement learning: Flow simulations are distributed across workers that collect agent experiences, while a master updates policy parameters from replay memory.The implementation uses the Smarties library and separates policy updating from interaction-data collection.
Reinforcement learning: V-RACER represents the policy with a neural network producing action-distribution statistics and state-value estimates, trained with Monte Carlo replay samples.Retrace supplies action-value estimates, and expectations are approximated from batches sampled from replay memory.
Reinforcement learning: ReF-ER stabilizes off-policy training by classifying experiences with importance weights and clipping gradients from samples outside the trust region.The trust-region bounds are [1/C, C], with β adjusted to maintain a target fraction of replay samples inside that region.
Training setup: Training uses channel flows at Reτ ≈2000, 4200, and 8000, with randomized initialization, wall-shear perturbations, and sampling locations in the log layer.Each learning iteration advances the simulation for 2δ/uτ, and the policy uses two hidden layers of 128 units with B = 512 Monte Carlo samples.
Wall-model formulation: Agents adjust modeled wall stress through multiplicative actions, with rewards based on improving mean wall-shear-stress predictions and reaching 1% accuracy.The states include instantaneous velocity, its wall-normal gradient, and the sampling-point location; the LES uses finite differences, fractional stepping, Runge-Kutta advancement, and the AMD SGS model.