Source-linked AI summary
A review on Deep Reinforcement Learning for Fluid Mechanics
Paul Garnier, Jonathan Viquerat, Jean Rabault, Aurélien Larcher, Alexander Kuhnle, Elie Hachem
TL;DR
Fluid-mechanics optimization and control involve nonlinear, high-dimensional problems with costly or unreliable conventional searches. The paper reviews DRL applications and coupling strategies, while presenting examples of its use in fluid mechanics. The reviewed results indicate robust, computationally practical possibilities, alongside unresolved high-Reynolds-number and learning-cost limitations.
Problem
Fluid-mechanics design and control face nonlinear, high-dimensional searches, costly sensitivity calculations, and limitations of some classical optimization methods.
Method
The paper introduces DRL, reviews fluid-mechanics applications and coupling methods, compares them with classical approaches, and describes recent test cases.
Results
The reviewed DRL algorithms proved robust to numerical noise; transfer learning, parallelization, and autoencoders supported faster convergence or reduced computational demands.
Takeaways & Limitations
DRL can be coupled with existing CFD solvers through high-level interfaces with minimal investment, supporting optimization and control tasks.
Takeaways & Limitations
High-Reynolds-number applications remain to be achieved, while representative learning can require around 24 hours on a modern CPU.
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) has recently been adopted in a wide range of physics and engineering domains for its ability to solve decision-making problems that were previously out of reach due to a combination of non-linearity and high dimensionality. In the last few years, it has spread in the field of computational mechanics, and particularly in fluid dynamics, with recent applications in flow control and shape optimization. In this work, we conduct a detailed review of existing DRL applications to fluid mechanics problems. In addition, we present recent results that further illustrate the potential of DRL in Fluid Mechanics. The coupling methods used in each case are covered, detailing their advantages and limitations. Our review also focuses on the comparison with classical methods for optimal control and optimization. Finally, several test cases are described that illustrate recent progress made in this field. The goal of this publication is to provide an understanding of DRL capabilities along with state-of-the-art applications in fluid dynamics to researchers wishing to address new problems with these methods.
1 Introduction
Fluid-mechanics optimization involves nonlinear, high-dimensional searches where exhaustive evaluation is often infeasible and classical methods can be costly or locally trapped. The paper reviews the still-limited DRL literature and introduces DRL concepts, applications, coupling choices, and future possibilities.
- Nonlinear, high-dimensional design spaces can make exhaustive evaluation of fluid-mechanics configurations unfeasible, motivating automatic optimization procedures.
- CFD optimization often incurs high computational costs for sensitivity calculations, while gradient-based methods may lack robustness and fall into local optima.
- Applications include multiphase microfluidics, aerodynamic shape optimization, conjugate heat transfer, biomechanics, energy, and marine technologies.
- Despite its potential, DRL applications in computational and experimental fluid dynamics remain limited and include flow control, process optimization, and shape optimization.
- The paper introduces modern RL and deep-learning basics, reviews fluid-mechanics applications, examines algorithms and reward shaping, and outlines future DRL–fluid-dynamics possibilities.
2 Deep Reinforcement Learning
This section introduces the mathematical foundations of reinforcement learning, explains its combination with deep learning, and surveys several DRL algorithms.
- The section presents reinforcement learning through a Markov Decision Process before detailing deep learning integration and surveying DRL algorithms.
2.1 Reinforcement Learning
Reinforcement learning models an agent interacting with an environment to choose actions that maximize discounted cumulative reward, using model-free or value- and policy-based approaches.
- An RL agent receives observations and rewards from an environment and seeks actions that maximize cumulative reward over an episode.
- A trajectory is a sequence of experienced states and actions, while discounted cumulative reward weights temporally distant rewards using γ ∈[0, 1].
- Model-free algorithms interact directly with the environment and are the most commonly used algorithms within the DRL community.
- Value-based methods estimate value functions to select high-value actions, whereas policy-based methods directly optimize a parameterized policy.
- Policy-based methods naturally handle high-dimensional action spaces and can learn stochastic policies, but they may become trapped in local minima.
- Actor-critic methods combine policy optimization with Q-function evaluation to address low-quality actions being obscured in vanilla policy-gradient averaging.
2.2 Deep Reinforcement Learning
Deep reinforcement learning uses neural networks to approximate value functions or produce action distributions, overcoming the poor scalability of tabular representations for large state and action spaces.
- Neural networks replace tables when state or action spaces are too large to scale with tabular value estimates.
- An artificial neuron computes w · x + b and applies an activation function σ to produce its output.
- Fully connected networks link every neuron in one layer to all neurons in the following layer, with weights and biases defining their adjustable parameters.
- Neural-network learning adjusts weights and biases to reduce a loss function, typically using stochastic gradients estimated through back-propagation.
- Convolutional layers extract local features from image patches and build increasingly sophisticated feature hierarchies.
2.3 DRL algorithms
The section introduces major DRL algorithm families, from value-based DQN methods to policy-gradient, actor-critic, and clipped policy optimization approaches. These methods address value estimation, exploration, variance, and overly large policy updates through distinct design choices.
- Deep Q-networks: DQN uses an artificial neural network to map states and actions to estimated Q-values instead of storing a Q-value table.The network is trained with a loss function based on the Bellman optimality equation.
- Deep Q-networks: Vanilla deep Q-learning balances exploration and exploitation with ε-greedy action selection, gradually reducing random actions during learning.With probability ε the agent selects a random action; otherwise it chooses the action with the highest Q-value.
- Deep Q-networks: Experience replay, fixed target networks, Double DQN, and prioritized replay address forgetting, correlated experiences, moving targets, overestimation, and poorly learned events.These improvements respectively reuse past experiences, stabilize targets, separate action selection from target estimation, and emphasize experiences with high training loss.
- Policy gradients: Policy-gradient methods optimize neural-network policy parameters using trajectory-derived advantage estimates, reducing the variance of full discounted-return gradients.The advantage compares an action’s expected cumulative reward with the average reward of actions available in the same state.
- Actor-critic methods: Actor-critic methods update a policy-based actor and value-based critic in parallel, using time-difference updates and an advantage estimate for policy improvement.The critic evaluates action quality and supplies targets used to update both value estimates and the actor.
- Trust-region and proximal policy optimization: PPO clips policy-ratio updates to prevent steps far from the previous policy, while preserving favorable updates indicated by positive advantages.Clipping limits large changes that could damage or collapse policy performance.
2.4 A conclusion on DRL
DRL has become an established and increasingly widespread approach for complex tasks, while its fluid-mechanics literature remains limited. Its agnostic interface, transfer-learning robustness, and parallel-learning capabilities make it promising for optimization and design in computational or experimental environments.
- Field growth: DRL’s ability to learn complex tasks is established, and publications using it have increased across multiple research fields.Figure 3 tracks publications mentioning deep reinforcement learning per year since 2010 and marks important algorithms or achievements.
- Capabilities: DRL supports control and optimization in both continuous and discrete contexts through a well-defined state, action, and reward interface.The environment may be a numerical simulation or a real-life experiment.
- Capabilities: Transfer learning can make agents train faster in similar environments, a strength for fluid mechanics where computational time is limiting.The paper contrasts this reuse of trained behavior with regular optimization techniques such as adjoint methods.
- Capabilities: DRL libraries can exploit parallel learning with close-to-perfect scaling on available resources, supporting optimization and design processes.The stated promise applies to computational or experimental environments.
3 Applications
The reviewed applications span swimming, rigid-body control, active flow control, and aerodynamic optimization, using DRL with varied fluid-mechanics couplings. Reported results include reduced energy expenditure, high-performance flow design, effective drag suppression, accelerated training, and agreement with classical optimization.
- Applications: The review surveys DRL applications across flow control, process optimization, shape optimization, and fluid–rigid-body systems.The applications include swimmer coordination, fluid-jet control of rigid bodies, active cylinder-wake control, and control-cylinder placement.
- Synchronised swimming: 20% higher swimming efficiency than the baseline of multiple solitary swimmers was achieved for a DRL-controlled follower fish.The leader follows a prescribed gait, while the follower learns a strategy in a two-dimensional viscous incompressible flow.
- Flow shape design: PMR over 90% was obtained for different target flows, and transfer learning reduced retraining time compared with an untrained agent.The reported comparison identifies DRL as a viable alternative to genetic algorithms for the reviewed flow-shape-design task.
- Active flow control: DRL-controlled jets suppressed around 93% of vortex-shedding drag while requiring typically 0.6% of the base-flow mass flow rate.The resulting wake modification was similar to boat-tailing, with an apparent increase in the recirculation bubble.
- Accelerated learning: Parallel multi-environment learning achieved a training speedup factor of up to around 60 without sensibly reducing learning ability.For simulation counts dividing the ANN learning frequency, perfect scaling was obtained and training was formally equivalent to serial learning.
- Classical comparison: For the two-cylinder optimization, DRL found the same optimal control-cylinder positions as the classical adjoint method at Re = 40 and Re = 100.At Re = 10, initial agents trained for approximately 5 hours with 17 CPUs, while transfer learning enabled later agents to train in 10 hours on one CPU.
4 Conclusion
The review finds that DRL can be coupled with CFD solvers with minimal investment and supports diverse optimization and control tasks. It also identifies unresolved challenges in high-Reynolds-number, highly turbulent, nonlinear flows and high-dimensional action spaces.
- 4 Conclusion: DRL algorithms can couple to existing numerical CFD solvers with minimal investment, enabling diverse optimization and control tasks.The review attributes this accessibility to the high-level interfaces of existing libraries.
- 4 Conclusion: The reviewed algorithms proved robust to possible numerical noise.
- 4 Conclusion: Transfer learning and autoencoders improved computational efficiency by accelerating policy convergence.Transfer learning reused agents from similar situations, while autoencoders reduced network size through compact, rich observations.
- 4 Conclusion: High-Reynolds-number applications remain to be achieved.
- 4 Conclusion: DRL capabilities and robustness in highly turbulent, nonlinear flows, and convergence in high-dimensional action spaces, remain unexplored.