Source-linked AI summary
Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
Colin Vignon, Jean Rabault, Ricardo Vinuesa
TL;DR
Active flow control offers promising engineering prospects, but applying reinforcement learning to increasingly complex fluid-dynamics problems raises field-specific challenges. This review surveys RL and DRL frameworks, focuses on DRL-based AFC, and highlights current limitations, upcoming milestones, and open questions.
Problem
Active flow-control problems have promising engineering prospects, while applying reinforcement learning to increasingly complex fluid-dynamics settings remains challenging.
Method
The review provides an overview of RL and DRL frameworks and their advances, then examines DRL applications to active flow control.
Results
The review identifies current challenges and limitations of DRL algorithms in active flow control and outlines potential milestones and open questions.
Takeaways & Limitations
The review frames DRL-based active flow control around fluid-dynamics challenges, prospective milestones, and questions for the fluid-mechanics community.
Takeaways & Limitations
Value-based and policy-based reinforcement-learning methods remain limited in the classical reinforcement-learning discussion.
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) has been applied to a variety of problems during the past decade, and has provided effective control strategies in high-dimensional and non-linear situations that are challenging to traditional methods. Flourishing applications now spread out into the field of fluid dynamics, and specifically of active flow control (AFC). In the community of AFC, the encouraging results obtained in two-dimensional and chaotic conditions have raised interest to study increasingly complex flows. In this review, we first provide a general overview of the reinforcement-learning (RL) and DRL frameworks, as well as their recent advances. We then focus on the application of DRL to AFC, highlighting the current limitations of the DRL algorithms in this field, and suggesting some of the potential upcoming milestones to reach, as well as open questions that are likely to attract the attention of the fluid-mechanics community.
I. INTRODUCTION
This introduction frames DRL as a promising approach for active flow control, where nonlinear, high-dimensional flows challenge traditional methods. It reviews RL/DRL foundations, applications, limitations, and future milestones for coupling DRL with AFC.
- DRL combines deep neural networks with reinforcement learning to address nonlinear, high-dimensional problems.
- RL agents interact with environments by receiving observations, selecting actions, and maximizing rewards; model-free methods operate directly from partial observations without an analytical system description.
- Chaotic flows are nonlinear and high-dimensional because of the governing Navier–Stokes equations and continuous flow state space, making local-linearization methods difficult to apply.
- RL applications in fluid mechanics include modeling, CFD solvers, shape optimization, separation control, drag reduction, wake control, and turbulent-channel skin-friction reduction.
- AFC actively influences separation, turbulence, or heat transfer toward a desirable flow state, whereas passive control does not require an energy supply.
- The review presents RL and DRL frameworks, surveys DRL advances and AFC applications, and highlights fluid-dynamics challenges, upcoming milestones, and open questions.
B. Model-based methods
Model-based methods simplify the environment by extracting dominant flow dynamics, reducing computation but potentially producing sub-optimal control in highly turbulent or high-Reynolds-number regimes. Model-free methods avoid environmental assumptions but can be computationally heavy because they interact directly with a Navier–Stokes solver.
- Reduced-order modeling can efficiently suppress large-scale structures and reduce drag in a three-dimensional turbulent bluff-body wake.
- Model-based methods construct simplified representations of the environment to capture the main dynamical processes governing the flow.
- The computational gain from simplifying the environment is especially relevant for flow control.
- At high Reynolds numbers, approximate models can yield sub-optimal solutions because they only partially represent turbulence.
- Recent work explores both model-based and model-free methods to balance model efficiency against the limitations of approximate models.
- Model-free methods directly interact with the environment without assuming an environmental model, but may require computationally heavy Navier–Stokes simulations.
1. Markov Decision Process
Reinforcement learning models sequential decision-making with states, actions, rewards, transition probabilities, and policies. Its methods differ by value versus policy optimization, feedback structure, and whether learning reuses data from other policies or past episodes.
- 1. Markov Decision Process: A Markov decision process is defined by states, actions, rewards, and transition probabilities describing how actions change states.
- 1. Markov Decision Process: A policy maps states to action choices or action probability distributions, while its parameters determine how the policy is modified.
- 1. Markov Decision Process: The Q-function measures expected cumulative reward after taking an action in a state, whereas the value function evaluates trajectories from the state alone.
- 1. Markov Decision Process: Value-based methods learn the Q-function and derive a policy, whereas policy-based methods directly optimize the policy.
- 3. Closed-loop and open-loop methods: Closed-loop control uses regular state feedback to adapt actions, while open-loop control does not rely on the current environment state.
- 4. On-policy and off-policy online and offline methods: Off-policy methods can reuse data generated by other or earlier policies, whereas on-policy methods cannot reuse data produced by a previous policy version.
- 4. On-policy and off-policy online and offline methods: Online learning updates policies in real time and uses current states, but requires computational resources that may not be available.
5. Value-based methods
Value-based reinforcement-learning methods estimate action values and derive actions from the learned Q-function. Q-learning updates these estimates recursively toward the optimal Q-function, while policy-gradient methods instead optimize policy parameters using reward gradients.
- 5. Value-based methods: Value-based methods learn the Q-function and derive the optimal policy from it.
- 5. Value-based methods: The Q-table represents action values across state–action pairs and can be approximated by a neural network in general settings.
- 5. Value-based methods: Q-learning recursively updates estimated action values so that Qπ converges to Q∗.
- 5. Value-based methods: The greedy policy selects the action with the largest Q∗ value for the current state.
- 6. Policy-based methods: Policy-gradient methods optimize a parameterized policy by adjusting its parameters in the direction of the objective gradient.
- 6. Policy-based methods: Stochastic policy gradients estimate the objective from sampled trajectories and their average cumulative rewards.
- 6. Policy-based methods: Deterministic policy gradients use deterministic actions and therefore sample states rather than a range of actions and states.
D. Limitations of the classical reinforcement-learning methods
Classical RL methods using simple function approximators are limited in state and action-space complexity, while neural networks extend RL to high-dimensional and nonlinear problems. Experience replay accelerates value-based methods, but RL with neural networks still faces correlation, moving-target, instability, and divergence issues.
- Experience replay reduces agent–environment interactions and accelerates value-based RL methods by reusing past experience.Replay buffers also provide more uncorrelated input data and a more stabilized data distribution for learning.
- Classical RL algorithms using simple function approximators are limited to problems with low-dimensional state and action spaces.
- Neural networks act as universal function approximators that can address high-dimensional and nonlinear problems beyond traditional RL limitations.Their use in RL enhanced performance in multi-input multi-output problems.
- On-policy learning from sequential tuples creates correlations that can bias training and cause neural-network learning to fail.Small changes in Q can also substantially alter the data distribution.
- DQL remains vulnerable to instability because its neural network continually chases a changing target, especially with nonlinear function approximators.This instability can lead to divergence and reduce algorithm robustness.
2. Deep Q-networks
Deep Q-networks stabilize value-based deep RL by separating action-value estimation from a slowly updated target network. DQN variants further address high-dimensional environments, similar-valued actions, and overestimation caused by using one network for both action selection and evaluation.
- Dueling DQN splits estimation into advantage and state-value streams, improving treatment of high-dimensional environments and similar-valued actions.
- DQN uses a second, slowly reacting network to estimate targets and stabilize value-based learning.The target network is updated less frequently or through a slow iterative update, slowing target movement.
- Double DQL and double DQN separate action selection from action evaluation to address overestimation of expected cumulative reward.The two networks alternate roles in double DQL, while double DQN reuses the second network already present in DQN.
- Using one neural network for both selecting and evaluating actions can overestimate expected cumulative reward and adversely affect performance.
- DQN achieved strong results with high-dimensional observation spaces, while double DQN further enhanced value-based DRL performance in Atari games.
4. Challenges and openings
Value-based methods struggle with high-dimensional or continuous action spaces, motivating policy-based and actor–critic approaches. DDPG extends deterministic actor–critic control to such action spaces and has shown promising results in chaotic-system stabilization and flow control.
- DQN and related value-based methods perform poorly as action-space dimensionality increases because they repeatedly compute an argmax over actions.This limitation motivates methods that support continuous action spaces.
- Policy-based methods avoid the action argmax and can handle action spaces that are very large or continuous.They can also support either off-policy or on-policy strategies, unlike the earlier replay-based methods described here.
- Deep policy-gradient methods optimize a policy network from trajectory returns, while advantage functions can reduce variance and the number of required trajectories.
- Actor–critic methods combine a policy-producing actor with a critic that evaluates actions through a Q-function or advantage estimate.This combines policy-based action selection with value-based evaluation.
- DDPG uses a deterministic actor–critic policy with perturbative exploration to leverage high-dimensional and continuous action spaces.It is a model-free, off-policy actor–critic method.
- DDPG controllers stabilized a chaotic Kuramoto–Sivashinsky system and reduced skin-friction drag in channel flow, illustrating promising performance with high-dimensional action spaces.
4. Proximal policy optimization
PPO is an on-policy actor-critic algorithm designed to improve robustness over earlier deep reinforcement-learning methods. Its clipped updates, variants, and parallel implementations make it useful for continuous control and increasingly complex flow-control problems.
- PPO is an on-policy actor-critic method developed to address robustness limitations of DQN and vanilla policy-gradient algorithms.
- PPO simplifies TRPO through first-order optimization while retaining the same goal of constraining policy changes.Alternative PPO losses use KL penalties rather than clipping.
- Its clipped objective penalizes policy updates that move too far from the previous policy, with ε controlling the permitted modification range.The clipping function returns the likelihood ratio unchanged within [1 − ε, 1 + ε] and caps it outside that interval.
- With ε = 0.2, PPO outperformed A2C, vanilla policy gradients, TRPO, and CEM on almost all tested continuous-control environments.
- PPO variants address distinct control needs, including open-loop operation, adaptive exploration variance, and actuator reduction for complex problems.PPO-CMA expands exploration variance during search and shrinks it near the global optimum, while AS-PPO-CMA limits actuator count and cumulative-reward loss.
- Parallelization: Parallelizing training episodes can divide computational cost by approximately N, with one implementation dividing computation time by twenty.Additional parallel environments can offset the larger batches required by more complex problems.
- Parallelization: Distributed agents can accelerate DRL and stabilize value-based methods without replay buffers, while exploiting flow-control locality and translational invariance.Gorila was approximately twenty times faster than DQN with 100 parallel agents.
IV. DEEP REINFORCEMENT LEARNING FOR FLOW CONTROL
Active flow control can outperform passive approaches across a wider range of operating conditions, but it requires energy and is often more complex to implement. The review surveys AFC advances and focuses on DRL challenges and upcoming milestones.
- Flow-control strategies are divided into passive and active methods, with active control generally offering greater efficiency across operating conditions.Passive devices are simpler and require neither energy supply nor environmental feedback, whereas active methods require energy and are often more complex.
- Riblets achieve 5 to 9% drag reduction, while modern active methods can achieve reductions twice as large or more.
- The review presents recent advances in active flow control, particularly applications of DRL to AFC.
- It highlights current challenges in combining DRL with AFC and identifies possible upcoming milestones.
B. Active flow control
Active flow control uses energy and may incorporate data during control or beforehand, distinguishing reactive feedback from predetermined strategies. Predetermined methods can be efficient but their effectiveness is limited when they cannot respond to evolving flow dynamics.
- Active control requires energy, and its data may come from experiments or CFD simulations for reactive or predetermined control.
- Feedback control uses instantaneous flow evolution, whereas predetermined control applies a sequence fixed in advance without feedback.
- Predetermined spanwise wall oscillations produced about 40% turbulent-drag reduction, while blowing and suction produced over 10% aerodynamic-efficiency improvement.
- One study reported 25–35% drag reduction and 7% net energy saving for predetermined wall oscillations.The net saving accounts for the power required by the wall oscillations.
- Open-loop actuators can be efficient, but their efficiency range remains limited because they do not account for evolving flow dynamics.
2. Feedback control
Feedback control incorporates the instantaneous flow state and can improve performance, but conventional approaches face limitations in complex or high-Reynolds-number flows. The review positions data-driven methods, especially DRL, as promising alternatives for challenging AFC problems.
- A closed-loop process improved control performance by 20% over open-loop control in the AFC problem considered.
- Opposition control: Opposition-control drag reduction decreases as Reynolds number increases, limiting its alignment with more sophisticated turbulent-flow objectives.
- Optimal control theory: Optimal control directly uses flow equations and can handle many input degrees of freedom, but its time horizon and adjoint-equation computations require substantial resources.
- Data-driven methods: Data-driven methods are presented as promising because they could overcome limitations encountered by earlier control approaches.
- Optimal control theory: Optimal-control time horizons cannot be extended indefinitely because adjoint equations become unstable beyond a threshold, creating shortcomings for high-Reynolds-number flows.
- Data-driven methods: Genetic-programming AFC outperformed overall open-loop methods by approximately 20% in CFD control of a vibrating cylinder.
- Data-driven methods: DRL applications in AFC have overtaken most results from previous methods and are especially suitable for multi-input multi-output problems.
C. Deep reinforcement learning and active flow control: challenges and prospects
DRL has expanded from two-dimensional and chaotic AFC demonstrations toward more complex flows, with current prospects centered on turbulence, three-dimensionality, scalable architectures, and transferability. The review also identifies limitations involving training cost and the implementation of physical symmetries.
- Scientific prospects: DRL may both control complex flows and reveal previously unknown physical structures, as illustrated by a non-opposition-control strategy and studies of chaotic systems.The review presents this as a possible contribution to scientific knowledge beyond engineering control.
- Current applications: DRL has been applied to AFC tasks including body positioning, shape and movement control, flow separation, skin-friction reduction, and heat-transport problems.
- Current applications: PPO controlled a cylinder wake in two-dimensional flow, while later studies demonstrated efficiency across Re ∈[60;400], Re = 1000, and Re = 2000.The later strategy at Re = 2000 differed significantly from opposition control.
- Upcoming milestones: A first milestone is proving existing algorithms effective in highly turbulent conditions, including turbulent-boundary-layer instabilities.The review identifies increasing Reynolds number in the cylinder problem as one route toward this goal.
- Upcoming milestones: Three-dimensional flows, multi-agent control, parallelization, and actuator reduction are proposed to enlarge AFC problems without proportionally increasing training or actuator requirements.Invariances may support arbitrary numbers of actuators while limiting training-cost growth.
- Limitations and prospects: DRL strategies have transferred across flow conditions and tolerated input or output noise, but exploiting symmetries can require constrained or larger networks and more training data.A symmetry-reduced state-action space enhanced DDPG efficiency relative to the fully developed space.
V. CONCLUDING REMARKS
The review presents DRL as a methodological shift for active flow control because nonlinear, high-dimensional systems often require direct full-system treatment. It surveys DRL’s engineering potential while emphasizing that its full value and scientific impact remain under investigation.
- The review provides an overview of RL and DRL frameworks before assessing their potential for active flow control.
- The review highlights DRL as potentially useful for engineering applications and for discovering previously unknown underlying structures in physical systems.
- DRL is attractive for nonlinear, high-dimensional systems where linearization and many analytical tools are not representative or applicable.Its broader applicability comes at the cost of being very data intensive.
- The full value and impact of DRL in AFC remain unsettled while the community investigates how far these methods can be taken and what insights they may yield.