Source-linked AI summary
Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control
Jean Rabault, Miroslav Kuchta, Atle Jensen, Ulysse Reglade, Nicolas Cerardi
TL;DR
Active flow control lacks demonstrated applications of deep neural networks with reinforcement learning. This paper trains such an agent to control a cylinder wake with synthetic jets, observing up to around 8% drag reduction.
Problem
Deep neural networks and reinforcement learning had seen limited application in fluid mechanics, motivating their introduction to active flow control.
Method
A PPO-based deep reinforcement learning agent trains an artificial neural network to control two zero-net-mass-flow synthetic jets on a cylinder.
Results
Up to around 8% drag reduction is observed while the ANN controls the 2D Kármán vortex street.
Takeaways & Limitations
The results provide proof that deep reinforcement learning can solve the analytically intractable problem of active flow control in this setting.
Takeaways & Limitations
The modified flow configuration is unstable, requiring small corrections to remain near the learned state.
Abstract
from arXiv · showhide
We present the first application of an Artificial Neural Network trained through a Deep Reinforcement Learning agent to perform active flow control. It is shown that, in a 2D simulation of the Karman vortex street at moderate Reynolds number (Re = 100), our Artificial Neural Network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder. By interacting with the unsteady wake, the Artificial Neural Network successfully stabilizes the vortex alley and reduces drag by about 8%. This is performed while using small mass flow rates for the actuation, on the order of 0.5% of the mass flow rate intersecting the cylinder cross section once a new pseudo-periodic shedding regime is found. This opens the way to a new class of methods for performing active flow control.
1 Introduction
Active flow control is important for reducing drag but remains difficult because fluid-mechanical systems are nonlinear, high-dimensional, time-dependent, and costly to solve in real time. This work introduces Deep Reinforcement Learning with neural networks as a new approach to active flow control.
- Motivation: Drag reduction matters across industry, aviation, and biology because aerodynamic drag causes substantial energy losses.The introduction cites 20% of energy losses on modern heavy-duty vehicles as aerodynamic drag and identifies drag as the main energy-loss source for airplanes.
- Motivation: Passive drag-reduction methods have been extensively studied, including Micro Vortex Generators for controlling transition to turbulence.The passage distinguishes this approach from shark-inspired mechanisms and discusses benefits in reduced drag.
- Challenge: Active flow-control design is complex because sparse flow measurements must be converted into control strategies within nonlinear, high-dimensional, computationally costly domains.These characteristics hinder analytical solutions and real-time predictive strategies.
- Novelty: The study introduces Deep Neural Networks and Reinforcement Learning to active flow control for the first time.The introduction positions these methods as successful in fields including image analysis, speech recognition, and optimal control.
- Approach: The work applies Deep Reinforcement Learning using PPO and a Fully Connected Artificial Neural Network to control two synthetic jets in a 2D unsteady wake.The cylinder geometry provides a low-cost setting that retains time dependence, nonlinearity, and high dimensionality.
2 Methodology
The methodology couples a non-dimensional 2D cylinder-flow simulation with PPO-based continuous jet control, using flow measurements from velocity probes as observations and drag/lift-related rewards. The proof-of-concept uses deliberately low mesh density and Reynolds number for rapid laptop training, with temporally smoothed actuation to support learning and numerical stability.
- Flow simulation: The simulation models flow around a cylinder with diameter D = 1 in a domain of length L = 22 and height H = 4.1.The flow is non-dimensionalized, with no-slip walls and an outflow boundary at the right domain wall.
- Numerical implementation: The governing equations use IPCS with explicit nonlinear treatment and finite-element spatial discretization in FEniCS.Mesh density and Reynolds number were kept low to enable fast proof-of-concept training on a laptop.
- Actuation: Two 10° jets at θ1 = 90° and θ2 = 270° control the cylinder-side mass flow rates Q_i, i = 1, 2.The jets are normal to the cylinder wall and use parabolic-like velocity profiles that vanish at their edges.
- Observations and learning: The PPO agent receives horizontal and vertical velocity measurements from 151 probes distributed around the cylinder and wake.The observations are intended to provide detailed information about the flow configuration for control-strategy discovery.
- Observations and learning: PPO learns continuous jet control from observations, actions, and rewards based on time-averaged drag penalized by mean lift.PPO was selected for continuous control, lower mathematical complexity, faster operation than TRPO, and little to no metaparameter tuning.
- Training stabilization: Control values remain constant for 50 numerical time steps, approximately 7.5% of a vortex shedding period, and are made continuous between updates.These modifications address PPO’s difficulty learning time-correlated continuous signals and avoid pressure and velocity jumps in the incompressible solver.
3 Results
Across 10 training realizations, the ANN learned a stable and robust active-flow-control strategy that reduced drag and strongly suppressed fluctuations. The strategy used an initial transient actuation to reach a lower-drag pseudo-periodic regime, then maintained it with small corrections.
- Drag and fluctuation control: 8% drag reduction was observed with active flow control, while vortex-shedding fluctuations were drastically reduced.The controlled regime follows an initial transient actuation and becomes pseudo-periodic with very limited control.
- Learning robustness: 10 training realizations produced representative results, demonstrating that DRL/PPO consistently learned a stable and robust drag-reduction strategy.The reported results use deterministic prediction without exploration noise.
- Drag and fluctuation control: ⟨CD⟩≈3.205 was the baseline mean drag coefficient, compared with an actively controlled mean reported as around ⟨C′.The supplied passage truncates the controlled mean value after ⟨C′.
- Drag and fluctuation control: 0.0016 was the controlled drag-fluctuation amplitude, a factor of around 20 below baseline; lift fluctuations decreased by around 5.7.The actively controlled system’s shedding frequency was around 3.5% lower.
- Actuation strategy: 0.02 was the peak normalized control mass-flow norm during the transient, falling to around 0.006 after the pseudo-periodic regime was established.The transient peak was a factor 3 smaller than the maximum value allowed during training.
- Flow modification: The controlled flow modifies the two-dimensional wake and separation behavior, with a larger separated-wake area than in the baseline case.This comparison concerns the established pseudo-periodic controlled regime after the initial large transient actuation.
4 Conclusion
Deep Reinforcement Learning, specifically PPO, discovered an active flow-control strategy for synthetic jets that controls the 2D Kármán vortex street. The results establish DRL as a method for active flow control and motivate applications to more complex simulations and real-world experiments.
- 4 Conclusion: DRL, specifically PPO, discovered an active flow-control strategy for synthetic jets on a cylinder and controlled the 2D Kármán vortex street.The strategy emerged by optimizing a reward function while treating the flow as an environment for interaction.
- 4 Conclusion: DRL provides a proof of concept for solving the high-dimensional, analytically untractable problem of active flow control.ANNs efficiently approximate strongly nonlinear functions and can be trained through direct experimentation with the flow.
- 4 Conclusion: The methodology motivates applying DRL to more complex simulations, including realistic 3D LES/DNS on large computer clusters, and to real-world experiments.The proposed directions also include investigating transfer learning between realistic simulations and the real world.
7 Appendix B: Details of simulation environment
The simulation environment maps jet actions (Q1, Q2) to Navier–Stokes solutions using specified inflow, wall, outflow, and jet boundary conditions. It advances the flow numerically with IPCS and finite elements, then integrates cylinder drag and lift, including jet surfaces.
- Environment response: The environment responds to each action tuple (Q1, Q2) by computing a Navier–Stokes solution in the computational domain Ω.The agent’s actions therefore determine the simulated flow through the PDE solution.
- Boundary conditions: Boundary conditions impose inflow velocity on ΓI, no slip on ΓW, an outflow condition on ΓO, and prescribed jet velocities on Γ1 and Γ2.The outflow condition is −pn + Re−1(n · ∇u) = 0, while the jet condition is u = fQi.
- Jet actuation: The jet profiles fQi are radial velocity profiles that continuously join the cylinder’s no-slip boundary, with modulation determined by angular position and jet geometry.Each jet boundary is controlled by one scalar Qi; negative Qi denotes suction.
- Numerical solver: The Navier–Stokes system is advanced with the Incremental Pressure Correction Scheme, using tentative-velocity, pressure-projection, and velocity-correction steps.The temporal discretization uses step size δt, with p = 0 on ΓO and n · ∇p = 0 elsewhere during pressure projection.
- Spatial discretization: The IPCS equations use continuous quadratic velocity and continuous linear pressure finite elements on triangular cells, with system matrices assembled before the time loop.Only right-hand-side vectors are updated during time stepping.
- Force evaluation: Drag and lift are integrated over the entire cylinder surface after solving for u and p, including the jet surfaces.The reported force quantities therefore include contributions from the jet portions of the cylinder boundary.
8 Appendix C: Deep Reinforcement Learning, Policy Gradient method and PPO
Appendix C summarizes the Policy Gradient and PPO methods used for deep reinforcement learning. The framework trains an ANN policy through environment interaction, gradient-based optimization, continuous-action distributions, replay sampling, and PPO stabilization heuristics.
- Deep Reinforcement Learning framework: The DRL agent interacts with a complex environment through noisy partial observations, ANN-applied actions, and state-dependent rewards.The system’s detailed internal state is usually unavailable.
- Policy Gradient method: Policy Gradient methods directly optimize the action distribution π∗(at|ot) to maximize the long-term discounted reward.The discount factor satisfies 0 < γ < 1.
- Policy Gradient method: Unlike Q-learning, Policy Gradient methods model the policy directly with ANN weights and offer better stability and convergence for continuous control.The learning objective is to find ANN weights Θ that maximize expected return.
- Policy Gradient method: Gradient descent updates ANN weights from experimentally sampled environment trajectories, with adaptive batch methods such as Adagrad or Adadelta stabilizing optimization.Trajectories are represented as sequences of states, actions, and rewards.
- Continuous control: For continuous control, the ANN predicts parameters of a compact-support action distribution from ot; the present implementation uses a Γ distribution.The distribution at each timestep represents probabilities for the optimal action under the current belief.
- PPO and implementation: Replay-memory sampling provides mostly uncorrelated training data, while PPO uses gradient clipping to restrict policy updates and improve stability.TensorFlow supplies ANN and gradient-descent facilities, and Tensorforce implements the DRL algorithm.
9 Appendix D: baseline simulation of a half cylinder without vortex shedding
The appendix constructs a modified no-shedding baseline by simulating a centered half-cylinder configuration that suppresses vortex shedding. Mesh refinement validates this distinct baseline, yielding a virtual full-cylinder steady-state drag coefficient of CDs = 2.93 for comparison with active control.
- Baseline construction: The modified baseline simulates a cylinder placed symmetrically at the domain centerline to assess drag without vortex shedding.This configuration differs from the main article’s setup, which includes a slight asymmetry.
- Baseline construction: The centered configuration kills vortex shedding and produces a modified baseline for drag assessment.The simulation is validated through a mesh refinement analysis distinct from the full-domain analysis.
- Baseline result: CDs = 2.93 is the virtual full-cylinder drag coefficient inferred from the asymptotic half-cylinder drag without vortex shedding.This value provides a comparison point for estimating active flow control’s efficiency in reducing vortex-shedding drag effects.
10 Appendix E: control with partial system information
The appendix tests active-flow-control learning with substantially reduced observations, replacing the main-body configuration of 151 velocity probes with 5 or 11 pressure probes. Both partial-observation networks learn valid strategies, although their drag-reduction results are somewhat worse than with full flow information.
- Observation configurations: The main-body results use 151 velocity probes, while this appendix examines learning from much more partial observation.The appendix focuses on demonstrating partial-observation learning rather than systematically analyzing probe sensitivity.
- Observation configurations: The modified configurations use either 5 pressure probes near the cylinder or 11 pressure probes near the cylinder and near wake.The network size and structure otherwise remain unchanged from the main-body configuration.
- Control results: Reduced input information produces somewhat worse asymptotic drag reduction than full flow information, potentially because far-wake information and spatial sampling resolution are missing.The passage identifies both absent far-wake configuration information and lower spatial resolution as possible causes.
- Control results: Both 5-probe and 11-probe networks learn valid control strategies after training and deterministic evaluation.The control results are presented after a training phase similar to that used in the main body.
- Control results: 3.03 is the reported mean CD for the 5-probe case, compared with 2.99 for the 11-probe case.These values are described as somewhat worse than the results obtained with 151 velocity probes.