Source-linked AI summary
Accelerating Deep Reinforcement Learning strategies of Flow Control through a multi-environment approach
Jean Rabault, Alexander Kuhnle
TL;DR
The paper addresses the need to accelerate DRL training for more challenging AFC problems, where CFD dominates computation time. It parallelizes experience collection across independent simulations and finds effectively perfect scaling through 20 environments, with slightly reduced learning quality beyond the update period. These results support faster DRL studies of more sophisticated fluid-mechanics problems.
Problem
Applying DRL to more challenging AFC configurations requires speedups because CFD, rather than ANN or DRL training, dominates execution time.
Method
The paper collects DRL experiences simultaneously from several independent CFD environments using a straightforward multi-environment implementation.
Results
Perfect scaling is obtained theoretically and in practice up to 20 environments, while larger environment counts produce steps and slightly reduced learning quality.
Takeaways & Limitations
Parallel training supports applying DRL to more sophisticated and realistic fluid-mechanics problems and conducting parametric studies in reasonable time.
Takeaways & Limitations
Parallelizing the simple 2D CFD simulation itself produced limited speedups, typically below a factor of 2, because communication outweighed the small per-core problem size.
Abstract
from arXiv · showhide
Deep Reinforcement Learning (DRL) has recently been proposed as a methodology to discover complex Active Flow Control (AFC) strategies [Rabault, J., Kuchta, M., Jensen, A., Reglade, U., & Cerardi, N. (2019): "Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control", Journal of Fluid Mechanics, 865, 281-302]. However, while promising results were obtained on a simple 2D benchmark flow at a moderate Reynolds number, considerable speedups will be required to investigate more challenging flow configurations. In the case of DRL trained with Computational Fluid Dynamics (CFD) data, it was found that the CFD part, rather than training the Artificial Neural Network, was the limiting factor for speed of execution. Therefore, speedups should be obtained through a combination of two approaches. The first one, which is well documented in the literature, is to parallelize the numerical simulation itself. The second one is to adapt the DRL algorithm for parallelization. Here, a simple strategy is to use several independent simulations running in parallel to collect experiences faster. In the present work, we discuss this solution for parallelization. We illustrate that perfect speedups can be obtained up to the batch size of the DRL agent, and slightly suboptimal scaling still takes place for an even larger number of simulations. This is, therefore, an important step towards enabling the study of more sophisticated Fluid Mechanics problems through DRL.
1 Introduction
Active Flow Control is important but difficult because nonlinear, time-dependent, high-dimensional fluid dynamics make efficient strategies hard to find. Data-driven methods, especially DRL, are promising candidates, and this work extends DRL through parallel experience collection.
- Active Flow Control supports applications including vehicle and aircraft drag reduction and engine-combustion optimization.
- AFC strategy design is difficult because Navier–Stokes dynamics combine non-linearity, time-dependence, and high dimensionality.
- Prior AFC research spans reduced-order, harmonic-forcing, direct-adjoint, linearized-model, mode-tracking, realistic-actuation, and open-loop approaches.
- Industrial AFC remains challenging because robust, efficient algorithms must leverage available sensors and actuators despite disturbances and device imperfections.
- DRL is promising for nonlinear, high-dimensional AFC and can scale in parallel to large amounts of data or computational power.
- The present work extends earlier DRL-based AFC research by illustrating parallelization and providing a flexible implementation.
2 Methodology
The methodology combines a cylinder-flow CFD environment, a PPO-based DRL controller, and parallel collection of experiences from independent simulations. It uses physically motivated action timing and smoothing, while targeting the CFD bottleneck with a straightforward multi-environment strategy.
- 2 Methodology: The new contribution is parallelizing DRL data collection, while the CFD simulation and core DRL methodology follow earlier work.
- 2.1 Simulation environment: The CFD environment is a nondimensional 2D cylinder flow at Re = 100 with two 10° side jets controlled by an ANN.
- 2.1 Simulation environment: The control reward combines drag reduction with lift penalization to discourage asymmetric wakes and consistently biased blowing.
- 2.2 Artificial Neural Network and Deep Reinforcement Learning algorithm: The controller uses 151 pressure probes as inputs and jet mass flow rates as outputs in a fully connected ReLU network trained with PPO.
- 2.2 Artificial Neural Network and Deep Reinforcement Learning algorithm: Actions update roughly 10 times per vortex-shedding period, while CFD control is smoothed at each numerical timestep to maintain continuity.
- 2.3 Parallelization of the data collection for the DRL algorithm: Training speed is limited by CFD, which consumes about 99.7% of computation time; independent environments therefore collect training data in parallel.
- 2.3 Parallelization of the data collection for the DRL algorithm: A straightforward multi-environment approach is preferred for expensive fluid simulations and moderately short learning processes over massive-scale distributed DRL methods.
3 Results and discussion
Using environments that divide the 20-episode update period preserves serial learning while producing perfect speedup. Over-parallelizing beyond that period remains effective but introduces stepped learning curves and slightly reduced learning quality.
- With 1, 2, 5, 10, or 20 synced environments, learning curves collapse together and performance remains strictly identical to serial training.These environment counts divide TL = 20 episodes.
- The divider configurations achieve perfect speedup, with the factor equal to the number of environments.
- Using 32 or 60 environments over-parallelizes data collection beyond TL by up to a factor of 3, while retaining satisfactory speedup but slightly reducing learning quality.Clear steps appear in the learning curves because synchronization is not enforced.
- Near-simultaneous episode completions trigger several ANN updates in quick succession, producing larger cumulative policy updates and stepped learning curves.For 60 environments, three network updates can occur in short succession.
- Although later updates may use data collected under an older policy, the results indicate that these slightly off-policy updates do not harm learning consistency or stability.The authors attribute this empirical robustness partly to PPO’s iterative approximate optimization and multiple small updates.
4 Conclusion
The multi-environment implementation parallelizes experience collection for DRL-based AFC, achieving perfect scaling through 20 environments and speedups around 60 with larger batches. These results support applying the approach to more sophisticated fluid-mechanics problems and combining it with simulation parallelization.
- For environment counts dividing the ANN update period, parallel training is effectively equivalent to serial training and scales perfectly up to 20 times.
- With more environments than the update period, off-policy updates and timing effects create steps in learning curves, but PPO remains robust and speedups reach around 60.
- The authors expect a trade-off between extreme over-parallelization and increasingly deteriorating results relative to the same effective number of serial training steps.
- The parallelization is presented as a milestone toward applying DRL/PPO to more sophisticated and realistic Fluid Mechanics problems.
- Combining DRL data-collection parallelism with simulation parallelization could allow CFD-based AFC training to scale to thousands of CPUs.
7 Appendix B: learning with and without lift penalization
Without lift penalization, DRL can increase reward through biased blowing that reduces drag while producing large mean lift. Adding lift penalization suppresses this cheating strategy and promotes vortex-shedding control.
- Without lift penalization, DRL discovers biased jet blowing that reduces drag but produces a large lift.The jets consistently blow in one direction at maximum strength after a given time.
- With lift penalization, the jets control vortex shedding satisfactorily rather than exploiting biased blowing.The penalization assigns negative reward to large lift biases.
- Without penalization, the wake becomes asymmetric and the velocity excess above the cylinder creates a large mean lift.
8 Appendix C: effect of action frequency and control smoothing
Learning depends on matching the action-update period to the flow dynamics and interpolating smoothly between updates. Intermediate update frequencies enable learning, while reasonable interpolation choices produce stable results.
- 8.1 Effect of the action update frequency: The action-update period Ta must lie between the CFD timestep dt and the vortex-shedding period TK.The required ordering is TK > Ta > dt.
- 8.1 Effect of the action update frequency: At Ta/TK = 0.5%, exploration is too brief to modify the Karman vortex street, preventing learning.
- 8.1 Effect of the action update frequency: At Ta/TK = 100%, action updates are too slow to control vortex dynamics, whereas a sweet spot occurs around Ta/TK = 10%.Around 10%, updates are fast enough for control but slow enough for exploration to affect the system.
- 8.2 Effect of the smoothing law: Interpolation between action updates must be smooth and continuous to avoid unphysical fluid accelerations and numerical instability.
- 8.2 Effect of the smoothing law: Reasonable exponential-decay parameters support satisfactory learning, while linear interpolation also works but may be slightly less efficient.The paper uses exponential decay throughout unless stated otherwise.