PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust flow control and optimal sensor placement using deep reinforcement learning

Romain Paris, Samir Beneddine, Julien Dandois

arXiv:2006.11005v2physics.flu-dyn

TL;DR

The paper develops deep-reinforcement-learning control for drag reduction in a simulated 2D cylinder flow, addressing nonlinear control and practical sensor requirements. It combines PPO-CMA with sparsity-seeking sensor optimization and evaluates robustness to Reynolds-number variation and measurement noise. The resulting policy reduces drag by 18.4% at Re = 120, remains near-optimal over Re = 100–216, and achieves state-of-the-art performance with five sensors.

  • Problem

    Nonlinear high-dimensional flow control remains difficult for local linear methods, motivating more flexible control approaches.

  • Method

    The paper uses PPO-CMA deep reinforcement learning with pressure observations and introduces sparsity-seeking training to optimize sensor number and placement.

  • Results

    18.4% drag reduction is achieved at Re = 120 with 12 wake-pressure sensors, while comparable performance is obtained across other Reynolds numbers.

  • Takeaways & Limitations

    A five-sensor layout preserves state-of-the-art performance, and a policy trained at Re = 120 remains near-optimal for Re in [100,216].

  • Takeaways & Limitations

    The paper’s control-policy interpretation is limited because no simple action pattern was identified and simpler linear controllers did not reproduce the observed efficiency.

Abstract

from arXiv · show

This paper focuses on a drag-reducing control strategy on a 2D-simulated laminar flow past a cylinder. Deep reinforcement learning algorithms have been implemented to discover efficient control schemes, using two synthetic jets located on the cylinder's poles as actuators and pressure sensors in the wake of the cylinder as feedback observation. The present work focuses on the efficiency and robustness of the identified control strategy and introduces a novel algorithm (S-PPO-CMA) to optimise the sensor layout. An energy-efficient control strategy reducing drag by 18.4% at Reynolds number 120 is obtained. This control policy is shown to be robust both to the Reynolds number in the range [100,216] and to measurement noise, enduring signal to noise ratios as low as 0.2 with negligible impact on performance. Along with a systematic study on sensor number and location, the proposed sparsity-seeking algorithm has achieved a successful optimisation to a reduced 5-sensor layout while keeping state-of-the-art performance. These results highlight the interesting possibilities of reinforcement learning for active flow control and pave the way to efficient, robust and practical implementations of these control techniques in experimental or industrial systems.

1. Introduction

Active flow control has moved beyond passive devices and locally linear methods toward nonlinear deep reinforcement learning, while practical deployment still requires fewer, better-placed sensors.

  • 1. Introduction: Nonlinear high-dimensional flow systems remain difficult for local linear control methods, which are often applied despite limitations.These approaches are best suited to weakly nonlinear systems and commonly rely on reduced-order models or local linear approximations.
  • 1. Introduction: Deep reinforcement learning offers a flexible route to nonlinear active flow control through learned feedback policies.Machine-learning methods use neural-network flexibility and interactions with the controlled environment to address complex control problems.
  • 1. Introduction: Reducing sensor requirements while preserving control performance is essential for transferring DRL flow control to experimental and industrial settings.Fewer sensors reduce hardware requirements, potential failure modes, and computational demands in embedded real-time systems.
  • 1. Introduction: Prior cylinder-flow DRL studies used 151 and 236 probes, motivating further optimization of sensor number and location.The paper positions sensor-layout reduction as a continuation and improvement of existing DRL control work.
  • 1. Introduction: The paper combines a simulated case study, DRL control, a sparsity-seeking sensor-optimization algorithm, and robustness analyses.The work addresses control efficiency, Reynolds-number variation, measurement noise, and optimal sensor layouts.

2. Description of the flow configuration and numerical methods

The study simulates quasi-incompressible laminar flow past a cylinder, controls it with reciprocal pole jets, and observes wake pressure using sensors. The numerical setup defines the uncontrolled vortex-shedding reference and drag-gain metric used for evaluation.

  • 2. Description of the flow configuration and numerical methods: The case study is a 2D cylinder flow with non-dimensional diameter D = 1, computed in a domain spanning 10D.The cylinder is centered at the origin, with the flow aligned along the x axis.
  • 2. Description of the flow configuration and numerical methods: The solver uses compressible Navier–Stokes equations at low Mach number, making density fluctuations negligible and the flow quasi-incompressible.The reference configuration is Re = 120, with Reynolds number varied elsewhere in the study.
  • 2. Description of the flow configuration and numerical methods: Reciprocal blowing and suction through two 6°-wide pole jets inject only cross-stream momentum with zero net mass flux.Each command is held for Δt = 0.25 non-dimensional time units and ramped over 20 iterations to avoid abrupt boundary changes.
  • 2. Description of the flow configuration and numerical methods: Wake sensors record pressure fluctuations at the end of every control step, while drag and lift coefficients are computed from the cylinder force.The pressure measurement is the local non-dimensional static pressure minus the reference inflow static pressure.
  • 2.2. Uncontrolled flow: At Re = 120, the uncontrolled flow has drag coefficient 1.379 with fluctuation amplitude 0.018 and Strouhal number St = 0.18.Across the studied range, the 2D flow remains laminar and does not undergo an additional stability bifurcation.
  • 2.2. Uncontrolled flow: The drag-gain metric measures controlled drag reduction relative to the base-flow reference, with 100% corresponding to complete suppression of vortex shedding.The baseline drag is decomposed into constant base-flow drag and an unsteady correction before defining the gain.

3. Reinforcement learning algorithms

The study uses actor–critic reinforcement learning for drag-minimising flow control, then introduces sparse PPO-CMA to remove redundant sensor inputs while preserving control performance.

  • 3.1. A short description of on-policy reinforcement learning: The agent observes partial flow states, selects bounded actions, and is trained to maximise cumulative reward, with drag minimisation as the control objective.Pressure sensors provide observations, while actions are constrained to the valid interval [−2, 2].
  • 3.2. Standard PPO-CMA: PPO-CMA addresses PPO’s premature exploration-variance collapse by adapting the action covariance matrix without clipping the surrogate objective.The actor outputs both the action mean and covariance used to sample actions.
  • 3.3. Sparse surrogate actor: S-PPO-CMA first trains a conventional PPO-CMA policy, then learns a sparse surrogate actor that selects relevant observations and discards redundant sensor information.The sparse actor is initialised from the reference actor and trained using outputs from the pretrained actor and critic.
  • 3.3. Sparse surrogate actor: The stochastic gated layer controls each observation component continuously, passing the measurement, mixing it with a substitute, or replacing it according to a learned gate.Gate parameters are trainable, and the expected number of active observation components defines the SGL’s L0 complexity.
  • 3.3. Sparse surrogate actor: The sparse actor balances sensor reduction against control performance through a regularisation parameter that penalises correlated, redundant observations.The regularisation term penalises gate parameters associated with observations correlated with other measurements.

4. Results and discussion

The learned controller reduces drag efficiently at Re = 120, remains effective across Reynolds-number and noise variations, and can be paired with a substantially sparser sensor layout. Flow analysis links the controlled state to altered recirculation and frequency content, while the policy’s detailed action mechanism remains difficult to interpret.

  • 4.1. Control performance and efficiency: 18.4% drag reduction is achieved at Re = 120, with a drag gain µCx around 100.6%.The controlled drag stabilises below the baseline after the initial transient.
  • 4.1. Control performance and efficiency: Actuation power peaks at 22% of baseline drag power initially but averages below 0.3% during the stabilised phase.The total energy trade-off becomes beneficial 13 time steps after control begins.
  • 4.2. Analysis of the controlled flow: The policy’s action pattern remains difficult to interpret, and simpler linear controllers did not reproduce the observed control behaviour.This suggests that the observed efficiency may require nonlinear control complexity, although the paper does not establish a definitive mechanism.
  • 4.2. Analysis of the controlled flow: The controlled flow lengthens the instantaneous recirculation bubble by more than 2.5 times, while the dominant stabilised-phase frequencies remain below the natural shedding frequency.The control uses frequencies St1 = 0.11 and St2 = 0.14, avoiding the highest resolvent-gain frequency; the selection remains unexplained.

5. Conclusion

The study develops robust nonlinear flow control and reduces the required sensor layout while preserving strong drag-reduction performance. It also identifies extensions toward more complex cases as future work.

  • 18.4% drag reduction is achieved at Re = 120 using PPO-CMA with 12 wake pressure sensors.Comparable performance is achieved across other Reynolds numbers considered.
  • Training at Re = 120 yields near-optimal drag reductions across Re in [100; 216], matching policies trained separately at those Reynolds numbers.The robustness is partly attributed to the case study’s non-dimensionalisation scheme.
  • S-PPO-CMA reduces the layout to 5 sensors while keeping state-of-the-art performance.The algorithm selects relevant sensors and discards redundant or irrelevant ones.
  • Future work includes extending the approach to larger sensor layouts for more complex cases.Improving the present case could require different mean-flow configurations and lower energy efficiency.

Appendix A. PPO, PPO-CMA and S-PPO-CMA learning algorithms

The appendix describes PPO’s advantage-based policy updates, trajectory collection, and clipped objective, then contrasts PPO-CMA’s use of two surrogate objectives and learned policy standard deviation.

  • PPO: The advantage Aπ(at, st) compares an action’s return with the critic’s expected state value Vπ(st).Generalised Advantage Estimation provides a more stable advantage estimate through λ_GAE.
  • PPO: The PPO objective clips the probability ratio between the updated and old policies using ε to constrain policy updates.The clipping function limits the ratio to [1 − ε, 1 + ε].
  • PPO: PPO collects trajectories under the current policy and estimates advantages with Generalised Advantage Estimation before updating the policy.The algorithm iterates until the policy converges.
  • PPO-CMA: PPO-CMA uses two unclipped surrogate objectives and trains the actor’s policy standard deviation as an additional output.The standard deviation is trained twice per update phase with different losses.

P P OCMA and Lµ

PPO-CMA uses a history buffer and covariance-related stabilization, while S-PPO-CMA adds sparsity training that penalizes correlated observations and adapts sensor weights.

  • PPO-CMA: PPO-CMA estimates its objective from a randomly sampled history buffer containing information from the past H training epochs.The method extends PPO with covariance matrix adaptation.
  • PPO-CMA: The PPO-CMA training loop collects trajectories, estimates advantages with GAE, appends returns and advantages to history, and samples data for updates.This preserves PPO’s trajectory-based optimization while adding the history-buffer procedure.
  • S-PPO-CMA: S-PPO-CMA adds a sparsity loss with a diagonal Tikhonov matrix that penalizes correlations between observations.The correlation threshold δ_corr determines which observation relationships are penalized.
  • S-PPO-CMA: The observation transformation uses a slowly updated average baseline to avoid unequal gradient updates caused by differing observation means.Batch-average corrections can introduce bias for small batches.

Appendix B. Numerical hyper-parameters

The appendix points to Table 3 for the main numerical parameters of the simulated case and learning algorithm.

  • Table 3 presents the additional numerical parameters used for the simulated case and learning algorithm.
Loading 2006.11005v2…