PaperlayerView exact revision on arXiv

Source-linked AI summary

A Two-Stage, Model-Based Reinforcement Learning Approach for Active Flow Control of Bluff Body Wakes

Aayushman Sharma, Suman Chakravorty

arXiv:2609.08436v1eess.SYcs.SY

TL;DR

The paper tackles how to stabilize an unknown, unstable equilibrium using partial observations and no explicit model. It combines information-state-based nonlinear transfer with local ARMA-LQR regulation, achieving complete lift suppression and a 44% total-drag reduction in a cylinder-wake simulation at Re = 100.

  • Problem

    The desired steady wake is an unknown, unstable equilibrium that is not visited in open loop, making stabilization from partial observations without an explicit model a challenge.

    Source: §I · p. 1, A. The Unknown Target Equilibrium · p. 3

  • Method

    An information-state iLQR transfer drives the flow toward the unknown equilibrium neighborhood, then a locally identified ARMA model supplies an infinite-horizon LQR regulator.

    Source: §I · p. 2, D. Discussion · p. 7

  • Results

    44% total-drag reduction, from uncontrolled C_D = 1.417 to C_D = 0.799, accompanies complete lift suppression using eight surface pressure sensors at Re = 100.

    Source: §VI · pp. 7–8

  • Takeaways & Limitations

    The two-stage controller combines global nonlinear transfer and asymptotic stabilization without an adjoint solver, reduced-order model, or full-state access.

    Source: §VI · p. 7

Abstract

from arXiv · show

This paper develops a data-driven, output-feedback approach to the infinite-horizon optimal control of high-dimensional nonlinear systems with unknown and unstable equilibria, using sparse partial observations. The approach builds on the transfer-plus-regulation decomposition of the infinite-horizon problem: a finite-horizon nonlinear transfer drives the system into a region where the dynamics are well-approximated by a linear model about the unknown operating point, and an infinite-horizon linear regulator identified within that region completes stabilization. We extend this framework to the partially observed setting by combining an ARMA-based information-state construction with a two-stage control architecture: an iterative linear quadratic regulator (iLQR) approach on the information state drives the system to the equilibrium neighborhood, discovered implicitly without prior knowledge of the target, and a locally identified time-invariant ARMA model provides the infinite-horizon regulator for asymptotic stabilization. The method requires no adjoint solver, reduced-order model, or full-state access. We validate the approach on high-fidelity Navier-Stokes simulations of the cylinder wake at $\mathrm{Re}=100$ using only eight surface pressure sensors, an order of magnitude fewer than recent model-based RL methods. The controller achieves complete suppression of vortex-shedding-induced lift oscillations and a $44\%$ reduction in total drag relative to the uncontrolled baseline.

I. INTRODUCTION

The paper addresses stabilization of an unknown, unstable wake equilibrium using sparse observations, without a model. It presents a data-driven controller and demonstrates suppression of wake-induced lift oscillations and drag reduction at Re = 100.

A. Cylinder Wake Flow Dynamics

  • Above Re ≈ 47, the steady wake loses stability and gives way to periodic vortex shedding that increases drag and induces oscillatory lift.The cylinder wake is used as a benchmark for closed-loop suppression of unsteady wake dynamics.Source: §I · p. 1, A. Cylinder Wake Flow Dynamics · p. 2

I. INTRODUCTION

  • The control problem is to stabilize an unknown, unstable steady wake using partial observations and without access to a model.The target is not visited in open-loop operation, so a known-setpoint formulation does not apply.Source: §I · p. 1
  • Existing approaches either require adjoint or full-state data, are sample-inefficient, or use dense sensors and provide no closed-form feedback guarantees.The cited model-based RL approaches use up to 151 probes.Source: §I · p. 1
  • At Re = 100, the controller completely suppresses lift oscillations and reduces total drag by 44% relative to the uncontrolled baseline using eight surface pressure sensors.The evaluation uses a high-order PyFR simulation.Source: §I · p. 2

B. Partial Observation Model

The controller uses sparse surface-pressure measurements rather than the inaccessible full flow field. Its objective is to stabilize the unknown operating point while minimizing drag through a transfer stage followed by local regulation.

B. Partial Observation Model

  • Eight equally spaced surface-pressure sensors provide the measurements, which are mapped to two-dimensional drag and lift force-surrogate observations for feedback.The full velocity and pressure fields are not accessible in the practical setting considered.Source: B. Partial Observation Model · p. 2
  • The flow objective is an infinite-horizon output-feedback policy that stabilizes the unknown operating point while minimizing drag.The paper decomposes the objective into finite-horizon nonlinear transfer and infinite-horizon linear regulation.Source: B. Partial Observation Model · pp. 2–3
  • The computational configuration uses a spectral-element mesh and antisymmetric actuation at ±70° around the cylinder.Fig. 2 shows the full domain and a near-cylinder view of the mesh, sensors, and actuator profiles.Source: Figure 2 · p. 3

III. CHALLENGES IN DATA-DRIVEN FLOW CONTROL

The target wake is unstable and unknown, while sparse pressure sensing limits direct state-based control. The paper uses force surrogates and an information state to pair nonlinear transfer with local regulation.

A. The Unknown Target Equilibrium

  • The desired steady symmetric wake is an unstable fixed point that is never visited in open-loop operation, leaving its high-dimensional state location inaccessible without continuation methods.A standard tracking formulation requiring the explicit target state is therefore infeasible.Source: A. The Unknown Target Equilibrium · p. 3

B. Data-Driven Force Estimation and the Two-Stage Strategy

  • Pressure-based drag and lift surrogates provide feedback coordinates because exact force coefficients require full surface pressure and shear-stress distributions.The target is expressed through zero lift and reduced drag, without requiring the underlying target state.Source: B. Data-Driven Force Estimation and… · p. 3, A. The Unknown Target Equilibrium · p. 3
  • Stacking a short history of outputs and inputs yields an approximately Markov information state for local input-output model identification and Riccati-based recursions.The construction is stated to hold under mild observability conditions.Source: B. Data-Driven Force Estimation and… · p. 3
  • Stage I uses information-state iLQR to transfer the shedding flow into the unknown equilibrium neighborhood, and Stage II uses a locally identified time-invariant ARMA model for infinite-horizon LQR stabilization.The extension addresses an unknown target, sparse outputs, and data-based identification of both transfer and local model.Source: B. Data-Driven Force Estimation and… · pp. 3–4

IV. INFORMATION-STATE BASED DATA-DRIVEN FLOW CONTROL APPROACH

Sparse surface pressures are converted into force surrogates, while short output-and-control histories form an information state for finite-horizon iLQR transfer toward the unknown equilibrium neighborhood.

A. Force Coefficient Estimation from Sparse Pressure Sensors

  • Eight-sensor pressure measurements are quadrature-based surrogates for lift and drag that omit skin-friction drag, which is dominated by pressure drag at Re = 100.The observation used for control is formed from these two force-coefficient estimates.Source: A. Force Coefficient Estimation from… · p. 4

B. Information-State Formulation

  • An ARMA model expresses output deviations using finite histories of past outputs and controls, which are stacked into an augmented information state for control.The model uses short histories (q = 3); its time-varying matrices are identified from rollout perturbations using SVD-truncated least squares.Source: B. Information-State Formulation · p. 4
  • POD-iLQR alternates forward simulation, ARMA linear-system identification, backward dynamic programming, and control updates over a 50-step horizon.The horizon is 25 s at a control timestep of 0.5 s.Source: B. Information-State Formulation · p. 4
  • At the end of Stage I, the flow reaches the unknown equilibrium neighborhood, with terminal drag serving as an empirical indicator rather than the final regulated drag level.The nonlinear optimizer discovers this terminal drag implicitly, without prior knowledge of the equilibrium state.Source: Transfer Outcome · p. 5

C. Stage II: ARMA-LQR Local Stabilization

Stage II identifies a local time-invariant ARMA model near the Stage I terminal operating point and uses an LQR regulator to complete stabilization.

C. Stage II: ARMA-LQR Local Stabilization

  • A time-invariant companion model is fitted by SVD-truncated least squares across transitions from short, perturbed rollouts initialized at the Stage I terminal snapshot.The local model supplies the regulation stage without requiring knowledge of the full equilibrium state.Source: LTI System Identification · p. 5, LQR Design · p. 5
  • The identified local model supports an infinite-horizon LQR regulator whose output-feedback law drives the information state and force coefficients toward the zero setpoint.The gain is computed by solving the Discrete Algebraic Riccati Equation; the weights give greater emphasis to lift and relax the control penalty.Source: LQR Design · p. 5
  • The regulation reference is fixed at zero, and the local model is identified from rollouts within the equilibrium neighborhood reached by Stage I.In force-coefficient coordinates, the reference is (0, 0).Source: LQR Design · p. 5, LTI System Identification · p. 5

A. Experimental Setup

The experiments use high-fidelity PyFR simulations of a two-dimensional cylinder wake at Re = 100, with antisymmetric body-force actuation and eight surface-pressure sensors.

A. Experimental Setup

  • The simulated flow has state dimension 106,338, with control updates every 0.5 s and five solver steps per control interval.The mesh has 3,427 elements and uses solution polynomial order p = 3.Source: A. Experimental Setup · p. 6
  • A single scalar sets the antisymmetric amplitudes of two cross-stream body-force actuators, while eight equally spaced surface-pressure sensors provide observations.The sensors are used to compute pressure-based force-coefficient approximations.Source: A. Experimental Setup · p. 6
  • All experiments start from a fully developed vortex-shedding state obtained by running the uncontrolled simulation to t = 220 s.The POD-iLQR and ARMA-LQR parameter sets are reported in Tables I and II.Source: A. Experimental Setup · p. 6

B. Uncontrolled Flow Baseline

At Re = 100, the uncontrolled cylinder wake settles onto a periodic vortex-shedding limit cycle that provides the force-coefficient baseline.

B. Uncontrolled Flow Baseline

  • The wake settles onto a globally stable limit cycle characterized by periodic Bénard-von Kármán vortex shedding.The shedding frequency is approximately 0.164 Hz.Source: B. Uncontrolled Flow Baseline · p. 6
  • For uncontrolled flow at Re = 100, the pressure-approximated time-averaged drag coefficient is ≈1.05, peak lift amplitude is ≈0.31, and Strouhal number is ≈0.164, forming the drag-reduction baseline.Source: B. Uncontrolled Flow Baseline · p. 6, Figure 3 · p. 6
  • Because the flow naturally returns to the vicinity of the shedding initial condition when control is inactive, explicit state resets between iLQR iterations are unnecessary.The approach can in principle run online with one perturbed-flow rollout and periodic resets to the vortex-shedding flow.Source: B. Uncontrolled Flow Baseline · p. 6

C. Optimal Controls

The two-stage controller first transfers the shedding flow toward the unknown equilibrium neighborhood, then regulates it to a symmetric steady wake. Together, the stages suppress lift oscillations and reduce total drag by 44%.

C. Optimal Controls

  • Over 50 Stage I steps, POD-iLQR suppresses peak lift amplitude from about 0.31 to near zero while drag decreases from its shedding-state mean.The true drag coefficient falls from about 1.42 during the transfer.Source: Stage I: POD-iLQR Transfer · p. 6
  • Across seven iLQR iterations, episodic cost decreases monotonically, while Stage I uses sign-alternating inputs and Stage II settles to persistent forcing.Stage I spans N = 50 steps.Source: Figure 4 · p. 7
  • Stage II ARMA-LQR drives drag and lift to steady values within about 50 further steps, maintaining the unstable equilibrium with a near-constant forcing output.The output is approximately −0.63 U∞²D.Source: Stage II: ARMA-LQR Stabilization · p. 7
  • 44% lower total drag (C_D: 1.417 to 0.799) accompanies complete vortex-street suppression under the two-stage controller.The drag value is from full-surface pressure and viscous integration; the steady wake is symmetric.Source: Stage II: ARMA-LQR Stabilization · p. 7

D. Discussion

The results support the two-stage decomposition: nonlinear transfer and local regulation are each needed for full stabilization. The reported Re = 100 outcome uses sparse pressure sensing, while pressure-based drag feedback omits viscous drag.

D. Discussion

  • Complete vortex-shedding suppression and a 44% total-drag reduction are achieved using eight pressure sensors without an adjoint solver, reduced-order model, or full-state access.The reported total drag is based on full-surface pressure and viscous integration.Source: D. Discussion · p. 7
  • Reported D changes from 1.05 to 0.4; the gap between C*D and bC*D reflects omitted viscous contribution, while at Re = 100 pressure drag dominates circular-cylinder total drag, yielding a reliable feedback signal.Source: D. Discussion · p. 7
  • The stages are complementary: ARMA-LQR alone yields only about 27% drag reduction with residual shedding, while removing regulation lets the flow return to shedding.The finite-horizon transfer reaches the equilibrium neighborhood but does not guarantee regulation.Source: D. Discussion · p. 7
  • The regulator targets zero lift as the equilibrium signature and treats drag as a soft performance term, so drag reduction is an outcome rather than a hard setpoint.Lift receives substantially greater weight than drag, especially in Stage II.Source: D. Discussion · p. 7
  • The main text reports Re = 100; additional Re = 150 validation is omitted for space, while repository results report the same qualitative behavior and 49.2% total-drag reduction.The repository includes scripts, parameters, force histories, and flow visualizations.Source: D. Discussion · p. 7

VI. CONCLUSIONS

The paper concludes that its two-stage output-feedback framework suppresses lift oscillations and reduces total drag using sparse pressure measurements. It identifies online model refinement and higher-Reynolds-number testing as future directions.

VI. CONCLUSIONS

  • At Re = 100, the framework completely suppresses lift and reduces total drag 44% (1.417 to 0.799) using eight pressure sensors without full-state access.The architecture also avoids an adjoint solver and reduced-order model.Source: §VI · pp. 7–8
  • Stage I navigates the nonlinear landscape that a local regulator cannot traverse, while Stage II supplies the asymptotic stabilization a finite-horizon optimizer cannot guarantee..Source: §VI · p. 7
  • Future work proposes continuously refining ARMA models across physical rollouts and scaling the approach to higher Reynolds numbers.The paper connects these directions to online learning and richer dynamics.Source: §VI · p. 7
  • The pressure-surrogate drag near 0.40 differs from the full-surface value 0.799 because the surrogate omits viscous contribution.Source: §VI · p. 8
Loading 2609.08436v1…