Source-linked AI summary
Control of chaotic systems by Deep Reinforcement Learning
Michele Alessandro Bucci, Onofrio Semeraro, Alexandre Allauzen, Guillaume Wisniewski, Laurent Cordier, Lionel Mathelin
TL;DR
Controlling chaotic nonlinear systems is challenging when physical models or full-state measurements are unavailable or impractical. This paper applies model-free deep reinforcement learning to the Kuramoto-Sivashinsky equation, and reports stabilization around unstable fixed solutions across tested initial conditions. The authors identify realistic turbulent-flow configurations as a direction for further application while noting unresolved measurement and state-space challenges.
Problem
Physical models may be unavailable or too slow for control, motivating methods that can work with limited observations in nonlinear systems.
Source: §1 · p. 2Method
The authors use model-free deep reinforcement learning with localized sensors and actuators to control the one-dimensional Kuramoto-Sivashinsky system.
Source: §5 · pp. 17–18Results
The controllers drove and stabilized chaotic KS dynamics around unstable fixed solutions, with performance essentially unchanged across the tested initial conditions.
Source: §5 · p. 18Takeaways & Limitations
The results support investigating RL control in more realistic configurations such as turbulent boundary layers and mixers.
Source: §5 · p. 19Further applications must address limited and noisy observables, low measurement sampling frequency, high-dimensional state-action spaces, and unknown or non-stationary time delays.
Source: §5 · p. 19
Abstract
from arXiv · showhide
Deep Reinforcement Learning (DRL) is applied to control a nonlinear, chaotic system governed by the one-dimensional Kuramoto-Sivashinsky (KS) equation. DRL uses reinforcement learning principles for the determination of optimal control solutions and deep Neural Networks for approximating the value function and the control policy. Recent applications have shown that DRL may achieve superhuman performance in complex cognitive tasks. In this work, we show that using restricted, localized actuations, partial knowledge of the state based on limited sensor measurements, and model-free DRL controllers, it is possible to stabilize the dynamics of the KS system around its unstable fixed solutions, here considered as target states. The robustness of the controllers is tested by considering several trajectories in the phase-space emanating from different initial conditions; we show that the DRL is always capable of driving and stabilizing the dynamics around the target states. The complexity of the KS system, the possibility of defining the DRL control policies by solely relying on the local measurements of the system, and their efficiency in controlling its nonlinear dynamics pave the way for the application of RL methods in control of complex fluid systems such as turbulent boundary layers, turbulent mixers or multiphase flows.
1. Introduction
The introduction motivates adaptive, observation-based control and presents reinforcement learning as a data-driven approach for nonlinear systems. The paper tests deep RL on chaotic Kuramoto-Sivashinsky dynamics and evaluates control quality and robustness.
1. Introduction
- The introduction situates control research amid environmental and energy-efficiency motivations, including potential reductions in emissions and oil consumption.Source: §1 · p. 2
- Control strategies can modify system responses open-loop or as a function of system observations, with the aim of improving a chosen objective.Source: §1 · p. 2
- Model-based control can be unavailable or too slow, while reduced-order models may lose robustness and accuracy when control is applied.Source: §1 · p. 2
- Data-driven control can avoid physical models and use partial observations, but existing approaches may offer limited performance or require extensive trial-and-error training.Source: §1 · p. 2
- The proposed RL approach learns closed-loop nonlinear control from agent-environment interactions, using neural networks to approximate value functions and policies.RL can use limited sensor information rather than a full state vector; value-function estimation may improve training speed and robustness to small perturbations.Source: §1 · pp. 3–4
- The paper evaluates its methodology by control quality and robustness to perturbations in the system and sensor measurements.Source: §1 · p. 4
- The KS equation provides a chaotic fluid-flow testbed for control methods intended for more complex systems such as turbulent mixers and boundary layers.The model exhibits features including traveling waves and can reach a chaotic regime for some configurations.Source: §1 · p. 4
2. From nonlinear optimal control to Reinforcement Learning
This section connects nonlinear optimal control to reinforcement learning through value functions and dynamic programming. It moves from the continuous HJB formulation to a discrete Bellman framework that supports learning from interaction.
2. From nonlinear optimal control to Reinforcement Learning
- The section develops the connection between optimal control and RL, deriving the HJB equation and its discrete-time Bellman counterpart.Source: §2 · p. 5
2.1. Definition of the control problem
- The control problem maps sensor measurements to actuator commands to optimize an objective, with the plant comprising the system, actuators, and sensors.The controller is framed as an optimal policy mapping measurements to control actions.Source: §2.1 · pp. 5–6
2.2. Hamilton-Jacobi-Bellman equation
- For nonlinear dynamics, dynamic programming seeks a PDE for the optimal value function whose action satisfies the state equation.The HJB equation is sufficient for optimality; over the whole state space, with a continuously differentiable value function, it is necessary and sufficient.Source: §2.2 · pp. 6–7
2.3. Bellman equation
- The discrete Bellman equation relates a value to immediate reward and discounted future value, forming the basis for RL inference from interaction.The framework uses expectations for stochastic transitions, and neural networks can approximate value and policy functions when state and action spaces are large.Source: §2.3 · pp. 7–9
- The Q-function measures long-term reward from a state-action pair followed by a policy.Source: §2.3 · p. 8
3. Reinforcement Learning
RL learns an optimal policy from interactions with the environment and can use localized observations rather than the full state. The paper presents DDPG as a model-free Actor–Critic method for continuous control.
3. Reinforcement Learning
- RL can parameterize its policy and value function from observable measurements, enabling learning with partial state information.For the KS application, the observations are pointwise measurements.Source: §3 · p. 9
3.1. Several classes of RL algorithms
- Actor-only methods optimize a policy, critic-only methods approximate the value function, and Actor–Critic methods use the critic to improve the actor.Actor–Critic uses two neural networks and supports parameterized policies for continuous action spaces.Source: §3.1 · pp. 9–10
3.2. Deep Deterministic Policy Gradient as an actor-critic algorithm for RL
- DDPG adapts deterministic policy gradients to deep networks, updating policy parameters along the action-value gradient rather than globally maximizing it.Its actor and critic are updated using shared discounted-reward expectations; exploration combines two noise processes with different time scales.Source: §3.2 · pp. 10–12
4. Control of the Kuramoto-Sivashinsky model
The paper tests model-free RL control of chaotic Kuramoto–Sivashinsky dynamics, targeting unstable invariant solutions with localized actuators and partial sensor measurements. The policies reach targets across transitions and varied initial conditions.
4.1. Dynamics of the Kuramoto-Sivashinsky equation
- The one-dimensional KS equation provides a spatiotemporally chaotic test bed; at domain length L = 22, dynamics visit E1 and E2 and approach E3 without reaching it.E1, E2, and E3 are unstable fixed points used as control targets.Source: §4 · p. 12, Figure 3 · p. 13, §4.1 · p. 14
4. Control of the Kuramoto-Sivashinsky model
- The KS equation is used as a nonlinear, one-dimensional model for fluid-flow control and exhibits traveling waves and chaotic regimes.The paper describes it as a test bed for the proposed control strategy.Source: §4 · p. 12
4.1.1. Numerical Simulations
- The numerical simulations use periodic boundaries, 64 Fourier collocation points, a third-order semi-implicit Runge–Kutta scheme, and time step 0.05.The spatial periodicity supports Fourier projection and spatial derivatives.Source: §4.1.1 · p. 14
4.2.2. Results
- Across three target transitions, policies reached targets in about 10 time-units; all ten random-start E3 trials converged, with one taking about 40.The transition tests and robustness runs are shown in Figures 4 and 5; the random-start tests use policy πE3.Source: Figure 4 · p. 15, Figure 5 · p. 15, §4.2.2 · p. 17
4.2.1. Implementation of the control policies
- The DDPG actor maps four sensor measurements to four actuator signals, while the critic takes eight inputs and outputs a scalar value.The networks use two hidden layers each; the actor’s maximum output amplitude is 0.5.Source: §4.2.1 · p. 16
4.2. Controlled system
- The controller targets E1, E2, or E3 using four equispaced Gaussian actuators and four staggered sensors measuring local velocity.The actuator centers are at 0, L/4, L/2, and 3L/4; sensors are located halfway between them.Source: §4.2.2 · p. 16, §4.2 · p. 16
5. Conclusions
The paper uses DDPG-based deep reinforcement learning to control chaotic Kuramoto–Sivashinsky dynamics with restricted actuators and local measurements. The controllers stabilize unstable fixed solutions across tested initial conditions, motivating extension to more realistic flow configurations.
5. Conclusions
- DDPG approximates the value function and control policy, with cost-landscape knowledge near trajectories and a non-episode-based off-policy framework.The authors state that this approach brings robustness to perturbations and can potentially require less training data.Source: §5 · p. 17
- Across ten randomly chosen initial conditions, control reached target E3 in about 15 time-units in all but one case, which took about 40.Five trajectories are shown in Figure 6; the remaining trajectories appear in gray in its phase-space panel.Source: Figure 6 · p. 18
- Model-free DRL stabilizes chaotic Kuramoto–Sivashinsky dynamics around unstable fixed solutions using limited, localized actuators and sensors.The tested regime is chaotic, which the authors distinguish from periodic or quasi-periodic dynamics.Source: §5 · pp. 17–18
- The authors suggest extending local-measurement DRL control to turbulent boundary layers and mixers, while noting unresolved challenges in realistic configurations.These include limited and noisy observables, low sampling frequency, high-dimensional state-action spaces, and unknown, non-stationary time-delays.Source: §5 · pp. 18–19
Appendix A. Solving the linear optimal problem: Variational and HJB approaches
Appendix A relates the linear-quadratic regulator problem to variational optimality conditions and the Hamilton-Jacobi-Bellman equation. It derives the Riccati formulation and notes that large systems often require alternative solution approaches.
Appendix A. Solving the linear optimal problem: Variational and HJB approaches
- For a linear, time-invariant state-space model, LQR minimizes a quadratic objective penalizing state energy and control effort.The system matrix comes from spatial discretization; actuator and sensor distributions are represented by B and C.Source: Appendix A · pp. 19–20
- The variational approach minimizes the objective subject to the state equation, yielding coupled direct and adjoint equations and an optimality condition.The direct equation is marched forward in time and the adjoint equation backward; the control is updated using the gradient.Source: Appendix A · p. 20
- The appendix derives the optimal control from the Hamiltonian and expresses the value function using an unknown symmetric positive-definite matrix.Substitution yields a differential Riccati equation; its time-invariant limit is the continuous-time algebraic Riccati equation.Source: Appendix A · pp. 20–21
- The LQR optimal control gain follows from the Riccati solution, and the separation principle connects LQR with linear-quadratic-Gaussian estimation.The time-invariant gain is obtained as T approaches infinity.Source: Appendix A · p. 21
- For systems with more than 10^4 degrees of freedom, the Riccati equation is not directly solvable, so direct-adjoint methods are often used instead.Finite sliding-window optimization leads to model predictive controllers.Source: Appendix A · p. 21