Source-linked AI summary
Balanced Active Subspaces for Optimal Control via Trajectory Optimization
Emmanuel S. Ameh, Samuel E. Otto
TL;DR
High-dimensional nonlinear optimal control is constrained by the difficulty of solving the HJB equation and approximating its value function. BASOC balances state variance and value-function sensitivity using state and costate data from locally optimal trajectories. On the considered nonlinear examples, BASOC reduced coordinates approximate optimal value functions and feedback laws better than the other methods, and for LQR it yields a snapshot-based approximation of initial-condition balancing.
Problem
Method
BASOC uses state and costate samples from locally optimal trajectories to balance state variance against value-function sensitivity in an oblique projection.
Source: §1 · p. 2Results
On the considered nonlinear examples, BASOC reduced coordinates approximate optimal value functions and feedback laws, whereas other methods can yield substantially poorer approximations.
Source: §6 · p. 19Takeaways & Limitations
For LQR problems, BASOC provides a snapshot-based approximation of initial-condition balancing for the optimally controlled closed-loop system.
Source: §6 · p. 19The BASOC Petrov–Galerkin method is intrusive because it requires direct access to the full-order governing equations; non-intrusive operator-inference approaches are left for future work.
Source: §4.1 · p. 8
Abstract
from arXiv · showhide
The curse of dimensionality associated with solving the Hamilton-Jacobi-Bellman equation for the value function remains a key obstacle for real-time optimal feedback control of high-dimensional nonlinear dynamical systems. This is especially severe in aerospace applications such as relaminarizing unsteady turbulent flows and recovering from aerodynamic stall where nonlinear dynamics play a significant role. We introduce Balanced Active Subspaces for Optimal Control (BASOC), a model reduction framework which seeks to find directions in state space along which the value function is most sensitive, and the distribution of states exhibits maximum variance along optimal trajectories. BASOC leverages trajectory optimization and the Pontryagin minimum principle to obtain state and co-state (value function gradient) data along locally optimal trajectories for constructing state and gradient covariance matrices. Balancing these matrices provides an oblique projection for approximating the value function, which can also be used to construct reduced-order models and surrogate feedback control laws. We show that for linear-quadratic optimal control problems, BASOC becomes equivalent to a snapshot method for initial condition balancing of the optimally controlled closed-loop realization. The framework is demonstrated on a nonlinear three-dimensional system where POD and balanced truncation are known to perform poorly, and on the nonlinear complex Ginzburg-Landau equation to demonstrate feasibility for larger-scale systems.
1. Introduction.
High-dimensional nonlinear optimal control is hindered by the cost of approximating value functions, motivating methods that exploit low-dimensional structure. BASOC uses optimal-trajectory information to balance value-function sensitivity with state variance, while addressing limitations of state-focused and adjoint-focused reduction methods.
1. Introduction.
- The cost of representing, computing, and storing value functions obstructs optimal control and reinforcement learning for high-dimensional nonlinear systems.The introduction highlights turbulent separated flows, aerodynamic stall recovery, and flow relaminarization as motivating aerospace examples.Source: §1 · p. 1
- The paper asks whether value functions for nonlinear shear-dominated flows have low-dimensional active variables and whether those variables can be exploited for control.The motivation draws on rapidly decaying singular values and low-dimensional active-variable approximations in LQR problems [49, 9, 57, 11].Source: §1 · p. 2
- BASOC constructs an oblique projection that jointly captures state directions of high value-function sensitivity and high variance along optimal trajectories.It uses Pontryagin costates as generalized value-function gradients and explains how the projection can support reduced-order models and surrogate feedback laws.Source: §1 · p. 2
- Among the compared methods—POD, BT, POD-Adjoint, and BASOC—only BASOC and BT respect arbitrary linear state transformations, owing to their constructions via balancing.Source: §1 · p. 2
1.1. Related work.
- State-trajectory POD can miss steep value-function gradients, while state ROMs can miss low-variance but dynamically important flow features [21, 31, 15, 52, 47].Adjoint POD based on optimal costates can approximate the value function but may poorly approximate states, limiting its suitability for dynamics ROMs and reduced-order observers.Source: §1.1 · p. 3
2. Preliminaries on Optimal Control.
The preliminaries formulate an infinite-horizon optimal control problem whose value function gives the optimal cost-to-go and, when differentiable, an optimal feedback law. The Pontryagin Minimum Principle supplies costate trajectories linked to value-function gradients, providing BASOC with sensitivity information.
2. Preliminaries on Optimal Control.
- The setup uses continuous-time, infinite-horizon dynamics, admissible controls, a running cost, and an output map later used to design output-based compensators.The value function is the infimum of the cost over admissible controls; when differentiable, it provides an optimal feedback law.Source: §2 · pp. 3–4
- The value function is difficult to approximate, compute, and store for high-dimensional systems, including fluid models with state vectors of 10^6 dimensions or more.This is attributed to the curse of dimensionality and the difficulty of solving the HJB equation in high dimensions.Source: §2 · p. 4
2.1. Pontryagin Minimum Principle.
- The Pontryagin Minimum Principle gives first-order necessary optimality conditions, with the costate enforcing the dynamics and evolving backward from a transversality condition.At differentiable points, the costate equals the value-function gradient; more generally, the relation is expressed through generalized gradients.Source: §2.1 · pp. 4–5
3. Balanced Active Subspaces for Optimal Value Functions.
BASOC builds balanced coordinates for the optimal value function from state and costate samples along locally optimal trajectories. Its projection balances state variance against value-function sensitivity and supports reduced approximations of the value function.
3. Balanced Active Subspaces for Optimal Value Functions.
- When trajectory optimization finds only locally optimal solutions, the resulting samples correspond to state-constrained optimal value functions.The sampling measure may represent states such as separated or turbulent flow that are to be driven toward attached or laminar flow.Source: §3 · p. 5
- The state covariance represents variance along optimal trajectories, while the costate covariance represents value-function sensitivity along those trajectories.The costate is used as the value-function gradient sample in this construction.Source: §3 · p. 5
- Balancing these covariances jointly retains directions with large state variance and strong value-function sensitivity.The balancing transformation diagonalizes the two covariances, and truncation retains the leading balanced coordinates.Source: §3 · p. 6
- The projection yields balanced active variables as a principled basis for reduced-order approximations of the value function.For Gaussian state distributions, the balancing projection minimizes the stated upper bound on mean-square approximation error.Source: §3 · p. 7
- BASOC balances empirical state and costate covariances from sampled locally optimal trajectories to form an oblique projection and reduced variables.Initial states are drawn from a measure, trajectories are computed by forward-backward sweep, and their state and costate snapshots are assembled for the projection.Source: §3 · pp. 5–7
4. Exploiting BASOC Projections and Variables.
BASOC projections can reduce the full-order dynamics to Petrov–Galerkin models and support surrogates in reduced coordinates. In the LQR setting, BASOC coincides with initial-condition balancing of the optimally controlled closed-loop system under the stated conditions.
4.1. Petrov-Galerkin modeling.
- BASOC projections can produce r-dimensional Petrov–Galerkin ROMs, whose construction requires direct access to the full-order governing equations.The projection itself can be computed using less intrusive automatic-differentiation methods for forward-backward sweeping.Source: §4.1 · p. 8
- A rank-r square-root balancing projection is the unique rank-r spectral projector of WcWo associated with its largest r eigenvalues.The uniqueness result is stated for the rank-r truncation of the slim SVD.Source: §4.1 · p. 9
- Under the stated LQR assumptions, the rank-r BASOC and balancing projections coincide when their singular values have a gap at r.This relates BASOC to initial-condition balancing of the optimally controlled closed-loop system.Source: §4.1 · pp. 9–10
- In the LQR case, state and costate covariances along optimal trajectories correspond to the controllability Gramian and observability Gramian used for balancing.The costates are gradients of the value function along optimal trajectories.Source: §4.1 · p. 10
4.2. Surrogates for suboptimal control.
- Value-function and feedback-law surrogates can be fitted in reduced BASOC coordinates, but the authors do not adopt online optimization because it can be costly.They also leave direct selection of balanced variables for the feedback law using second derivatives for future work.Source: §4.2 · pp. 10–11
5. Numerical Examples and Evaluation.
The numerical examples test BASOC projections and their use in feedback control, reduced-order forecasting, and state estimation on a challenging nonlinear toy model and the CGL equation.
5.1. Challenging nonlinear toy problem.
- For the toy model, BASOC coordinates make the value function nearly single-valued along optimal trajectories, unlike POD and BT; POD Adjoint is comparable to BASOC.The rapidly decaying x3 influences x1 and x2 through quadratic interactions, while POD overlooks x3 and BT considers only linear dynamics.Source: §5.1 · pp. 11–12, Figure 1 · p. 12
5.2. Complex Ginzburg-Landau Equation.
- For the 440-state CGL model, BASOC singular values decay rapidly, and BASOC, BT, and POD Adjoint separate high- and low-value trajectories in three coordinates, unlike POD.The BT projection uses an unstable-system balancing method because the equilibrium is unstable.Source: §5.2 · pp. 13–14, Figure 3 · p. 13, Figure 4 · p. 14
5.3. Full-state feedback control.
- On CGL, BASOC feedback surrogates beat full-state fits by about an order of magnitude; on the toy model, rank-2 BASOC and POD Adjoint fits also outperform full-state fits.BT and state POD perform poorly on the toy problem.Source: Figure 6 · p. 15, Figure 5 · p. 15, §5.3 · p. 16
5.4. Open-loop ROM-based forecasting.
- Toy-model BASOC and POD Adjoint ROMs predict outputs with orders-of-magnitude lower normalized error than BT and POD; for CGL, BASOC is strongest at the highest tested dimension.Because the prescribed input drives CGL toward an unstable equilibrium, forecast accuracy is assessed by duration of agreement rather than asymptotic stability.Source: §5.4 · pp. 16–17, Figure 7 · p. 16, Figure 8 · p. 17
5.5. Open-loop state estimation with EKF-ROM.
- In toy-model EKFs, BASOC, POD, and POD Adjoint estimate similarly, with BASOC slightly lower errors and BT higher errors; on CGL, BASOC leads early while BT leads later.State POD and BT improve substantially within the EKF compared with open-loop forecasting, likely due to output-based corrections.Source: §5.5 · pp. 17–18, Figure 9 · p. 18
5.6. Output-based closed-loop feedback control via EKF-ROM.
- For output-based closed-loop control, BASOC has the lowest mean relative cost error on the toy model; on CGL it is best or near-best, with BT comparable overall.BT and state POD perform poorly on the toy model; the POD-based and POD Adjoint CGL compensators also perform poorly, likely reflecting their feedback-surrogate and estimator performance, respectively.Source: Figure 10 · p. 18, Figure 11 · p. 19, §5.6 · p. 19
6. Conclusion.
BASOC jointly balances value-function sensitivity and state variance, and is equivalent in LQR problems to snapshot-based initial-condition balancing of the optimally controlled closed-loop system. On the nonlinear toy model and CGL equation, its reduced coordinates approximate value functions and feedback laws, while BASOC-based ROMs improve forecasts over POD and balanced truncation in nonlinear regimes.
6. Conclusion
- BASOC jointly captures state-space directions of value-function sensitivity and high state variance through an oblique projection.The projection supports reduced-order modeling and control.Source: §6 · p. 19
- For linear-quadratic problems, BASOC becomes a snapshot-based approximation of initial-condition balancing for the optimally controlled closed-loop system.Source: §6 · p. 19
- BASOC-based ROMs provide accurate forecasts with improvements over balanced truncation and POD in nonlinear regimes, with the largest advantages on the nonlinear toy model.That toy model has strongly nonlinear dynamics specifically designed to defeat POD.Source: §6 · p. 19
- BASOC reduced coordinates approximate optimal value functions and feedback laws on the nonlinear toy model and CGL equation, while other methods can be substantially poorer on these examples.POD on optimal adjoint snapshots can approximate the value function accurately, but may produce poor higher-dimensional ROMs when states do not lie in the span of the gradients.Source: §6 · p. 19
- Future work will test BASOC on larger nonlinear flow-control problems, use second-order value-function information for feedback-law subspaces, and seek performance guarantees.Source: §6 · p. 20
Appendix A. Forward–Backward Sweep Algorithm.
The forward–backward sweep method optimizes a control trajectory by alternating forward state simulation with a backward adjoint solve, then returns optimal state, control, adjoint, cost, and value-function data. Quadrature supports the cost and gradient calculations, and the reported optimization time differs substantially between the toy and CGL models.
Appendix A. Forward–Backward Sweep Algorithm
- The sweep alternates a forward state rollout with a backward adjoint simulation, then passes the cost and control gradient to an optimizer for an update.The iteration continues until the control-cost gradient norm meets the stopping tolerance.Source: Appendix A · p. 20
- After convergence, the algorithm computes the optimal state, control, adjoint, cost, and value function along the trajectory.Source: Appendix A · p. 20
- The cost gradient is obtained in closed form using an adjoint formulation, while Gauss–Radau quadrature approximates continuous integrals.The reported optimization took about 1 minute per toy-model trajectory and about 15 minutes per CGL trajectory; Figure 12 plots cost and gradient-norm convergence.Source: Appendix A · pp. 20–21, Figure 12 · p. 21
Appendix B. State Estimation and Output-Based Closed-Loop Control.
The appendix uses the same extended Kalman filter reduced-order model (EKF-ROM) estimator for open-loop state estimation and output-based closed-loop control. At each fixed time step it propagates the full and reduced systems, updates the reduced-state estimate from measurements, and evaluates the surrogate control law at that estimate.
Appendix B. State Estimation and Output-Based Closed-Loop Control
- The same EKF-ROM estimator is used for both open-loop state estimation and output-based closed-loop control.Source: Appendix B · p. 21
- The discrete-time EKF uses a fixed step of Δt = 0.01 and propagates reduced states with the ROM flow map, approximated using SciPy’s RK45 integrator.The flow-map Jacobian is approximated to first order using a forward Euler discretization and a sufficiently small time step.Source: Appendix B · pp. 21–22
- The filter propagates the full-order system and reduced estimate, uses measurements to update the estimate and covariance, then applies the surrogate control law to the updated estimate.Process and measurement noises are assumed uncorrelated, zero-mean Gaussian, with covariance matrices Q̂ and R̂.Source: Appendix B · p. 22