Source-linked AI summary
Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations
Tenavi Nakamura-Zimmerer, Qi Gong, Wei Kang
TL;DR
High-dimensional HJB equations make optimal feedback control difficult because direct discretization suffers from the curse of dimensionality and existing methods may be restrictive or locally valid. The paper uses grid-free PMP-generated data, physics-informed neural networks, and adaptive sampling to approximate semi-global value functions and compute candidate feedback controls in real time. The framework is demonstrated on a six-dimensional nonlinear rigid body and systems up to 30 dimensions, while larger-scale applicability and constrained problems remain open challenges.
Problem
Direct HJB discretization becomes intractable as state dimension grows, while existing high-dimensional approaches can require restrictive structure or remain local to nominal trajectories.
Method
The method trains neural networks on causality-free PMP data, uses costate information and adaptive sampling, and avoids state-space discretization and expensive PDE residuals.
Results
The framework learns semi-global value functions and candidate feedback controllers for a six-dimensional nonlinear rigid body and systems with dimensions up to 30.
Takeaways & Limitations
The trained neural networks support real-time feedback control over large state domains and can generate richer data near difficult value-function regions.
Takeaways & Limitations
The method’s performance on problems larger than 30 dimensions remains unresolved, and extensions to free-final-time problems and state or control constraints remain future work.
Abstract
from arXiv · showhide
Computing optimal feedback controls for nonlinear systems generally requires solving Hamilton-Jacobi-Bellman (HJB) equations, which are notoriously difficult when the state dimension is large. Existing strategies for high-dimensional problems often rely on specific, restrictive problem structures, or are valid only locally around some nominal trajectory. In this paper, we propose a data-driven method to approximate semi-global solutions to HJB equations for general high-dimensional nonlinear systems and compute candidate optimal feedback controls in real-time. To accomplish this, we model solutions to HJB equations with neural networks (NNs) trained on data generated without discretizing the state space. Training is made more effective and data-efficient by leveraging the known physics of the problem and using the partially-trained NN to aid in adaptive data generation. We demonstrate the effectiveness of our method by learning solutions to HJB equations corresponding to the attitude control of a six-dimensional nonlinear rigid body, and nonlinear systems of dimension up to 30 arising from the stabilization of a Burgers'-type partial differential equation. The trained NNs are then used for real-time feedback control of these systems.
1. Introduction.
High-dimensional HJB control is difficult because grid-based discretization scales exponentially with state dimension. The paper proposes a data-driven, grid-free neural-network framework that uses causality-free data generation, physics-informed training, and adaptive sampling to approximate semi-global solutions and feedback controls.
- The discretized HJB problem grows exponentially with state dimension, making direct solution intractable for moderately large systems.
- Existing approximate HJB methods can be dimension-limited, difficult to verify, locally valid, or restricted by problem structure.
- The method generates initial open-loop optimal trajectories with a causality-free PMP-based algorithm, then trains a neural network on the resulting value-function data.
- Training incorporates value-function gradient information, estimates data needs, and adaptively adds samples where the value function is difficult to learn.
- The framework produced a six-dimensional rigid-body controller and handled Burgers’-type systems with dimensions 10, 20, and 30 without simplifying their dynamics.
- The demonstrated capabilities include semi-global HJB solutions, empirically validated accuracy, progressive data generation, and computationally efficient real-time nonlinear feedback control.
2. A causality-free method for HJB equations.
The paper replaces grid-based HJB solution with neural-network approximation trained from causality-free PMP boundary-value problems. This enables high-dimensional, semi-global feedback control while retaining explicit caveats about BVP optimality and scalability.
- The desired feedback controller must work for arbitrary measured states, unlike an open-loop control valid only for one initial condition.
- The value function is the optimal cost-to-go, and its state gradient determines feedback control through Hamiltonian minimization.
- PMP characteristics convert the HJB problem into two-point boundary-value problems whose solutions provide value, gradient, and control data along trajectories.
- Using solved BVP data to train a neural network is grid-free, avoiding the rapidly growing number of sparse-grid BVP solves required by interpolation.
- The method assumes BVP solutions are optimal, although PMP may also yield sub-optimal solutions and intersecting characteristics can create non-smooth value functions.
- A partially trained neural network closes the loop to generate additional trajectories and warm-start BVP solves, reducing data-generation cost without parameter tuning.
3. Neural network approximation of the value function.
The method trains feedforward neural networks to approximate the HJB value function from trajectory data, while regularizing the state gradient using costate information. Automatic differentiation supports efficient gradient evaluation for real-time feedback computation.
- The workflow samples initial conditions, solves BVPs to generate value data, trains a neural network, and evaluates it on independent data.
- Feedforward neural networks represent the value function as a composition of nonlinear layers, using tanh activations in hidden layers.
- Physics-informed training models the costate alongside the value function instead of computing expensive PDE residuals over collocation points.
- Automatic differentiation computes exact state gradients without finite-difference approximations and makes their evaluation inexpensive after graph compilation.
- Gradient regularization encourages physically consistent value-function gradients, which are essential because the optimal control depends explicitly on the state gradient.
- Validation uses relative mean absolute error for value predictions and relative mean L2 error for gradients, emphasizing regions requiring substantial control effort.
- At runtime, the controller evaluates the neural-network gradient and solves the Hamiltonian control optimization, which can be analytic for control-affine systems with convex running cost.
4. Adaptive sampling and model refinement.
The method progressively enlarges training data while refining a neural-network value-function model, using gradient-based convergence tests to guide sampling. It combines physics-informed training, adaptive sample-size selection, and trajectory-based data generation to build models beyond nominal neighborhoods.
- Adaptive data generation: The method uses costate information and progressive data generation to make training effective when each new data point requires solving an expensive boundary-value problem.This addresses the computational cost of generating large data sets for adequately representing the value function.
- Model refinement: Small data sets are initially used to train the neural network, after which progressively larger sets are generated and training continues until convergence conditions are satisfied.The approach uses L-BFGS for the initial low-fidelity model and enlarges the data set in successive rounds.
- Convergence testing: Validation data can assess model accuracy, but generating enough validation samples may be too expensive and validation tests do not directly guide sample-size selection.The paper therefore distinguishes final accuracy validation from the statistically motivated test used between training rounds.
- Convergence testing: The convergence test compares sample-based gradient statistics with expected-gradient behavior, using root mean square error and an L1 gradient norm to select subsequent sample sizes.The L1 denominator is less sensitive to outliers and makes the test less likely to propose unreasonably large sample sizes.
- Convergence testing: The convergence criterion indicates whether more data is unlikely to improve the optimizer’s expected gradient, but does not by itself establish that the trained model is good.The test concerns first-order optimality over the domain rather than overall model quality.
- Adaptive data generation: Warm-starting boundary-value solves with approximate trajectories supports rich data generation across the full domain rather than only near a nominal trajectory.The resulting data set can contain points from the entire state domain, enabling a high-fidelity model of the value function.
5. Application to rigid body attitude control.
The rigid-body application models the initial-time value function for six-dimensional satellite attitude control and evaluates neural-network accuracy, adaptive training, solver convergence, and closed-loop behavior. Physics-informed learning and adaptive sampling achieve accurate value and gradient predictions with substantially fewer data, while the resulting controller stabilizes the system under measurement noise.
- Problem setup: The six-dimensional satellite state uses Euler-angle attitude variables and body-frame angular velocity, with torque inputs governing the rigid-body dynamics.The model has n = 6 spatial dimensions and m = 3 control inputs.
- Value-function learning: The study approximates the initial-time value function V(0, v, ω) with a neural network and implements the resulting control as a time-independent moving-horizon feedback law.The network takes attitude and angular velocity as inputs, while each control evaluation assumes t = 0.
- Value-function learning: With 512 data points, the neural network achieves better accuracy than the sparse-grid characteristics method using 44,698 points, making the approach about 90 times more data-efficient.With 8192 data points, the neural network is almost four times as accurate as the sparse-grid method; physics-informed learning is required for this performance.
- Adaptive data generation: Adaptive sampling passes its convergence test after seven rounds with 2110 samples, reaching value-function accuracy 3.3 × 10^-4 and gradient prediction accuracy 1.6 × 10^-3.The adaptive model’s gradient predictions are as accurate as those from a fixed 4096-sample training set, while using fewer data.
- Solver convergence: Time-marching with at least K = 4 intervals reliably solves the difficult sampled boundary-value problems, whereas solving over the full interval without an initial guess converges very rarely.The adaptive procedure also relies on hyperparameter choices that may need adjustment between optimization rounds.
6. Application to control of Burgers’-type PDE.
The paper applies its adaptive deep-learning framework to Burgers’-type PDE control after Chebyshev pseudospectral discretization, producing value-function approximations in dimensions up to 30. The learned models accurately predict value functions and gradients, while adaptive NN-assisted data generation supports scalable training and real-time stabilization.
- Problem formulation: The reaction term αXe^βX destabilizes the origin, making stabilization more challenging.The reported parameters include α = 1.5 and β = −0.1.
- Problem formulation: The Burgers’-type PDE is converted into an n-dimensional ODE-constrained optimal-control problem using Chebyshev pseudospectral collocation.The discretization uses internal Chebyshev points, differentiation matrices, and Clenshaw-Curtis quadrature while automatically enforcing boundary conditions.
- Learning setup: The framework approximates time-dependent value functions for n = 10, 20, and 30 over the initial-condition domain −2 ≤ x_j ≤ 2.Time dependence allows the resulting controls to be implemented directly or with a moving horizon.
- Learning setup: Training begins with relatively few BVP-generated trajectories, then expands the data set through additional trajectory generation and adaptive sampling.The initial data are generated from uniformly sampled initial conditions, with subsets used when data sets become large.
- Results: The trained NNs accurately predict both the value function and its gradient, including for the 30-dimensional problem.Validation uses independently generated trajectories from 50 randomly selected initial conditions.
- Results: Training effort scales reasonably with dimension, and low-fidelity NNs can be trained quickly to improve subsequent data generation.The paper notes that generating data becomes the most expensive computation as n increases.
- Results: NN-assisted warm starts reduce the convergence-time burden of time marching as dimension increases, while online control evaluations remain millisecond-scale.For n = 30, the NN controller stabilizes the unstable origin from both tested initial conditions and produces controls close to BVP-derived optimal controls.
7. Conclusion.
The framework combines causality-free, physics-informed learning with adaptive data generation to solve high-dimensional HJB equations and design candidate feedback controllers. Demonstrations reach 30 dimensions, while larger problems and constrained or nonsmooth settings remain open challenges.
- The method avoids finite-difference gradients and strict restrictions on system dynamics while using costate data to train physically consistent models from small data sets.The authors report that this supports better controllers and enables validation of model accuracy.
- Adaptive data generation produces semi-global data sets across dynamic states and concentrates samples near complicated value-function regions or high-control-effort areas.This supports more accurate neural-network models and better-performing controllers.
- The framework designs candidate optimal feedback controllers for a six-dimensional nonlinear rigid body and solves HJB equations in systems up to 30 dimensions using limited data.Empirical validation indicates that the neural networks approximate the value function well.
- Extensions to free-final-time problems and state or control constraints remain challenging because they can produce non-unique PMP solutions and non-smooth value functions.The authors identify these extensions as important future development areas.