Source-linked AI summary
Approximate Optimal Trajectory Tracking for Continuous Time Nonlinear Systems
Rushikesh Kamalapurkar, Huyen Dinh, Shubhendu Bhasin, Warren Dixon
TL;DR
Continuous-time nonlinear optimal tracking remains an open extension of ADP methods developed mainly for regulation. The paper transforms the time-varying tracking problem into a time-invariant one, then uses neural-network approximation and Lyapunov analysis to obtain ultimately bounded tracking and convergence to an approximate optimal policy.
Problem
Extending approximate dynamic programming from optimal regulation to continuous-time nonlinear optimal tracking is a non-trivial open problem because the value function and controller are time-varying.
Method
The paper transforms the tracking problem into a time-invariant optimal control problem and applies an ADP policy-evaluation and policy-improvement architecture.
Results
The developed controller guarantees ultimately bounded tracking of the desired trajectory and ultimately bounded error between the approximate and optimal policies.
Takeaways & Limitations
The method provides an approximate solution to infinite-horizon optimal tracking for control-affine nonlinear systems and can be applied to high-dimensional nonlinear dynamical systems using simple methods.
Abstract
from arXiv · showhide
Approximate dynamic programming has been investigated and used as a method to approximately solve optimal regulation problems. However, the extension of this technique to optimal tracking problems for continuous time nonlinear systems has remained a non-trivial open problem. The control development in this paper guarantees ultimately bounded tracking of a desired trajectory, while also ensuring that the controller converges to an approximate optimal policy.
I. INTRODUCTION
ADP and RL methods have largely addressed infinite-horizon regulation, while continuous-time nonlinear optimal tracking remains difficult because the relevant value function and controller vary with time. The paper addresses this obstacle by transforming the problem to a time-invariant form and proves bounded tracking with approximate optimal-policy convergence.
- Background: ADP commonly uses function approximation to seek approximate solutions of the Bellman or HJB equation in continuous-state or continuous-time settings.The HJB equation is the continuous-time counterpart of the Bellman equation.
- Background: Infinite-horizon regulation with quadratic cost is the most common ADP problem because its value function is time-invariant.Time invariance makes function approximation applicable to the value function.
- Problem: Continuous-time tracking approaches use value functions and controllers that are time-varying functions of tracking error, creating an approximation challenge on a noncompact time domain.Neural networks can only approximate functions on compact domains, making the time-varying value function and controller difficult to approximate from tracking error alone.
- Contribution: The paper includes the desired trajectory and tracking error as neural-network inputs, requiring solution of a different HJB equation with a time-varying feedback component.This controller structure is identified as fundamentally different from previous results.
- Contribution: A system transformation converts the time-varying tracking problem into a time-invariant optimal control problem suitable for neural-network approximation.The HJB equation includes the value function’s partial derivative with respect to the desired trajectory.
- Result: Lyapunov analysis proves ultimately bounded tracking and convergence of the controller to an approximate optimal policy.The result directly addresses both tracking performance and approximate optimality.
II. FORMULATION OF TIME-INVARIANT OPTIMAL CONTROL PROBLEM
The paper formulates tracking for control-affine nonlinear systems by defining the tracking error and imposing regularity, boundedness, and trajectory assumptions. A concatenated state transformation then converts the time-varying tracking problem into a time-invariant optimal control problem with admissible policies and a local cost.
- System and objective: The system is control-affine, with locally Lipschitz dynamics, state x, and control input u.The dynamics satisfy f(0)=0 and have unique solutions for bounded initial conditions and controls.
- System and objective: The control objective is to track a bounded continuously differentiable desired signal xd using the error e = x − xd.The open-loop error dynamics are obtained from this definition.
- Assumptions: The formulation assumes bounded full-column-rank g with bounded, locally Lipschitz pseudoinverse g+ and a bounded desired trajectory generated by locally Lipschitz hd.The desired trajectory satisfies ẋd = hd(xd) and hd(0)=0.
- Transformation: A steady-state policy ud corresponding to xd is used in defining the transformed dynamics.The transformation incorporates the desired-trajectory dynamics and steady-state control policy.
- Transformation: The concatenated state ζ ∈ R2n transforms the tracking problem into a time-invariant optimal control problem with dynamics F and G and control μ.The transformed system preserves local Lipschitz properties under the stated assumptions.
- Optimal-control formulation: The transformed optimal policy μ* minimizes a local cost r subject to the transformed dynamics, with positive-definite weighting matrices defining the quadratic cost.The admissible policy set is denoted Ψ.
III. APPROXIMATE OPTIMAL SOLUTION
The approximate solution uses the HJB equation, neural-network representations of the optimal value function and policy, and separate critic and actor weight estimates. Adaptive updates based on Bellman-error minimization, persistent excitation, and Lyapunov-oriented bounds support the subsequent convergence and stability analysis.
- Optimal solution: The optimal value function satisfies the HJB equation, whose zero Hamiltonian condition yields the closed-form optimal policy.The policy is defined under the existence of a minimizing policy and V* ∈ C1 with V*(0)=0.
- Neural-network approximation: On compact χ ⊂ R2n, the optimal value function is represented by a neural network plus reconstruction error.The activation function is bounded and continuously differentiable, and the ideal weights are bounded.
- Neural-network approximation: Approximate value-function and policy representations use separate estimates of the ideal neural-network weights.Separate estimates reflect the Bellman error’s linear dependence on value-function weights and nonlinear dependence on policy weights.
- Controller and Bellman error: The approximate controller is obtained by substituting the approximate value function and policy into the transformed optimal-control relations.The resulting approximate Hamiltonian differs from the optimal Hamiltonian by the Bellman error.
- Adaptive updates: Value-function weights are updated by normalized least squares with exponential forgetting, while policy weights follow the critic weights through adaptive laws.The updates use positive adaptation gains and a forgetting factor λ ∈ (0, 1).
- Analysis: Persistent excitation and bounded-system estimates are used to analyze weight-error dynamics and establish bounds needed for stability analysis.The perturbed value-weight dynamics are compared with a nominal exponentially convergent system.
IV. STABILITY ANALYSIS
The analysis addresses the challenge that the transformed optimal value function is not positive definite by relating time-invariant and time-varying representations. Lemmas and Lyapunov arguments establish boundedness properties used for stability analysis.
- Stability analysis: The main contribution is a transformation enabling the optimal policy and optimal value function to be analyzed for tracking.
- Stability analysis: The transformation expresses the optimal value function as time-invariant in ζ, while its equivalent time-varying form supports tracking-error analysis.The time-varying representation is shown to be positive definite and decrescent in the tracking error.
- Stability analysis: The optimal value function is not positive definite in ζ because Q is positive semidefinite, creating a technical obstacle for Lyapunov analysis.
- Stability analysis: The candidate Lyapunov function combines the optimal value function with actor and critic weight errors to support the subsequent stability theorem.
- Stability analysis: Lemma 1 shows that a positive definite function bounded and continuous uniformly in time is bounded above by a continuous positive definite function and is therefore decrescent.
- Stability analysis: Lemma 2 bounds the optimal value function above and below by class K functions of the tracking-error norm on a closed ball.The bounds hold for all times and errors within the specified ball.
A. Gain conditions and gain selection
The paper selects controller gains under sufficient conditions and uses an iterative procedure to preserve a compact operating set. Theorem 1 then guarantees ultimate boundedness of tracking and policy error.
- A. Gain conditions and gain selection: If the gain-selection condition is not met, increasing NN neurons can reduce approximation error and support revised gain selection.
- A. Gain conditions and gain selection: The sufficient gain conditions depend on initial conditions and on compact sets used for function-approximation and Lipschitz bounds.
- A. Gain conditions and gain selection: Algorithm 1 iteratively selects gains and ensures that the selected compact set Z remains admissible for the analysis.
- A. Gain conditions and gain selection: Theorem 1 states that, under Assumptions 1–3 and sufficient gain conditions, the controller and update laws guarantee ultimately bounded tracking and policy error.
- A. Gain conditions and gain selection: The Lyapunov analysis uses a function combining the optimal value function with actor and critic weight errors, then bounds its derivative on Z.
- A. Gain conditions and gain selection: Over each interval of length T, the Lyapunov function decreases under the stated conditions, including the bound −ηcψϖ7.
- A. Gain conditions and gain selection: The iterative procedure maintains bounded system state, with supt∈[0,∞) ∥Z(t)∥2 ≤ Z.
V. CONCLUSION
The paper presents an ADP policy evaluation and improvement approach for infinite-horizon optimal tracking in control-affine nonlinear systems, transforming the time-varying problem into a time-invariant one. Lyapunov analysis establishes ultimately bounded tracking and estimation, while the approximation remains subject to basis-function choice, persistent excitation, and exact model knowledge.
- The approach approximately solves infinite-horizon optimal tracking for control-affine nonlinear systems with quadratic cost.
- A system transformation converts the tracking problem's time-varying value function into a time-invariant optimal control problem.
- Lyapunov analysis establishes ultimately bounded tracking and estimation for the developed controller.
- The method can address high-dimensional nonlinear dynamical systems using simple polynomial basis functions and sinusoidal probing signals.
- Approximation accuracy depends on basis-function selection, and the result hinges on system states being persistently exciting.
- Computing the desired control requires exact model knowledge, while avoiding that control with a multilayer neural network remains a future challenge.