Source-linked AI summary
Adaptive dynamic programming using Lyapunov function constraints
Thomas Göhrt, Pavel Osinenko, Stefan Streif
TL;DR
ADP approximates the infinite-horizon value function, but guaranteeing closed-loop stability with approximation structures is challenging. This paper incorporates a Lyapunov function constraint and uses Hessian-free gradient descent for actor and critic optimization. The authors report convergence to prescribed vicinities and improved cost over a nominal controller in a simulation study.
Problem
ADP approximates the infinite-horizon value function, but guaranteeing closed-loop stability when using approximation structures is a challenge.
Source: §I · pp. 1–2Method
The approach incorporates a Lyapunov function constraint into ADP and uses time-varying-gain gradient descent to avoid inverse-Hessian calculations.
Source: §I · p. 2, §VI · p. 6Results
The authors report convergence to prescribed vicinities of the time-varying optima and cost improvement over a nominal controller in a simulation study.
Source: §VI · p. 6Takeaways & Limitations
The case study compares the proposed critic-based controller with a nominal controller across 2601 initial conditions.
Source: §V · p. 6, §VI · p. 6The demonstrated comparison is a simulation case study using a specified utility function and nominal stabilizing controller.
Source: §V · p. 5, §VI · p. 6
Abstract
from arXiv · showhide
This work is concerned with a stabilizing adaptive dynamic programming (ADP) approach to approximate solution of a given infinite-horizon optimal control problem. Since the latter problem cannot, in general, be solved exactly, a parametrized function approximator for the infinite-horizon cost function is introduced in ADP (so called ``critic''). This critic is used to adapt the parameters of the function approximator. The so called ``actor'' in turn derives the optimal input of the system. It is a notoriously hard problem to guarantee closed-loop stability of ADP due to the use of approximation structures in the control scheme. Since at least stabilizability is always assumed in the analyses of ADP, it is justified to invoke a respective Lyapunov function. The proposed ADP scheme explicitly uses the said Lyapunov function to simultaneously optimize the critic and guarantee closed-loop stability. A Hessian-free optimization routine is utilized for the actor and critic optimization problems. Convergence to prescribed vicinities of the optima is shown. A computational study showed significant performance improvement for the critic-based approach compared a nominal stabilizing controller for a range of initial conditions.
I. INTRODUCTION
The introduction motivates ADP as an online alternative to solving the difficult infinite-horizon control problem directly, while emphasizing the challenge of guaranteeing stability with approximators. It proposes Lyapunov-constrained ADP with Hessian-free optimization; the preliminaries state the assumptions and notation used in the analysis.
I. INTRODUCTION
- The HJB equation is difficult to solve analytically, and backward state-space discretization can be unsuitable for online control, motivating ADP value-function approximations [1],,,.The approximator is the critic; the actor derives the controller by optimizing the HJB in terms of the critic.Source: §I · p. 1
- A central ADP challenge is closed-loop stability, while some existing analyses assume the optimization is analytically solvable at all times,,,.The introduction also notes that critic learning and system-model acquisition can involve exhaustive offline phases.Source: §I · p. 1
- The proposed ADP approach uses a Lyapunov function constraint for closed-loop stability and time-varying-gain gradient descent to avoid inverse-Hessian calculations.The authors report convergence to prescribed vicinities and cost improvement over a nominal controller in a case study.Source: §I · p. 2
II. PRELIMINARIES
- The setup assumes a feedback controller and radially unbounded Lyapunov function, and notes that the global assumption can be relaxed to a local one.The introduction describes the proposed scheme as pure online learning of the critic under a stabilizability constraint.Source: §I · p. 1, §II · p. 2
III. PROBLEM STATEMENT
The problem statement formulates actor and critic optimization problems for the ADP scheme. The actor problem incorporates a Lyapunov-based constraint through a barrier relaxation, while the critic objective uses an integral form of squared Bellman error.
III. PROBLEM STATEMENT
- The actor optimization is relaxed using a barrier function, and its solution is generally only suboptimal for the original infinite-horizon problem.The relaxation uses a positive, decreasing time-varying parameter μ(t).Source: §III · p. 2
- The critic objective uses interval reinforcement, the integral of squared Bellman error over the moving interval [t−T, t].The parameter w is held constant over that interval.Source: §III · pp. 2–3
- The stated task is to find update rules for both the actor optimization problem (10) and the critic optimization problem (11).The section also identifies analysis of the resulting problems and update rules as a goal.Source: §III · p. 3
IV. UPDATE RULES AND ANALYSIS
The section introduces update rules and analyzes the proposed optimization problems, including their convexity, feasibility, stability, and convergence properties.
IV. UPDATE RULES AND ANALYSIS
- The section covers update-rule formulation and analysis of the optimization problems, including convergence of optimization variables to vicinities of optimal parameters.It also considers convexity, closed-loop stability, and feasibility.Source: §IV · p. 3
A. Convexity analysis
The convexity analysis establishes strong convexity for the critic and actor objectives under their respective assumptions. It also notes that the critic problem is unconstrained and therefore has no feasibility issue.
A. Convexity analysis
- Under persistent excitation, the critic cost function is strongly convex with parameter δ1.The critic problem is unconstrained because the dynamics can be included directly in J, so feasibility is not an issue.Source: A. Convexity analysis · p. 3
- Under Assumption 1, the actor cost function is strongly convex with parameter m1.The analysis relies on the affine inequality constraint, strongly convex input cost, and convex barrier term.Source: A. Convexity analysis · p. 3
B. Stability analysis
The stability analysis shows that, under Assumption 1, the state converges to the set where k(x) ≤ γ. The proof uses the Lyapunov derivative and LaSalle’s invariance principle.
B. Stability analysis
- Under Assumption 1, the state converges to G := {x ∈ R^n : k(x) ≤ γ}.The theorem considers a continuously differentiable input along the trajectory.Source: B. Stability analysis · p. 3
- The convergence region G is defined by the condition k(x) ≤ γ.The proof notes that k is positive definite, making G compact.Source: B. Stability analysis · p. 3
- Outside G, the Lyapunov derivative is negative, and LaSalle’s invariance principle is used to establish convergence.The proof places the zero-derivative set within G.Source: B. Stability analysis · p. 3
C. Update rules
The update rules use gradient descent with specially tuned, time-varying gains for actor and critic optimization. This avoids inverse-Hessian calculations while targeting convergence to prescribed vicinities.
C. Update rules
- Gradient descent with specially tuned gains is proposed to avoid computing the inverse Hessian.The paper motivates this choice by the potential expense of inverse-Hessian-based tracking controllers.Source: C. Update rules · pp. 3–4
- The selected gradient-descent routine is presented as sufficient for this study, while other optimization routines are possible.The passage notes alternatives, particularly from regret analysis.Source: C. Update rules · p. 4
- The actor and critic optimization problems each have a defined update rule.The rules are identified as updates for the actor problem (10) and critic problem (11).Source: C. Update rules · p. 4
D. Feasibility analysis
The feasibility analysis shows that the actor input remains feasible along the system trajectory when it starts feasible. The update dynamics move it away from the constraint boundary.
D. Feasibility analysis
- The analysis establishes that the actor input remains feasible along the system trajectory.Feasibility is with respect to optimization problem (10).Source: D. Feasibility analysis · p. 4
- If the actor input starts feasible, it remains feasible for all t ≥ 0 under the stated update rule.The result holds for every initial state under Assumption 1.Source: D. Feasibility analysis · p. 4
- At the constraint boundary, the update is shown to move the actor input away from the infeasible region.The proof examines inputs approaching the boundary z(x,u) = γ.Source: D. Feasibility analysis · p. 4
- The barrier-function gradient tends to infinity at the boundary, supporting the boundary-feasibility argument.The proof also uses that z is linear in u and that the cost is globally defined and twice continuously differentiable.Source: D. Feasibility analysis · p. 4
E. Convergence analysis
With suitable gains, the actor and critic parameters converge to prescribed vicinities of their respective optima. The analysis establishes these convergence results and notes that smaller vicinities require larger gains.
E. Convergence analysis
- With suitably chosen gains, actor and critic parameters converge to prescribed vicinities of their respective optima.The theorem asserts the existence of gains α and β under Assumptions 1 and 2.Source: E. Convergence analysis · p. 4
- When the actor error exceeds ε1, the actor-side Lyapunov function has a negative derivative.This is one of the conditional decrease results used in the convergence analysis.Source: E. Convergence analysis · p. 5
- When the critic error exceeds ε2, the critic-side Lyapunov function has a negative derivative.This is the corresponding conditional decrease result for the critic.Source: E. Convergence analysis · p. 5
- The joint error state converges to the set E, with actor and critic components approaching their respective convergence sets.The proof uses the separate decreases of V1 and V2 and concludes convergence to prescribed vicinities B.Source: E. Convergence analysis · p. 5
- Smaller choices of ε1 and ε2 require larger gains α and β.The constants and functions used to calculate the gains are known, so the gains can be calculated online.Source: E. Convergence analysis · p. 5
V. CASE STUDY AND DISCUSSION
The case study compares the proposed controller with nominal and global-optimal controllers over multiple initial conditions, using performance-mark cost differences and example trajectories.
V. CASE STUDY AND DISCUSSION
- The case study's system is asymptotically stable under u := −x1 − x2, and its utility is r = x⊤x + 0.25u2.Source: §V · p. 5
- The proposed controller using (23) and (24), with gains from Theorem 2, is compared with the nominal controller and the converse-approach optimal controller.The proposed controller's initial condition is v(x(0)).Source: §V · p. 5
- The simulation evaluates 2,601 initial conditions, with x1 and x2 ranging from −5 to 5 in steps of 0.2.Source: §V · p. 6
- Fig. 1 compares performance-mark cost differences across initial conditions: the proposed approach versus the nominal controller on the left and the global optimal controller on the right.The passage describes the comparison but gives no direction or magnitude for the differences.Source: §V · p. 6, Figure 1 · p. 6
- Fig. 2 presents state, input, α, and β trajectories for x(0) = (−1 1)⊤.Source: Figure 2 · p. 6
VI. CONCLUSION
The conclusion presents a Lyapunov-constrained ADP formulation with stability guarantees and a time-varying gradient method that avoids inverse-Hessian calculations. Its convergence analysis establishes convergence to prescribed vicinities of time-varying optima, and simulations report cost improvement over a nominal controller.
VI. CONCLUSION
- The ADP formulation modifies the actor optimization problem using a Lyapunov-function constraint to provide stability guarantees.Source: §VI · p. 6
- A time-varying gradient-descent method avoids calculating the inverse Hessian during optimization.Source: §VI · p. 6
- The analysis covers convexity, feasibility, existence of solutions, and convergence, with convergence to prescribed vicinities of time-varying optima.Source: §VI · p. 6
- Simulations over different initial conditions show cost improvement compared with a nominal controller.Source: §VI · p. 6