PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems

Yu Jiang, Zhong-Ping Jiang

arXiv:1401.0020v4math.DSmath.OC

TL;DR

Optimal control for uncertain nonlinear polynomial systems is difficult because HJB equations are hard to solve and conventional ADP approximations can be computationally burdensome and non-global. The paper replaces HJB solution with SOS-based relaxed policy iteration, then applies it online using real-time measurements. The resulting method provides suboptimal control with global asymptotic stabilization while avoiding neural-network approximation.

  • Problem

    HJB equations are difficult to solve for nonlinear systems, while existing ADP approximations can have high computational complexity and fail to achieve global stabilization.

  • Method

    The paper relaxes the HJB equation into an optimization problem, solves policy iterations with SOS/SDP programs, and implements them online from real-time measurements without perfect dynamics knowledge.

  • Results

    The proposed method finds a suboptimal solution to continuous-time nonlinear optimal control problems and produces a globally stabilizing control policy without neural-network approximation.

  • Takeaways & Limitations

    Global ADP can provide computational benefits for data-driven adaptive optimal control of nonlinear polynomial systems while retaining global asymptotic stabilization.

Abstract

from arXiv · show

This paper presents a novel method of global adaptive dynamic programming (ADP) for the adaptive optimal control of nonlinear polynomial systems. The strategy consists of relaxing the problem of solving the Hamilton-Jacobi-Bellman (HJB) equation to an optimization problem, which is solved via a new policy iteration method. The proposed method distinguishes from previously known nonlinear ADP methods in that the neural network approximation is avoided, giving rise to significant computational improvement. Instead of semiglobally or locally stabilizing, the resultant control policy is globally stabilizing for a general class of nonlinear polynomial systems. Furthermore, in the absence of the a priori knowledge of the system dynamics, an online learning method is devised to implement the proposed policy iteration technique by generalizing the current ADP theory. Finally, three numerical examples are provided to validate the effectiveness of the proposed method.

I. INTRODUCTION

The paper develops global adaptive dynamic programming for uncertain continuous-time nonlinear polynomial systems, replacing difficult HJB solution procedures with parameterizable optimization and online learning. It avoids neural-network approximation while targeting globally stabilizing suboptimal control.

  • ADP addresses optimal-control problems without known system dynamics by learning approximations of the cost function online.
  • Neural-network ADP can require many basis functions, creating computational burden and making basis selection difficult for unknown target functions.
  • Neural-network approximations are generally effective only on compact state-space sets, limiting their use for global stabilization.
  • The paper relaxes solving the HJB equation into an optimization problem whose solutions can be easily parameterized.
  • A relaxed policy iteration method formulates each iteration as a sum-of-squares program equivalent to a semidefinite program, directly seeking a suboptimal original-control solution.
  • Online learning implements the iterative scheme using real-time measurements without perfect system knowledge, avoiding neural networks while achieving global asymptotic stabilization.

C. Conventional policy iteration

Conventional policy iteration alternates policy evaluation and improvement to approach the optimal HJB solution, but analytical evaluation is difficult and approximation methods remain computationally and geographically limited. The relaxed formulation instead provides feasible Lyapunov-certified policies and an upper-bound interpretation of the cost function.

  • C. Conventional policy iteration: Conventional policy iteration evaluates a cost function by solving a partial differential equation, then improves the control policy.
  • C. Conventional policy iteration: Under the stated assumptions, conventional iteration produces nonincreasing cost functions, globally stabilizing policies, and convergence to the optimal solution when the policies converge.
  • C. Conventional policy iteration: Analytical policy-evaluation solutions are non-trivial, while neural-network and Galerkin approximations generally provide acceptable results only on compact state-space sets.
  • III. SUBOPTIMAL CONTROL WITH RELAXED HJB EQUATION: The relaxed optimal-control problem replaces the HJB equality with an inequality over a compact region where performance improvement is emphasized.
  • III. SUBOPTIMAL CONTROL WITH RELAXED HJB EQUATION: A feasible relaxed solution yields a globally stabilizing control policy and an upper bound on the closed-loop cost for every initial state.
  • III. SUBOPTIMAL CONTROL WITH RELAXED HJB EQUATION: The optimal value function is a global optimal solution of the relaxed problem, while any feasible solution can serve as a Lyapunov function.
  • III. SUBOPTIMAL CONTROL WITH RELAXED HJB EQUATION: The relaxed cost-function variable may overestimate the actual cost associated with its induced policy, but it remains easier to parameterize and supports stability analysis.

IV. SOS-BASED POLICY ITERATION FOR POLYNOMIAL SYSTEMS

The paper replaces a difficult polynomial nonnegativity constraint with SOS constraints, reducing computational burden while enabling policy iteration for polynomial systems.

  • SOS constraints reduce the computational burden of enforcing polynomial inequality constraints in the policy-iteration problem.For polynomials of degree at least four, deciding non-negativity is NP-hard; SOS programs provide a tractable restriction.

A. Polynomial parametrization

The method parameterizes polynomial value functions and control policies, then alternates SOS-based policy evaluation and improvement. Under stated assumptions, each iteration is feasible, globally stabilizing, and converges pointwise to a polynomial limit, although initialization is nontrivial.

  • Polynomial parametrization: Polynomial value functions and control policies make the Lie-derivative and policy terms polynomial, enabling SOS-based optimization.The construction restricts Vi to R[x]2,2r and produces polynomial updates ui+1.
  • Initialization: The algorithm requires an initial pair (V0, u1) with V0 positive definite and L(V0, u1) SOS.This initialization is assumed before policy iteration begins.
  • Policy iteration: Each iteration performs SOS-based policy evaluation followed by policy improvement, then repeats the two-step cycle.The policy-evaluation program solves for pi, while policy improvement updates the control policy from the value-function gradient.
  • Theoretical properties: The SOS program is well-defined, has a nonempty feasible set, and produces globally asymptotically stabilizing closed-loop policies under the stated assumptions.The resulting value functions belong to P and satisfy the theorem’s associated inequalities.
  • Convergence: The sequence Vi(x0) converges for every initial state to a polynomial V* in R[x]2,2r ∩ P.The limit is bounded between the optimal value function Vo and the initial value function V0.
  • Limitations: For general nonlinear systems, a suitable initial pair may not exist and must be found before executing the algorithm.Finding such a pair amounts to solving bilinear matrix inequalities.
  • Extensions: The method can extend to polynomial-fraction policies and nonconstant R(x) when α(x) is selected so det(R(x)) divides α(x).Under this divisibility condition, the SOS iteration’s solvability and convergence properties are unchanged.

V. GLOBAL ADAPTIVE DYNAMIC PROGRAMMING FOR UNCERTAIN POLYNOMIAL SYSTEMS

The online-learning extension addresses the need for unavailable system dynamics by implementing policy iteration from real-time data with bounded exploration noise.

  • The online method replaces perfect knowledge of f and g with real-time measurements, avoiding prior system identification.The method calculates the policy-iteration parameters directly from online data.
  • A bounded time-varying exploration noise e is added to the feedback policy for learning.The resulting input is ui + e during online implementation.

A. Forward completeness

Forward completeness guarantees that the learning system’s trajectories remain defined for all nonnegative time despite bounded exploration noise. The online parameters are then recovered from real-time data under a rank condition.

  • Forward completeness: Forward completeness means solutions exist for all t ≥ 0 for every initial condition and input signal.This property is needed because exploration noise is present during learning.
  • Forward completeness: If ui is globally stabilizing and Vi−1 satisfies the required nonpositivity condition, the noisy system is forward complete.The result follows from the Lyapunov structure established for the policy-iteration scheme.
  • Forward completeness: Every policy produced by the proposed iteration yields a closed-loop system that remains forward complete under bounded exploration noise.This proposition holds under Assumptions 2.2 and 4.1.
  • Online parameter recovery: The quantities L(V, ui) and R−1gT∇V are represented parametrically so their coefficients can be inferred without identifying f and g.The online equations are obtained by integrating along measured trajectories.
  • Online parameter recovery: Real-time online data directly determine lp and Kp through the integrated relations used by the learning method.These parameters encode the value-function and control-policy representations.
  • Implementation: The practical implementation iterates the online learning procedure, with the flowchart specifying its execution sequence.The method is presented as an ADP-based online learning algorithm.
  • Identifiability: A rank condition analogous to persistency of excitation is required for unique parameter determination and convergence.The number of collected data intervals must satisfy qi ≥ qi0.

1) Initialization:

The method initializes with a feasible pair (V0, u1), then applies the current policy with exploration noise while collecting data until the data matrix has full column rank.

  • Initialize with a pair (V0, u1) satisfying Assumption 4.1, then set the iteration index to i = 1.
  • Apply the exploratory control u = ui + e and compute Φi, Ξi, and Θi until Φi has full column rank.

3) Policy evaluation and improvement:

Policy evaluation and improvement are formulated as SOS-based optimization iterations using online measurements, with theoretical feasibility and convergence properties. Numerical examples report globally stabilizing suboptimal policies and comparisons with initial or exact policies.

  • 3) Policy evaluation and improvement:: Each policy-improvement step finds an optimal pair (pi, Ki+1) through an SOS program.The objective is to minimize cT p over p and K.
  • 3) Policy evaluation and improvement:: The online iteration updates ui+1 from Ki+1 and repeats the data-collection and optimization steps.
  • 3) Policy evaluation and improvement:: Theorem 5.6 states that the online optimization problem has a nonempty feasible set and that the sequences {Vi} and {ui} inherit properties 2)-5) of Theorem 4.3.
  • 3) Policy evaluation and improvement:: A locally stabilizing learning policy may fail to ensure forward completeness or global attraction, as illustrated by ẋ = θx^3 + u with u1 = −x.
  • 3) Policy evaluation and improvement:: In the first simulation, convergence is attained after five iterations, producing a suboptimal policy and cost function compared with exact and initial policies.The policy is updated every five seconds, with convergence defined by |pi − pi−1| ≤ 10^-3.
  • 3) Policy evaluation and improvement:: For the fault-tolerant system, the method seeks to reduce the cost while preserving global asymptotic stability in Ω, improving on a robust policy whose optimality was not well addressed.The performance comparison is conducted in Ω = {x | x ∈ R^2 and |x| ≤ 1}.
  • 3) Policy evaluation and improvement:: The second simulation applies online learning for seven updates and obtains a suboptimal, globally stabilizing control policy with an associated cost function.

C. An active suspension system

The active-suspension example applies online learning to a quarter-car model with uncertain parameters, seeking improved performance while retaining global asymptotic stability. After convergence, the learned controller is evaluated against the uncontrolled system and the initial cost function.

  • C. An active suspension system: The quarter-car model represents car-body and wheel-assembly positions, velocities, and masses, with stiffness and damping parameters defining the suspension.
  • C. An active suspension system: The uncertain-parameter system is assumed globally asymptotically stable without control, while the proposed method targets lower performance cost and retained global stability.
  • C. An active suspension system: The learning experiment uses bounded exploration noise and reaches convergence after 10 iterations, starting from an initial policy u1 = 0.
  • C. An active suspension system: The resulting controller is described as suboptimal and globally stabilizing for the active suspension system.
  • C. An active suspension system: After an impulse disturbance at t = 120, the learned closed-loop system is compared with the original system without control.
  • C. An active suspension system: Restricting x3 ≡ x4 ≡ 0 for visualization, the learned cost function V10 is reported to be significantly lower than the initial V0.Both cost functions contain 65 monomials in their full forms.

VII. CONCLUSIONS

The paper presents a global, data-driven ADP method for nonlinear polynomial systems that replaces PDE-solving policy iteration with convex optimization and yields globally stabilizing policies. Its extension to more general nonlinear systems and uncertainty classes remains under investigation.

  • VII. CONCLUSIONS: The proposed global ADP method targets data-driven adaptive optimal control of nonlinear polynomial systems and produces globally stabilizing control policies.It is presented as a global method rather than a semiglobal or local stabilization approach.
  • VII. CONCLUSIONS: Each policy-iteration step solves a convex optimization problem instead of a partial differential equation, directly obtaining a suboptimal continuous-time optimal-control solution.The method can also be viewed as directly solving Hamilton-Jacobi inequalities used in H∞ control.
  • VII. CONCLUSIONS: The method avoids neural-network approximation and is reported to overcome slow convergence and loss of global asymptotic stability while providing computational benefits.Conventional schemes may require many basis functions for accurate approximation on compact sets.
  • VII. CONCLUSIONS: Extending the methodology to more general deterministic or stochastic nonlinear systems and systems with parametric or dynamic uncertainties remains future work.These extensions are identified as being under current investigation.

APPENDIX A SUM-OF-SQUARES (SOS) PROGRAM

The appendix formulates sum-of-squares programs as convex optimization problems equivalent to semidefinite programs, then proves key policy-iteration properties including value-function bounds and global stabilization.

  • SUM-OF-SQUARES PROGRAM: An SOS program is introduced as a convex optimization problem with polynomial SOS constraints.The constraints require specified polynomials to be sums of squares.
  • SUM-OF-SQUARES PROGRAM: SOS programs are equivalent to semidefinite programs, which optimize linear functions subject to linear matrix inequality constraints.The conversion can be performed manually or with tools such as SOSTOOLS and YALMIP.
  • Policy-Iteration Properties: Under the stated assumptions, conventional policy iteration maintains the value-function bound Vi(x) ≥ Vᵒ(x) and establishes monotonic inequalities across iterations.The lemma also states conditions involving L(Vi−1, ui) and L(Vi, ui+1).
  • Policy-Iteration Properties: Induction shows that each updated policy remains globally stabilizing when the iterated value function is a valid Lyapunov function.The proof identifies Vi+1 as belonging to the class P and uses it to establish global stabilization.
  • Policy-Iteration Properties: If a pair satisfying the relevant conditions exists, uniqueness of the HJB solution implies it equals the optimal value and control pair.The proof concludes V* = Vᵒ and u* = uᵒ.
Loading 1401.0020v4…