PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-based reinforcement learning for infinite-horizon approximate optimal tracking

Rushikesh Kamalapurkar, Lindsey Andrews, Patrick Walters, Warren E. Dixon

arXiv:1506.00685v1eess.SYcs.SYmath.OC

TL;DR

Infinite-horizon optimal tracking for uncertain nonlinear systems is challenging because trajectory-dependent value functions are difficult to approximate and existing approaches may require persistence of excitation. The paper uses model-based reinforcement learning with a concurrent-learning identifier to simulate Bellman-error experience beyond observed trajectories. Stability analysis establishes bounded tracking error and convergence of the learned policy to a neighborhood of the optimal policy, with simulations demonstrating the technique.

  • Problem

    For infinite-horizon tracking, the value function depends explicitly on time and cannot be approximated with arbitrary accuracy by universal function approximators over an unbounded domain; online methods also use persistence of excitation that cannot generally be guaranteed a priori.

    Source: §I · p. 1

  • Method

    A concurrent-learning system identifier estimates the desired steady-state controller and drift dynamics, enabling model-based reinforcement learning to evaluate Bellman error at unexplored states.

    Source: §I · p. 1, §VIII · p. 7

  • Results

    Lyapunov-based analysis establishes bounded states, ultimately bounded tracking error, and convergence of the learned policy to a neighborhood of the optimal policy; simulations demonstrate the technique.

    Source: §VII · p. 6, §VIII · p. 7

  • Takeaways & Limitations

    The method combines a steady-state controller estimate, augmented-state stationary formulation, and Bellman-error extrapolation for approximate online infinite-horizon tracking.

    Source: §VIII · p. 7

    Assumption 4 cannot generally be guaranteed a priori, although it can be monitored online; sufficient conditions verifiable a priori remain future work.

    Source: §VIII · p. 7

Abstract

from arXiv · show

This paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. Model-based reinforcement learning is used to relax the persistence of excitation condition. Model-based reinforcement learning is implemented using a concurrent learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.

I. INTRODUCTION

The introduction motivates online optimal tracking for continuous-time nonlinear systems, where continuous state-action spaces and time-varying trajectories complicate RL value approximation. It proposes model-based RL with a concurrent-learning identifier to estimate the steady-state controller and simulate Bellman-error experience beyond the observed trajectory.

I. INTRODUCTION

  • Continuous state-action spaces have slowed progress in RL-based optimal control compared with finite state-action spaces and stationary environments.The introduction notes that RL methods have been used for deterministic optimal regulation, while infinite-horizon tracking has also been studied online and offline.Source: §I · p. 1
  • For infinite-horizon tracking, the value function depends explicitly on time and cannot be approximated with arbitrary accuracy over an unbounded domain by universal function approximators.When the desired trajectory is the output of an autonomous system, the value function can instead be represented as stationary using the state augmented with that trajectory.Source: §I · p. 1
  • A concurrent-learning identifier and model-based RL address unknown drift dynamics while enabling off-trajectory Bellman-error evaluation.The approach is intended to relax persistence of excitation to finite excitation and approximate the optimal policy without an added exploration signal.Source: §I · pp. 1–2
  • The prior stationary reformulation in requires exact model knowledge because its steady-state tracking controller depends on the system model.Extending it to uncertain systems requires estimating that controller and accounting for estimation error in stability analysis.Source: §I · p. 1
  • The stated contributions include approximate model inversion for the steady-state controller and simulation evidence of policy approximation without an added exploration signal.The method uses a concurrent-learning-based identifier and seeks to relax persistence of excitation to finite excitation.Source: §I · pp. 1–2

II. PROBLEM FORMULATION AND EXACT SOLUTION

The problem formulation recasts tracking as an infinite-horizon regulation problem using the tracking error and the difference between actual and desired steady-state control. Under stated assumptions, this yields autonomous augmented-state dynamics and a stationary optimal control problem.

II. PROBLEM FORMULATION AND EXACT SOLUTION

  • The system is control-affine, with locally Lipschitz drift and control-effectiveness functions, and the objective is to track a time-varying desired trajectory.The tracking error is defined as e = x − xd.Source: §II · p. 2
  • A quadratic infinite-horizon cost in tracking error and actual input is infinite when the steady-state tracking input is nonzero, so the formulation instead penalizes tracking error and control mismatch.The mismatch is between the actual control signal and the desired steady-state control.Source: §II · p. 2
  • The assumptions specify bounded control effectiveness with full column rank and a bounded desired trajectory satisfying a matching condition that permits a steady-state tracking control.That control is expressed using the right pseudoinverse of the control-effectiveness matrix.Source: §II · p. 2
  • Defining μ as actual control minus desired steady-state control allows the system dynamics to be written in autonomous form using an augmented state.The augmented state ζ concatenates the tracking error and desired trajectory.Source: §II · p. 2
  • The problem is online infinite-horizon optimal regulation: it simultaneously synthesizes and uses a control signal to minimize a cost while tracking the desired trajectory; assuming an optimal policy exists, that policy is characterized by the value function.Source: §II · p. 2

III. BELLMAN ERROR

The section formulates Bellman error as the residual for approximate actor–critic value and policy functions, then explains how model-based experience can extend learning beyond the observed trajectory. A concurrent-learning identifier supports this extrapolation while relaxing the persistence-of-excitation requirement.

III. BELLMAN ERROR

  • Because the exact HJB solution is generally infeasible, actor–critic approximations substitute parameterized estimates of the optimal value function and policy into the HJB equation to define Bellman error.The critic estimates value-function parameters and the actor estimates policy parameters.Source: §III · p. 3
  • Approximation seeks actor–critic parameters that minimize Bellman error over a compact operating domain, since exact value-function basis functions are generally unavailable.The section describes uniform approximation of the value function and policy over that domain as desirable.Source: §III · p. 3
  • Integral Bellman error and derivative-estimation methods avoid direct drift-model knowledge, but their estimates depend on the state trajectory and can be evaluated only along it.Consequently, these methods require persistence of excitation for convergence; integral RL with experience replay relaxes this to finite excitation.Source: §III · p. 3
  • A learned drift model permits Bellman error to be evaluated at unexplored off-trajectory states, simulating experience alongside data stored from the trajectory.A concurrent-learning identifier uses recorded data to relax the persistence-of-excitation requirement for parameter convergence.Source: §III · pp. 3–4

IV. SYSTEM IDENTIFICATION

The system identifier approximates unknown drift dynamics with a neural network and updates its weights using concurrent learning. Its history stack supports finite-excitation learning and can be recorded online before switching to the developed controller.

IV. SYSTEM IDENTIFICATION

  • The drift dynamics are approximated by a neural network with estimated weights, and a concurrent-learning identifier updates those weights online.The identifier uses state-derivative data and a positive learning gain.Source: §IV · p. 4
  • A history stack of recorded state-action pairs supports the identifier’s concurrent-learning update.The stack need not be available in advance; it can be recorded over a finite interval when the system states are sufficiently exciting.Source: §IV · p. 4
  • During stack collection, the controller from can be used before switching once to the controller developed here.The passage states that stability of the resulting switched system follows from stability of the individual subsystems.Source: §IV · p. 4

V. VALUE FUNCTION APPROXIMATION

The intractable optimal value-function and policy problem is approximated with neural networks over a compact operating domain. Online learning adjusts their weights while accounting for drift-model estimates and the estimated steady-state controller.

V. VALUE FUNCTION APPROXIMATION

  • The optimal value function is represented by a neural network over a compact operating domain, replacing an intractable minimization with finite-dimensional weight estimation.The representation includes a function reconstruction error.Source: §V · p. 4
  • Neural-network weight estimates approximate both the optimal value function and the optimal policy.The section describes separate critic and actor weight estimates.Source: §V · p. 4
  • Online weight selection minimizes the approximation error for a given drift estimate while the identifier is updated and the control law maintains stability.The optimization is coupled to drift-model learning rather than performed independently.Source: §V · p. 4
  • The stability analysis includes the error between the estimated and desired steady-state tracking controllers.The passage relates the controlled tracking-error trajectories to those of the reformulated system.Source: §V · pp. 4–5

VI. SIMULATION OF EXPERIENCE

Model-based simulation of experience approximates Bellman-error minimization using a finite set of points in the operating domain. The learning updates use the errors evaluated at those points alongside the instantaneous Bellman errors.

VI. SIMULATION OF EXPERIENCE

  • Simulation of experience minimizes a squared sum of Bellman errors over finitely many state-space points because computing the supremum is generally intractable.The approximation relies on an assumption about a finite set of points in the operating domain.Source: §VI · p. 5
  • The weight-update laws use the finite-point Bellman errors together with instantaneous Bellman errors.The section defines normalized terms and adaptation parameters used by the update laws.Source: §VI · p. 5

VII. STABILITY ANALYSIS

The stability analysis uses Lyapunov functions and sufficient conditions on gains, approximation capacity, and Bellman-error extrapolation points. Under the stated assumptions and selections, the system states remain bounded, tracking error is ultimately bounded, and the policy converges to a neighborhood of optimal.

VII. STABILITY ANALYSIS

  • The analysis treats the optimal value function as a Lyapunov candidate when the state penalty is positive definite.The passage states that this makes the optimal value function positive definite for the concatenated system under the optimal policy.Source: §VII · p. 5
  • The proof selects a candidate Lyapunov function and bounds its time derivative using the identifier and learning-update relations.The candidate function includes state-estimation and parameter-estimation errors.Source: §VII · pp. 5–6
  • Satisfying the sufficient stability condition requires enough neural-network neurons and Bellman-error extrapolation points.The condition also depends on the chosen compact set and approximation errors; increasing neuron count and learning gains can reduce the stated constant for a fixed set.Source: §VII · p. 6
  • The controller and learning laws ensure bounded system states, ultimately bounded tracking error, and policy convergence to a neighborhood of the optimal policy under the theorem’s conditions.The conditions include Assumptions 2–4 and selections of gains, neural-network neurons, and Bellman-error extrapolation points.Source: §VII · p. 6
  • Invoking Theorem 4.18 in, trajectories Z(t) satisfying the cited initial-norm condition are bounded for all t ∈ R and have lim sup as t→∞ of ∥Z(t)∥ at most vl.Source: §VII · p. 7

VIII. CONCLUSION

The paper presents a concurrent-learning implementation of model-based reinforcement learning for approximate online optimal tracking, using a system identifier to support Bellman-error extrapolation. Assumption 4 is not generally guaranteed a priori, though it can be monitored online.

VIII. CONCLUSION

  • A concurrent-learning implementation of model-based reinforcement learning provides an approximate online solution to infinite-horizon tracking for nonlinear continuous-time control-affine systems.The approach uses a system identifier to remove dependence of the desired steady-state controller on drift dynamics and to simulate experience through Bellman-error extrapolation.Source: §VIII · p. 7
  • The desired steady-state controller and state augmentation formulate feasible and stationary optimal control problems for tracking.The system identifier also facilitates Bellman-error extrapolation.Source: §VIII · p. 7
  • Assumption 4 cannot generally be guaranteed a priori, but oversampling with N ≫ L can heuristically meet it and its satisfaction can be monitored online.Threshold-based point selection can preserve rank; verifying sufficient conditions a priori remains future work.Source: §VIII · p. 7
Loading 1506.00685v1…