PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-policy reinforcement learning for $ H_\infty $ control design

Biao Luo, Huai-Ning Wu, Tingwen Huang

arXiv:1311.6107v3eess.SYcs.LGmath.OCstat.ML

TL;DR

The paper addresses nonlinear continuous-time H∞ control when the internal system model is unknown and the HJI equation is difficult to solve analytically. It develops an off-policy RL method that learns from data generated by arbitrary control and disturbance signals, with NN-based implementation and convergence analysis. The method is tested on a linear F16 aircraft plant and a nonlinear rotational/translational actuator problem, achieving the reported H∞ performance in the latter.

  • Problem

    Nonlinear H∞ control requires solving an HJI equation, but the equation is generally difficult to solve analytically and model-based methods require an unavailable or costly system model.

  • Method

    The paper derives an off-policy RL method that learns the HJI solution from data generated by arbitrary control and disturbance signals, using an actor-critic structure and least-square NN weight updates.

  • Results

    The method is tested on a linear F16 aircraft plant and a nonlinear rotational/translational actuator problem, with the actuator experiment reaching rd(t)=3.7024(< γ = 6).

  • Takeaways & Limitations

    The proposed NN-based off-policy RL method learns the HJI solution and H∞ control policy from system data rather than requiring the internal mathematical model.

Abstract

from arXiv · show

The $H_\infty$ control design problem is considered for nonlinear systems with unknown internal system model. It is known that the nonlinear $ H_\infty $ control problem can be transformed into solving the so-called Hamilton-Jacobi-Isaacs (HJI) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, model-based approaches cannot be used for approximately solving HJI equation, when the accurate system model is unavailable or costly to obtain in practice. To overcome these difficulties, an off-policy reinforcement leaning (RL) method is introduced to learn the solution of HJI equation from real system data instead of mathematical system model, and its convergence is proved. In the off-policy RL method, the system data can be generated with arbitrary policies rather than the evaluating policy, which is extremely important and promising for practical systems. For implementation purpose, a neural network (NN) based actor-critic structure is employed and a least-square NN weight update algorithm is derived based on the method of weighted residuals. Finally, the developed NN-based off-policy RL method is tested on a linear F16 aircraft plant, and further applied to a rotational/translational actuator system.

I. INTRODUCTION

The paper frames nonlinear H∞ control as an HJI-equation problem that is difficult to solve analytically, especially when the internal system model is unknown. It introduces an off-policy RL approach that learns from system data rather than requiring the evaluating policies or a mathematical model.

  • Motivation: H∞ control seeks disturbance attenuation in the L2-gain setting, while nonlinear H∞ design can be transformed into solving an HJI equation.The desired controller should stabilize the closed-loop system and keep the L2-gain at or below a prescribed level γ.
  • Motivation: The HJI equation is a nonlinear PDE that is difficult or impossible to solve analytically and may lack global analytic solutions even in simple cases.
  • Related work: Existing approximation methods use policy iterations, successive PDE approximations, Galerkin approximation, or related policy-iteration schemes for HJI equations.
  • Contribution: The paper introduces an off-policy RL method for nonlinear continuous-time H∞ control with an unknown internal system model.Its stated organization covers problem formulation, off-policy learning for nonlinear and linear systems, and simulation studies.
  • Problem setting: For unknown internal dynamics, data-based methods are needed because the iterative HJI equations require the full system model.The system model f(x) is explicitly assumed unknown in the paper's problem formulation.

III. MOTIVATION FROM INVESTIGATION OF RELATED

Existing H∞ control iterations rely on the full system model and evaluating policies, creating practical limitations for unknown-model systems. These limitations motivate an off-policy approach that can reuse arbitrary data-generation policies.

  • The iterative H∞ control equations require the full system model, including the internal dynamics.
  • Model-based policy-iteration schemes approximate the HJI equation through linear PDEs, but require nested updates of control, disturbance, and value-function iterations.
  • On-policy methods evaluate costs using data generated by the evaluating control and disturbance policies, which may be unavailable or only approximately known.
  • On-policy learning usually requires adjustable disturbance signals, restricting implementation in many real systems.
  • On-policy exploration is restricted because only evaluating policies generate data, while several implementations also require multiple neural networks and time-consuming online learning.
  • The paper therefore proposes off-policy reinforcement learning for unknown internal dynamics to improve data-generation flexibility and address these implementation drawbacks.

IV. OFF-POLICY REINFORCEMENT LEARNING FOR H∞

The paper derives an off-policy reinforcement-learning method for H∞ control design and proves its convergence, then develops a neural-network actor-critic structure for implementation.

  • The method targets H∞ control design through off-policy reinforcement learning and includes a convergence proof.
  • A neural-network-based critic-actor structure is developed for implementation.

A. Off-policy reinforcement learning

The off-policy formulation replaces model-dependent PDE learning with an integral equation that uses arbitrary input data. Its equivalence to the model-based iteration establishes convergence to the HJI solution.

  • A. Off-policy reinforcement learning: The system is rewritten around the current control and disturbance policies, with arbitrary input deviations represented explicitly.
  • A. Off-policy reinforcement learning: Integrating the resulting derivative relation over a time interval yields the off-policy learning equation.
  • A. Off-policy reinforcement learning: The value function can be learned from arbitrary input signals rather than the evaluating control and disturbance policies.
  • A. Off-policy reinforcement learning: Theorem 1 states that the off-policy equation is equivalent to the linear PDE used in the model-based iteration.
  • A. Off-policy reinforcement learning: This equivalence supports the convergence proof by showing that the off-policy iteration has the same solution as the linear PDE.
  • A. Off-policy reinforcement learning: The paper concludes that the off-policy iteration converges to the HJI solution while using system data instead of the unknown internal dynamics.

B. Implementation based on neural network

The paper uses a critic NN to approximate the HJI solution and derives a least-squares weight update from weighted residuals and sampled system data. Algorithm 2 learns offline from arbitrary control and disturbance signals, then produces the H∞ policy for real-time use.

  • NN structure: The critic NN approximates the unknown value function over a compact domain using a finite-dimensional activation-function basis.The approximation is motivated by the high-order Weierstrass theorem.
  • Weight update: Weighted residuals force the HJI residual toward zero in an average sense, yielding a least-squares update for the critic NN weights.Monte Carlo integration replaces computationally expensive numerical inner products over the domain.
  • Data requirements: The sampled data must be sufficiently rich for Z(i) to have full column rank, which can be encouraged with larger samples, varied initial states, and rich input signals.Persistent excitation is useful but not necessary and remains difficult to achieve.
  • Algorithm 2: Algorithm 2 collects real system samples, initializes critic weights, iteratively updates them, and stops when successive weights differ by at most ξ.After convergence, the learned weights define the H∞ control policy.
  • Off-policy learning: Off-policy learning evaluates the current cost function using data generated by different control and disturbance signals, including exploratory or random policies.The method does not require adjustable disturbances and reduces the accumulated error associated with approximate on-policy data.
  • Practical implementation: The policy is learned offline, reused for real-time control, and trained from a collected dataset that can be repeatedly reused without additional data.Only one critic NN is required because the control and disturbance policies are obtained from its weights.

C. Convergence analysis for NN-based off-policy RL

The convergence analysis establishes conditions under which the critic and actor NNs converge, and then proves uniform convergence of the learned HJI solution and H∞ policy.

  • Assumptions: Theorem 2 assumes complete activation functions, uniform approximation of the value function and its gradient, and linear independence of activation-function differences.These conditions support convergence of the critic and actor approximations.
  • Proof strategy: The convergence proof establishes the required linear independence of transformed activation-function gradients and derives approximation results for the NN scheme.The argument uses a contradiction and follows related first-order linear PDE results.
  • NN convergence: Theorem 2 implies convergence of both the critic NN and actor NNs under its stated assumptions.The actor policies are constructed from the learned critic representation.
  • Uniform convergence: Theorem 3 proves that the NN-based off-policy RL algorithm converges uniformly to the HJI solution and the H∞ control policy.The theorem is stated for sufficiently large iteration index i and hidden-layer size L.
  • Control implications: With the limiting H∞ policy, the disturbance-free closed-loop system is locally asymptotically stable, while disturbed-output behavior is analyzed through the H∞ formulation.The stability statement is local and specifically concerns w(t) ≡ 0.

V. OFF-POLICY REINFORCEMENT LEARNING FOR LINEAR H∞CONTROL

For linear systems, the HJI equation reduces to an algebraic Riccati equation, allowing the off-policy method to learn the unknown matrix parameter through a simpler least-squares scheme. F16 simulations track critic-weight convergence.

  • Linear formulation: For the linear system, the HJI equation becomes an algebraic Riccati equation, whose stabilizing solution yields V*(x) = xT Px and the linear H∞ policy.The result requires a stabilizing solution P ≥ 0.
  • Off-policy learning: The linear off-policy method collects samples to compute regression terms and learns the unknown parameter vector vec(P(i+1)) without cost-function approximation.The resulting least-squares scheme is simpler than the nonlinear NN formulation.

VI. SIMULATION STUDIES

The simulation studies test the developed NN-based off-policy RL method on a linear F16 aircraft plant and apply it to a nonlinear rotational/translational actuator benchmark.

  • Simulation scope: The method is evaluated on a linear F16 aircraft plant and then applied to the nonlinear rotational/translational actuator system.These are the two stated simulation-study settings.

A. Efficiency test on linear F16 aircraft plant

The off-policy RL method learns the F16 plant’s ARE solution from sampled data and converges to the ideal critic weights in five iterations across tested sampling intervals.

  • Plant and setup: The F16 plant is modeled as a linear continuous-time aircraft system with angle of attack, pitch rate, and elevator deflection states.The control input is elevator actuator voltage, while disturbance represents wind gusts on angle of attack.
  • Plant and setup: The critic NN uses six activation functions to represent the quadratic value function V*(x) = xT Px for the linear system.The ideal weight vector is [1.6573, 2.7908, −0.3322, 1.6573, −0.3608, 0.4371].
  • Convergence: Five iterations are required for the critic NN weight vector to converge to the ideal values under the baseline simulation settings.The experiment uses 100 samples, random input noise in [0,0.1], Δt = 0.1s, and stopping criterion ξ = 10−7.
  • Convergence: Five iterations remain sufficient when Δt is varied from 0.2s to 0.5s, indicating insensitivity of Algorithm 2 to this parameter.The critic weights converge to the ideal values for every tested interval.

B. Application to the rotational/translational actuator nonlinear benchmark problem

The method is applied to the nonlinear RTAC benchmark, where coupled rotational and translational dynamics challenge control design; the critic weights converge rapidly and the resulting controller satisfies the prescribed gain level.

  • Nonlinear RTAC benchmark: The RTAC benchmark couples rotational and translational motions, creating a nonlinear test problem for control methods.The experiment sets R = 1 and γ = 6.
  • Learning setup: The nonlinear HJI solution is learned with a 20-dimensional critic NN using 300 samples and random input noise in [0,0.5].The simulation uses Δt = 0.033s and stopping criterion ξ = 10−7.
  • Learning outcome: Three iterations are sufficient for the critic NN weight vector to converge rapidly in the RTAC experiment.The converged weights are then used to compute the H∞ control policy.
  • Closed-loop evaluation: The closed-loop experiment evaluates state trajectories, control input, and disturbance attenuation under a decaying random disturbance signal.The disturbance is w(t) = 0.2r1(t)e−0.2tcos(t), with r1(t) ∈ [0,1].
  • Closed-loop evaluation: The attenuation ratio converges to 3.7024, below γ = 6, indicating the designed controller achieves the prescribed L2-gain performance level.This conclusion is drawn from the reported behavior of rd(t) as time increases.

VII. CONCLUSIONS

The paper concludes that NN-based off-policy RL can learn HJI solutions for unknown-model continuous-time H∞ control from data generated by arbitrary policies. Simulations on F16 and RTAC systems support its effectiveness.

  • Conclusions: The proposed method solves continuous-time H∞ control problems with unknown internal system models using system data rather than model dynamics.Its convergence to the HJI solution is established theoretically.
  • Conclusions: The method’s off-policy data can be generated with arbitrary control and disturbance signals instead of the evaluating policy.This enables direct learning from real-system information while avoiding identification of the internal dynamics.
  • Conclusions: The actor-critic implementation uses one NN for cost-function approximation and a least-square scheme for updating its weights.The method is tested on both linear F16 and nonlinear RTAC systems.
Loading 1311.6107v3…