PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-Triggered Practical Fixed-Time Integral Reinforcement Learning for Unknown Nonlinear Systems

Tien Dat Vu, My Nguyen Bach, Minh Doan

arXiv:2610.00800v2eess.SYcs.SY

TL;DR

Unknown nonlinear dynamics make optimal feedback construction difficult, while networked control also faces timing and communication requirements. The paper combines integral drift identification, inverse-optimal fixed-time design, replay-based critic learning, and event-triggered implementation; it reports practical fixed-time stability and exclusion of Zeno behavior.

  • Problem

    Unknown nonlinear dynamics obstruct direct construction of optimal feedback, and existing approaches do not jointly address practical fixed-time learning and closed-loop stability with event-triggered implementation.

    Source: §I · p. 1

  • Method

    The framework reconstructs the drift from integral data, constructs an inverse-optimal running cost, and combines current data with experience replay in an event-triggered critic-only learning law.

    Source: §I · p. 1, C. Critic-Only Integral Reinforcement… · p. 3

  • Results

    The analysis establishes practical fixed-time critic convergence and closed-loop stability under event-triggered implementation, with a positive inter-event-time bound excluding Zeno behavior.

    Source: §I · pp. 1–2, §IV · p. 8

  • Takeaways & Limitations

    The reported framework achieves practical fixed-time learning and closed-loop stability without persistent excitation, while excluding Zeno behavior.

    Source: §IV · p. 8

Abstract

from arXiv · show

This paper develops an event-triggered fixed-time integral reinforcement learning framework for optimal control of unknown nonlinear systems. An integral data-driven identifier is first used to reconstruct the unknown dynamics, after which an inverse-optimal formulation is employed to construct a fixed-time running cost. A learning law satisfying the practical fixed-time property is then derived. Previously collected data, or data obtained during a finite excitation interval, are stored in an experience-replay buffer and incorporated into the weight-update law. This avoids the persistent-excitation condition, which is often difficult to satisfy in practical operation. To reduce communication and control updates, an event-triggered mechanism is introduced. The paper shows that, under the event-triggered implementation, the closed-loop system still achieves practical fixed-time stability, while the proposed triggering rule guarantees the exclusion of Zeno behavior. Finally, a nonlinear example is presented to verify the theoretical results developed in the paper.

I. INTRODUCTION

The introduction frames optimal control of unknown nonlinear systems as a challenge involving unavailable dynamics, fixed-time guarantees, and communication constraints. It presents an event-triggered fixed-time IRL framework and formulates the goal of learning the drift and designing an optimal policy.

I. INTRODUCTION

  • Unknown nonlinear dynamics complicate HJB-based optimal feedback and uniform-time guarantees, while networked control also requires reducing transmissions without sacrificing performance.The paper identifies these challenges in increasingly complex cyber–physical systems.Source: §I · p. 1
  • Existing work leaves unresolved the joint provision of unknown-dynamics reconstruction, practical fixed-time critic convergence, practical fixed-time closed-loop stability, and event-triggered implementation.The introduction contrasts these gaps with prior event-triggered IRL and fixed-time learning results,,,.Source: §I · p. 1
  • The proposed framework identifies the unknown drift from integral data, constructs an inverse-optimal running cost, and uses replay data in a critic-only learning law without persistent excitation.Its event-triggered sample-and-hold implementation is analyzed for practical fixed-time closed-loop stability and a positive inter-event-time bound excluding Zeno behavior.Source: §I · pp. 1–2

A. Problem Formulation

  • The problem formulation considers a control-affine system with unknown drift and known input matrix, aiming to learn the drift and drive the state to the origin within a uniform finite settling time.The stated setting assumes locally Lipschitz dynamics on a compact region containing the origin and uniquely existing solutions while they remain there.Source: A. Problem Formulation · p. 2

B. Neural Approximation of the Unknown Dynamics

The unknown drift is represented on the compact operating set by a neural basis approximation with unknown ideal weights and a bounded approximation error.

B. Neural Approximation of the Unknown Dynamics

  • The drift is approximated by prescribed continuously differentiable basis functions, unknown ideal weights, and a residual approximation error.The basis vector is zero at the origin, and the basis functions and residual error are bounded on the compact set.Source: B. Neural Approximation of the Unknown… · p. 2
  • The identified drift uses the same prescribed basis vector with estimated weights.The estimate is written as ˆf(x) = ˆW_f^⊤σ_f(x).Source: B. Neural Approximation of the Unknown… · p. 2

C. Integral Data-Driven Identification

Integral measurements over sampled intervals produce data for a finite-data least-squares estimate of the unknown drift weights. Under a finite-excitation information condition, the estimate is unique and the identification error is bounded.

C. Integral Data-Driven Identification

  • Integrating the approximated dynamics over sampling intervals yields data segments relating state increments and integrated basis functions to the unknown weights.The data relation includes an integrated approximation-error term.Source: C. Integral Data-Driven Identification · p. 2
  • The estimated weights are obtained by fitting the accumulated integral data in the least-squares sense.The cumulative information matrix is updated by adding each new data segment's basis-vector contribution.Source: C. Integral Data-Driven Identification · p. 2
  • Finite excitation requires the accumulated information matrix to be positive definite after a finite number of data segments, equivalently giving a full-row-rank data matrix.Under this condition, the least-squares minimizer is unique and the identification error is uniformly bounded on the operating set.Source: C. Integral Data-Driven Identification · p. 2

A. Fixed-Time Optimal-Control Formulation

The formulation uses a designer-selected value function and prescribed decay condition to specify fixed-time behavior, then constructs a cost for which that value function is optimal. The minimizing policy follows from the HJB equation for the identified model.

A. Fixed-Time Optimal-Control Formulation

  • A designer-selected positive-definite value function encodes the desired closed-loop behavior through a decay condition that yields fixed-time stability with an initial-condition-independent settling-time bound.The condition uses positive coefficients and exponents satisfying 0 < γ1 < 1 < γ2.Source: A. Fixed-Time Optimal-Control… · p. 2
  • The running-cost terms are constructed so the selected value function is optimal and its corresponding policy satisfies the prescribed decay condition.The performance index is defined for this purpose.Source: A. Fixed-Time Optimal-Control… · p. 2
  • For the identified model, the HJB equation and a positive control Hessian yield a unique minimizing policy.The passage states that ∂²H/∂u² = 2R > 0.Source: A. Fixed-Time Optimal-Control… · pp. 2–3

B. Reverse Construction of the Fixed-Time Optimal Cost

The section constructs an inverse-optimal cost and feedback for a designer-selected value function with prescribed fixed-time decay. It selects the control minimizing R-weighted effort among controls satisfying that decay constraint.

B. Reverse Construction of the Fixed-Time Optimal Cost

  • The designer-selected value function is assigned a fixed-time decay condition, which the inverse-optimal construction uses to define a compatible optimal cost and policy.The policy is constructed for the identified model, and the feasibility condition ensures the construction is well defined away from the origin.Source: B. Reverse Construction of the… · p. 3
  • Among controls satisfying the prescribed decay constraint, the construction chooses the one minimizing the R-weighted control effort.The control constraint is ∇Vd^T g u = −σd; the resulting minimizer is used to construct the inverse-optimal cost.Source: B. Reverse Construction of the… · p. 3
  • The selected control is used to construct L1 and L2 so the Hamiltonian-minimizing policy coincides with the decay-enforcing control.The minimized Hamiltonian is then used to impose the HJB condition.Source: B. Reverse Construction of the… · p. 3

C. Critic-Only Integral Reinforcement Learning

This section builds a critic-only integral reinforcement learning law from current and stored trajectory data. Under its recorded-regressor and bounded-approximation-error assumptions, the critic-weight error converges in fixed time to a bounded residual set.

C. Critic-Only Integral Reinforcement Learning

  • The critic estimate represents the value function using basis functions and weights, with an explicit approximation-error term.The resulting critic-generated policy uses the gradient of this estimated value function.Source: C. Critic-Only Integral Reinforcement… · p. 3
  • The behavior input is implemented by holding the event-sampled target policy, so its difference from the continuously evaluated policy is the event-induced error.That error enters the learning identity as ∇V^T g e_u.Source: C. Critic-Only Integral Reinforcement… · p. 3
  • Integral Bellman relations yield a normalized residual whose current and recorded samples provide data for updating the critic weights.Stored samples are normalized regressors and residuals; the approximation residuals are assumed bounded.Source: C. Critic-Only Integral Reinforcement… · p. 4
  • The critic update combines the current normalized Bellman residual with stored-sample residuals using a two-power learning law.The powers use exponents below and above one, and the stored-data regressors are assumed to satisfy the stated excitation condition.Source: C. Critic-Only Integral Reinforcement… · p. 4
  • Under the recorded-regressor condition and bounded normalized approximation error, the critic-weight error converges in fixed time to a bounded residual set.When the approximation error is zero, the paper recovers exact fixed-time convergence.Source: C. Critic-Only Integral Reinforcement… · pp. 4–5

D. Event-Triggered Implementation and Composite Stability

The event-triggered implementation holds the control between update instants and incorporates the resulting policy error into a composite stability analysis. The triggering rule yields practical fixed-time stability, and bounded rates together with a positive triggering margin provide a positive lower bound on inter-event times.

D. Event-Triggered Implementation and Composite Stability

  • At each event instant, the state and critic estimate update the critic-only policy, whose control signal is held until the next event.The resulting implementation error is the difference between the held and continuously evaluated policies.Source: D. Event-Triggered Implementation and… · p. 5
  • The composite stability analysis combines state and critic errors with a measure of event-induced control error.The event measure includes the instantaneous policy error and its accumulated energy since the most recent event.Source: D. Event-Triggered Implementation and… · pp. 5–6
  • Under the stated assumptions and trigger, the composite quantity satisfies a two-power decay inequality with a bounded residual, establishing practical fixed-time stability.The residual collects continuous-learning, history, and triggering contributions; if it is zero, exact fixed-time convergence is recovered.Source: D. Event-Triggered Implementation and… · pp. 5–6
  • Outside the residual set, the decay inequality gives a settling-time bound independent of the initial condition.The paper states that exact fixed-time convergence follows when the total residual is zero.Source: D. Event-Triggered Implementation and… · pp. 6–7
  • If the policy is continuously differentiable on the compact operating set and the state and critic-weight rates are bounded, every nonterminal inter-event interval has a uniform positive lower bound.Consequently, infinitely many events cannot accumulate in finite time, excluding Zeno behavior.Source: D. Event-Triggered Implementation and… · p. 7

E. Simulation Setup

The simulation setup specifies a three-state control-affine nonlinear plant with identity input matrix and a quadratic desired value function. It uses the identified drift to construct an inverse-optimal running cost and feedback policy.

E. Simulation Setup

  • The example uses a three-dimensional plant with a nonlinear drift and identity input matrix.The state and control each have three components.Source: E. Simulation Setup · p. 7
  • The desired value function is chosen as one half the squared state norm.The setup defines r2(x) as the sum of the squared state components.Source: E. Simulation Setup · p. 7
  • Using the identified drift, the example specifies an inverse-optimal running cost and its associated feedback policy.The policy is defined to be zero at the origin, and the text states that it establishes the desired fixed-time convergence property.Source: E. Simulation Setup · p. 7

F. Simulation Results

The simulation reports fixed-time closed-loop behavior across five initial conditions, close tracking by the event-triggered input, and convergence of critic weights and value-function error. The triggering results show positive observed inter-event times, while Zeno exclusion is supported analytically.

F. Simulation Results

  • All five trajectories reach a small neighborhood of the origin within about 4 s, consistent with the fixed-time bound Tmax = 4 s.The initial-state norms range from approximately 0.5 to 3.0, with comparable settling times.Source: Figure 2 · p. 7, F. Simulation Results · p. 8
  • The event-triggered sample-and-hold input closely tracks the continuously evaluated policy, with an inset discrepancy of only the order of 10^-3.The first input reaches approximately −0.97 near 0.5 s and approaches zero by about 2.7 s.Source: Figure 3 · p. 7, F. Simulation Results · p. 8
  • The critic weights converge to [0.5, 0.5, 0.5, 0, 0, 0]⊤, exactly representing the selected value function under the chosen basis.The main critic transient disappears within approximately 15–20 s.Source: Figure 4 · p. 7, F. Simulation Results · p. 8
  • The value-function approximation error becomes visually negligible within about 2 s for all five cases.Source: Figure 5 · p. 8, F. Simulation Results · p. 8
  • Observed inter-event times range from approximately 5 × 10^-3 s to 1.7 × 10^-1 s, while Zeno exclusion follows analytically from Lemma 4.The observed minimum equals the integration step, so the simulation alone indicates only the absence of numerical event accumulation.Source: Figure 8 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

IV. CONCLUSION

The paper presents event-triggered fixed-time integral reinforcement learning for optimal control of unknown nonlinear systems. It reports practical fixed-time learning and closed-loop stability without persistent excitation, with Zeno behavior excluded.

IV. CONCLUSION

  • The framework targets optimal control of unknown nonlinear systems through event-triggered fixed-time integral reinforcement learning.The paper reports simulation results verifying its effectiveness.Source: §IV · p. 8
  • Integral data reconstruct the unknown drift for an inverse-optimal fixed-time control problem.Source: §IV · p. 8
  • Experience replay supports practical fixed-time convergence of a critic-only learning law without persistent excitation, while event triggering preserves practical fixed-time closed-loop stability and excludes Zeno behavior.Source: §IV · p. 8
Loading 2610.00800v2…