Fixed-Time Integral Reinforcement Learning for Saturated Nonlinear Multi-Agent Systems Under FDI Attacks

Motivation

• FDI signals can corrupt measurements, exchanged information, or control commands, motivating zero-sum differential-game modeling of secure formation control.The associated optimality condition is characterized by the HJI equation.Source: §I · p. 1
• Unknown nonlinear drift makes direct HJI solution intractable, while integral reinforcement learning removes the drift from finite-window learning residuals.Nonquadratic input utilities embed symmetric command constraints and yield bounded hyperbolic-tangent policies.Source: §I · p. 1

Research gap

• Existing fixed-time secure reinforcement learning is limited to comparatively simple second-order models, leaving unknown nonlinear dynamics and coupled secure-control channels unresolved.The paper identifies the joint treatment of unknown drift, graph coupling, same-channel FDI, bounded commands, and fixed-time critic learning as challenging.Source: §I · p. 1

Approach

• The proposed framework decomposes graph-coupled errors into drift, secure-control, and adversarial channels, then uses a two-power state penalty and integral Bellman–Isaacs residual.Recorded data preserve excitation as trajectories approach the desired formation.Source: §I · p. 1

Contributions

• The paper proposes a cost construction, critic update law, unified stability analysis, and resilient controller for unknown nonlinear systems under FDI attacks.These contributions target fixed-time parameter-error convergence and stable, robust closed-loop behavior.Source: §I · p. 1

Secure game formulation

• The defender minimizes the cost, while disturbances and same-channel FDI signals act as maximizing inputs in the local zero-sum game.The original game cost remains independent of the unknown value function.Source: §III · p. 3

Saturation-aware HJI

• Symmetric input constraints are embedded directly in the HJI equation through a saturation-compatible utility.This design produces admissible bounded secure policies within the prescribed input set.Source: §III · p. 3

Graphical game

• The local graphical game separates self and neighboring secure inputs from locally and remotely entering adversarial channels.Neighboring secure policies are treated as fixed coupling signals when each follower’s local HJI equation is formed.Source: A. Graphical Game and Fixed-Time… · p. 3

Fixed-time condition

• A fixed-time stability theorem converts a two-power dissipation inequality with residual term Δ_i into practical fixed-time convergence.The settling-time bound is independent of the initial value, and Δ_i = 0 yields exact fixed-time convergence.Source: A. Graphical Game and Fixed-Time… · p. 4

Cost-induced shaping

• The running cost is designed so mixed powers of the measurable coordination error generate the required fixed-time structure directly through the HJI equation.This avoids placing the unknown value function in the running cost and makes the construction cost-induced rather than externally assumed.Source: A. Graphical Game and Fixed-Time… · p. 4

Inverse-optimal interpretation

• Inverse recovery offers a systematic alternative for constructing a fixed-time-compatible penalty, but requires matching, positivity, saturated-policy admissibility, and neighboring-policy consistency checks.The paper retains a simpler sufficient condition because inverse-cost synthesis is not its main contribution.Source: A. Graphical Game and Fixed-Time… · pp. 4–5

HJI formulation

• The secure utility penalizes defender effort, while negative quadratic disturbance terms represent the maximizing roles of disturbances and same-channel FDI attacks.The resulting HJI equation is nonlinear and generally analytically intractable, motivating learning-based approximation.Source: A. Graphical Game and Fixed-Time… · p. 5

• The unknown value function and gradient are approximated on a compact trajectory-relevant region using an RBF critic with bounded approximation residuals.Source: B. Integral Bellman–Isaacs Residual · pp. 5–6
• A finite data window converts the HJI relation into a computable identity based on feature differences and an approximate integral running term.Source: B. Integral Bellman–Isaacs Residual · p. 6
• When the critic weights, approximation, and policies are ideal, the residual is zero, so residual minimization recovers the saddle-point condition along the trajectory.Source: B. Integral Bellman–Isaacs Residual · p. 6
• The integral Bellman–Isaacs residual enforces the HJI equation along measured trajectories without requiring explicit knowledge of the unknown drift.Source: B. Integral Bellman–Isaacs Residual · p. 6
• A finite experience stack stores feature differences and running terms, replacing persistent excitation with a requirement that recorded data contain enough independent information.Source: B. Integral Bellman–Isaacs Residual · p. 6

• The residual formulation avoids pointwise model-based HJI errors because the unknown drift appears in the unavailable pointwise equation.Source: C. Fixed-Time Critic Update · p. 6
• The update is derived from residual gradients whose current and replay regressors are feature differences, enabling direct learning from finite-window and recorded data.Source: C. Fixed-Time Critic Update · pp. 6–7
• The fixed-time critic update combines current and recorded residuals with leakage, using two powers to accelerate learning near zero and avoid slow learning for large residuals.Source: C. Fixed-Time Critic Update · pp. 6–7
• Because the learned control policy depends on the critic gradient, bounding critic-weight error in fixed time prevents an arbitrarily long learning transient from extending the closed-loop convergence bound.Source: C. Fixed-Time Critic Update · p. 7
• Experience replay makes initial excitation sufficient for critic learning, even when future formation trajectories no longer remain persistently exciting.Source: C. Fixed-Time Critic Update · p. 7

• The critic-weight estimation error is practically fixed-time stable, and its convergence-time bound is independent of the initial critic-weight error.Source: §IV · pp. 7–9
• Bounded Bellman–Isaacs approximation, integration, and leakage terms form a residual constant that does not depend on the initial critic-weight error.Source: §IV · p. 9
• The critic Lyapunov estimate uses two powers of the weight error plus a bounded perturbation term to establish practical fixed-time convergence.Source: §IV · p. 9
• The formation proof combines critic convergence with the cost-induced HJI decay to obtain practical boundedness of the leader-referenced error.Source: §IV · pp. 9, 12
• The leader-rooted formation error and critic-weight error reach bounded residual sets within fixed times, with the formation bound given by the maximum follower convergence time.Source: §IV · pp. 9, 12

• The simulation models four nonlinear followers with unknown drift, external disturbances, secure control inputs, and actuator-side FDI signals relative to one leader.Source: A. Simulation Setup · p. 13
• The actuator-side FDI attack begins at t = 8 s, while the leader and desired follower offsets define the leader-referenced formation task.Source: A. Simulation Setup · p. 13
• Each follower uses a six-node Gaussian RBF critic with normalized coordination-error inputs over the admissible state domain.Source: A. Simulation Setup · p. 13
• Replay data are collected during the pre-attack interval t ∈[0, 8] s and retained afterward to preserve initial-excitation information for critic learning.Source: A. Simulation Setup · p. 13
• The numerical fixed-time comparison scales initial formation-error perturbations by s ∈ {0.5, 1.0, 1.5, 2.0, 2.5}.Source: A. Simulation Setup · p. 13

• The observed settling times remain between 9.53 and 10.76 s as the initial-error scale increases from 0.5 to 2.5.The corresponding initial norm grows from approximately 30 to above 150, yielding a common numerical bound.Source: B. Simulation Results · p. 13, Figure 5 · p. 14
• The six critic weights for each follower reach bounded constant values in approximately 5.7 s without subsequent drift.Source: B. Simulation Results · p. 13, Figure 3 · p. 14
• Formation position errors are driven close to the origin while velocity errors remain small after the transient.The initial position errors span approximately [-5, 6] along x and [-4, 5] along y.Source: B. Simulation Results · p. 13, Figure 4 · p. 14
• Secure inputs remain within |u_ci,ℓ| < 40, with peak magnitudes of approximately 39 and 32 along the x- and y-channels.Source: B. Simulation Results · p. 14, Figure 6 · p. 14
• Bounded tracking errors are maintained under nonlinear dynamics, persistent disturbances, actuator FDI attacks, and input constraints.FDI signals reach approximately 0.67 after activation at t_a = 8 s, while disturbances remain within approximately [-0.063, 0.063].Source: B. Simulation Results · p. 14, Figure 7 · p. 14, Figure 4 · p. 14

• The comparison constants in the stability certificate exist on the relevant compact region but cannot yet be systematically computed from available system information.Their relationship with the cost-shaping coefficients is therefore not fully constructive.Source: §VI · p. 15
• Bounded approximation and adversarial residuals imply practical fixed-time convergence to a prescribed neighborhood rather than exact convergence to the origin under persistent perturbations.Source: §VI · p. 15
• The cost-shaping coefficients are conservatively selected by empirical tuning and affect both the Bellman–Isaacs residual and critic learning.Source: §VI · p. 15
• The admissibility certificate is local and does not establish that every subsequent critic-only policy update preserves admissibility.Source: §VI · p. 15
• Data-driven admissibility preservation during online learning and constructive evaluation of the HJI comparison constants remain future-work directions.Source: §VI · p. 15

• The framework handles unknown dynamics, disturbances, FDI attacks, and bounded inputs in nonlinear leader–follower formation control.Source: §VII · p. 15
• A nonquadratic utility enforces input constraints, while a two-power cost and replay-based critic update drive critic and formation errors to compact residual sets in fixed time.The settling times are independent of initial conditions.Source: §VII · p. 15
• Simulations confirmed resilient formation tracking under persistent disturbances and time-varying FDI attacks.Source: §VII · p. 15

• The construction certifies and initializes an admissible HJI branch without requiring an exact finite-dimensional Koopman model.Online critic-only integral Bellman–Isaacs learning continues directly from finite-window residuals.Source: Appendix A · pp. 15–16
• The lifted coordinate η_i = L_Ki(χ_i) preserves the local coordination-error origin because η_i = 0 if and only if χ_i = 0.Source: Appendix A · p. 15
• Stored trajectory windows identify a lifted nominal model through informative data satisfying rank(R_Ki) = N_Ki + m.The identified dynamics include a finite-data fitting residual and lifting or identification error term.Source: Appendix A · p. 15
• A validation bound with α_Ki := λ_min(Q_Ki) − 2∥P_i∥ρ_Ki > 0 yields a locally stabilizing initial policy and an invariant admissible region.The resulting Lyapunov derivative is negative away from the origin.Source: Appendix A · pp. 15–16
• The initial policy produces exponential decay of the lifted state and converges to zero while satisfying the saturation-compatible input constraint.Source: Appendix A · p. 16