PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Secure Learning Control for Large-scale Multirobots under Stealthy Actuator Attacks

Xinglong Zhang, Qingwen Ma, Cong Li, Hui Yin, Changxin Zhang, Yueying Wang, Wei Pan, Xin Xu

arXiv:2609.06896v1cs.RO

TL;DR

Distributed multirobot control lacks secure performance guarantees under malicious, stealthy actuator attacks, despite the promise of combining RL and DMPC. DSLC learns attack and defense policies as closed-loop policies in a distributed differential-game framework. The paper reports formation and trajectory-tracking benefits under attacks, scalability to 1000 robots in simulation, and real-world tests across several coordination scenarios.

  • Problem

    Distributed learning control for multirobot systems lacks provable performance guarantees, and secure cooperative performance optimization under stealthy actuator attacks remains challenging.

    Source: §I · p. 1

  • Method

    DSLC models attackers and defenders as players in a differential-game DMPC framework and learns their defense and attack policies online in analytic closed-loop form.

    Source: §I · p. 2

  • Results

    DSLC controls multirobots to track reference trajectories under attacks where DRHRL and DRMPC exhibit formation disorder and tracking failure, and simulations demonstrate scalability to 1000 robots.

    Source: §I · p. 2, B. Comparative Results · p. 9

  • Takeaways & Limitations

    The framework is presented as generalizable across formation and containment scenarios, with real-world experiments on multiple wheeled robots and real-time optimization targeted to small-scale onboard processors.

    Source: §I · p. 2, D. Discussion and Limitations · pp. 14–15

    The stated real-time implementation target is small-scale onboard computing processors.

    Source: D. Discussion and Limitations · p. 15

Abstract

from arXiv · show

Distributed learning control for multirobot systems (MRS) offers significant flexibility in presence of uncertainties but lacks provable performance guarantees. A promising direction involves integrating reinforcement learning (RL) into distributed model predictive control (DMPC), leveraging the strengths of RL in nonlinear policy design and the receding-horizon replanning capabilities of DMPC. However, ensuring secure control within such a learning framework under malicious cyber attacks, particularly stealthy ones, remains a critical challenge, because the distributed policies generation depends on information exchange among neighbors, where compromised agents can rapidly influence the behavior of others through the communication network. This article proposes a distributed secure learning control (DSLC) framework for large-scale MRS under malicious, stealthy actuator attacks. Our framework offers two key features: (i) a unified approach that enables secure learning control across various coordination scenarios and (ii) a game-theoretic distributed learning-based predictive control strategy that learns how to balance the attacker and defender through a differential-game based DMPC framework. Specifically, DSLC employs a distributed attacker-actor-critic architecture to learn the optimal defense and attack policies online within each prediction interval. Unlike numerical optimization-based controllers that calculate open-loop control sequences, our method simultaneously generates adversarial attack policies and corresponding defense policies in analytical closed-loop form. The defense policies could be directly generalized to MRS with varying scales and diverse actuator attack probabilities. The effectiveness and scalability of DSLC are validated through comprehensive simulations and real-world experiments in multiple wheeled robots via various control tasks.

I. INTRODUCTION

The introduction frames secure performance optimization under stealthy actuator attacks as an unresolved challenge for distributed multirobot control. It presents DSLC as a unified learning-based predictive-control framework and situates it against limitations in RL, DMPC, and secure-control approaches.

B. Distributed MPC for MRS

  • Traditional DMPC relies on nonlinear dynamics knowledge and repeated online numerical optimization, creating deployment challenges for large teams on resource-constrained robots.Prior RL-based closed-loop DMPC policies improve computational efficiency but are limited to a specific formation task and do not address attacks or external disturbances.Source: §I · p. 1, B. Distributed MPC for MRS · p. 3

I. INTRODUCTION

  • Stealthy deception attacks can manipulate actuator signals while evading anomaly- or residual-based detection, and existing secure methods leave real-time cooperative performance optimization unresolved.The introduction contrasts these attacks with denial-of-service attacks, which disrupt communication and are described as relatively easier to detect.Source: §I · pp. 1–2
  • DSLC targets large-scale multirobot control under stealthy actuator attacks, combining distributed secure learning with cooperative performance optimization across formation and containment tasks.The framework is presented as generalizable across general formation, affine formation, and containment scenarios.Source: §I · p. 2
  • DSLC models attackers and defenders as adversarial players in a finite-horizon differential game and learns their policies online in analytic closed-loop form.The approach decomposes the optimization into smaller subproblems and uses a forward-in-time iteration to support deployment across robot-team sizes and attack probabilities.Source: §I · p. 2

A. RL for Multirobot Control

  • Prior RL approaches for multirobot control generally lack online learning and formal stability guarantees, and the cited works do not address malicious attacks within distributed learning control.The paper positions DSLC as addressing performance optimization with theoretical guarantees under adversarial cyber threats.Source: A. RL for Multirobot Control · p. 3

C. Secure Control for MRS under Attacks

  • Existing secure-control methods for denial-of-service and deception attacks mainly emphasize stability and robustness, while attack-time performance optimization remains an identified gap.The paper characterizes cooperative optimal control under malicious stealthy attacks as its response to that gap.Source: C. Secure Control for MRS under Attacks · p. 3

III. PROBLEM FORMULATION

The problem formulation models robot coordination errors under actuator attacks and poses secure control as a finite-horizon defender-attacker game within continuous-time DMPC. Its setup covers multiple coordination scenarios and uses receding-horizon optimization.

III. PROBLEM FORMULATION

  • The formulation introduces continuous-time kinematic models for multirobot systems and a differential-game DMPC problem for control under malicious actuator attacks.The local robot model uses position, heading, velocity, acceleration, and yaw-rate quantities.Source: §III · p. 3, A. Kinematic Models of MRS under… · p. 4

A. Kinematic Models of MRS under Malicious Attacks

  • A unified consensus-error state is defined for general formation, affine formation, and containment control using neighboring robot and leader information.The notation specifies neighbor sets and, for the relevant coordination cases, leader-state and Laplacian information.Source: A. Kinematic Models of MRS under… · p. 4
  • Actuator attacks enter each robot’s input channel as an added signal gated by a Bernoulli variable, with attack probability βi.The combined applied input is the defender input plus the gated malicious signal.Source: A. Kinematic Models of MRS under… · p. 4
  • The attack model focuses on stealthy actuator signals that can preserve nominal dynamical consistency and use smooth profiles to avoid detection.The signals are not assumed to be limited to low-magnitude noise; the proposed approach optimizes defense against a worst-case attack strategy.Source: A. Kinematic Models of MRS under… · p. 4
  • The stability analysis assumes a structured stabilizing feedback policy exists for the overall system.The local policies use neighboring states.Source: A. Kinematic Models of MRS under… · p. 4

B. Differential-game based DMPC Problem

  • At each decision time, the defender minimizes and the attacker maximizes a finite-horizon cost in the continuous-time DMPC formulation.The formulation includes a prediction horizon and a positive-definite terminal penalty matrix.Source: B. Differential-game based DMPC Problem · pp. 4–5
  • Traditional game-theoretic DMPC repeatedly solves nonlinear numerical optimization problems, which is computationally intensive for large-scale systems.Only the actions for the implementing interval are applied before the optimization is repeated at the next time.Source: B. Differential-game based DMPC Problem · p. 5

IV. DISTRIBUTED SECURE LEARNING CONTROL FRAMEWORK

DSLC learns analytic closed-loop defense and worst-case attack policies in a distributed, forward-in-time process. Its policy-learning formulation uses local information and a terminal penalty designed to support closed-loop stability.

IV. DISTRIBUTED SECURE LEARNING CONTROL FRAMEWORK

  • DSLC learns analytic closed-loop defense policies and worst-case attack policies instead of computing open-loop sequences through numerical optimization.Policy learning proceeds forward in time within each prediction interval, reducing computation compared with backward-in-time dynamic programming.Source: §IV · p. 5

A. Game-theoretic Finite-horizon Distributed Policy Learning

  • Each robot’s local defense and attack policies can be computed using neighboring-state information through their connection to the local cost gradient.The framework uses distributed policy learning for the differential-game DMPC problem.Source: A. Game-theoretic Finite-horizon… · pp. 5–6
  • Policy iteration alternates policy evaluation and improvement to solve the game-theoretic HJI equation within each prediction interval.The integral RL formulation permits evaluation at discrete instants while retaining interval-based prediction accuracy.Source: A. Game-theoretic Finite-horizon… · p. 6
  • The integral formulation decomposes the prediction-interval min-max problem into sequential subproblems solved by heuristic forward-in-time policy iteration.The passage describes this procedure as taking place at successive time steps within the interval.Source: A. Game-theoretic Finite-horizon… · p. 6
  • A terminal penalty matrix is designed with candidate terminal control policies after linearizing the local model around the origin.The passage identifies the terminal penalty as important for stability under receding-horizon optimization.Source: A. Game-theoretic Finite-horizon… · p. 6
  • The terminal penalty condition supports recursive feasibility and closed-loop stability within the distributed policy-iteration framework.The choices of defender and attacker control weights must satisfy the stated condition; the text notes that the attacker weight cannot be arbitrarily large.Source: A. Game-theoretic Finite-horizon… · p. 6

B. Distributed Attacker-Actor-Critic Implementation

Each robot uses a local attacker–actor–critic pair to approximate its value function and attack and defense policies. Continuously updating network weights supports policy generation within each prediction interval with reduced computational load.

B. Distributed Attacker-Actor-Critic Implementation

V. SIMULATION STUDIES

The simulation studies evaluate DSLC under actuator attacks, including tests at scales up to 1,000 robots. The parameter-setting subsection specifies empirical choices and constraints, attack-probability settings, and performance indices.

V. SIMULATION STUDIES

  • Simulations evaluate DSLC under actuator attacks and assess scalability and deployment capability for multirobot systems up to 1,000 robots.Source: §V · p. 7

A. Parameter Setting and Evaluation Criteria

B. Comparative Results

Across simulations and real-robot experiments, DSLC maintains control under actuator attacks and extends policies learned from two robots to systems of up to 1,000 robots. In real-world formation tasks, it outperforms DRHRL across attack probabilities, including when affine formations must navigate obstacles.

B. Comparative Results

  • Under varying attack probabilities, DSLC maintained trajectory tracking as robot numbers varied, while DRHRL and DRMPC showed formation disorder and tracking failure under attack.The comparative simulations also report lower IAE and ITAE for DSLC, and significantly lower Jc in most cases.Source: B. Comparative Results · p. 9

C. Scalability Validation of DSLC under Attacks

  • Policies learned from two robots stabilized systems of 100 and 1,000 robots under varied attacks, demonstrating DSLC’s scalability in simulation.The authors report that training on small systems can reduce computational and deployment complexity.Source: C. Scalability Validation of DSLC… · p. 9, Figure 5 · p. 11

VI. REAL-WORLD EXPERIMENTS

  • Real-world tests used wheeled robots with simulation-learned defense policies implemented at 50 Hz, while local-state attackers injected deception signals into individual actuators.The platform used vision-based localization and a host computer to transmit commands over WiFi 6.Source: §VI · p. 9

A. General Formation Control under Attacks

  • In real-world general-formation tests with four or six robots, DSLC preserved formation and trajectory tracking at β = 0, 0.5, and 1, while DRHRL showed disruptions and collisions under attack.The reported IAE values for DSLC were lower than DRHRL’s at β = 0.5 and β = 1.Source: A. General Formation Control under… · pp. 9–10

B. Affine Formation Control under Attacks

  • In affine-formation experiments, DSLC scaled formations to pass obstacles and outperformed DRHRL, whose robots experienced formation disorder and collisions.DSLC’s IAE values were lower than DRHRL’s at attack probabilities β = 0.5 and β = 1.Source: B. Affine Formation Control under… · pp. 10–12

C. Containment Control under Attacks

In containment experiments under actuator attacks, DSLC guided followers into the leaders’ convex hull while maintaining safe distances, including after another leader joined. DRHRL did not maintain containment under the reported attack conditions.

C. Containment Control under Attacks

  • The containment tests used five real robots and one virtual robot, with another leader joining after 10 seconds in the reported experiment.Experiments covered attack probabilities β = 0.5 and β = 1.Source: C. Containment Control under Attacks · p. 12
  • Under β = 0.5 and β = 1, DSLC maintained containment and safe distances, while DRHRL showed collisions and failed to keep the follower inside the convex hull.When another leader joined, DSLC guided the follower into the updated convex hull; the reported IAE values were lower than DRHRL’s under attack.Source: C. Containment Control under Attacks · p. 13

D. Discussion and Limitations

DSLC is reported to generalize across formation and containment tasks under actuator attacks, with simulations and experiments supporting its scalability. The discussion identifies assumptions and validation boundaries involving communication networks, attack probabilities, and robot platforms.

D. Discussion and Limitations

  • Across formation and containment, DSLC maintained coordinated behavior and reconfigured under attack, while DRHRL failed obstacle avoidance or containment, supporting broad-scenario generalizability.The comparisons cover affine formation at β = 0.5 and β = 1 and containment at β = 0.5.Source: D. Discussion and Limitations · p. 14, Figure 19 · p. 14, Figure 20 · p. 14

VII. CONCLUSIONS

  • The conclusion attributes scalability to optimization decomposition and heuristic policy iteration, with closed-loop defense policies deployable across scales and attack probabilities without retraining.The discussion says this supports real-time distributed optimization on small onboard processors.Source: D. Discussion and Limitations · p. 15, §VII · p. 15

D. Discussion and Limitations

  • The framework assumes a static communication network without delays; switching topologies and closed-loop robustness with delays remain future work.Source: D. Discussion and Limitations · p. 15
  • Policy learning assumes known attack probability β, which may be difficult to estimate; robust control for unknown β ∈[0, 1] remains unresolved.A defense policy designed for one probability can still be effective under mismatched probabilities, as Fig. 26 demonstrates.Source: D. Discussion and Limitations · p. 15
  • Although formulated for kinematic MRS and potentially applicable to ground and aerial robots, experimental validation is limited to wheeled ground robots.Source: D. Discussion and Limitations · p. 15

APPENDIX A

The appendix describes task functions for general formation and states a conditional closed-loop stability result. Its proof uses the value-function decrease to establish asymptotic stability under the stated feasibility and matrix conditions.

A. Functions in Section III-A

  • The appendix describes functions for cooperative control tasks, including general formation, and defines relative-state notation for robot and reference positions.Source: A. Functions in Section III-A · p. 17

B. Closed-loop Stability Analysis

C. Learning Convergence and Practical Stability Verification

Under persistent excitation and bounded neural-network assumptions, the learning analysis establishes uniformly ultimately bounded approximation errors. Because learned policies may be suboptimal, the paper proposes a practical condition for checking closed-loop stability.

C. Learning Convergence and Practical Stability Verification

  • With persistent excitation and Assumption 2, the critic, attacker, and defender approximation-error matrices are uniformly ultimately bounded.Assumption 2 requires bounded network weights, approximation errors, activation functions, and associated gradients on a compact set.Source: C. Learning Convergence and Practical… · p. 18
  • The convergence proof analyzes neural-network learning with a Lyapunov function built from per-robot terms and bounded actor and defender inputs.The stated input bounds account for actuator constraints on real robots.Source: C. Learning Convergence and Practical… · pp. 18–19
  • The practical stability check requires J(t) ≤ Jb(t) at each sampled time t0 + k tc.Jb can be chosen as a monotonically decreasing function with Jb(∞) = 0; the learned policy's cost need not decrease monotonically.Source: C. Learning Convergence and Practical… · p. 19
  • The section also includes trajectory figures for two- and four-robot formations under attacks and six robots with varying attack probabilities.The six-robot figure specifies four different vectors of robot attack probabilities.Source: Figure 23 · p. 19, Figure 24 · p. 20, Figure 25 · p. 20

D. Auxiliary Simulation Results

Auxiliary simulations examine DSLC under varied robot attack probabilities and formation sizes, with further cases reported in Appendix B.

D. Auxiliary Simulation Results

  • Six-robot simulations examine DSLC under diverse attack probabilities across robots.The section points to Fig. 23 for these cases.Source: D. Auxiliary Simulation Results · p. 21
  • Two- and four-robot formation comparisons under attacks complement the six-robot scenario shown in Fig. 4.The comparisons are presented in Figs. 24 and 25.Source: D. Auxiliary Simulation Results · p. 21
  • Appendix B presents additional simulations on deploying defenses across attack probabilities and against alternative destabilizing attackers.The section identifies these as further secure-control cases.Source: D. Auxiliary Simulation Results · p. 21

A. Learned Weights for Two-Robot Formation under β = 0.5, 1

The appendix reports learned two-robot actor and attacker weights, then examines defense-policy transfer, computational scaling, experimental task outcomes, and performance against an unseen destabilizing attack.

A. Learned Weights for Two-Robot Formation under β = 0.5, 1

  • For two-robot formation, the appendix reports learned actor and attacker weights under attack probabilities β = 0.5 and 1.It presents weights separately for each probability case.Source: A. Learned Weights for Two-Robot… · p. 21

B. Auxiliary Results and Analysis

Loading 2609.06896v1…