PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology

Xiao Sizhe, Dong Lijing, Bai Rui, Tan Xin

arXiv:2609.08802v1cs.RO

TL;DR

Cooperative navigation with changing topologies remains difficult when MARL must provide strict safety using limited sensing. This paper combines LiDAR-based CBLF action screening with structured actor–critic learning, and reports zero collisions across arrival-distance configurations alongside improved performance over baselines. Its reported platform deployment and experiments support the framework’s engineering feasibility and cooperative stability under dynamic topologies.

  • Problem

    Cooperative MARL must handle dynamic coordination and strict safety, while reward penalties, global-state assumptions, and abstract constraints do not readily provide physical safety under sensing constraints.

    Source: §I · p. 1

  • Method

    The framework uses LiDAR-based CBLF screening with a value-reconstructed attention actor and a GAT centralized critic to support safe navigation under changing topologies.

    Source: §I · p. 1, C. Attention-based Actor Design · p. 5

  • Results

    The framework outperformed MADDPG, MATD3, and MASAC in training efficiency and final navigation performance, and maintained zero collisions across arrival-distance configurations.

    Source: B. Comparative Result Analysis · p. 6

  • Takeaways & Limitations

    Deployment on physical differential-drive robots is reported as evidence of engineering feasibility, cooperative stability under dynamic topologies, and robust safety.

    Source: §I · p. 1

Abstract

from arXiv · show

This paper presents a graph-based safe multi-agent reinforcement learning (MARL) framework for cooperative navigation with time-varying topology. To address the critical challenge of ensuring safety in environments with sensing constraints, a safety-decoupled mechanism is introduced through a Control Barrier-Like Function (CBLF) action screening layer. This mechanism bridges the gap between discrete LiDAR perception and continuous safety constraints, ensuring that physical safety constraints are strictly satisfied regardless of the learning progress. Building upon this safety foundation, a unified structural architecture is proposed, integrating a attention-based actor and a Graph Attention Network (GAT) centralized critic. The actor utilizes a value vector reconstruction mechanism that explicitly encodes relative geometric relations through a collaborative tracking error matrix, enabling scale-insensitive policy learning under time-varying communication topologies. Meanwhile, the GAT-based critic models evolving interaction structures for accurate global value estimation. The proposed framework is validated on real differential-drive robot platforms, and experimental results demonstrate superior stability and safety in dynamic scenarios with limited fields-of-view.

I. INTRODUCTION

Cooperative navigation needs to handle changing interactions and strict physical safety, challenges that reward-based MARL and common sensing or network assumptions do not fully resolve. The paper proposes a graph-based safe MARL framework combining LiDAR-based action screening with structured actor–critic learning.

I. INTRODUCTION

  • Reward penalties alone do not ensure physical safety, while existing safe MARL often assumes global state or abstract constraints that are difficult to map to real sensors.With local sensing and changing neighbor sets, conventional MLP-based actors and critics can also struggle to encode geometric relations and estimate global values.Source: §I · p. 1
  • The proposed framework combines LiDAR-based CBLF action screening with an attention-based actor and GAT centralized critic for cooperative navigation under time-varying topology.The actor uses value-vector reconstruction to encode relative geometry, while the critic models evolving interaction topologies.Source: §I · p. 1
  • The framework was deployed on differential-drive robot platforms, with the paper reporting cooperative stability under dynamic topologies and robust safety in scenarios.The supplied contribution passage describes real-world deployment and engineering feasibility.Source: §I · p. 1

A. Cooperative Navigation Using Reinforcement Learning

  • Prior cooperative-navigation work includes graph, attention, communication, social-awareness, and hierarchical methods, but many approaches rely on predefined interaction structures or lack structured prior knowledge.The paper positions knowledge-embedded attention and a GAT critic as a response to these limitations under partial observability and dynamic topologies.Source: A. Cooperative Navigation Using… · p. 2

B. Safe MARL

Safe MARL methods can depend on abstract constraints or accurate models that are difficult to apply in perception-constrained settings. This paper motivates LiDAR-based CBLFs, while the problem formulation specifies differential-drive robots navigating with local sensing and limited communication.

B. Safe MARL

  • Existing safe MARL methods use penalty optimization, constraint projection, CBFs, or MPC, but abstract representations and accurate-model assumptions can limit use with realistic perception constraints.The paper proposes LiDAR-based perception with CBLFs to construct geometrically verifiable safety constraints.Source: B. Safe MARL · p. 2

III. PROBLEM FORMULATION

  • The system comprises differential-drive robots with actuator saturation, 180° forward LiDAR discretized into 36 beams, and a time-varying communication topology.Agents exchange state information only when their relative distance is below a predefined communication range.Source: §III · pp. 2–3
  • Agents must reach designated targets before Tmax without collisions; any collision or failure to reach targets by the time limit ends the episode.The architecture figure depicts a centralized GAT critic alongside an actor whose outputs pass through LiDAR-based CBLF screening.Source: §III · p. 2, Figure 1 · p. 3

IV. THE GRAPH-BASED SAFE MULTI-AGENT REINFORCEMENT LEARNING

The framework combines a CBLF-based action-screening layer with structured actor–critic learning for cooperative navigation. Its formulation uses local observations over a distance-dependent, time-varying communication graph and optimizes team navigation while respecting collision and timing constraints.

IV. THE GRAPH-BASED SAFE MULTI-AGENT REINFORCEMENT LEARNING

A. Multi-agent Reinforcement Learning Framework

  • A distance-dependent disk model defines communication edges, producing time-varying topology and local observations that include LiDAR, own kinematics, and knowledge-embedded attention.The joint policy seeks expected discounted cumulative reward while satisfying collision-free requirements within Tmax.Source: A. Multi-agent Reinforcement Learning… · p. 3
  • Agent rewards combine task completion, navigation efficiency, and motion constraints, with task-related feedback based on terminal states and navigation reward encouraging progress and team convergence.Distance-reduction terms include individual, team-average, and team-maximum reductions.Source: A. Multi-agent Reinforcement Learning… · pp. 3–4
  • Obstacle penalties use tiers based on minimum LiDAR range, increasing in penalty as the nearest measured range decreases.Motion penalties suppress unstable rotation and stagnation, including penalties for high angular velocity and combinations of low linear and high angular velocity.Source: A. Multi-agent Reinforcement Learning… · p. 4

B. Geometrically Verifiable Action Screening via LiDAR-based CBLF

The action-screening mechanism transforms the policy’s raw control into a geometrically feasible safe input by checking future occupancy against local LiDAR geometry.

B. Geometrically Verifiable Action Screening via LiDAR-based CBLF

1) Kinematic Trajectory Prediction:

The robot predicts its pose after a defined horizon from a control input using differential-drive kinematics. This prediction supports checking potential spatial occupancy before execution.

1) Kinematic Trajectory Prediction:

2) Geometrically Verifiable Safety Function:

The LiDAR-based CBLF evaluates whether the safety disk around a predicted state fits within the perceived environment. If a policy action is unsafe, a discrete search selects a physically consistent correction within actuator limits.

2) Geometrically Verifiable Safety Function:

  • The LiDAR-based CBLF is a one-step discrete safety filter suited to discrete spatial sampling and a non-differentiable min-projection operator.It guarantees the subsequent discrete state lies in the geometrically verifiable safe set when h(s) ≥0.Source: 2) Geometrically Verifiable Safety… · p. 4
  • The safe passageway requires the entire sector around the predicted state to accommodate the safety radius within boundaries detected by raw LiDAR rays.Source: Figure 2 · p. 4
  • A state is geometrically safe when h(s) ≥0, meaning the safety disk fits within the sensor’s valid field-of-view and clears detected obstacles.The disk is centered at the predicted position, and h(s) measures its boundary’s distance from perceived constraints.Source: 2) Geometrically Verifiable Safety… · p. 4

3) Discrete Safety-Decoupled Optimization:

  • When the policy action has h(s) < 0, discrete search explores candidate actions by discretizing linear velocity and aligning angular velocity with safe LiDAR beams.Search expands iteratively from the nearest beam, respecting actuator limits while prioritizing safety.Source: 3) Discrete Safety-Decoupled… · p. 4

C. Attention-based Actor Design

The actor combines local observations with attention-based coordination features. Its attention uses collaborative tracking errors as values, while a communication-range mask accommodates time-varying topology.

C. Attention-based Actor Design

  • Each agent’s input combines 36-dimensional LiDAR ranges and five-dimensional kinematic states, which are linearly projected into a compact local representation.The representation characterizes individual controllability and obstacle-avoidance constraints.Source: C. Attention-based Actor Design · pp. 4–5
  • Value-reconstructed attention uses state embeddings for queries and keys and relative-pose tracking errors for values, injecting geometric guidance into coordination.Each Eij = (∆xij, ∆yij, ∆θij) represents relative pose; the diagonal is zero.Source: C. Attention-based Actor Design · p. 5
  • A masking matrix accommodates time-varying communication topologies within sensing range Rc when computing structural features through attention.Source: C. Attention-based Actor Design · p. 5
  • The policy concatenates local and structural features to sample each agent’s control action.The passage states that this dual-pathway architecture remains scalable and robust across varying agent counts while maintaining high-precision coordination in dynamic environments.Source: C. Attention-based Actor Design · p. 5

D. GAT-based Critic

The centralized critic uses a graph attention network to represent agents and their interactions. It aggregates neighbor features to produce value estimates for centralized training and global credit assignment.

D. GAT-based Critic

  • The critic represents each agent as a dynamic-graph node with a 44-dimensional feature vector combining LiDAR, self-state, and relative-goal information.The vector includes 36-dimensional LiDAR data, five-dimensional self-state, and three-dimensional relative-goal information.Source: D. GAT-based Critic · p. 5
  • The GAT computes attention coefficients between agents and aggregates neighbor features to capture inter-agent influence.Neighbor sets are defined within communication range Rc.Source: D. GAT-based Critic · p. 5
  • Multi-head attention and shared GAT parameters filter task-relevant swarm structure and produce value estimates for all agents for global credit assignment.Source: D. GAT-based Critic · p. 5

V. SIMULATION AND EXPERIMENTAL RESULTS

The method is evaluated in simulation and on physical robots, with comparisons across arrival distances and component ablations. It outperforms the cited baselines, maintains zero collisions across arrival-distance configurations, and shows a safety–efficiency trade-off as swarm size increases.

A. Evaluation Metrics and Experimental Design

  • Evaluation uses 100 test trials, Success Rate, Collision Rate, and Average Steps, with MAPPO as a baseline and four TurtleBot3 robots for physical validation.The simulation retains the TurtleBot3's non-holonomic drive limits and LiDAR characteristics.Source: A. Evaluation Metrics and Experimental… · p. 5, §V · p. 5
  • Experiments vary arrival distance, ablate action screening, the attention-based actor, and the GAT critic, and test physical robustness under sensor noise and unmodeled dynamics.The training reset procedure leaves agents less robust to sudden obstacles, including leading agents that stop at their goals.Source: A. Evaluation Metrics and Experimental… · pp. 5–6

B. Comparative Result Analysis

  • The proposed method outperforms MADDPG, MATD3, and MASAC in training efficiency and final navigation, and is the only method with zero collisions across arrival-distance configurations.Table I summarizes the baseline comparisons and core-component ablations across three arrival distances.Source: B. Comparative Result Analysis · p. 6, Table I · p. 6
  • When leading agents stop and become unexpected obstacles, CBLF screens actions into a geometrically verifiable safe domain, preserving the minimum safety radius during evasive maneuvers.The trajectory analysis describes the attention-based actor coordinating navigation before this sudden-obstacle challenge.Source: B. Comparative Result Analysis · p. 6
  • Success rate declines from 88.0% to 68% as N increases, while the proposed approach maintains zero collisions across configurations.The paper attributes the decline primarily to congestion near target areas, where the safety layer prevents trailing agents from entering crowded regions.Source: B. Comparative Result Analysis · p. 6, Table II · p. 6

C. Ablation Study Analysis

The ablation analysis links quantitative results with learning-curve trends to examine the CBLF screening layer, attention actor, and GAT critic. Without CBLF screening, collision rates rise and reward learning is volatile despite occasional high rewards.

C. Ablation Study Analysis

  • The ablation study links Table I's quantitative data with Fig. 3's learning-curve trends to examine CBLF screening, the attention-based actor, and the GAT-based critic.Source: C. Ablation Study Analysis · pp. 6–7
  • In the navigation snapshots, stopped leading agents become obstacles for trailing agents, and CBLF screening corrects velocities to support a collision-free path to the remaining targets.Source: Figure 4 · p. 7

1) CBLF Screening Layer:

  • Without CBLF screening, collision rate reaches 45% at the 0.3m arrival distance, while reward learning is volatile despite occasional high rewards.The authors describe CBLF as filtering unsafe actions and providing a physical inductive bias for policy convergence.Source: 1) CBLF Screening Layer · p. 7

2) Attention-based Actor:

The framework combines value-reconstructed attention and a GAT critic with CBLF action screening; the conclusion reports improved performance over MASAC and MATD3, especially with sudden obstacles. Four-robot experiments report smooth navigation and no physical collisions during close-proximity maneuvers.

2) Attention-based Actor:

  • Removing value-reconstructed attention increases average steps and yields stagnant rewards and severe instability after 6000 epochs.The attention mechanism uses the cooperative tracking error matrix Eij to capture spatial sensitivity for precision navigation across dynamic topologies.Source: 2) Attention-based Actor · p. 7

3) GAT-based Critic:

  • Without the GAT critic, learning is flatter and does not reach the high-reward regime within the same training duration; success rates also drop across three arrival distances.The passage attributes the critic's value estimation to adaptive weighting of neighboring influences and reduced gradient variance in the MAPPO update.Source: 3) GAT-based Critic · p. 7

D. Real World Experiments

  • Four TurtleBot3 robots with 360° 2D LiDAR, IMUs, and wheel encoders were tested in an indoor arena using fused onboard-sensor state estimates.The robots started from known fixed positions, enabling real-time position and orientation estimates.Source: D. Real World Experiments · p. 7
  • In the real-world validation, the attention-based actor produced smooth paths amid dense interactions, and CBLF screening prevented collisions during close-proximity midstage crossover.The figure caption reports that the robots reached their targets while maintaining safe inter-agent distance.Source: Figure 5 · p. 7, D. Real World Experiments · pp. 7–8

VI. CONCLUSION

  • The framework significantly outperforms MASAC and MATD3, particularly in environments with sudden obstacles, while reporting zero collisions during learning and execution.The conclusion also identifies large-scale swarm navigation in highly uncertain dynamic environments as future work.Source: §VI · p. 8

Related papers

Loading 2609.08802v1…