Source-linked AI summary
Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, Chuchu Fan
TL;DR
The paper tackles safe multi-agent control when learning-based systems lack formal guarantees and must scale beyond centralized coordination. It jointly learns decentralized control policies and barrier certificates, adding permutation-invariant architectures and online policy refinement. The approach outperforms other methods on safety and task completion, generalizing from 8-agent training to scenarios with up to 1024 agents.
Problem
Learning-based autonomous systems lack formal guarantees and generalization capability, while centralized multi-agent control does not scale effectively.
Method
The framework jointly learns decentralized control policies and control barrier functions, using quantity-permutation invariant architectures and online policy refinement.
Results
The method outperforms previous approaches in safety and task completion and generalizes from 8-agent training to scenarios with up to 1024 agents.
Takeaways & Limitations
Decentralized barrier-certificate learning provides a scalable approach for safe control across unseen scenarios and varying numbers of agents.
Takeaways & Limitations
The generalization bound applies to open-loop training data and does not cover closed-loop testing under distribution shift.
Abstract
from arXiv · showhide
We study the multi-agent safe control problem where agents should avoid collisions to static obstacles and collisions with each other while reaching their goals. Our core idea is to learn the multi-agent control policy jointly with learning the control barrier functions as safety certificates. We propose a novel joint-learning framework that can be implemented in a decentralized fashion, with generalization guarantees for certain function classes. Such a decentralized framework can adapt to an arbitrarily large number of agents. Building upon this framework, we further improve the scalability by incorporating neural network architectures that are invariant to the quantity and permutation of neighboring agents. In addition, we propose a new spontaneous policy refinement method to further enforce the certificate condition during testing. We provide extensive experiments to demonstrate that our method significantly outperforms other leading multi-agent control approaches in terms of maintaining safety and completing original tasks. Our approach also shows exceptional generalization capability in that the control policy can be trained with 8 agents in one scenario, while being used on other scenarios with up to 1024 agents in complex multi-agent environments and dynamics.
1 INTRODUCTION
The paper addresses the scalability and safety limitations of learning-based multi-agent control by jointly learning decentralized control policies and control barrier functions. Experiments show strong safety, task completion, scalability, and generalization across agent counts and environments.
- Learning-based autonomous systems lack formal guarantees and generalization, complicating dependable safety-critical multi-agent control.
- Centralized strategies struggle to scale, while uncoordinated decentralized strategies often lack safety and performance guarantees.
- The framework jointly learns decentralized multi-agent control policies and control barrier functions from data, with generalization error guarantees.
- Policies trained with 8 agents operate with up to 1024 agents while maintaining low collision rates across complex environments and dynamics.
- The method outperforms leading learning-based and traditional planning approaches in task completion and safety.
- Quantity-permutation invariant neural architectures handle changing numbers and orderings of neighboring agents.
2 PRELIMINARIES
The preliminaries define control barrier functions for certifying safety in dynamical systems, then extend the setup to decentralized multi-agent systems using each agent’s local neighborhood observations.
- Control Barrier Functions: A control barrier function defines a safe superlevel set that can keep system states from entering the dangerous set.The safe set is represented by {s | h(s) ≥ 0}, while the dangerous set is S_d.
- Control Barrier Functions: The barrier condition requires ∇_s h · f(s, u) + α(h) ≥ 0 for states satisfying h(s) ≥ 0.Here, α is a strictly increasing class-K function with α(0) = 0.
- Control Barrier Functions: If the initial state lies in the barrier-certified set and the condition holds under policy π, the state remains there for all future time.This establishes that the state does not enter the dangerous set under the policy.
- Multi-Agent Formulation: In the multi-agent formulation, each agent has its own state and dynamics, while the joint system state contains all N agent states.Agent i evolves according to ṡ_i(t) = f_i(s_i(t), u_i(t)).
- Multi-Agent Formulation: Each agent’s safety depends jointly on its state and a local observation containing the states of its neighborhood agents.The observation dimension varies with the number of neighboring agents, so the state-observation space includes variable-sized local inputs.
- Multi-Agent Formulation: The paper seeks decentralized policies π_i(s_i(t), o_i(t)), with one policy per agent and no central coordinating controller.This design is intended to support scaling to very large numbers of agents.
3 LEARNING FRAMEWORK FOR MULTI-AGENT DECENTRALIZED CBF
The framework uses decentralized control barrier functions that depend only on each agent’s local state and observation, allowing shared safety contracts to guarantee multi-agent safety without centralized coordination. It jointly learns control policies and barrier functions from trajectories, with statistical generalization guarantees limited to the open-loop training distribution.
- Decentralized control barrier functions: Decentralized CBFs depend on each agent’s local state and observation rather than the joint state of all agents.This avoids the exponential state-space growth of a centralized barrier function.
- Decentralized control barrier functions: If every agent satisfies the same decentralized CBF contract, the multi-agent system remains safe.The safety guarantee follows because each local barrier remains nonnegative and enforces a minimum neighborhood distance.
- Joint learning formulation: The learning objective jointly selects barrier functions and control policies whose trajectory-based margins satisfy the decentralized CBF conditions.The optimization requires yi(τ_i, h_i, π_i) ≥ γ for every sampled training trajectory and agent.
- Joint learning formulation: Statistical learning theory provides generalization guarantees for unseen open-loop data drawn from the training distribution.The bound depends on function-class richness, the margin γ, and the number of training samples.
- Joint learning formulation: Closed-loop generalization during deployment remains untreated because testing can induce distribution shift from the open-loop training data.The authors leave a systematic closed-loop generalization guarantee for future work.
- Joint learning formulation: The computational graph is decentralized because each agent uses the same per-agent computation.The figure displays only agent i’s graph, with the other agents sharing that structure.
4 SCALABLE LEARNING OF DECENTRALIZED CBF IN PRACTICE
The practical method jointly trains neural controllers and barrier certificates with safety and goal-reaching losses, using quantity-permutation invariant encoders for variable neighborhoods. It further refines actions online when learned CBF conditions are violated, supporting deployment across changing agent populations and scenarios.
- Joint loss functions: The practical learner combines decentralized CBF losses with a goal-reaching loss to train controllers and barrier certificates together.The final objective is L = L_c + ηL_g, with η = 0.1 in the experiments.
- Joint loss functions: The safety loss enforces three CBF conditions, while the goal-reaching term matches the learned action to a reference control input.The reference input may come from classical controllers such as LQR or PID.
- Training procedure: The policy is trained on-policy by repeatedly collecting state-observation pairs under the current system and updating network parameters with gradient descent.The implementation samples 128 pairs per update, using learning rate 10^-3 and weight decay 10^-6.
- Scalable architectures: Quantity-permutation invariant encoders map variable-size, order-independent neighbor observations to fixed-length representations.Row-wise max pooling preserves a fixed feature dimension as the number and ordering of neighboring agents change.
- Spontaneous online policy refinement: Online refinement minimizes CBF-condition violations by perturbing the neural policy’s action during testing.The action increment is initialized at zero and iteratively updated; its norm is penalized to discourage large changes.
- Experimental environments: The experiments use Navigation, Predator-Prey, and Nested-Rings 2D environments to evaluate the practical framework.Navigation and Predator-Prey come from the multi-agent particle environment, while Nested-Rings comes from prior multi-agent control work.
5 EXPERIMENTAL RESULTS
Experiments evaluate safety and task performance across 2D and 3D multi-agent environments, comparing MDBC with learning-based and model-based baselines. MDBC maintains strong safety and reward as agent counts increase, including tests with up to 1024 agents after training with 8.
- Baselines: The evaluation compares MDBC with MAMPS, PIC, MADDPG, and S2M2 across multi-agent control tasks.MAMPS, PIC, and MADDPG are learning-based baselines, while S2M2 is a model-based safe motion planner for drone tasks.
- Evaluation criteria: Safety rate is the primary evaluation criterion, complemented by average reward measuring task completion.Safety rate tracks whether agents remain in the safe set; reaching a goal gives +10 reward, while entering the dangerous set gives -1.
- 2D tasks: 32 agents: MDBC maintains high safety rate and average reward in 2D tasks, while other methods perform much worse.Figure 4 reports results after convergence, averaged over 10 independent trials.
- Safety limitation: MDBC does not guarantee 100% safety because decentralized CBF conditions may fail on some testing observations through generalization error.The paper reports this limitation alongside the strong empirical safety results.
- 3D tasks: 32 agents: MDBC maintains high reward and safety rate in 3D drone tasks, while competing methods perform worse.The 3D environments include Maze and Tunnel, where drones must reach randomly selected targets without collisions.
- Generalization: 8 agents trained: MDBC generalizes across environments and agent counts to tests with up to 1024 agents.Additional agents are added by copying the trained 8 agents’ neural-network parameters.
6 CONCLUSION
The paper learns decentralized control barrier functions jointly with safe multi-agent control policies. Experiments show that the approach scales to arbitrary agent counts and generalizes to unseen, complex environments.
- Approach: The method jointly learns decentralized control policies and control barrier functions as safety certificates.The framework is supported by a theoretical generalization bound and practical techniques for learning.
- Findings: Experiments show that the method outperforms previous methods while scaling to an arbitrary number of agents.The conclusion also reports remarkable generalization to unseen and complex multi-agent environments.
- Findings: The approach demonstrates remarkable generalization capabilities to unseen and complex multi-agent environments.
A PROOF OF PROPOSITION 1
The proof connects local decentralized CBF conditions to global multi-agent safety. It uses the minimum of individual CBFs as a global certificate and relies on invariance of each local safe set.
- Local certificate conditions: The decentralized CBF conditions require each agent’s local barrier derivative to satisfy the certificate inequality.The condition is stated for every local state-observation pair in the safe region.
- Invariance: If the initial local state-observation pairs are safe and the CBF conditions hold, each local barrier remains nonnegative over time.This invariance prevents each agent from entering its dangerous set.
- Global safety: Because every agent remains outside its dangerous set, the whole multi-agent system is safe.The argument applies the local safety result to all agents simultaneously.
- Regularity: The safety argument permits nonsmooth local CBFs whose time derivatives exist almost everywhere, using generalized gradients when needed.The guarantee does not require the derivative to exist everywhere.
- Global certificate: The global CBF is constructed as the minimum of the individual local CBF values.Each local CBF depends on an agent’s projected state and neighborhood observation.
B GENERALIZATION ERROR BOUND OF THE DECENTRALIZED CBF
The framework provides probabilistic generalization guarantees for learned decentralized CBFs and policies under specified function classes. The guarantee applies across all agents, while its practical use depends on aligning training and testing trajectory distributions.
- Generalization guarantee: The error probability ϵ_i measures violations of decentralized CBF conditions for agent i over randomly sampled trajectories, including trajectories not seen during training.This definition treats ϵ_i as the generalization error.
- Generalization guarantee: The bound guarantees that decentralized CBF generalization error is upper bounded for all N agents with probability at least 1 −δ.The proposition assumes bounded outputs and feasibility of the learning problem.
- Generalization guarantee: For Lipschitz parametric or reproducing-kernel Hilbert-space function classes, complexity bounds can yield vanishing generalization errors as sample size increases.The argument relies on further bounding the Rademacher complexity of the relevant function classes.
- Practical implementation: The implementation samples training data online under the current controller and iteratively updates the controller to reduce training–testing distribution shift.The authors state that the distributions become almost negligible in shift at the steady stage, enabling use of the proposition.
C MODEL DYNAMICS
The experiments use distinct dynamic models for 2D ground robots and 3D drones. Ground robots use a double-integrator model for navigation and predator-prey tasks, while the nested-rings task uses a model from prior work.
- Ground robots: 2D ground robots use a double-integrator state s_i = [x_i, y_i, v_x,i, v_y,i] for navigation and predator-prey tasks.The nested-rings task instead uses the model from Rodríguez-Seda et al. (2014).
- Drones: 3D drone experiments use a separate dynamics model specified in the supplementary material.The cited passage introduces the drone dynamics but does not provide the full equations.
D SUPPLEMENTARY EXPERIMENT
Supplementary experiments examine scalability, online policy refinement, and learned CBF behavior. Policies trained with 8 agents generalize to scenarios with up to 1024 agents while maintaining high safety and reward, and online refinement further improves safety.
- Generalization: Safety rate and average reward remain high as the number of agents grows exponentially.The reported result concerns 2D ground-robot experiments omitted from the main paper.
- Generalization: 8-agent training generalizes to testing scenarios with up to 1024 agents in 2D tasks.Figure 8 reports this training and testing scale directly.
- Online policy refinement: Online policy refinement further improves safety, while refinement is required on fewer than 17% of testing steps.The ablation reports promising safety even without refinement and a small proportion of refined steps.
- Learned CBF visualization: The learned CBF is visualized in a two-agent Maze environment using relative distance and relative velocity to interpret collision-avoidance behavior.Positive relative velocity indicates that the agents are approaching each other.