Source-linked AI summary
Decentralized Safe Multi-Agent Reinforcement Learning via Predictive Shielding
Yacine El Yamani, Hanna Krasowski, Elena Vanneaux
TL;DR
Separately trained robots can face safety and performance challenges when deployed together, and reactive shields may limit performance near unseen obstacles. This paper combines decentralized predictive shielding with model-based learning and a communication-free conflict-resolution protocol; it reports better task completion than shielding baselines and faster computation than independent learning approaches.
Problem
Separately trained agents deployed together may face collisions and performance challenges, while common multi-agent RL approaches assume joint training and shared objectives.
Source: §1 · p. 1Method
The framework combines decentralized predictive shields using learned models and local sensing with a communication-free stochastic protocol for resolving symmetric livelocks.
Source: §1 · p. 1Results
The proposed method outperforms shielding baselines in task completion and independent learning approaches in computation time.
Source: §8 · p. 6Takeaways & Limitations
The framework supports safe online adaptation of independently trained agents while minimizing livelocks in shared environments.
Source: §1 · p. 1, §8 · p. 6The prediction horizon must currently be chosen before deployment, and the impact of model mismatch between training and deployment remains unclear.
Source: §8 · p. 6
Abstract
from arXiv · showhide
Environments are increasingly populated by multiple robots performing independent tasks with limited prior knowledge of each other. Deploying such multi-agent systems presents significant challenges. Specifically, shifts in deployment states compared to training data can lead to poor policy performance and compromised safety. While safety shields exist to mitigate these risks, they are typically reactive, which degrades performance near unseen obstacles,and centralized, limiting their scalability. To address this, we propose a decentralized framework that integrates predictive shielding with model-based finite horizon Q-learning. This approach allows agents to safely adapt their pre-trained policies during deployment. Furthermore, to mitigate livelocks in symmetric scenarios, we introduce a communication- free protocol for conflict resolution
1. INTRODUCTION
The paper targets safe deployment of separately trained agents that share an environment without joint training. It combines decentralized predictive shields with a communication-free protocol to preserve safety and address performance loss and symmetric livelocks.
1. INTRODUCTION
- Separately pretrained agents may collide when deployed together, while common multi-agent RL approaches assume joint training and shared objectives.The considered robots perform distinct tasks independently in a shared physical space.Source: §1 · p. 1
- Individual shields are composed using an assume–guarantee paradigm to ensure overall system safety without explicit agent communication.The individual shield replaces unsafe proposed actions with a provably safe backup policy; simple rules such as stopping can degrade task performance.Source: §1 · p. 1
- Predictive shields use a learned environment model and local sensing to optimize behavior several steps ahead, addressing task-performance loss from reactive shields with trivial backups.The approach adapts infinite-horizon RL to static environments and finite-horizon RL to dynamic constraints.Source: §1 · p. 1
- A communication-free stochastic protocol addresses livelocks in symmetric scenarios by randomly having an agent follow its desired path aggressively or yield.The predictive shield also aims to adapt to unseen constraints and reduce inter-agent interference.Source: §1 · p. 1
- The framework equips non-communicating heterogeneous agents with decentralized predictive shields, aiming to improve group reward while preserving hard safety guarantees.The paper also reports scalability, low online computational cost, and robust performance in dense multi-agent environments.Source: §1 · p. 1
2. PRELIMINARIES
The preliminaries frame multi-agent decision-making through Dec-POMDPs and independent Q-learning, then introduce shields that replace unsafe actions with safe backups. They also state a deterministic-transition simplification for the analysis.
2. PRELIMINARIES
- A Dec-POMDP represents multi-agent decision-making under uncertainty and partial observability, with agent, state, action, reward, observation, transition, and discount components.The transition function is assumed deterministic here to ease explanation and analysis, though the approach could extend to probabilistic settings.Source: §2 · p. 2
- Independent Q-learning gives each agent its own Q-table and simultaneous ε-greedy action selection, but it does not always converge.It requires no coordination or communication and scales well with the number of agents.Source: §2 · p. 2
- Safety shields accept safe agent actions and replace unsafe ones with a precomputed, provably safe backup policy.A safety specification requires the system state to remain in the safe set at every execution time.Source: §2 · p. 2
- A decentralized post-posed shield ensures safety for a winning region when every set in its sequence remains within the safe set.The preliminaries introduce a sequence of sets for this safety condition.Source: §2 · p. 2
3. PROBLEM STATEMENT
The problem is safe, performant deployment of agents trained independently for separate tasks, then run simultaneously without communication. The objective combines safety constraints with improved cumulative reward across agents.
3. PROBLEM STATEMENT
- Each agent is trained independently in an environment without other agents, using a finite discrete MDP with position and task-related state features.The position subspace is relevant to checking safety because conflicts arise there.Source: §3 · p. 2
- Training yields an optimal value-action function and a sample-based model containing the agent’s observed transitions.The deployment setting puts all agents together in shared state and action spaces without communication; each uses local observations, including nearby agents within sensing range.Source: §3 · p. 2
- The deployment transition functions are assumed unchanged from training, and task rewards remain relevant because the agents’ tasks are assumed unchanged.The paper also assumes a decentralized communication-free safety shield, while leaving its design outside the problem statement’s scope.Source: §3 · pp. 2–3
- The objective is safe and performant deployment, improving cumulative reward across agents because shielded pretrained policies need not maximize that reward.The shield is assumed to guarantee safety during deployment.Source: §3 · pp. 2–3
- A joint state is safe only when every agent state is safe, with unsafe positions defined by obstacles or agents occupying the same position.The agents must remain in the safe set throughout execution.Source: §3 · p. 2
4. MODEL PREDICTIVE SHIELD
The predictive shield locally replans within each agent’s observability range, using separate procedures for static and dynamic safety constraints. Model-based finite-horizon independent Q-learning handles dynamic obstacles while the integrated architecture separates static and dynamic checks to reduce unnecessary computation.
4.1 Predictive shield for static safety constraints
- For static constraints, agents simulate reachable states within a local planning region, with the horizon kstat limited by visibility.Model-based Q-learning updates values for safe sampled transitions and applies a backup policy with a penalty when an action is unsafe.Source: §4.1 · p. 3
4.2 Predictive shield for dynamic obstacles
- For dynamic obstacles, agents use time-dependent Q-values over a finite horizon and maximize worst-case reward because they cannot predict other agents’ trajectories.The dynamic horizon kd is no greater than the static horizon kstat; only agents within the 2kd observability range are considered.Source: §4.2 · p. 3
4.3 Model-Based Finite-Horizon Independent Q-Learning
- MB-FH-IQL simulates an agent’s own evolution with its internal model while assuming worst-case evolution of other agents.The integrated architecture separates static and dynamic constraints because infinite-horizon learning converges faster in stationary settings and distinct horizons can reduce computation.Source: §4.3 · pp. 3–4
- The full shield checks for static violations within kstat and dynamic violations within kd, running finite-horizon learning only when dynamic violations are predicted.The static Q-table is used as the terminal cost for the dynamic procedure.Source: §4.3 · p. 4
5. COMMUNICATION-FREE CONFLICT RESOLUTION PROTOCOL
The protocol addresses livelocks that can remain in decentralized, densely occupied or symmetric configurations. When the nominal action is predicted unsafe, it probabilistically chooses between following the nominal policy and a conservative two-step sequence that discourages reversals.
5. COMMUNICATION-FREE CONFLICT RESOLUTION PROTOCOL
- The protocol is introduced to mitigate livelocks that may arise in densely occupied or symmetric configurations despite predictive shielding.Its conservative branch temporarily deviates from the goal and discourages an immediate return to the previous state.Source: §5 · p. 4
- When the nominal action is predicted unsafe, the protocol chooses between two consecutive nominal-policy steps and a conservative two-step coordination sequence.The nominal-action trigger is based on the static single-agent Q-table.Source: §5 · p. 4
- A logarithmically decaying switching probability based on coordination count differentiates agents in similar configurations without communication or role assignment.The count measures how frequently an agent has been involved in coordination situations.Source: §5 · p. 4
- The conservative branch excludes the nominal action on its first step, encouraging an alternative safe action that can break symmetry.Its constrained action selection excludes designated actions from the safe-action set.Source: §5 · p. 4
- On the second step, the protocol excludes actions that would immediately return the agent to its previous state, reducing short oscillatory behavior.The two-step deviation temporarily guides the agent toward an intermediate objective.Source: §5 · p. 4
6. DEC-POMDP ALGORITHM
The control architecture uses predictive shielding as its primary layer to filter actions and mitigate dynamic interference while preserving goal-directed behavior. When the nominal policy is infeasible in crowded or symmetric situations, it switches to a stochastic conflict-resolution strategy while retaining a guarantee that unsafe actions are never executed.
6. DEC-POMDP ALGORITHM
- When the nominal policy is infeasible in a crowded or symmetric situation, the architecture switches to the Conflict Resolution Protocol.The protocol uses stochastic switching between a conservative multi-agent policy and the nominal trajectory to minimize livelocks.Source: §6 · p. 5
- The flowchart depicts algorithms and their executions per time step, with the predictive shield filtering actions as the primary control layer.The shield is described as mitigating dynamic interference while preserving goal-directed behavior.Source: §6 · p. 4, Figure 1 · p. 5
7. EXPERIMENTS
Experiments compare the proposed method with independent learning and shielding baselines, then examine how prediction horizon performs across free-space and static-obstacle settings. A mid-length dynamic horizon is most effective with static obstacles, while longer horizons can reduce performance in dense settings.
7. EXPERIMENTS
- The benchmark compares IQL, MIS, DMPS, and Dyna-Q Shield with the proposed method across three canonical scenarios, using decentralized safety constraints for shielding methods.The listed comparison table covers horizon, pretraining time, and deployment time; its caption does not provide result values.Source: §7 · p. 5, Table 1 · p. 6
- On a 10 × 10 multi-agent path-finding grid, across 45 random initializations, increasing agent density in free space or static-obstacle density with four agents (kstat = 7) generally reduced the fraction reaching goals.Source: §7 · p. 5
- With static obstacles, k_dyna = 3 is most effective; in free space, k_dyna = 1 has a slight edge and lower computational complexity.The authors attribute drops with excessively long horizons to accumulated uncertainty, which can cause freezing or excessive path deviations.Source: Figure 3 · p. 5, §7 · p. 5
- Across 45 random initializations, tests vary agent density in free space and static-obstacle density with four agents and k_stat = 7.Increasing either agent density or static-obstacle density generally lowers the fraction of agents reaching their goals.Source: §7 · p. 5
8. CONCLUSION
The paper concludes that its decentralized framework combines predictive shielding with communication-free conflict resolution for safe deployment of independently trained agents. Experiments report better task completion than shielding baselines and lower computation time than independent learning approaches, while several deployment choices and assumptions remain open.
8. CONCLUSION
- Experiments report better task completion than shielding baselines and lower computation time than independent learning approaches.The conclusion provides no numerical effect sizes in the supplied passage.Source: §8 · p. 6
- The framework combines predictive safety shielding and a communication-free conflict-resolution protocol to adapt policies online while guaranteeing safety and minimizing livelocks.Source: §8 · p. 6
- The prediction horizon must be selected before deployment, worst-case estimation makes the method conservative, and model mismatch effects remain unclear.The authors identify adaptive horizon selection and further investigation of model mismatch as future work.Source: §8 · p. 6