PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning

Hongrui Zheng, Aditya Singh, Rahul Mangharam

arXiv:2610.13737v1cs.LGcs.AI

TL;DR

Shared team constraints measure violations without identifying which agents’ actions influence them. SafeDiSTL uses STL robustness gradients and differentiable dynamics to allocate running-maximum cost increments across agents and decision times. Across three case studies, it shows strong improvement in handling shared constraints over other state-of-the-art MARL baselines.

  • Problem

    Shared team-level violation costs measure constraint outcomes but do not distinguish which agents’ actions most influence those violations.

    Source: §1 · p. 2

  • Method

    SafeDiSTL uses STL robustness gradients through differentiable dynamics to allocate shared running-maximum cost increments across agents at their original decision times.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    Across payload transport, connected navigation, and networked traffic control, SafeDiSTL shows strong improvement in handling shared constraints over other state-of-the-art MARL baselines.

    Source: §7 · pp. 12–13

  • Takeaways & Limitations

    The framework provides agent-specific learning signals for shared constraints, addressing the need for explicit policy updates when responsibility is asymmetric.

    Source: §1 · p. 2

    SafeDiSTL relies on robustness gradients through system dynamics, which may be hard to obtain in contact-rich scenarios, and the paper does not benchmark higher-fidelity robotic scenarios or policy transfer to real robots.

    Source: §7 · p. 13

Abstract

from arXiv · show

Cooperative multi-agent tasks often impose constraints on a shared objective or collective safety. A team level violation cost function can measure this constraint outcome, but does not distinguish which agents' actions most strongly influence that violation. We present SafeDiSTL, a constrained multi-agent reinforcement learning framework that allows for credit assignment even with joint constraint measurements. We use Signal Temporal Logic (STL) robustness to measure episode-level violation through running-maximum cost increments. Smooth full-episode robustness and differentiable dynamics provide action sensitivities, which allocate each cost increment across agents at the same decision time. These sensitivities account for delayed effects when weighting agents' shares of each cost increment. We study the framework in cooperative payload transport, connected navigation, and networked traffic control tasks. SafeDiSTL achieves 1.6-5.1x the safe-success rate in transport, improves navigation safe-success by 11.1%, and delivers 50-68% more throughput in traffic when compared to constrained baselines. Results demonstrate a strong improvement of the learned policy in dealing with shared constraint while retaining the performance of unconstrained policies.

1 Introduction

Shared team constraints measure collective violations but do not identify which agents’ actions drive them, making uniform penalties a poor fit for asymmetric responsibilities. SafeDiSTL uses STL robustness gradients to allocate shared costs to agents and decision times, then incorporates those costs into decentralized policy optimization.

1 Introduction

  • A shared violation cost alone does not indicate which agents should update their policies, and uniform penalties can hinder task learning when agents have asymmetric responsibilities.In the payload example, equal responsibility can cause deadlock despite different desired agent updates.Source: §1 · p. 2
  • STL robustness gradients through differentiable dynamics capture how individual actions influence later constraint outcomes and allocate cost increments at their original timestep.Cost advantages then propagate learning signals to earlier decisions.Source: §1 · p. 2
  • SafeDiSTL introduces agent- and time-specific attribution of shared team constraints using STL robustness gradients while preserving total cost.It incorporates attributed costs into policy optimization for decentralized execution.Source: §1 · pp. 2–3
  • The paper evaluates SafeDiSTL on cooperative tasks, comparing task performance and constraint satisfaction with other baselines.Source: §1 · p. 3

2 Related Work

Prior work addresses constrained MARL and multi-agent credit assignment through coordinated updates, structured interaction models, and attribution methods. SafeDiSTL focuses on allocating shared constraint costs from trajectory sensitivity rather than allocating reward credit or online safety-filter obligations.

Constrained MARL

  • MACPO and IsCPO coordinate policy updates for shared constraints, while SafeDiSTL explicitly allocates trajectory violation cost according to agent influence.MACPO and MAPPO-Lagrangian use constrained trust-region updates and Lagrangian optimization, respectively; IsCPO updates agents sequentially in random order.Source: §2 · p. 3
  • Ling et al. use aggregate local-state and action counts for scalable constraint credit assignment, whereas SafeDiSTL targets coupled geometric configurations and action-dependent dynamics.Source: §2 · p. 3

Multi-agent Credit Assignment

  • COMA uses counterfactual baselines, QPD integrated gradients, and STAS temporal redistribution plus Shapley attribution for cooperative reward, whereas SafeDiSTL differentiates STL robustness through dynamics to attribute shared constraints.Source: §2 · p. 3

3 Preliminaries

Signal Temporal Logic (STL) expresses trajectory requirements, and its quantitative robustness measures the degree of satisfaction or violation. Smooth robustness approximations and differentiable dynamics enable gradients from trajectory robustness to states and policy actions.

3.1 Signal Temporal Logic

  • STL specifies spatiotemporal requirements over trajectories, including the requirement to always avoid collision between a payload and obstacles.Source: §3.1 · pp. 3–4
  • STL formulas combine predicates with negation, conjunction, and the temporal until operator; eventually and always are defined using these operators.An atomic predicate’s truth depends on a function of the trajectory state.Source: §3.1 · p. 4
  • Robustness assigns a real-valued margin to a trajectory, with positive values indicating satisfaction and negative values indicating violation.Smooth max and min approximations enable automatic differentiation through robustness, and differentiable dynamics extend gradients to policy actions.Source: §3.1 · p. 4

4 Problem Formulation

The problem formulation models decentralized agents with a shared task reward and shared episode cost subject to a budget. Its Lagrangian treatment distinguishes policy updates from the multiplier update, which continues to use the original shared cost.

4.1 Constrained Cooperative Markov Game

  • The cooperative multi-agent MDP is extended to a constrained Markov Game with a nonnegative shared cost and an episode cost budget.Source: §4.1 · p. 4
  • The model uses decentralized agent policies and specifies the joint state, each agent’s observation, and the joint action.Source: §4.1 · p. 4
  • The team objective combines shared task reward and shared cost over trajectories of a finite episode horizon.Source: §4.1 · pp. 4–5

4.2 Lagrangian SafeMARL

  • A Lagrangian formulation trades reward against excess cost using a nonnegative multiplier.The canonical projected primal-dual update ascends in policy parameters and updates the multiplier against the cost budget.Source: §4.2 · p. 5
  • MACPO uses sequential constrained trust-region subproblems, whereas MAPPO-L combines reward and cost advantages in a PPO-style surrogate with nonnegative multiplier updates.Source: §4.2 · p. 5
  • SafeDiSTL attributes shared trajectory cost across agents and decision times for actor updates, while retaining unallocated shared cost for multiplier updates.Source: §4.2 · p. 5

5 SafeDiSTL

SafeDiSTL decomposes shared constraint costs into agent-specific signals using full-episode STL sensitivities. It integrates those signals into constrained policy optimization while retaining the original shared cost for multiplier updates.

5 SafeDiSTL

  • Full-episode STL sensitivities allocate each running-maximum cost increment across agents at the same decision time while preserving the increment and total episode cost.The sensitivity weights reflect effects propagated through the episode, while the cost assigned remains local to each timestep.Source: §5 · p. 5, §5.1 · pp. 6–7

5.1 Decomposing the Shared Constraint

  • Only violations that exceed the previous running maximum add cost, and the undiscounted increments telescope to the episode violation cost.The undiscounted sum preserves peak violation regardless of when it occurs.Source: §5.1 · p. 6
  • At each decision time, agent weights are normalized from full-episode action-sensitivity magnitudes, with uniform weights when all sensitivities vanish.The sensitivity differentiates smooth episode robustness through the dynamics while holding other sampled actions fixed, capturing delayed effects.Source: §5.1 · pp. 6–7

5.2 Constrained Policy Optimization

  • Constrained policy optimization combines shared reward advantages with each agent’s attributed cost advantages, while local cost critics use allocated costs.A consistency penalty encourages the mean local-critic prediction to agree with the centralized cost critic.Source: §5.2 · p. 7
  • The shared multiplier update uses mean exact episode cost against the budget, not an individual agent’s attributed cost.Attribution changes agents’ learning signals, while the dual update retains the original shared constraint.Source: §5.2 · pp. 7–8

6 Experiments

Across cooperative transport, connected navigation, and networked traffic, the experiments compare SafeDiSTL with reward-only and constrained baselines. SafeDiSTL balances constraint satisfaction and task performance across five scenarios, with task-specific results in all three benchmarks.

6.1 Experimental Setup

  • Evaluations use six training seeds, common resets, and equal scenario-specific simulation-step budgets across the three benchmarks.The budgets are 15M steps for transport and navigation and 20M for traffic; SafeDiSTL uses running-maximum increment costs, while other methods use equivalent per-step costs.Source: §6.1 · pp. 8–9, §6 · p. 8

6.2 The Constraint-performance Tradeoff

  • Across all five scenarios, SafeDiSTL occupies the top-right constraint-satisfaction and performance quadrant, while other methods trade away MAPPO performance.The comparison uses constraint-satisfaction metrics against performance normalized to MAPPO.Source: Figure 2 · p. 9, §6.2 · p. 10

6.3 Cooperative Transport

  • SafeDiSTL achieves the highest mean safe-success rates in transport: 35.6 ± 9.6%, 16.2 ± 11.0%, and 3.5 ± 2.3% with one, two, and three obstacles.MAPPO-L is the closest competitor at 7.0 ± 3.7%, 4.5 ± 0.9%, and 2.2 ± 1.0%, respectively; safe success requires reaching the goal without obstacle contact.Source: Figure 3 · p. 9, §6.3 · p. 11

6.4 Connected Navigation

  • In connected navigation, SafeDiSTL has the highest mean goal success (71.88±6.13%) and safe-success (43.88±4.40%); MAPPO-L has higher constraint satisfaction (56.64 ± 10.61% versus 54.69 ± 8.33%).MACPO reaches 4.43 ± 1.96% safe successes, while MAPPO has the highest mean return but almost always violates the constraint.Source: §6.4 · p. 11

6.5 Networked Traffic Control

  • SafeDiSTL retains 71.4% of MAPPO throughput, with core vehicle-hours at 8.8% and time above core limits at 0.3% of MAPPO.MACPO and MAPPO-L retain 47.6% and 42.5% of MAPPO throughput, respectively, while reducing constraint violation to the same level.Source: §6.5 · p. 12

7 Limitations and Conclusions

SafeDiSTL decomposes shared constraints for agent-specific credit assignment, and the paper reports improved handling of shared constraints across three case studies. Its benchmark results rely on robustness gradients through system dynamics, while the appendix details the constrained-policy baselines.

7 Limitations and Conclusions

  • Across three case studies, SafeDiSTL shows strong improvement in handling shared constraints over other state-of-the-art MARL baselines.The framework uses STL robustness gradients to decompose team constraints while preserving total cost.Source: §7 · pp. 12–13
  • SafeDiSTL relies on robustness gradients through system dynamics, which may be difficult to obtain in contact-rich scenarios.The authors suggest finite differences as an option for nondifferentiable dynamics, at the cost of more simulation calls.Source: §7 · p. 13

A Baseline Optimization Details

  • Equation 6 is a first-order Lagrangian template, not the update rule for every constrained policy optimizer.CPO solves its constrained trust-region subproblem multipliers anew at each iteration, unlike the stateful multiplier update.Source: A Baseline Optimization Details · p. 17

A.1 MACPO

  • MACPO updates decentralized actors sequentially, using reward and cost surrogates subject to cost constraints and a KL trust region.Its subproblem multipliers are solved anew for each actor update.Source: §A.1 · p. 17
  • Practical neural-policy updates do not guarantee constraint satisfaction on every trajectory, despite MACPO’s cost-reducing recovery step when an update is infeasible.The stated qualifications include sampling error, local approximations, and finite line searches.Source: §A.1 · p. 17

A.2 MAPPO-Style Lagrangian Surrogates

  • The MAPPO-style surrogate combines reward and shared-cost advantages, while SafeDiSTL substitutes attributed cost advantages for the broadcast shared-cost signal.The experimental MAPPO-L baseline updates independent actors sequentially and resets nonnegative multipliers at each actor’s optimization phase.Source: §A.2 · pp. 17–18
  • Clipping and finite stochastic primal–dual updates do not enforce exact feasibility at every iteration.The first-order implementation also does not inherit CPO’s trust-region bounds or SafeMARL’s theoretical sequential-update guarantees.Source: §A.2 · p. 18

B Supplementary Transport Results

The supplied passages identify supplementary result tables and their relation to figures, but do not provide their numerical outcomes.

B Supplementary Transport Results

  • Table 1 provides the numerical results plotted in Figure 3.The passage does not state the table’s values.Source: B Supplementary Transport Results · p. 18
  • Table 2 contains supplementary navigation results, including safe return and final-20 distance.Its caption defines safe success as joint goal achievement and whole-episode connectivity.Source: Table 2 · p. 19
  • Table 3 reports full traffic metrics relative to MAPPO, with higher completed trips and lower values in other columns preferred.The caption specifies mean ± SD over six seeds and 1,000 common reset scenarios per policy.Source: Table 3 · p. 19

C Supplementary Navigation Results

The supplementary navigation table adds metrics to the Figure 6 results, while the traffic table retains the full metrics shown in Figure 8.

C Supplementary Navigation Results

  • Table 2 provides the numerical results plotted in Figure 6, together with safe return and final-20 distance.The caption defines safe success as requiring both goal achievement and whole-episode connectivity.Source: C Supplementary Navigation Results · p. 19

D Supplementary Traffic Results

E Ablation Studies

The ablations vary attribution, cost formulation, and multiplier updates; the supplied passage describes their design but not their outcomes.

E Ablation Studies

  • The ablations compare uniform attribution with sensitivity-based attribution, max-incremental cost with step cost, and fixed-step with Adam multiplier updates.“Ours” retains sensitivity-based attribution with max-incremental cost.Source: E Ablation Studies · p. 19
  • Max-incremental cost uses the same incremental maximum cost as SafeDiSTL, while step cost uses positive per-transition violation.Labels without “+ Adam” use fixed-step multiplier updates.Source: E Ablation Studies · p. 19
  • Ablation results are presented in Tables 4, 5, and 6, with the “Ours” rows repeated from the main tables.The passage says those rows use the same evaluation protocols and seed-level aggregation.Source: E Ablation Studies · p. 19

F Robustness Differentiation

SafeDiSTL differentiates smoothed full-episode robustness through differentiable dynamics to obtain action sensitivities, while exact costs and satisfaction tests remain unchanged. The resulting attribution has implementation requirements and modeling limits.

F Robustness Differentiation

  • Masked normalization assigns zero weight to inactive entries and falls back to uniform weights when active sensitivities are uninformative; complete episodes are needed to finalize attribution targets.Allocation assumes a nonempty active set at each cost-bearing transition, and nonfinite active derivatives are flagged.Source: F Robustness Differentiation · p. 20
  • Full-episode robustness sensitivities guide cost shares assigned locally at each timestep, with differentiable dynamics providing action derivatives.The derivative norm is taken over each agent’s action coordinates.Source: F Robustness Differentiation · pp. 19, 21
  • Attribution uses sensitivity magnitudes, discarding direction; its weights depend on the model, action parameterization, and smoothing scale.Conserving scalar cost does not make the attributed policy gradient an unbiased estimator of the original shared-cost gradient.Source: F Robustness Differentiation · p. 21
  • The smooth surrogate supplies derivatives only; exact cost and satisfaction tests remain authoritative, while log-sum-exp smooths the always-conjunction whose exact minimum can concentrate derivatives.The smoothing parameter β is positive.Source: F Robustness Differentiation · p. 21

G SafeDiSTL Optimization and Implementation Details

The implementation pairs shared and agent-specific cost learning signals with separate local and shared constraint handling. It specifies task constraints, environments, and policy inputs for transport, navigation, and traffic benchmarks.

G.1 Cost advantages and critic losses

  • The hybrid cost critic uses a centralized predictor for team-mean cost and local predictors for attributed costs; its learned outputs are not an exact decomposition by architecture.GAE uses rollout-time predictions and fixed unnormalized targets during critic fitting, with cost targets ending at completed episodes.Source: §G.1 · pp. 21–22

G.2 Actor normalization and individual constraints; G.3 Shared multiplier update

  • Local obstacle and workspace costs retain separate critics, budgets, and dual updates, while the shared multiplier tracks mean exact unallocated episode cost.The local budgets use active-transition deficits; the shared multiplier has a finite practical cap.Source: §G.2 · p. 22, §G.3 · p. 23

H.1 Collaborative Transport

  • Transport uses three force-controlled agents to move a shared package around obstacles, with package-clearance violations as the shared constraint and package-goal distance defining success.The planar episodes last 500 control steps across randomized obstacle layouts.Source: §H.1 · p. 23

H.2 Connected Navigation

  • Navigation evaluates four agents crossing staggered obstacle barriers while maintaining buffered whole-episode connectivity; safe goal success requires both connectivity checks and the final-20-step goal criterion.The checked bottleneck must remain at or below 3.325 m, and disconnection does not terminate an episode.Source: §H.2 · pp. 23–24

H.3 Networked Traffic Control

  • Traffic control uses six peripheral agents to meter destination-specific flows into a shared core, with the core occupancy limit defining the shared constraint.The task uses 240 control intervals, and its reward is completed trips per interval divided by 100.Source: §H.3 · p. 24

I.1 Cooperative Transport

  • Transport actors observe agent and package state, goal-relative position, obstacles, and teammates; observations have 19+2K dimensions, with shared actors for SafeDiSTL and MAPPO variants.Policies use two 64-unit tanh layers and a two-dimensional Gaussian-mean head; MAPPO-L and MACPO use separate actor parameters per agent.Source: §I.1 · pp. 24–25

I.2 Connected Navigation

  • Navigation actors receive 72-dimensional observations covering teammates, communication packets, obstacles, and corridor boundaries; shared-policy variants use shared actors, unlike MAPPO-L and MACPO.The feed-forward actors use two 64-unit tanh layers and explicit packet contents and ages rather than recurrent state.Source: §I.2 · p. 25

I.3 Networked Traffic Control

  • Traffic actors receive 31 coordinates covering local queues and demand plus broadcast core queues and exit factors; their six-dimensional admission outputs mask the self-destination control.SafeDiSTL and MAPPO variants share one actor across agents, while MAPPO-L and MACPO use independent actors.Source: §I.3 · pp. 25–26
Loading 2610.13737v1…