Source-linked AI summary
Permutation Robustness Is Not Enough: Action Collapse in Multi-Agent Transformer Policies
Amit Thakur, Mukesh Singhal
TL;DR
Multi-agent teams are unordered, but transformer policies process agents as ordered tokens, so the paper examines how order affects cooperative navigation policies. It evaluates permutation consistency together with action-collapse diagnostics and reports that regularization can improve consistency while excessive strength can homogenize actions. The findings support assessing task return, order robustness, and differentiated behavior together.
Problem
Transformer policies process agents as ordered token sequences even though multi-robot teams are exchangeable, and learned identity or position embeddings can tie behavior to token slots.
Source: §1 · p. 1Method
The paper evaluates multi-agent transformer policies using task return, permutation-consistency measures, and action-collapse diagnostics.
Source: §1 · p. 2Results
Takeaways & Limitations
Evaluation should report task return, permutation metrics, and diversity, same-action, and maximum-frequency diagnostics together.
Source: §7 · p. 8, §8 · p. 10Experiments cover simulated cooperative navigation with a centralized actor observing joint observations, and the N = 4 experiments are a seed-0 stress test rather than a multi-seed benchmark.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Transformer policies are attractive for multi-agent robot learning because self-attention can model interactions among agents. However, multi-agent teams are unordered, while transformers typically process agents as ordered token sequences. We study how this mismatch affects cooperative navigation policies under agent-order permutations. Our results show that low permutation error alone can be misleading: policies may appear robust simply because all agents choose the same action. We therefore evaluate policies using both permutation-consistency metrics and action-collapse diagnostics, including action diversity, same-action fraction, and maximum action frequency. A PPO-ID baseline yields non-collapsed behavior but remains order-sensitive, while strong equivariance regularization can still induce homogeneous behavior. A weak equivariance penalty improves the robustness while preserving more diverse actions for teams with \(N=3\) agents, whereas teams with \(N=4\) agents require substantially smaller regularization weights. These findings suggest that multi-agent transformer policies should be evaluated not only by return and permutation robustness, but also by whether they maintain non-collapsed, differentiated multi-agent behavior.
1 Introduction
Multi-robot teams are exchangeable, but transformers process agents as ordered tokens, creating a mismatch that can make policies depend on arbitrary agent order. The paper tests whether policies coordinate robustly or exploit that ordering, and argues permutation consistency must be assessed alongside task performance and action diversity.
1 Introduction
- Learned identity or position embeddings can improve specialization but also attach behavior to token slots rather than physical agents.Robot order may change across communication, batching, logging, dynamic team membership, or deployment interfaces.Source: §1 · pp. 1–2
- The central question is whether transformer-based multi-agent policies learn coordination or agent ordering, since relabeling the same physical team should not change its commands.The desired equivariance is for input shuffles to produce corresponding output shuffles.Source: §1 · pp. 1–2
- Low permutation error alone can be misleading because a policy that gives every agent the same action can appear permutation-consistent.The paper therefore evaluates return, permutation consistency, and non-collapsed action behavior jointly.Source: §1 · p. 2
- The paper formalizes permutation sensitivity with logit, policy-distribution, action, and return metrics and introduces action-diversity, same-action-fraction, and maximum-frequency diagnostics.It reports a non-collapsed PPO-style baseline, a robustness-diversity tradeoff under equivariance regularization, and a four-agent test of team-size dependence.Source: §1 · p. 2
2 Related Work
Prior cooperative MARL work addresses coordination and credit assignment, while transformer and set-architecture work motivates attention and permutation symmetry. This paper focuses on order stability and evaluates behavior beyond return alone.
2 Related Work
- Centralized-training methods such as MADDPG [8], COMA, VDN [15], and QMIX [11] address credit assignment and coordination; PPO-based training is also a practical baseline [19].This paper uses a PPO/MAPPO-style regime to study transformer policies’ stability to agent ordering.Source: §2 · p. 2
- Transformers process token sequences, while learned position or agent-ID embeddings generally break permutation equivariance; set architectures explicitly enforce permutation symmetry.Robot teams have an exchangeability symmetry: changing the list order should not change the physical command assigned to each robot.Source: §2 · p. 2
- Attention has been used for MARL interaction modeling, but return-focused evaluations can miss order sensitivity and same-action collapse.The paper complements return with distributional consistency, executed-action consistency, and action diversity diagnostics.Source: §2 · pp. 2–3
3 Problem Formulation
The paper formulates cooperative multi-agent learning with shared rewards and a policy whose agent actions are sampled separately but conditioned on joint observations. Its transformer maps each observation to a token and produces per-agent action logits, with equivariance specifying how outputs should change under reordered inputs.
3 Problem Formulation
- Agents receive observations in a cooperative partially observable Markov game, take joint actions, and share a reward optimized over trajectories.The notation defines the policy as stochastic and the objective’s expectation over trajectories generated by executing it.Source: §3 · p. 3
- The policy maps each agent observation to a token, adds learned identity and position embeddings, and transforms the sequence into contextualized agent representations.Each representation is used to choose that agent’s action, and a linear actor head produces action logits.Source: §3 · p. 3
- The joint policy factorizes over sampled agent actions, although each action distribution can depend on all agents through transformer attention.Permutation equivariance requires logits to reorder in the same way as the input agents; slot-tied embeddings can violate this condition.Source: §3 · pp. 3–4
4 Diagnostics and Training Objective
The section defines measures of ordering sensitivity and action collapse, then incorporates optional equivariance regularization into PPO-style multi-agent training. Its evaluation combines ordering conditions with performance, robustness, and collapse diagnostics.
4 Diagnostics and Training Objective
- Permutation evaluation compares return under normal, random, and reverse orderings and measures logit, policy, or greedy-action disagreement after outputs are unpermuted.The training method adds equivariance regularization, including an optional pairwise action-distribution term.Source: §4 · pp. 4–5
- Action-collapse diagnostics include normalized action diversity, same-action fraction, and maximum action frequency across agents and timesteps.Complete same-action collapse has normalized diversity Ddiv = 1/N.Source: §4 · p. 4
- PPO-style training uses a centralized transformer actor and critic, with a probability-ratio objective and rollout-based updates.The algorithm collects rollouts, computes GAE advantages and returns, applies PPO clipped and value losses, and optionally evaluates pairwise diversity.Source: §4 · pp. 4–5
- A standard entropy bonus encourages exploration and discourages premature deterministic action collapse, while the centralized critic predicts rollout-return targets.More accurate critic predictions can reduce policy-gradient variance through better advantage estimates.Source: §4 · p. 5
5 Experimental Setup
The experiments use cooperative navigation to distinguish order sensitivity, useful specialization, and homogeneous action collapse. They compare PPO variants under controlled agent-token reorderings and include a four-agent stress test.
5 Experimental Setup
- The cooperative navigation task rewards covering landmarks while avoiding inefficient configurations, and requires behavior conditioned on other agents and landmarks.The task is used to distinguish order sensitivity, useful specialization, and homogeneous action collapse.Source: §5 · p. 5
- Each observation becomes a 128-dimensional token processed by two four-head self-attention blocks with learned agent-ID and sequence-position embeddings.A centralized critic estimates team value; each agent's action distribution is conditioned on all agent observations.Source: §5 · p. 6
- Evaluation compares canonical order with fresh random permutations and reversed order, then unpermutes outputs to the physical-agent order before computing metrics.This isolates ordering sensitivity from changes in physical state.Source: §5 · p. 6
- The three-agent comparison tests PPO-ID, weak and stronger equivariance penalties, and stronger regularization paired with policy-diversity regularization.Models are evaluated over three seeds, with hyperparameters selected jointly for return, permutation error, and non-collapsed action statistics.Source: §5 · p. 6
- The four-agent seed-0 stress test sweeps much smaller regularization coefficients after preliminary runs showed stronger collapse under the three-agent setting's coefficient.The test assesses whether qualitative tradeoffs persist as team size grows.Source: §5 · p. 6
6 Results
Across three- and four-agent experiments, equivariance regularization improves permutation consistency but can reduce action diversity, with its effects sensitive to team size. The results pair robustness measures with action-collapse diagnostics.
6 Results
- PPO-ID has Ddiv = 0.649 ± 0.047 and Fsame = 0.187 ± 0.108, but random action disagreement remains 0.498 ± 0.052.It provides a non-collapsed baseline that remains order-sensitive.Source: §6 · pp. 6–7, Figure 1 · p. 7
- At λeq = 0.01, random equivariance error falls from 0.293 to 0.009 and action disagreement from 0.498 to 0.196, while action diversity falls from 0.649 to 0.425.Same-action fraction rises from 0.187 to 0.739, showing the robustness-diversity tradeoff.Source: Figure 1 · p. 7, §6 · p. 7
- The stronger λeq = 0.05 and Eq+Pair settings have excellent permutation metrics but high same-action fractions, so low equivariance error alone does not establish non-collapsed behavior.Pairwise policy-diversity regularization helps only partially in this setting.Source: §6 · p. 7
- For N = 4, λeq = 5 × 10^-4 gives the best return but more homogeneous behavior, while 2.5 × 10^-4 preserves healthier diversity but has worse return.The stress test reports a return-diversity tradeoff among the weaker coefficients.Source: §6 · p. 7, Figure 2 · p. 7
- Ddiv = 0.25, Fsame = 1, and Fmax = 1 at λeq = 0.01 for N = 4, indicating complete collapse under the coefficient used for N = 3.The four-agent stress test reports this outcome in Table 2 and Figure 2.Source: Figure 2 · p. 7, §6 · p. 7, Table 2 · p. 8
- Figure 3 places desirable policies toward low permutation error and high action diversity, while its lower-left region marks policies that are robust but collapse-prone.The figure frames robustness and diversity as joint evaluation dimensions.Source: Figure 3 · p. 8
7 Discussion and Limitations
The experiments show that equivariance regularization trades order consistency against differentiated behavior, and its useful strength depends on team size. The authors recommend evaluating task utility, permutation consistency, and action collapse together, while considering architectures and objectives that preserve coordination without forcing identical actions.
7 Discussion and Limitations
- Equivariance regularization can improve permutation consistency but overly large weights can push agents toward homogeneous actions, and useful scales do not transfer naively across team sizes.PPO-ID learns differentiated behaviors that can depend on agent order.Source: §7 · p. 8
- A useful evaluation suite combines task return, order-sensitivity metrics, and action-diversity or collapse diagnostics rather than treating symmetry as a single score.The authors select weak regularization because it improves consistency without fully destroying differentiated behavior.Source: §7 · pp. 8–9
- For evaluation, test a nontrivial ordering intervention and report distributional consistency, action disagreement, and a collapse diagnostic.In the experiments, Ddiv = 1/N and Fsame = 1 indicate that all agents choose the same action at every step.Source: §7 · p. 9
- For N = 3, λeq = 0.01 reduces random equivariance error by more than an order of magnitude relative to PPO-ID while preserving moderate action diversity; for N = 4, that coefficient collapses completely.Useful N = 4 values are two orders of magnitude smaller, and still expose a tradeoff between return and non-collapse.Source: §7 · p. 9
- Equivariance is not undesirable, but set pooling or relative geometric attention may provide it by construction without a strong output-level penalty.The authors propose task-aware diversity objectives that encourage spreading over landmarks without forcing different actions when the same motion is appropriate.Source: §7 · p. 9
Limitations
- The experiments are limited to simulated cooperative navigation, use a centralized actor with joint observations, and include only a seed-0 N = 4 stress test.Pairwise action diversity is a simple anti-collapse regularizer; future work should explore task-aware diversity, landmark assignment, relative geometric attention, and equivariant-by-construction policies.Source: Limitations · p. 9
8 Conclusion
Robustness metrics alone can mislead: PPO-ID remains non-collapsed but order-sensitive, while equivariance regularization can reduce permutation error and induce same-action collapse. The paper concludes that evaluation should consider task return, relabeling consistency, and whether agents retain differentiated behavior.
8 Conclusion
- PPO-ID produces non-collapsed behavior but remains order-sensitive, while equivariance regularization reduces permutation error and can induce same-action collapse.The conclusion recommends reporting task return, permutation metrics, and action-collapse diagnostics together.Source: §8 · p. 9
- Identity embeddings can improve specialization yet create slot-specific behavior, while equivariance penalties can remove slot dependence and useful distinctions among agents.The paper frames robust coordination as requiring symmetry-aware modeling and safeguards against trivial invariant behavior.Source: §8 · p. 10
A Appendix
The appendix provides reproducibility details and mathematical explanations while using its tables as guides to interpreting collapse and regularization metrics.
A Appendix
- The appendix focuses on reproducibility and mathematical explanations, with tables serving as interpretation guides for collapse and regularization metrics.Source: A Appendix · p. 12
A.1 Additional Experimental Details
The appendix specifies the cooperative-navigation setup, policy architecture, PPO settings, permutation-evaluation procedure, and regularization coefficients used in the experiments.
A.1 Additional Experimental Details
- The experiments use cooperative navigation with a shared team reward, three agents and landmarks in the main setting, and four of each in the stress test.Each run uses a fixed-horizon discrete-action episode; evaluation covers 200 episodes for each ordering mode.Source: §A.1 · p. 12
- Each agent observation becomes a 128-dimensional token processed by two four-head self-attention blocks, with ID and positional embeddings added.The actor maps final tokens to discrete-action logits, and the critic estimates centralized value from the transformer representation.Source: §A.1 · p. 12
- PPO uses rollout length 1024, four epochs, minibatches of 256, learning rate 3 × 10^-4, γ = 0.99, λ = 0.95, clip ϵ = 0.2, entropy coefficient 0.01, value coefficient 0.5, and gradient norm 0.5.Source: §A.1 · p. 12
- Permutation evaluation compares normal and permuted logits after unpermuting the outputs, and likewise unpermutes executed actions to match commands to physical agents.Source: §A.1 · p. 12
- For N = 3, the selected weak setting is λeq = 0.01, while λeq = 0.05 illustrates collapse-prone behavior; the N = 4 stress test spans λeq values from 2.5 × 10^-4 to 5 × 10^-2.Source: §A.1 · p. 12
A.2 Additional Mathematical Explanation
Self-attention without slot-specific information is permutation equivariant, but identity or positional embeddings tied to sequence slots can break that property. Low equivariance error can also arise from identical actions rather than differentiated coordination, motivating joint reporting of collapse and permutation metrics.
A.2 Additional Mathematical Explanation
- Without slot-specific information, self-attention is permutation equivariant; learned identity or positional embeddings tied to sequence slots can break this property.The explanation traces how permuting tokens transforms attention and notes that fixed slot embeddings do not permute with the input.Source: §A.2 · pp. 12–13
- Identical logits across agents yield zero equivariance error, even when greedy actions are identical at most timesteps and differentiated coordination has not been learned.The passage motivates reporting Ddiv, Fsame, and Fmax alongside return and permutation consistency.Source: §A.2 · p. 13
- At the collapse floor, Ddiv is 1/3 for N = 3 and 1/4 for N = 4 when all agents choose the same action at every timestep.In that case, the number of unique actions is one and Ddiv = 1/N.Source: §A.2 · p. 13
A.3 Additional Tables
- Table 3 is an interpretation guide for collapse metrics.Its caption identifies its scope but does not state specific metric interpretations.Source: Table 3 · p. 13