Source-linked AI summary
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, Yaodong Yang
TL;DR
The paper asks how cooperative MARL can be modeled with sequence models to address the difficulty of joint multi-agent policy optimization. It introduces MAT, an encoder-decoder architecture using advantage decomposition and sequential policy search, trained online rather than from demonstrations. Across several benchmarks, MAT reports superior performance and data efficiency to MAPPO and HAPPO, with strong few-shot generalization on unseen tasks despite changes in agent number.
Problem
Cooperative MARL must coordinate individual policy improvements and joint updates, while prior sequence-model RL methods such as Decision Transformer use pre-collected offline data.
Method
MAT uses an encoder-decoder sequence model and the multi-agent advantage decomposition theorem to turn joint policy optimization into sequential policy search, trained online in an on-policy fashion.
Results
MAT achieves superior performance and data efficiency to MAPPO and HAPPO across StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football, with strong few-shot generalization on unseen tasks.
Takeaways & Limitations
MAT provides a cooperative MARL solution with linear complexity, monotonic performance improvement, and parallel policy learning during training.
Takeaways & Limitations
Parameter sharing can produce exponentially worse suboptimal outcomes, while HAPPO’s sequential update scheme cannot run in parallel.
Abstract
from arXiv · showhide
Large sequence model (SM) such as GPT series and BERT has displayed outstanding performance and generalization capabilities on vision, language, and recently reinforcement learning tasks. A natural follow-up question is how to abstract multi-agent decision making into an SM problem and benefit from the prosperous development of SMs. In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the task is to map agents' observation sequence to agents' optimal action sequence. Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL. Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multi-agent problems and, most importantly, endows MAT with monotonic performance improvement guarantee. Unlike prior arts such as Decision Transformer fit only pre-collected offline data, MAT is trained by online trials and errors from the environment in an on-policy fashion. To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks. Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and HAPPO. Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents. See our project page at https://sites.google.com/view/multi-agent-transformer.
1 Introduction
MARL remains difficult because agents must improve individual policies while coordinating joint updates for team benefit. MAT reframes cooperative MARL as sequence modeling, using an encoder-decoder architecture and sequential policy search to provide efficient online learning with a monotonic improvement guarantee.
- CTDE methods ease coordination by allowing agents to access global information and opponents’ actions during training.
- Existing methods such as COMA, MADDPG, QMIX, MAPPO, PR2, and GR2 extend centralized critics, policy gradients, value decomposition, trust regions, or recursive reasoning to MARL.
- Sequence models have achieved strong performance and few-shot generalization across language and other sequential tasks, motivating their application beyond NLP.
- Transformer-based offline RL methods show potential for long-horizon credit assignment and reward sparsity, but Decision Transformer relies on pre-collected offline data.
- MAT transforms cooperative MARL joint policy optimization into sequential policy search with an encoder-decoder model trained online, yielding linear complexity and a monotonic performance improvement guarantee.
2 Preliminaries
This section formulates cooperative MARL, motivates multi-agent advantage decomposition for credit assignment, and reviews MARL and Transformer-based approaches. It contrasts simultaneous action selection with sequential updates that enable agent-wise policy optimization.
- 2.1 Problem Formulation: Cooperative MARL models agents acting from local observations while receiving a shared joint reward under a Markov-game formulation.The joint observation and action spaces combine the corresponding spaces of all agents, and the transition kernel governs subsequent observations.
- 2.2 Multi-Agent Advantage Decomposition Theorem: Shared rewards create credit-assignment difficulties because individual agents cannot deduce their contributions to team success or failure.This difficulty contributes to high-variance multi-agent policy-gradient estimates and motivates local value functions and counterfactual baselines.
- 2.2 Multi-Agent Advantage Decomposition Theorem: The multi-agent advantage decomposition theorem decomposes joint action advantages across an ordered permutation of agents without requiring additional assumptions.It supports analyzing interactions and decomposing the joint value signal into agent- and predecessor-conditioned contributions.
- 2.2 Multi-Agent Advantage Decomposition Theorem: Sequentially selecting actions with positive agent-wise advantages yields a positive joint advantage while searching action spaces additively rather than multiplicatively.The resulting search complexity is Pn i=1 |Ai| instead of the joint action-space size |A| = Qn i=1 |Ai|.
- 2.3 Existing Methods in MARL: MAPPO shares parameters across agents, whereas HAPPO updates heterogeneous-agent policies sequentially using the advantage decomposition theorem and a monotonic improvement guarantee.Parameter sharing can impose θi = θj and lead to exponentially-worse sub-optimal outcomes; HAPPO’s sequential updates cannot run in parallel.
- 2.4 The Transformer Model: Transformers use encoder-decoder sequence modeling with autoregressive decoding and attention, motivating sequential representations for multi-agent interactions.Existing MARL work applies Transformer-based representations, while the sequential paradigm orders agents so later decisions account for preceding actions.
3 The Surprising Connection Between MARL and Sequence Models
MAT reframes cooperative MARL as sequential decision making by decomposing joint advantages and modeling agents’ observations and actions with a Transformer. This yields linear complexity, parallel policy updates, flexible agent handling, and the MAT encoder-decoder architecture.
- Theorem-based sequentialization: The multi-agent advantage decomposition theorem makes maximizing each agent’s local advantage equivalent to maximizing the joint advantage when agents observe predecessors’ actions.This motivates assigning agents an arbitrary decision order and selecting each agent’s action sequentially.
- Computational benefits: Sequence modeling changes MARL complexity growth with the number of agents from multiplicative to additive, yielding linear complexity.
- Computational benefits: Transformer attention allows ground-truth actions to be batched for simultaneous policy updates, improving training speed compared with one-by-one updates.This addresses the computational cost of sequential policy updates required for monotonic improvement in HAPPO.
- Flexible agent modeling: Sequence models provide a unified solution for heterogeneous agents and varying agent numbers through flexible sequence lengths rather than treating each configuration as a separate task.
- MAT architecture: MAT implements this paradigm with an encoder-decoder architecture that maps agent observation sequences to sequential, autoregressive action predictions.Masked attention restricts each agent to accessing only preceding agents’ actions during training.
4 The Multi-Agent Transformer
MAT uses an encoder-decoder Transformer to map joint observations to agent actions while preserving sequential dependencies among agents. Its training combines value-function approximation, masked autoregressive decoding, and parallelizable clipped objectives.
- Encoder: MAT encodes joint observations with self-attention blocks that represent agent information and high-level interrelationships among agents.Residual connections and MLPs are included in each encoder block.
- Encoder: The encoder is trained to approximate value functions by minimizing empirical Bellman error, using a target network updated every few epochs.The target-network parameters are non-differentiable during optimization.
- Decoder: The decoder uses masked self-attention so each agent’s action depends only on preceding decoded actions, maintaining the sequential update scheme.A second masked attention function links action representations with encoded observation representations.
- Decoder: The decoder outputs each agent’s action distribution from representations of preceding joint actions and encoded observations, trained with a clipped PPO objective.Actions are generated autoregressively during inference but computed in parallel during training from stored preceding actions.
- Optimization: MAT’s clipping objectives can be computed in parallel during training while retaining a monotonic joint-return improvement guarantee.The model avoids waiting for predecessor updates and does not require their updated action distributions for importance sampling.
- Optimization: MAT requires permuting the sequential update order at every iteration to ensure the limiting joint policy has no agent incentivized to change its policy.This requirement is connected to reaching a Nash equilibrium.
5 Experiments and Results
MAT is evaluated across cooperative MARL benchmarks and few-shot transfer settings, with experiments emphasizing its architecture, generalization, and comparisons against established methods.
- Performance on Cooperative MARL Benchmarks: MAT outperforms MAPPO and HAPPO in almost all reported cooperative MARL tasks, covering both homogeneous and heterogeneous-agent settings.The evaluated benchmarks include SMAC, Multi-Agent MuJoCo, Bi-DexHands, and Google Research Football.
- Performance on Cooperative MARL Benchmarks: MAT also outperforms MAT-Dec, highlighting the importance of its decoder architecture.MAT-Dec uses decentralized actors while keeping the encoder fixed for comparison with CTDE methods.
- MAT as Excellent Few-shot Learners: MAT achieves strong zero-shot and few-shot generalization on unseen SMAC and Multi-Agent MuJoCo tasks after pre-training.The evaluation varies the amount of online data and compares pre-trained MAT with MAT trained from scratch.
- MAT as Excellent Few-shot Learners: 10% data few-shot MAT achieves higher rewards than a 100% data model trained from scratch on HalfCheetah with the same disabled joints.The authors attribute this result to pre-training providing more favorable initial weights and reducing the likelihood of bad local optima.
6 Conclusion
The paper connects cooperative MARL with generic sequence modeling through MAT, a Transformer-like encoder-decoder that sequentializes joint policy optimization. Experiments report strong performance against MAPPO and HAPPO, while the authors identify multimodal sequence models as a future direction.
- Conclusion: MAT unifies cooperative MARL in a Transformer-like encoder-decoder model using advantage decomposition to turn joint optimization into sequential decision making.The resulting autoregressive formulation establishes the paper’s central connection between MARL and sequence modeling.
- Conclusion: MAT demonstrates strong empirical performance on three challenging benchmarks against state-of-the-art MARL methods including MAPPO and HAPPO.
- Conclusion: The authors propose extending multi-agent learning to large multimodal sequence models in future work.This direction is motivated by the established connection between MARL and generic sequence models.
A Algorithm Details
Algorithm 1 alternates environment interaction with encoder-decoder training. Observations are encoded, actions are generated autoregressively, rewards are stored, and both networks are updated from sampled experience.
- Algorithm Initialization: The algorithm initializes encoder and decoder parameters, a replay buffer, the step size, batch size, agent count, episode count, and episode length.
- Inference Phase: During inference, the encoder maps the agents’ observations into representations that are passed to the decoder.
- Inference Phase: The decoder generates agents’ actions autoregressively, using preceding actions while iterating through the agents.
- Environment Interaction: The procedure executes joint actions, collects rewards, and inserts observation-action-reward transitions into the replay buffer.
- Training Phase: Training samples minibatches, computes encoder values and the joint advantage with GAE, evaluates encoder and decoder losses, and updates both networks by gradient descent.
B Hyper-parameter Settings for Experiments
The experiments use documented hyper-parameter settings for each algorithm and benchmark domain, with baseline implementations and selected PPO settings standardized for comparison.
- General Settings: Baseline implementations follow their official repositories, with hyper-parameters retained at their original best-performing settings.
- General Settings: PPO epochs and PPO clip are unified to 10 and 0.05 across SMAC scenarios for pre-training and few-shot fine-tuning.
- SMAC: The SMAC domain provides common settings for MAT, MAT-Dec, MAPPO, and HAPPO alongside separate settings for MAT variants and baseline methods.
- Other Domains: Multi-Agent MuJoCo, Bi-DexHands, and Google Research Football each have documented common or domain-specific hyper-parameter tables.
C Details of Experimental Results
MAT outperforms baseline methods across homogeneous and heterogeneous multi-agent benchmarks, including SMAC, Google Research Football, and Multi-Agent MuJoCo. It also shows few-shot generalization and improved computational efficiency in larger-agent settings.
- MAT significantly outperforms baseline methods on HalfCheetah tasks with disabled joints, emphasizing capability for heterogeneous-agent tasks.
- MAT consistently outperforms rival methods on SMAC, indicating strong modeling capability for homogeneous-agent tasks.
- MAT outperforms MAPPO and HAPPO on Google Research Football tasks with 2–4 agents.
- MAT exhibits powerful few-shot generalization on multi-agent MuJoCo tasks when parts of the robot fail.
- Sequence-modeling methods MAT and MAT-Dec achieve superior few-shot performance over MAPPO on SMAC tasks.
- MAT has better computational efficiency than HAPPO in scenarios with 10M environment steps, especially with larger numbers of agents.
D Ablation studies
The ablations examine agent identity and architectural components in MAT. Results support one-hot agent-id encoding and the complete encoder-decoder Transformer architecture.
- Observation encoding: Agent-id encoding improves the representation choice for settings where agents are shuffled at every iteration.The ablation compares one-hot agent identifiers with sequence-position encodings; Figure 14 confirms the implementation choice.
- Model architecture: The complete Transformer architecture achieves the best performance across homogeneous and heterogeneous scenarios.The comparison includes encoder-decoder, encoder-only, decoder-only, and GRU implementations.
- Model architecture: The ablations emphasize the advantage of the Transformer and the necessity of encoder-decoder architectures.