Source-linked AI summary
Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, Hongyao Tang
TL;DR
Cooperative MARL must learn coordinated decentralized policies despite private observations and limited communication, while existing value decompositions impose assumptions without sufficient theoretical grounding. Qatten derives a general global-to-individual Q-value relation and implements it with multi-head attention, achieving the best performance on almost all StarCraft maps with interpretable attention weights.
Problem
Existing value decomposition methods impose assumptions between Qtot and individual Qi values and do not explicitly model each agent’s impact on the whole system.
Method
Qatten derives a general decomposition formula for Qtot in terms of Qi and uses multi-head attention to model agent-level importance during Q-value mixing.
Results
Qatten obtains the best performance on almost all maps in the StarCraft MARL benchmark, with attention analysis providing intuitive explanations of the weights.
Takeaways & Limitations
Attention-based decomposition can represent agent-specific contributions to global value mixing while supporting decentralized policy optimization.
Takeaways & Limitations
Future work identifies explicit exploration for difficult tasks and adaptation to settings with hundreds of agents as open directions.
Abstract
from arXiv · showhide
In many real-world tasks, multiple agents must learn to coordinate with each other given their private observations and limited communication ability. Deep multiagent reinforcement learning (Deep-MARL) algorithms have shown superior performance in such challenging settings. One representative class of work is multiagent value decomposition, which decomposes the global shared multiagent Q-value $Q_{tot}$ into individual Q-values $Q^{i}$ to guide individuals' behaviors, i.e. VDN imposing an additive formation and QMIX adopting a monotonic assumption using an implicit mixing method. However, most of the previous efforts impose certain assumptions between $Q_{tot}$ and $Q^{i}$ and lack theoretical groundings. Besides, they do not explicitly consider the agent-level impact of individuals to the whole system when transforming individual $Q^{i}$s into $Q_{tot}$. In this paper, we theoretically derive a general formula of $Q_{tot}$ in terms of $Q^{i}$, based on which we can naturally implement a multi-head attention formation to approximate $Q_{tot}$, resulting in not only a refined representation of $Q_{tot}$ with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies. Extensive experiments demonstrate that our method outperforms state-of-the-art MARL methods on the widely adopted StarCraft benchmark across different scenarios, and attention analysis is further conducted with valuable insights.
1 Introduction
Cooperative MARL requires coordinated decentralized policies under partial observation and communication constraints, while centralized learning scales poorly. Value decomposition addresses this through individual Q-values, and Qatten derives a theoretically grounded attention-based alternative.
- Motivation: Centralized learning can use joint observations and common rewards but scales poorly because the joint action space grows exponentially with agent count.Partial observations and communication constraints also necessitate decentralized policies.
- Motivation: Independent decentralized learners are difficult to train because global rewards create non-stationarity and obscure other agents’ changing behaviors.CTDE mitigates this by learning a centralized value function to guide decentralized policies, although centralized critics face scalability challenges.
- Related Work: VDN assumes Qtot is the additive sum of individual Q-values, whereas QMIX uses a nonlinear implicit mixing network to address VDN’s limitations.These methods impose structural assumptions on the relationship between global and individual values.
- Contribution: Qatten derives a decomposition formula for Qtot and uses multi-head attention to model each agent’s importance when mixing individual Q-values.The approach is intended to refine global-value representation and support decentralized policy maximization.
- Contribution: The paper evaluates Qatten on the StarCraft benchmark and reports superior performance against popular MARL algorithms, alongside attention-weight analysis.The analysis examines how agent properties affect mixing weights.
2 Background
The paper formulates cooperative partially observed Markov games in which agents optimize discounted returns from local information. It also situates attention as a mechanism for selectively weighting agent information in multiagent learning.
- Markov Games: A Markov game extends a Markov decision process with multiple agents, joint actions, state transitions, and agent reward functions.Each agent receives a local observation and learns a policy mapping observations to an action distribution.
- Markov Games: When all agents receive the same reward, the Markov game becomes fully cooperative because an action beneficial to one agent is also beneficial to the others.The agents optimize expected discounted returns, with γ as the discount factor.
- Markov Games: In partially observed settings, agents can use action-observation histories instead of current observations, with recurrent neural networks added to local policies or Q-functions.This provides a history-based input for decentralized decision-making.
- Value Decomposition: Value decomposition methods rely on decentralisability, or Individual-Global-Max, to connect decentralized individual action selection with global value maximization.The supplied formulation identifies this property as important for VDN and QMIX.
- Attention Mechanism: Attention maps a query and key-value pairs to an output by weighting values according to query-key compatibility.Multi-head attention allows focus on information from different representation sub-spaces.
3 Theoretical Analysis of Global and Individual Q-values
The analysis expresses the global Q-value as a function of individual Q-values near a maximal action point. Its expansion motivates approximating agent-dependent coefficients with attention, while nonlinear interactions can be represented through individual values and their combinations.
- Global and Individual Q-values: The global Q-value Qtot(s,⃗a) can be viewed as a function of individual Q-values by applying the Implicit Function Theorem.The joint action and state determine the global value before this functional reparameterization.
- Assumptions: The analysis assumes no independent agent, so changing an individual Q-value affects the global Q-value, though the effect may be positive or negative.The corresponding partial derivative is not identically zero, while isolated zero points are allowed.
- Theoretical Result: Theorem 1 states that, for continuous actions and no independent agent, Qtot has a local expansion in individual Q-values with state-dependent constants and coefficients.The expansion is derived near a maximal action point with the state fixed.
- Proof Intuition: Products such as QiQj can be reformulated using first-order terms and constants, allowing individual Q-values to approximate the global Q-value effectively.The proof sketch extends this approximation idea to other nonlinear higher-order terms.
- Theoretical Result: The expansion’s coefficients encode higher-order cross terms, while their superexponential decay supports truncating the representation in practice.λi,h depends on order-h partial derivatives and corresponds to cross terms such as QiQj.
- From Theory to Qatten: Qatten uses attention as a universal function approximator for the decomposition coefficients, motivating its multi-head mixing architecture.The paper connects this choice to the Transformer Universal Approximation Theorem.
4 The Practical Implementation of Qatten
Qatten implements theoretically motivated value decomposition with recurrent agent Q-networks and an attention-based mixing network. The design models agent-level contributions, preserves monotonicity, and supports consistent decentralized action selection.
- Architecture: Qatten combines recurrent agent Q-networks with an attention-based mixing network that transforms individual Q-values into Q_tot.The mixing network also receives global state and individual agent features.
- Attention mixing: Key-value memory approximates per-agent coefficients, using global-state queries and individual-feature keys to model each agent’s impact.Similarity values between global and individual embeddings are passed into a softmax.
- Attention mixing: Multiple attention heads approximate different orders of partial derivatives, and their head Q-values are summed to form the mixed value.The first term c(s) is learned by a neural network that takes the global state as input.
- Policy consistency: Qatten’s monotonicity yields the IGM property, enabling tractable joint-action maximization and consistency between centralized and decentralized policies.This supports off-policy learning with decentralized action selection.
- Weighted heads: Weighted head Q-values use state-dependent nonnegative weights to relax attention-imposed bounds and increase Qatten’s representation ability.The weights are obtained with an absolute activation from a two-layer network using the global state.
5 Experimental Evaluation
Qatten is evaluated on StarCraft II decentralized micromanagement tasks across easy, hard, and super hard scenarios, with additional ablation and attention analyses. It generally matches or outperforms competing methods, while weighted head Q-values improve difficult-scenario performance and attention weights reflect agent roles.
- Experimental Settings: Qatten is evaluated in the SMAC StarCraft II benchmark across easy, hard, and super hard scenarios.Results are averaged over 5 runs with different seeds.
- Easy Scenarios: On easy scenarios, Qatten achieves competitive performance with QMIX and other popular methods, including heterogeneous and asymmetric tasks.The reported comparison uses median test win rate.
- Hard Scenarios: On hard scenarios, Qatten consistently gains almost the best performance, and it consistently beats all other approaches across the four maps.In bane_vs_bane, Qatten performs much better than QMIX, which cannot learn steadily, because the four Banelings are vital to winning.
- Super Hard Scenarios: On super hard scenarios, Qatten exceeds QMIX by a large margin in MMM2 and reaches approximately 16% win rate on 3s5z_vs_3s6z after 2 million training steps.The super hard experiments augment Qatten with weighted head Q-values.
- Ablation Study: Weighted head Q-values improve Qatten-base on three difficult scenarios by flexibly adjusting head weights.The ablation is intended to capture more sophisticated relations between Q_tot and Q_i.
- Attention Analysis: Attention weights are nearly equally divided among homogeneous marines in 5m_vs_6m but differ more across agent types in 3s5z_vs_3s6z.The heat maps show steps increasing from top to bottom and agent IDs horizontally under each head.
6 Conclusion and Future Work
Qatten combines a theoretically derived Q-value decomposition with multi-head attention that models each agent’s impact on the global value. It achieves the best performance on almost all StarCraft maps, while future work targets exploration and larger-scale settings.
- Conclusion: Qatten derives a general decomposition formula relating global and individual Q-values.The method uses this formulation to model individual impacts at the agent level when transforming Q-values into Q_tot.
- Conclusion: Multi-head attention establishes the mixing network by explicitly modeling individual impacts on the whole system.
- Conclusion: Qatten achieves the best performance on almost all maps in the challenging StarCraft MARL benchmark.Attention analysis provides intuitive explanations of the attention weights.
- Future Work: Future work includes combining Qatten with explicit exploration mechanisms for difficult MARL tasks and adapting attention to settings with hundreds of agents.
A Proofs
The appendix expands Q_tot in terms of individual Q-values and applies local expansions to higher-order terms. It states mild convergence conditions and truncates the attention-based series for practical computation.
- Proofs: The proof expands Q_tot in terms of individual Q-values through higher-order terms.
- Proofs: The derivation applies the local expansion of each individual Q-value to the second-order term.
- Assumptions: Convergence of the series requires only mild conditions, such as boundedness or sufficiently small growth of partial derivatives with order.
- Implementation: For implementation, Qatten models coefficients with single-head attention and truncates the series at H heads because coefficients decay quickly with head index.
B Experimental Settings
The experiments use the SMAC StarCraft II setting, where agents receive partially observed local information while training can access a global state. Actions and rewards define cooperative micromanagement scenarios.
- Experimental Settings: The experiments follow SMAC environment settings and describe the environment details for completeness.
- Observations: Agents observe map information within a circular sight range of 9, making the environment partially observable.Agents can observe other units only when both are alive and within sight range.
- State: The global state contains relative coordinates, unit features, and allied-unit energy or cooldown information.
- Actions: Available actions include directional movement, attacking an enemy in shooting range, stopping, and no-op, with dead agents restricted to no-op.
- Rewards: Agents receive joint rewards for enemy damage, opponent kills, and winning the battle, with rewards scaled to a maximum cumulative value of around 20 per scenario.
B.4 Training Configurations
Training uses PyMARL with SMAC-aligned hyperparameters and reports map-dependent training schedules. The listed configuration includes Qatten’s mixing-network parameters.
- Training Schedule: Training takes about 8 to 18 hours depending on agent numbers and map features.
- Training Schedule: Experiments use about 2 million training steps, testing every 10,000 steps, and batches of 32 episodes from a 5,000-episode replay buffer.
- Exploration: The training procedure uses ε-greedy exploration.
- Implementation: Qatten is implemented in PyMARL using the same training and testing configurations as SMAC.The paper lists Qatten mixing-network parameters separately in Table 3.
B.6 Win Percentage Table of All Maps
Qatten achieves the best performance on almost all evaluated map scenarios, with results reported as median test win rates. The evaluation spans easy, hard, and super-hard scenarios with distinct strategic demands.
- Qatten obtains the best performance on almost all map scenarios.
- Median test win rates are reported for all methods across the scenarios presented in the paper.
- Hard scenarios: Hard scenarios require precise coordination, specialized counter-strategies, or managing unusually large action spaces.
C Extended Attention Analysis
Qatten’s attention analysis shows how agent roles and learned features influence the allocation of global value across agents and attention heads. The patterns differ between homogeneous and heterogeneous scenarios.
- 5m_vs_6m: In homogeneous 5m_vs_6m, allies receive similar attention weights because the marines play similar roles.
- 3s5z_vs_3s6z: On 3s5z_vs_3s6z, Head 1 focuses on output damage, while Heads 1 and 2 tend to focus on Stalker agents.
- Figure 5: Figure 5 orders attention heat-map steps from top to bottom and identifies agent ids horizontally under each head.