PaperlayerView exact revision on arXiv

Source-linked AI summary

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson

arXiv:2003.08839v2cs.LGcs.MAstat.ML

TL;DR

Cooperative multi-agent reinforcement learning needs decentralised policies, while centralised training can expose global state and remove communication constraints; how to connect the two remains unclear. QMIX uses a state-conditioned mixing network with non-negative weights to form a monotonic joint action-value from per-agent values, and the paper evaluates it with SMAC. On SMAC, QMIX outperforms IQL, VDN, and COMA in performance and learning speed, while the benchmark and PyMARL support standardised evaluation and further research.

  • Problem

    Learning joint action-values with extra state information is attractive for centralised training, but extracting decentralised policies from them remains unclear.

  • Method

    QMIX combines per-agent action-values through a state-conditioned mixing network with non-negative weights, enforcing monotonicity while enabling decentralised execution.

  • Results

    QMIX outperforms IQL, VDN, and COMA in absolute performance and learning speed on SMAC, with gains especially on harder and heterogeneous-agent tasks.

  • Takeaways & Limitations

    SMAC and PyMARL provide benchmark scenarios, implementations, and evaluation guidance for comparing cooperative deep multi-agent reinforcement learning methods.

  • Takeaways & Limitations

    QMIX cannot represent nonmonotonic value functions, so its representational constraints can prohibit learning an optimal policy.

Abstract

from arXiv · show

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

1. Introduction

QMIX addresses the challenge of learning decentralised policies from centralised action-values by constraining a richer factored representation to preserve decentralised greedy action selection. The paper also introduces SMAC and PyMARL to evaluate and support cooperative multi-agent reinforcement learning.

  • Motivation: Centralised action-values capture joint-action effects but are difficult to learn at scale and do not directly yield decentralised policies.Independent learning avoids centralised representations but cannot explicitly represent agent interactions and may fail to converge.
  • QMIX: QMIX represents a richer class of centralised action-values than VDN while retaining decentralised policy extraction through a monotonic mixing constraint.The mixing network combines per-agent values nonlinearly and uses positive weights, with state-conditioned weights produced by hypernetworks.
  • SMAC: SMAC introduces challenging StarCraft II micromanagement scenarios for centralised training with decentralised execution.Agents act from local observations while facing scripted opponents across diverse, partially observable scenarios.
  • Resources: The paper releases PyMARL and evaluation guidelines covering standardised metrics, sample efficiency, and computational requirements.PyMARL is modular, extensible, PyTorch-based, and includes implementations of several key multi-agent reinforcement learning algorithms.
  • Results: QMIX outperforms IQL, VDN, and COMA in absolute performance and learning speed on SMAC, especially on harder tasks and heterogeneous-agent tasks.Ablations identify state conditioning and flexible multi-layer mixing as necessary for consistent performance across tasks.

2. Related Work

Related work spans independent, centralised actor-critic, value-decomposition, communication, and coordination-graph approaches to cooperative multi-agent reinforcement learning. These methods differ in scalability, representational capacity, decentralisability, communication requirements, and optimisation difficulty.

  • Independent learning: Independent Q-learning is trivially decentralised but suffers instability from the non-stationarity caused by simultaneously learning and exploring agents.Despite lacking convergence guarantees in this setting, IQL can remain a strong practical baseline.
  • Centralised training: Centralised critics and actor-critic methods exploit additional agent information during training, but may be sample-inefficient or difficult to scale with many agents.COMA uses a centralised critic and counterfactual advantages for decentralised actors in cooperative settings.
  • Value decomposition: VDN enables decentralised execution by summing individual agent terms, but cannot represent the full range of centralised action-value functions or use extra state information during training.QMIX is proposed as a richer alternative within the same broad middle ground between independent and fully centralised methods.
  • Value decomposition: QTRAN can represent a larger class of joint action-values than QMIX, but its optimisation is difficult and performs poorly on complex SMAC environments.The exact linear optimisation is prohibitively expensive, so QTRAN uses a soft L2-penalty approximation.
  • QMIX extensions: QMIX’s monotonicity constraint limits representational capacity, motivating extensions such as MAVEN, ASN, communication-based methods, and SMIX(λ).MAVEN addresses limitations with a shared latent space but requires initial-state access and communication at the first timestep.
  • Benchmarks: SMAC converts StarCraft II into a strictly decentralised, locally partially observable cooperative multi-agent benchmark.It complements existing work by focusing on decentralised micromanagement rather than centralised control of the full game.

3. Background

The paper formulates cooperative multi-agent reinforcement learning as a decentralised partially observable decision process and studies centralised training with decentralised execution. It reviews deep Q-learning, recurrent histories, independent learning, and value decomposition as foundations for QMIX.

  • Dec-POMDP: A cooperative Dec-POMDP specifies global states, joint actions, transitions, rewards, observations, agent histories, and discounting.Each agent selects actions from its own action-observation history while the environment evolves from the joint action.
  • CTDE: Under centralised training with decentralised execution, learning accesses global state and all local histories, while each deployed policy uses only its own history.This separates training information from execution information.
  • Deep Q-learning: Deep Q-learning estimates action-values with a neural network trained from replayed transitions by minimising squared temporal-difference error.The target uses a periodically updated target network and a greedy next-action value.
  • Partial observability: Recurrent deep Q-networks allow agents in partially observable settings to condition on longer action-observation histories.LSTM and GRU architectures are used to support learning over longer timescales.
  • Value-based methods: Independent Q-learning decomposes a multi-agent problem into simultaneous single-agent problems but lacks convergence guarantees under changing agent policies.VDN instead represents a joint action-value as a sum of per-agent value functions, enabling decentralised greedy selection.

4. QMIX

QMIX extends value decomposition with a monotonic mixing architecture that preserves consistency between centralised joint-action selection and decentralised per-agent argmax decisions. State-conditioned hypernetworks produce the mixing parameters, enabling richer value functions while retaining tractable decentralised execution.

  • QMIX: QMIX represents Qtot by combining per-agent Qa values through a monotonic function, extending VDN beyond simple summation.The representable class includes nonlinear monotonic combinations of individual value functions in fully observable settings.
  • QMIX: Monotonicity ensures that the global greedy joint action matches individual argmax choices over each agent’s Qa.This makes decentralised action selection possible without exhaustive evaluation of exponentially many joint actions.
  • QMIX: Agent networks estimate Qa from each agent’s observation history, including the current observation and previous action; shared networks may also receive agent IDs.The overall architecture combines these agent networks with the mixing network and hypernetworks.
  • QMIX: The mixing network uses non-negative weights, with hypernetworks generating state-conditioned layer parameters while biases remain unrestricted.Conditioning the mixing weights on the global state allows Qtot to depend on extra state information without passing that state directly through the monotonic network.
  • QMIX: QMIX computes off-policy targets using a DQN-style loss, while the monotonic structure makes Qtot maximisation linear in the number of agents.The mixing network supports centralised training and can be discarded for fully decentralised execution after training.
  • QMIX: QMIX cannot perfectly represent nonmonotonic value functions, although its broader representation can still recover optimal policies and improve bootstrapped Q-value estimates.The paper reports optimal-policy recovery in an environment without per-timestep coordination and stronger performance than VDN on SMAC.

5. Two-Step Game

The two-step cooperative matrix game tests whether VDN and QMIX can represent joint-action values accurately enough to recover the optimal strategy. QMIX represents the value function accurately and achieves the optimal reward, whereas VDN selects a suboptimal first action.

  • Game setup: The two-step game evaluates VDN and QMIX on a cooperative task where Agent 1 selects the subsequent matrix game before both agents act.The first-step choice determines State 2A or State 2B, while second-step actions produce global rewards from the corresponding payoff matrix.
  • Experimental design: 5000 episodes with full exploration isolate representational capacity as the limiting factor in the learned state-action value functions.ϵ = 1 ensures that every available game state is eventually explored.
  • Results: QMIX accurately represents the joint-action value function, whereas VDN cannot because QMIX has higher representational capacity.The learned Qtot values are reported in Table 4 for both methods.
  • Results: VDN selects Action A and receives reward 7, while QMIX recovers the optimal strategy and receives reward 8.The result follows from the methods’ learned joint-action values in the two-step game.
  • Interpretation: More accurate Q-value estimates matter for bootstrapping, and QMIX’s increased representational capacity directly leads to more accurate bootstrapped estimates.The paper presents this game as evidence connecting representational capacity with Q-value accuracy.

6. Random Matrix Games

Random single-step matrix games compare the accuracy of VDN and QMIX’s maximum joint-action values under full exploration. QMIX learns significantly more accurate maxima across varying numbers of agents and actions.

  • Experimental design: The experiment compares learned max_u Qtot(s, u) values because they are the only quantities used for bootstrapping in 1-step Q-learning.Single-step matrix games permit direct comparison of the learned maximum Q-values in isolation.
  • Experimental design: The maximum return is 10, while all other payoff values are sampled uniformly from [0, 10), with ϵ = 1 ensuring equal action sampling.Payoff matrices remain fixed within each run but differ across seeds.
  • Results: QMIX learns significantly more accurate maxima than VDN across varying numbers of agents and actions.Figure 4 reports medians across 10 runs with shaded 25%-75% quartiles; the dashed line at 10 marks the correct value.

7. The StarCraft Multi-Agent Challenge

SMAC is a StarCraft II micromanagement benchmark for decentralized multi-agent control under partial observability and demanding coordination requirements. It provides diverse scenarios, centralized training state information, and standardized evaluation of learning progress.

  • Benchmark design: SMAC focuses on micromanagement and assigns each unit to an independent agent that must control it from local information.The benchmark modifies StarCraft II micromanagement specifically for decentralized control.
  • Observations and state: Centralized training provides additional state information, including information about all units on the map, to support evaluation of centralized-training methods.Individual agents receive local observations, while the global state is available only during centralized training.
  • Benchmark challenges: Scenarios combine partial observability, up to 27 learning agents, heterogeneous units, long-term planning, and high-dimensional observations.These characteristics are presented as common in real-world multi-agent systems.
  • Benchmark contribution: SMAC integrates these challenges into a benchmark intended to measure progress in decentralized multi-agent control.The paper describes it as one of the first MARL benchmarks including all listed features.
  • Scenario structure: SMAC scenarios require agents to learn coordinated behaviors under partial observability while enemy units are controlled by the built-in StarCraft II AI.Each scenario is a confrontation between learned allied agents and scripted enemy units, with varying army composition and terrain.
  • Observations and state: Agents observe local map and unit features within a sight range, including positions, health, shields, unit types, terrain, and relevant action information.The observation space is partially observable because agents cannot observe units outside their sight range.
  • Actions: The action set includes movement, attacking, stopping, no-op, and healing for Medivacs, with 7 to 70 possible actions depending on the scenario.The discrete action space varies by scenario.
  • Evaluation: Performance is evaluated by mean win percentage over environment steps, using periodic evaluation episodes and reporting medians with 25-75% percentiles across five runs.Evaluation disables exploratory behavior and uses 32 episodes in practice.

8. PyMARL

PyMARL is an open-source framework designed to support rapid development and benchmarking of deep multi-agent reinforcement-learning algorithms. Its modular PyTorch-based implementation emphasizes extensibility, readable data handling, and efficient batching.

  • Framework purpose: PyMARL is a lightweight software engineering framework focused specifically on multi-agent reinforcement learning development and experimentation.It is designed with fewer abstractions than general reinforcement-learning frameworks such as RLLib.
  • Architecture: Its modular codebase supports rapid development of new algorithms and benchmarking against implementations of current deep MARL methods.Components can be isolated for testing purposes.
  • Implementation: PyMARL is built on PyTorch to accelerate deep neural-network execution and training while leveraging its broader ecosystem.The framework is intended for fast experimentation.
  • Availability: The PyMARL codebase is available at the project’s GitHub repository.The paper gives the repository URL explicitly.
  • Data handling: The framework encapsulates replay-buffer data in an accessible structure and maximizes batching during inference and learning for speedups over naive implementations.The data interface preserves access to the underlying PyTorch tensors.
  • Included algorithms: PyMARL includes implementations of QMIX, QTRAN, COMA, VDN, and IQL as benchmark algorithms.These implementations provide a common basis for deep MARL comparisons.

9. SMAC Results

On SMAC, QMIX achieves the strongest overall performance while the benchmark exposes challenges from partial observability, heterogeneous scenarios, and difficult coordination. Performance varies substantially across difficulty categories and methods.

  • QMIX achieves the highest overall test win percentage and is the best performer on up to eight scenarios during training.
  • QMIX, IQL, and VDN significantly outperform COMA, indicating greater sample efficiency for off-policy value-based methods.
  • Easy scenarios: QMIX exceeds 95% test win rate on all five Easy scenarios, while IQL and COMA perform poorly on four of five.
  • Hard scenarios: The Hard scenarios combine challenges including very large action spaces, asymmetric control, high variance, and delayed rewards.
  • Super Hard scenarios: Only QMIX makes meaningful progress on two of five Super Hard scenarios, where all algorithms otherwise show poor performance.

10. Analysis

The analysis attributes QMIX’s performance primarily to state-conditioned and flexible mixing-network parameterisation rather than nonlinearity alone. Additional experiments examine recurrence and critic architecture as contributors to performance.

  • Role of the Central State: Conditioning the mixing network on the central state is extremely important, but state information alone does not explain QMIX’s performance gains.
  • Role of the Central State: VDN-S improves over VDN but remains worse than QMIX, while QMIX-NS varies across scenarios and does not match QMIX.
  • Role of Nonlinear Mixing: QMIX learns approximately linear mixing across most SMAC timesteps, despite being capable of representing nonlinear functions.
  • Role of Nonlinear Mixing: QMIX’s flexible mixing-network parameterisation accounts for most of its advantage, although nonlinear mixing can still help in some scenarios.
  • Agent History: An RNN is unnecessary on easy 3s5z but crucial for learning effective kiting on harder 3s vs 5z.
  • Role of the COMA Critic Architecture: Changing COMA’s critic architecture has little performance effect, while adding state degrades performance in 2s3z.

11. Conclusion

QMIX enables end-to-end learning of decentralised policies in a centralised setting by imposing monotonicity on a state-informed mixing network. The paper also contributes SMAC and PyMARL, while identifying remaining challenges in expressivity, exploration, and benchmark difficulty.

  • Contributions: QMIX learns rich joint action-values that admit tractable decentralised decompositions by imposing a monotonicity constraint on the mixing network.The method uses centralised training while preserving decentralised action selection.
  • Contributions: SMAC provides 14 diverse cooperative combat scenarios targeting partial observability and high-dimensional inputs, and PyMARL implements several deep MARL algorithms.The released framework includes IQL, VDN, QMIX, QTRAN, and COMA.
  • Results: QMIX improves final performance over relevant deep multi-agent reinforcement learning algorithms, especially VDN.The paper analyzes this comparison through ablations and visualisations.
  • Results: A state-dependent bias and state-conditioned learned mixing of agent utilities are identified as crucial components of QMIX’s performance.The analysis finds that the learned mixing need only be linear to recover most performance, but its parametrisation matters.
  • Future work: Future work targets harder scenarios requiring more diverse units, terrain use, and higher coordination, alongside better exploration and broader joint-action-value representations.The paper also highlights the performance gap between value-based and policy-based methods as an open research direction.

B.3 Results

The Two Step Game results compare losses and final rewards across methods, while the broader benchmark setup covers symmetric, asymmetric, and micro-trick scenarios. Experiments use specified neural architectures, exploration schedules, and training procedures for the evaluated algorithms.

  • Two Step Game: Figure 27 reports loss for six methods on the Two Step Game, with the mean and 95% confidence interval shown across 30 independent runs.Table 8 reports each method’s final testing reward.
  • SMAC scenarios: SMAC includes symmetric battles, asymmetric battles with outnumbered enemy forces, and micro-trick tasks requiring coordinated use of terrain.Examples include countering unit types, considering enemy health, and walling off a choke point.
  • Environment: SMAC uses SC2LE’s raw API to provide unit-level observations and commands without graphical input, targeting decentralised multi-agent control rather than full-game mastery.The benchmark modifies allied-unit automatic responses and uses a difficult built-in game AI setting.
  • Implementation: Agent networks use recurrent DRQNs with GRU hidden states, parameter sharing, and linearly annealed independent ε-greedy exploration in the SMAC experiments.The mixing network has one 32-unit ELU hidden layer, while hypernetworks use one 64-unit ReLU hidden layer.
  • Implementation: COMA uses an independently specified feedforward critic and an eight-episode rollout procedure for critic and policy updates.The critic is updated backward through each episode before the agent policies are updated.

D.4 Table of Results

Table D.4 reports final median algorithm performance and heuristic-based mean test win percentages, while Table 9 reports test win rates for the various algorithms.

  • Table D.4: Table D.4 reports final median performance, defined as the maximum median across testing intervals within the last 250k training steps.It also reports mean test win percentage across 1000 episodes for the heuristic-based AI.
  • Table 9: Table 9 contains the test win rate percentage for the various algorithms.The table caption identifies the reported measure but does not provide values in the supplied passage.

Appendix E. Regression Experiment

The regression experiment fixes sampled states, agent Q-values, and targets while training only the mixing network to minimize target-output error.

  • Setup: States, agent Q-values, and targets remain fixed, so only the mixing-network parameters change during regression training.This isolates the mixing network’s fitting behavior.
  • Training: Training uses a 32-sample minibatch, a replay buffer of 200, a 32-dimensional mixing embedding, and L2 loss for 2000 timesteps.At each timestep, a state is added to the replay buffer before a training step.
Loading 2003.08839v2…