PaperlayerView exact revision on arXiv

Source-linked AI summary

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson

arXiv:1803.11485v2cs.LGcs.MAstat.ML

TL;DR

Cooperative multi-agent RL needs decentralised policies while centralised training can provide global state information for coordination. QMIX monotonically mixes local agent values into a richer joint value, and it outperforms established value-based methods on StarCraft II micromanagement tasks.

  • Problem

    Centralised joint action-values capture coordination but do not provide an obvious decentralised policy, while simpler factorizations limit representational complexity.

  • Method

    QMIX combines per-agent values through a complex non-linear mixing network constrained to be monotonic by non-negative weights.

  • Results

    QMIX improves final performance over other value-based multi-agent methods with less sophisticated factorization and independent Q-learning on StarCraft II micromanagement tasks.

  • Takeaways & Limitations

    Monotonic factorization provides a rich joint action-value while retaining tractable decentralised action selection.

  • Takeaways & Limitations

    QMIX cannot represent value functions that do not factorize as monotonic combinations of individual agent values.

Abstract

from arXiv · show

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

1. Introduction

Cooperative multi-agent RL must reconcile decentralised execution with the richer coordination available during centralised training. QMIX addresses this by monotonically mixing local agent values into a central value, enabling decentralised action selection and outperforming value-based baselines on StarCraft II tasks.

  • Motivation: Partial observability and communication constraints require policies based only on each agent’s local action-observation history.Decentralisation also avoids joint action spaces that grow exponentially with the number of agents.
  • Motivation: A centralised Qtot captures joint-action effects but is difficult to learn at scale and offers no obvious decentralised policy extraction.It conditions on global state and joint actions, whereas execution requires individual actions from local observations.
  • QMIX: QMIX enforces monotonicity between Qtot and per-agent Qa values, making global and individual argmax operations consistent.This avoids requiring VDN’s full additive factorisation while retaining decentralised greedy action selection.
  • QMIX: QMIX combines per-agent values through a complex non-linear mixing network rather than VDN’s simple sum.The mixing network uses positive weights to preserve consistency between centralised and decentralised policies.
  • Evaluation: QMIX outperforms IQL and VDN in absolute performance and learning speed across StarCraft II unit-micromanagement tasks.The paper reports particularly large gains on a task with heterogeneous agents.

2. Related Work

Related work spans independent decentralised learning, centralised coordination, communication-based methods, actor-critic hybrids, and value decomposition. QMIX is positioned within this landscape as a cooperative method that uses centralised training without requiring communication during execution or pre-specified dependencies.

  • Independent learning: Independent Q-learning decentralises trivially but suffers from instability and non-stationarity caused by changing policies of other agents.These approaches also cannot explicitly represent agent interactions.
  • Centralised learning: Centralised joint-action learning handles coordination but scales poorly because the joint action space grows exponentially with the number of agents.Coordination graphs improve scalability by decomposing global rewards, but require dependencies to be supplied in advance.
  • Communication: Communication-based methods exchange information between agents during execution, with some approaches additionally estimating individual agent rewards.CommNet uses a centralised network, while BicNet uses bidirectional RNNs in an actor-critic setting.
  • Actor-critic methods: COMA and related actor-critic methods combine centralised learning with decentralised execution but can require on-policy learning or become impractical with many agents.COMA uses a centralised critic and counterfactual advantages for multi-agent credit assignment.
  • Value decomposition: VDN enables centralised value learning with decentralised execution but uses only a sum of individual terms and ignores additional training-time state information.Its representation therefore covers only a limited class of centralised action-value functions.
  • Benchmarks: QMIX is evaluated in StarCraft II Learning Environment tasks because SC2LE is actively supported and provides a more stable testing environment than StarCraft.The benchmark is established in deep multi-agent RL, although some prior methods do not require decentralised execution.

3. Background

The background formalises cooperative partially observable multi-agent tasks and the centralised-training, decentralised-execution setting. It then contrasts deep Q-learning, independent Q-learning, and VDN’s additive value decomposition.

  • Dec-POMDPs: A cooperative multi-agent task is represented as a Dec-POMDP with states, joint actions, transitions, shared rewards, observations, agents, and discounting.The true environment state is distinct from the observations available to individual agents.
  • Decentralised policies: In partial observability, each agent conditions its stochastic policy on its own action-observation history, while the joint policy has a joint action-value function.The action-observation history captures the information available to decentralised decision-making.
  • Centralised training: Training may access all local histories and the global state, but execution restricts each learned policy to its own history.This is the centralised-training, decentralised-execution paradigm.
  • Deep Q-learning: Deep Q-learning learns neural-network action values by sampling replay transitions and minimising squared temporal-difference error.The target uses a periodically updated target network.
  • Independent Q-learning: Independent Q-learning decomposes the multi-agent problem into simultaneous single-agent problems but lacks convergence guarantees because other agents’ policies change.This changing environment creates non-stationarity during learning.
  • Value decomposition: VDN represents the joint action-value as a sum of individual Qa functions based only on each agent’s local history and action.Each agent can then select actions greedily from its own Qa, although Qa is technically a utility function rather than an expected-return value function.

4. QMIX

QMIX combines per-agent value functions through a state-conditioned monotonic mixing network, retaining decentralised action selection while representing richer centralised value functions than VDN. Its positive-weight architecture enables efficient centralised maximisation but excludes non-monotonic value functions.

  • QMIX generalises VDN by representing Qtot as a richer monotonic combination of individual value functions rather than a simple sum.
  • Consistency requires the global argmax of Qtot to match individual argmax operations on each Qa, not full factorisation.
  • Because the monotonic constraint supports decentralised greedy actions, Qtot maximisation for off-policy updates is linear in the number of agents.
  • Agent networks produce individual values, while a feed-forward mixing network combines them monotonically into Qtot using non-negative weights.
  • Hypernetworks condition mixing-network weights and biases on the global state, allowing Qtot to incorporate extra state information non-monotonically.
  • QMIX cannot perfectly represent value functions where an agent’s best action depends on other agents’ simultaneous actions, although it can approximate them more accurately than VDN.

5. Two-Step Game

A two-step cooperative matrix game tests whether VDN and QMIX can represent the required joint-action values. Under full exploration, QMIX recovers the optimal strategy and reward, whereas VDN learns a suboptimal one.

  • The game has two steps: Agent 1 selects the next matrix game, then both agents act and receive a global payoff.
  • 5000 episodes with ε = 1 ensure all game states are explored, isolating representational capacity as the remaining limitation.
  • QMIX accurately represents the joint-action value function, whereas VDN cannot.
  • Reward 8: QMIX recovers the optimal strategy, while VDN selects Action A and receives reward 7.

6. Experimental Setup

The experiments evaluate QMIX and ablations on decentralised StarCraft II micromanagement tasks, including tests of state information and non-linear mixing. The setup uses partial observability, centralised training information, and a heuristic comparison.

  • Environment: QMIX is evaluated on decentralised StarCraft II micromanagement problems where each agent controls an individual army unit.Combat scenarios place two groups of units symmetrically, with allied units controlled by decentralised agents and enemies controlled by the built-in StarCraft II AI.
  • Environment: Partial observability restricts agents to observing living allied and enemy units within their unit sight range.Agents cannot distinguish units that are dead from units that are out of range.
  • Ablations: The ablations test the influence of extra state information and the necessity of non-linear transformations in the mixing network.The experiments compare QMIX variants without hypernetworks and without a hidden mixing layer.
  • Ablations: QMIX-NS removes state conditioning from the mixing network while enforcing monotonicity by taking the absolute value of its weights.Its mixing-network weights and biases are learned without conditioning on the state.
  • Ablations: QMIX-Lin removes the mixing network’s hidden layer, while VDN-S adds a state-dependent term to a sum of agent Q-values.QMIX-Lin uses the state to perform a weighted sum, whereas VDN-S uses a one-hidden-layer network to produce the added state-dependent term.
  • Baselines: The evaluation includes a fully observable heuristic in which each agent attacks the closest enemy and continues targeting it until that unit dies.Afterward, the agent attacks the nearest remaining enemy.

7. Results

The results compare greedy decentralised policies using test win rates across StarCraft II combat maps and ablations. QMIX is the strongest overall method, with state information and non-linear mixing especially important for heterogeneous-agent scenarios.

  • Evaluation: The test win rate is the percentage of 20 greedy decentralised evaluation episodes in which all enemies are defeated within the time limit.Training is paused every 100 episodes for each evaluation run.
  • Main Results: IQL fails to learn a policy that consistently defeats the enemy in all scenarios.Its training is highly unstable because other agents change their behaviour during learning, creating environmental non-stationarity.
  • Main Results: VDN outperforms IQL in all scenarios and more consistently learns basic coordinated focus firing.On the 3m task, however, VDN cannot consistently defeat the enemy, whereas focus firing performs well on the 5m and 8m maps.
  • Main Results: QMIX is noticeably the strongest performer on all maps, particularly those with heterogeneous agent types.The largest gaps occur on 3s 5z and 1c 3s 5z, where VDN does not reach the heuristic’s performance.
  • Ablation Results: On heterogeneous-agent maps, good performance requires both central state information and non-linear value-function factorisation.QMIX-NS performs on par with or slightly better than VDN, while comparisons involving VDN-S and QMIX-Lin support the necessity of non-linear mixing for exploiting central state information.
  • Learned Policies: QMIX learns coordinated positioning strategies that protect Stalkers while Zealots block and attack enemy Zealots.VDN and VDN-S use less protective positioning in the described heterogeneous scenarios.

8. Conclusion

QMIX learns rich joint action-values that can be decomposed into decentralised per-agent values through a monotonic mixing constraint. Its representational scope is broader than VDN but excludes some non-monotonic value functions and is not guaranteed in partially observable settings.

  • 8. Conclusion: QMIX combines per-agent values into a rich joint action-value function using a monotonic mixing constraint.The mixing network is a universal approximator for monotonic functions in the fully observable setting.
  • 8. Conclusion: QMIX cannot necessarily represent value functions in a Dec-POMDP when local observations produce incorrect agent action orderings.The monotonicity constraint prevents correct representation when an agent’s preferred action conflicts with the centralised value ordering.
  • 8. Conclusion: QMIX expands beyond VDN by representing non-linear monotonic value functions that VDN cannot represent.VDN represents only a limited class of centralised action-value functions through a sum of individual agent terms.
  • 8. Conclusion: QMIX cannot perfectly represent value functions where an agent’s best action depends on the simultaneous actions of other agents.This limitation follows directly from the required monotonic factorisation.
  • 8. Conclusion: The two-step-game analysis compares Qtot for VDN, QMIX, QMIX-NS, QMIX-Lin, and VDN-S, alongside IQL’s Qa values.The supplied passages identify the compared value functions and the learned-value-function tables but do not report their numerical contents.

B.3. Results

The results section reports learning loss and final test reward for six methods on the Two Step Game, with inputs including local observations and global state features. The supplied passages do not provide the plotted or tabulated numerical outcomes.

  • B.3. Results: Table 6 reports the final testing reward for each method.The supplied passage identifies the table but does not include its values.
  • B.3. Results: Figure 5 reports loss for all six methods on the Two Step Game across 30 independent runs.It shows the mean and 95% confidence interval.

C.2. Architecture and Training

The StarCraft II architecture uses recurrent per-agent networks with shared training settings, while a separate mixing network combines agent values. Episodes have map-dependent time limits.

  • C.2. Architecture and Training: Each agent network is a DRQN with a GRU containing a 64-dimensional hidden state.A fully connected layer appears before and after the recurrent layer.
  • C.2. Architecture and Training: Agents independently select actions with epsilon-greedy exploration while epsilon anneals from 1.0 to 0.05 over 50k time steps.The discount factor is γ = 0.99.
  • C.2. Architecture and Training: Agent-network parameters are shared across agents, with a one-hot agent identifier appended to each observation.Parameter sharing speeds learning, and RMSprop is used for training.
  • C.2. Architecture and Training: Episode limits vary by map, ranging from 60 time steps on 3m and 5m maps to 200 on 1c 3s 5z.Episodes terminate when one army is defeated or the time limit is reached.
  • C.2. Architecture and Training: The mixing network has one hidden layer of 32 units with an ELU nonlinearity.A 32-unit ReLU hypernetwork produces the final mixing-network bias.

D. StarCraft II Results

StarCraft II results compare six methods and a heuristic algorithm across six combat maps, including ablations and unit-type-specific settings. The supplied passages identify the comparisons but not their numerical win rates.

  • D. StarCraft II Results: The results compare all six methods with a heuristic-based algorithm on six combat maps.Figure 6 reports win rates for IQL, VDN, and QMIX, while Figure 7 reports QMIX and its ablations.
  • D. StarCraft II Results: The heuristic-based algorithm’s test win-rate percentages are reported in Table 7.The supplied passage identifies the metric but does not provide its values.
  • D. StarCraft II Results: Unit type is included only on the 2s 3z, 3s 5z, and 1c 3s 5z maps.Cooldown is the time a unit must wait before firing again; shields regenerate after absorbing damage.
  • D. StarCraft II Results: The experiments use α = 0.99 without weight decay or momentum.
Loading 1803.11485v2…