Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Problem setting

• Decentralised policies are needed under partial observability and communication constraints, but centralised Qtot is difficult to learn and does not directly yield individual policies.Centralised training can expose additional state information and remove communication constraints, yet the best way to exploit it remains open.Source: §1 · p. 2

Related approaches

• IQL cannot explicitly represent agent interactions and may fail to converge, whereas VDN enables decentralised greedy policies but restricts Qtot complexity and ignores training-time state information.COMA and related centralised critics require on-policy learning and become impractical beyond a handful of agents.Source: §1 · p. 2

Method

• QMIX uses positive-weight nonlinear mixing networks, conditioned on training-time state information, to combine per-agent Qa values into a richer Qtot while preserving decentralised argmax consistency.This supports complex centralised action-value functions, scalable factorisation, and inexpensive individual action selection.Source: §1 · pp. 2–3

SMAC benchmark

• SMAC is a benchmark for centralised training with decentralised execution, built on StarCraft II to test coordinated micromanagement under partial observability and local observations.Its scenarios involve independent learning agents controlling units against the built-in StarCraft II AI and cover high-dimensional, coordination-intensive challenges.Source: §1 · pp. 3–4

PyMARL framework

• PyMARL is an open-source, modular PyTorch framework implementing key multi-agent RL algorithms and providing evaluation guidelines for metrics, sample efficiency, and computational requirements.It is intended as an extensible template for practical deep multi-agent reinforcement learning research.Source: §1 · p. 4

Contributions and results

• QMIX outperforms IQL, VDN, and COMA in absolute performance and learning speed, with larger gains on harder SMAC tasks and heterogeneous agents.Ablations identify state conditioning and flexible multi-layer mixing as necessary for consistent performance across tasks.Source: §1 · p. 4

2. Related Work

• Deep multi-agent reinforcement learning has progressed from tabular methods toward neural methods for high-dimensional spaces, with this paper focusing on fully cooperative centralised training and decentralised execution.In this setting, agents jointly maximise a shared reward while acting without centralised execution.Source: §2 · p. 5

2.1 Independent Learners

• Independent learners achieve decentralised execution but suffer non-stationarity, while hysteretic, lenient, and related methods improve robustness without using centralised state information or learning joint action-values.These approaches include Independent Q-learning, Hysteretic Q-learning, Leniency, and distributional extensions for distinguishing miscoordination from stochasticity.Source: §2.1 · p. 5

2.2 Centralised Execution

• Coordination graphs reduce centralised joint-action computation but require agent dependencies in advance, whereas QMIX and VDN correspond to fully disconnected graphs enabling decentralised execution without that prior knowledge.Centralised joint action-values avoid non-stationarity but scale exponentially with the number of agents; communication-based alternatives require centralised execution, unlike QMIX.Source: §2.2 · pp. 5–6

2.3 Centralised Training and Decentralised Execution

• Hybrid methods use centralised critics or value functions to train decentralised policies, but QMIX learns a factored joint Q-function directly, unlike policy distillation, VDN’s limited decomposition, and QTRAN’s costly optimisation.Actor-critic alternatives can face local minima, poor sample efficiency, scalability limits, or computational inefficiency; QTRAN represents more joint Q-functions but performs poorly on complex SMAC environments in practice.Source: §2.3 · pp. 6–7

2.4 Extensions of QMIX

• QMIX extensions address exploration, action semantics, communication, learning targets, transfer, skills, mixed actions, and applications, while COMIX extends factorisation to continuous actions and outperforms prior MADDPG on Multi-Agent Mujoco.MAVEN improves exploration in some SMAC scenarios; ASN requires environment-specific action knowledge; SMIX(λ) has the same representational class as QMIX; COMIX approximates greedy per-agent maximisation with the cross-entropy method.Source: §2.4 · pp. 8–9

2.5 PyMARL and SMAC

• SMAC converts StarCraft II into decentralised cooperative problems through strict decentralisation and local partial observability, using a more stable, developer-supported environment than StarCraft-based benchmarks.SMAC and PyMARL have subsequently been used to evaluate and compare numerous multi-agent reinforcement-learning methods.Source: §2.5 · p. 9

2.6 Hypernetworks and Monotonic Networks

• QMIX transforms the centralised state into positive weights for a second neural network, combining hypernetwork-like conditioning with monotonic-network constraints.The positive weights ensure the second network is monotonic in its inputs, connecting QMIX to prior work on hypernetworks and functionally restricted neural networks.Source: §2.6 · p. 9

3. Background

• A cooperative multi-agent task is modeled as a Dec-POMDP in which agents select joint actions, share rewards, and transition according to the environment dynamics.Each agent receives partial observations and acts from its own action-observation history, while the joint action-value function measures discounted future return.Source: §3 · pp. 9–10

3.1 Deep Q-Learning

• Deep Q-learning trains a neural action-value function from replayed transitions by minimizing squared temporal-difference error with a periodically updated target network.The target is y_DQN = r + γ max_u′ Q(s′, u′; θ−).Source: §3.1 · p. 10

3.2 Deep Recurrent Q-Learning

• Deep recurrent Q-networks condition on full action-observation histories, using recurrent architectures such as LSTMs or GRUs to learn over longer timescales.This addresses partial observability by incorporating information accumulated across observations and actions.Source: §3.2 · p. 10

3.3 Independent Q-Learning

• Independent Q-learning decomposes the multi-agent problem into simultaneous single-agent learners, but changing agent policies create non-stationarity and remove convergence guarantees.Despite this limitation, IQL can remain a strong practical baseline in mixed and competitive games.Source: §3.3 · p. 10

3.4 Value Decomposition Networks

• Value decomposition networks represent the joint action-value as a sum of per-agent utility functions conditioned on individual action-observation histories.This additive representation enables decentralised greedy action selection by each agent with respect to its own utility function.Source: §3.4 · pp. 10–11

4. QMIX

• QMIX represents Qtot as a monotonic combination of per-agent Qa values, so decentralized greedy actions agree with the centralized joint argmax.Non-negative mixing-network weights enforce monotonicity, making centralized maximization linear in the number of agents rather than exponential.Source: §4 · pp. 11, 13, 14
• QMIX uses agent DRQNs, a state-conditioned mixing network, and hypernetworks that generate non-negative mixing weights for centralized training and decentralized execution.Agent networks receive current observations and previous actions; hypernetworks condition the mixing weights on the global state, while the mixing network uses ELU nonlinearities.Source: §4 · pp. 12–13
• QMIX is trained end-to-end with a DQN-style loss on joint rewards, while the mixing network may be discarded after training for fully decentralized execution.Limited-communication settings can retain the mechanism by exchanging low-dimensional action-values and gradients between agents and the coordinating mixer.Source: §4 · p. 14

4.1 Representational Complexity

• QMIX represents any value function that factors into a nonlinear monotonic combination of individual value functions whose action orderings agree with the joint-action values.This follows from the mixing network’s universal approximation of monotonic functions and the universal approximation capacity of the agent networks.Source: §4.1 · p. 14
• QMIX cannot necessarily represent the true optimal Q-function in Dec-POMDPs when local observations cause incorrect individual action orderings relative to joint-action values.The monotonicity constraint also excludes value functions where an agent’s best action depends on the simultaneous actions of other agents; these are nonmonotonic functions.Source: §4.1 · pp. 14–16
• QMIX’s greater representational capacity improves bootstrapping estimates, enabling optimal-policy learning where VDN fails, more accurate Qtot maxima in matrix games, and significantly better SMAC performance than VDN.These benefits are attributed to more accurate Q-value estimates for target regression, despite QMIX’s inability to represent all value functions exactly.Source: §4.1 · p. 16

5. Two-Step Game

• The game has two steps: Agent 1 selects between games leading to State 2A or State 2B, then both agents act for a global payoff.The payoff matrices are shown in Table 3.Source: §5 · p. 16, Table 3 · p. 17
• Training lasts 5000 episodes with full exploration (ϵ = 1), ensuring all game states are eventually explored so representational capacity is the remaining limitation.Architecture, hyperparameters, and additional results are provided in Appendix B.Source: §5 · p. 17
• QMIX’s greater representational capacity accurately models the joint-action value function, whereas VDN cannot.The learned Qtot values in Table 4 expose this difference.Source: §5 · p. 17, Table 4 · p. 17
• QMIX recovers the optimal first-step strategy and receives reward 8, while VDN selects Action A and receives reward 7.These outcomes follow directly from the learned joint-action values.Source: §5 · p. 17
• The example demonstrates that accurate Q-value estimates are important for bootstrapping, linking QMIX’s representational advantage to more accurate bootstrapped estimates.The paper states that further evidence is provided in Section 6.Source: §5 · p. 17

Experimental objective

• The experiment compares max_u Q_tot(s, u) because it is the only learned quantity used for bootstrapping in one-step Q-learning.Single-step matrix games permit direct comparison of the learned maximum Q-values in isolation.Source: §6 · pp. 17–18

Experimental setup

• Each payoff matrix has maximum return 10, other values sampled uniformly from [0, 10), and epsilon set to 1 so all actions are sampled equally.Matrix values differ across seeds but remain fixed within each run.Source: §6 · p. 18

Results

• QMIX learns significantly more accurate maximum Q-values than VDN across varying numbers of agents and actions, due to its larger representational capacity.Figure 4 reports median max_u Q_tot(s, u) across 10 runs, with the correct value marked at 10.Source: §6 · p. 18, Figure 4 · p. 18

7. The StarCraft Multi-Agent Challenge

• SMAC provides a challenging benchmark for decentralised multi-agent control by requiring locally observed agents to learn cooperative combat under partial observability while exposing global state during training.Each unit is controlled by an independent agent with a limited field of view, while groups of agents battle an opposing scripted army.Source: §7 · pp. 19–20

7.1 Scenarios

• SMAC scenarios pit learned allied agents against a scripted enemy army, ending when one army dies or the time limit is reached, with win rate as the objective.Scenario configurations vary unit positions, numbers, types, and terrain, and are designed to require specific micromanagement techniques.Source: §7.1 · pp. 20–21, Table 5 · p. 21

7.2 State and Observations

• Agents receive local, sight-limited observations of nearby units, terrain, and allied actions, whereas centralised training additionally provides all-map unit information and global action histories.The sight range is nine, and each agent cannot distinguish a distant teammate from a dead teammate.Source: §7.2 · pp. 21–22

7.3 Action Space

• The action space includes directional movement, enemy attacks, stopping, and no-op actions, with 7–70 available actions depending on the scenario.Agents may attack only enemies within shooting range six; because sight range is larger, targets farther away require movement before firing.Source: §7.3 · p. 22

7.4 Rewards

• SMAC optimizes battle win rate using a default shaped reward based on damage, kills, and a winning bonus, while also providing a sparse ±1 win/loss alternative.The default reward is encouraged for fair comparisons across scenarios.Source: §7.4 · p. 22

7.5 Evaluation Methodology

• Evaluation requires decentralised execution from each agent’s own action-observation history, while permitting centralised training with shared observations, parameters, gradients, or global state.Standardised environment, map, game-AI, and episode-limit settings are required for comparable results.Source: §7.5 · pp. 22–23

7.6 Evaluation Metrics

• Performance is evaluated using mean win percentage over environment steps, based on 32 exploratory-free evaluation episodes across five independent training runs, with median performance and 25-75% percentiles reported.The number of independent runs and environment steps used in training are reported.Source: §7.6 · p. 23

• PyMARL is an open-source framework focused on fast, easy development and experimentation in multi-agent reinforcement learning, without distributed-training support.It uses fewer abstractions than RLLib and is tailored to multi-agent settings.Source: §8 · p. 23
• Its modular PyTorch codebase supports rapid algorithm development, benchmarking, and isolation of components for testing.The framework provides implementations of current deep MARL algorithms and leverages PyTorch’s execution and training ecosystem.Source: §8 · p. 23
• PyMARL improves deep MARL implementation usability through understandable code and an encapsulated buffer interface that preserves access to underlying PyTorch tensors.The design addresses challenges beyond single-agent deep reinforcement learning and simplifies data handling between components.Source: §8 · pp. 23–24
• Its batching-oriented inference and learning aims to provide significant speed-ups over more naive implementations, and it includes QMIX, QTRAN (Son et al., 2019), COMA (Foerster et al., 2018), VDN, and IQL.PyMARL is available at https://github.com/oxwhirl/pymarl.Source: §8 · pp. 23–24

Overall results

• QMIX achieves the highest overall test win percentage and is best on up to eight scenarios, while IQL, VDN, and QMIX outperform COMA.Figure 7 compares median test win percentage across all 14 scenarios and counts scenarios where each algorithm leads by at least 1/32.Source: Figure 7 · p. 24, §9 · p. 25

Benchmark difficulty

• The heuristic AI performs relatively poorly, showing that SMAC requires more complex behaviour than simply focus-firing the closest enemy.The heuristic attacks the closest enemy with the entire team until it dies, then selects the next closest enemy.Source: §9 · pp. 24–25

QTRAN comparison

• QTRAN fails on 3s5z and takes substantially longer than VDN and QMIX to reach strong performance on 2s3z, limiting further benchmarking.QTRAN-Base was slightly more performant and stable than QTRAN-Alt in preliminary experiments.Source: Figure 8 · p. 25, §9 · p. 25

Easy scenarios

• QMIX exceeds 95% test win rate on all five Easy scenarios, whereas IQL and COMA perform poorly on four of five.These scenarios serve as sanity checks, while the results demonstrate the advantage of learning a centralised but factored Qtot.Source: §9 · p. 25, Figure 9 · p. 26

Hard scenarios

• Hard scenarios expose varied challenges, including a huge action space, high variance, asymmetric control, and delayed rewards from prolonged kiting.In bane vs bane, IQL finds a winning strategy while other methods struggle; QMIX and VDN plateau on 5m vs 6m.Source: §9 · pp. 25–26, Figure 10 · p. 26

Super Hard scenarios

• Only QMIX makes meaningful progress on two of five Super Hard scenarios, indicating that exploration is a bottleneck for many cases.These scenarios provide a testbed for future multi-agent reinforcement-learning research.Source: Figure 11 · p. 27, §9 · p. 27

10. Analysis

• QMIX’s superior performance is explained mainly by the central state and flexible mixing-network parameterisation, while nonlinearity is not generally necessary on SMAC.The central state is important but insufficient alone; learned solutions are often mostly linear, indicating that parameterisation flexibility—not nonlinearity—accounts for most of QMIX’s advantage.Source: §10 · p. 27

10.1 Role of the Central State

• QMIX-NS outperforms VDN-S on two scenarios but performs poorly on 3s5z, showing that state conditioning and learned mixing each contribute differently.VDN-S improves over VDN but remains below QMIX; QMIX-NS can benefit from learned mixing over fixed summation on harder scenarios.Source: Figure 12 · p. 28, §10.1 · p. 28

10.2 Role of Nonlinear Mixing

• QMIX-Lin matches QMIX on 3s5z but underperforms on the other scenarios, whereas QMIX-2Lin performs similarly to QMIX, showing that extra-layer parameterisation matters more than nonlinearity.QMIX can still learn nonlinear functions when required, and nonlinear mixing remains slightly beneficial on 2c vs 64zg; however, an additional layer improves optimisation relative to a single linear layer.Source: Figure 13 · p. 29, §10.2 · pp. 29, 32, Figure 17 · p. 31

10.5 Role of Network Size and Depth

• Increasing VDN agent-network capacity improves performance, especially on MMM2, but VDN Wider and VDN Deeper still do not match QMIX with smaller agent networks.Increasing the VDN-S state-dependent bias has minimal impact over VDN-S, while the larger-agent comparisons are acknowledged as unfair because they use more expressive networks.Source: Figure 18 · p. 32, §10.5 · p. 35

10.3 Role of RNN in Agent Network

• An RNN is unnecessary on easy 3s5z but crucial on 3s vs 5z for learning effective kiting from action-observation history.This comparison uses QMIX with and without recurrent agent networks.Source: §10.3 · p. 33, Figure 22 · p. 34

10.4 Role of the COMA Critic Architecture

• Matching COMA’s critic architecture more closely to QMIX has little performance effect, and adding state information degrades COMA on 2s3z.The modified critic uses per-agent components, a mixing network, and variants with or without state access.Source: Figure 23 · p. 34, §10.4 · p. 34, Figure 24 · p. 35

11. Conclusion

• QMIX represents rich joint action-values through tractable per-agent decompositions by imposing monotonicity on its mixing network.This structure supports centralised training with decentralised policies and extra state information.Source: §11 · p. 35
• SMAC provides 14 diverse decentralised micromanagement combat scenarios testing partial observability and high-dimensional inputs, while PyMARL open-sources implementations of IQL, VDN, QMIX, QTRAN, and COMA.The authors plan harder scenarios with more diverse units, terrain features, and greater coordination demands.Source: §11 · pp. 36–37
• QMIX improves final SMAC performance over relevant deep multi-agent RL algorithms, especially VDN, with state-dependent bias and state-conditioned learned utility mixing crucial to performance.The learned mixing need only be linear to recover most performance, but its parametrisation remains important.Source: §11 · pp. 36–37
• Future research targets the value-based versus policy-based performance gap, richer coordinated joint-action Q-values, and exploration methods beyond epsilon-greedy.The paper notes that QMIX and VDN significantly outperform COMA in sample efficiency and final performance across all scenarios, while SMAC super-hard scenarios can benchmark exploration advances.Source: §11 · p. 37

Appendix A. Monotonicity

• The monotonicity analysis establishes that maximizing the mixing network can be performed through the per-agent action-value maximizers under the stated nonnegative-derivative condition.The appendix expresses Qtot as a function of Q1(τ1,u1), …, Qn(τn,un) and identifies its maximizer.Source: Appendix A · p. 46

B.1 Architecture and Training

• The experiments use shared-parameter single-hidden-layer DQN agent networks with 64 ReLU units, epsilon-greedy action selection, replay batches of 32, and a mixing network of 8 units.Training uses γ = 0.99, a 500-episode replay buffer, target updates every 100 episodes, learning rate 5 × 10^-4, and 10k timesteps; agents receive the full state.Source: §B.1 · pp. 46–47

Appendix B. Two Step Game

• The Two Step Game uses one-hot encoded states, beginning at State 1 and transitioning to State 2 or State 3 based on Agent 1’s first action.State 2 has an all-7s payoff matrix.Source: §B.1 · p. 47

B.2 Learned Value Functions

• Learned value functions for the Two Step Game are reported for VDN, QMIX, QMIX-NS, QMIX-Lin, and IQL in Tables 6 and 7.Table 6 reports Qtot for the value-decomposition methods, while Table 7 reports Qa for IQL.Source: §B.2 · p. 47, Table 6 · p. 47, Table 7 · p. 48

B.3 Results

• Two Step Game evaluation compares six methods using losses and final testing rewards, with losses summarized across 30 independent runs using means and 95% confidence intervals.Figure 27 presents the loss curves, while Table 8 reports final test rewards.Source: §B.3 · p. 48, Table 8 · p. 48

D.1 Scenarios

• SMAC includes symmetric countering scenarios, asymmetric battles requiring health-aware targeting, and micro-trick challenges demanding coordinated strategies such as walling or kiting.Examples include six Zealots versus 24 Zerglings in Corridor and three Stalkers versus five Zealots in 3s vs 5z.Source: §D.1 · p. 49

D.2 Environment Setting

• SMAC uses SC2LE for engine communication but defines cooperative multi-agent micro challenges in which each learning agent controls one military unit, rather than the full game.The raw API supplies non-graphical unit information and permits commands to individual unit IDs, supporting decentralised task design.Source: §D.2 · p. 49
• SMAC suppresses allied units’ automatic attack responses, uses modified sight and shooting ranges, and benchmarks decentralised control rather than mastery of the original StarCraft II game.Enemy units retain their default behaviour, and the game AI is set to level 7.Source: §D.2 · pp. 49–50

D.3 Architecture and Training

• SMAC agents use shared-parameter DRQNs with 64-dimensional GRU states, independent epsilon-greedy exploration annealed from 1.0 to 0.05, and Double Q-Learning for Q-learning variants.Training uses 5000-episode replay buffers, batches of 32 fully unrolled episodes, target updates every 200 episodes, and gamma = 0.99.Source: §D.3 · p. 50
• QMIX uses a 32-unit ELU mixing network and 64-unit ReLU hypernetworks whose outputs are made non-negative, while COMA uses a 128-unit critic architecture and separate rollout-based updates.All neural networks use RMSprop with learning rate 5 × 10−4; COMA anneals epsilon from 0.5 to 0.01 over 100k timesteps and trains from eight episodes.Source: §D.3 · p. 50
• QTRAN is evaluated with alternative centralized-Q architectures, network depths, and lambda settings, selecting the best configuration across all three tested scenarios; VDN-S varies the depth of its state-dependent bias network.The QTRAN search covers lambda_nopt in {0.1, 1, 10} with lambda_opt = 1, while VDN-S Bigger uses two 64-unit hidden layers instead of one.Source: §D.3 · pp. 50–51

D.4 Table of Results

• Table D.4 reports the final median performance of the tested algorithms.The final median is the maximum median across testing intervals within the last 250k training steps.Source: §D.4 · p. 51, Table 9 · p. 51
• The final median is computed over testing intervals within the last 250k of training.Specifically, it uses the maximum median across those intervals.Source: §D.4 · p. 51
• The heuristic-based ai is additionally evaluated by mean test win percentage across 1000 episodes.This metric is shown alongside the algorithms’ final median performance.Source: §D.4 · p. 51

Training

• Training uses a mixing embedding dimension of 32 for 2000 timesteps, with replay-buffer sampling and minibatch L2-loss minimization.The replay buffer has size 200, and each minibatch contains 32 samples.Source: Appendix E · p. 52