PaperlayerView exact revision on arXiv

Source-linked AI summary

Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid, Gregory Farquhar, Bei Peng, Shimon Whiteson

arXiv:2006.10800v2cs.LGcs.MAstat.ML

TL;DR

QMIX’s monotonic factorisation limits representation of coordination-dependent value functions, and its unweighted projection can recover a suboptimal policy even from Q*. The paper replaces that projection with weighted alternatives, proves correct maximal-action recovery in the tabular setting, and develops CW QMIX and OW QMIX, which improve coordination and robustness empirically while adding complexity and sometimes performing poorly in challenging domains.

  • Problem

    QMIX cannot represent joint action values where an agent’s action ordering depends on other agents’ actions, creating a limitation for same-timestep coordination.

  • Method

    The paper analyses QMIX as an unweighted projection operator, introduces two weighted projection schemes, and extends them to CW QMIX and OW QMIX with an unrestricted learned approximation of Q*.

  • Results

    The weighted projection recovers the correct maximal joint action for any Q, including Q*, while deep Weighted QMIX improves coordination and robustness on predator-prey and SMAC tasks.

  • Takeaways & Limitations

    Weighted QMIX lifts QMIX’s monotonic coordination restriction in theory and improves its empirical ability to coordinate within the evaluated cooperative MARL tasks.

  • Takeaways & Limitations

    The approach introduces extra complexity and can perform poorly in certain challenging domains; architecture choices for the learned Q* approximation can also affect performance.

Abstract

from arXiv · show

QMIX is a popular $Q$-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action $Q$-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the $Q$-learning targets and then projects them into the space representable by QMIX. This projection returns a representable $Q$-value that minimises the unweighted squared error across all joint actions. We show in particular that this projection can fail to recover the optimal policy even with access to $Q^*$, which primarily stems from the equal weighting placed on each joint action. We rectify this by introducing a weighting into the projection, in order to place more importance on the better joint actions. We propose two weighting schemes and prove that they recover the correct maximal action for any joint action $Q$-values, and therefore for $Q^*$ as well. Based on our analysis and results in the tabular setting, we introduce two scalable versions of our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW) QMIX and demonstrate improved performance on both predator-prey and challenging multi-agent StarCraft benchmark tasks.

1 Introduction

QMIX’s monotonic value-factorisation enables decentralised execution but cannot represent important nonmonotonic coordination dependencies. The paper formalises this limitation, shows why unweighted projection can produce suboptimal policies, and introduces weighted alternatives with scalable empirical variants.

  • QMIX combines per-agent utilities through a monotonic mixing function, enabling efficient maximisation and decentralised execution.
  • QMIX cannot represent value functions where an agent’s action ordering depends on other agents’ actions, limiting coordination within a timestep.
  • QMIX projects Q-learning targets into its representable function class by minimising unweighted squared error across joint actions.
  • Even with access to Q*, this projection can estimate the optimal joint action incorrectly and yield a suboptimal policy.The limitation is attributed to equal importance placed on approximating all joint actions.
  • Weighted projections emphasise better joint actions; two proposed weighting functions recover the correct maximal action for any joint action Q-values, including Q*.The paper then develops scalable CW QMIX and OW QMIX variants, which outperform QMIX on predator-prey and show improved exploration robustness across SMAC maps.

2 Background

The paper frames cooperative MARL as decentralised partially observable decision-making trained centrally with shared information but executed from local histories. QMIX estimates joint action values by monotonic mixing of per-agent utilities and learns them with Q-learning-style temporal-difference updates.

  • A cooperative Dec-POMDP models states, joint actions, transitions, rewards, observations, histories, policies, agents, and discounting.
  • In the idealised tabular setting, full observability makes the problem equivalent to an MMDP or standard MDP with the joint action space U^n.
  • Centralised training accesses the true state and all agents’ histories, whereas execution restricts each agent to its own action-observation history.
  • The Bellman optimality operator defines Q-learning targets using expected rewards and next states under the environment dynamics.
  • VDN sums per-agent utilities, while QMIX combines them through a state-dependent continuous monotonic function to estimate the joint action value.
  • Deep QMIX minimises squared temporal-difference error on replay-buffer samples, using target-network estimates and hypernetwork-generated nonnegative mixing weights.

3 QMIX Operator

The idealised QMIX operator computes Q-learning targets and projects them into the monotonic function class, but this projection can have non-unique solutions and incorrect greedy actions. These failures persist even with exact optimisation, complete state-action updates, and unrestricted representation within Qmix.

  • Operator definition: QMIX represents its restricted class Qmix as monotonic functions of tabular per-agent utilities.The analysis treats Qmix as the space of all joint-action values representable by monotonic mixing functions.
  • Operator definition: The tabular analysis assumes exact optimisation over every state and joint action, while allowing multiple objective-minimising solutions.A random q is returned when the optimisation does not have a unique solution; the setup removes exploration and optimisation-accuracy confounds.
  • Operator definition: The idealised operator separates target computation using T* from projection into Qmix.This decomposition defines QMIX as the composition of the projection operator and the optimal Bellman target operator.
  • Properties of T*: Applying the projection to Q* can yield multiple global minima and an incorrect argmax because Q* may not belong to Qmix.Table 1 illustrates distinct minimising Qtot values, while Table 2 illustrates a projected value function with the wrong maximal action.
  • Properties of T*: QMIX can underestimate the true optimal joint action, including −12 instead of 8 when its argmax is incorrect.When QMIX selects the correct argmax, it represents the maximum Q-value perfectly; underestimation occurs when the selected joint action is not truly optimal.
  • Properties of T*: These failure modes are fundamental to the restricted objective and are independent of compute, exploration, and mixing-function or utility parametrisation limits.The analysis exactly minimises the objective, updates every state-action pair, and assumes Qtot can be any member of Qmix.

4 Weighted QMIX Operator

Weighted QMIX changes QMIX’s projection by weighting joint actions, prioritising accurate estimates of better actions while retaining a monotonic representational class. In the idealised setting, the proposed weightings recover optimal action choices and support convergence to an optimal policy.

  • Weighted projection: Weighted QMIX introduces a projection operator that applies a weighting function to joint-action errors instead of treating all actions equally.Setting w(s, u) = 1 recovers the original QMIX projection.
  • Motivation: Uniform projection can underestimate the optimal action because errors across suboptimal actions outweigh improved estimation of the single optimal action.This produces a suboptimal policy even when the optimal action value is otherwise identifiable.
  • Idealised Central Weighting: Idealised Central Weighting down-weights every suboptimal action and, for some α > 0, guarantees argmax ΠwQ = argmax Q for any Q.The weighting requires computing the maximum over the joint action space.
  • Optimistic Weighting: Optimistic Weighting gives higher weight to actions underestimated relative to Q and guarantees the projected Qtot has the correct argmax for some α > 0.This provides a practical alternative to directly identifying the maximal joint action.
  • Deep-RL-oriented operator: Weighted QMIX learns an unrestricted approximation ˆQ∗ while using monotonic Qtot to suggest actions for tractable maximisation and decentralised execution.The unrestricted ˆQ∗ supplies richer value estimates, while Qtot remains useful for maximisation.
  • Guarantees: For Idealised Central or Optimistic Weighting, ˆQ∗ converges to Q∗ and the weighted projection recovers an optimal policy.The result preserves the restricted Qtot function class and changes the projection solution through weighting.

5 Deep RL Algorithm

The deep-RL implementation combines monotonic Qtot, an unrestricted ˆQ∗, and a weighting function in a partially observable setting. CW and OW QMIX use practical weighting procedures to focus learning on important joint actions and improve robustness and coordination performance.

  • Setting: Deep Weighted QMIX extends the idealised analysis to deep RL in a Dec-POMDP setting where agents do not observe the full state.The implementation combines neural-network function approximators with the weighted projection idea.
  • Architecture: Weighted QMIX has three components: monotonic Qtot, unrestricted joint-action ˆQ∗, and a weighting function w.Qtot supplies per-agent utilities and the mixing network from which decentralised policies are derived.
  • Targets: For each minibatch transition, yi uses ˆQ∗ evaluated at the next state and the action suggested by argmaxu′ Qtot(τ′, u′).Qtot’s monotonicity makes this maximisation tractable.
  • ˆQ∗: The unrestricted ˆQ∗ mixing network is not constrained to be monotonic, allowing a richer approximation of joint action values.It is conditioned on the state and agents’ action-observation histories and does not share agent-network parameters with Qtot.
  • CW QMIX: Centrally-Weighted QMIX approximates idealised central weighting locally by comparing each target yi with ˆQ∗ at Qtot’s suggested action.The method is used because the maximal joint action under ˆQ∗ is computationally infeasible to compute directly.
  • OW QMIX: Optimistically-Weighted QMIX uses an exact practical weighting that increases emphasis on joint actions whose current Qtot estimate is below yi.The algorithm is named Optimistically-Weighted QMIX (OW).
  • Effect of weighting: Separating joint-action weighting from replay-buffer behaviour reduces dependence on exploration frequency and encourages better policies across exploration strategies.QMIX instead weights joint actions proportionally to their execution by behaviour policies.

6 Results

Experiments evaluate Weighted QMIX on Predator Prey and SMAC scenarios, emphasizing coordination, exploration robustness, and limitations. Weighted QMIX solves Predator Prey faster than QTRAN and generally improves robustness, while performance can depend on exploration and architecture.

  • 6.1 Predator Prey: In Predator Prey, CW-QMIX and OW-QMIX solve the task faster than QTRAN, while QMIX, MADDPG, MASAC, and QPLEX fail to solve it.QMIX, MADDPG, and MASAC achieve negative or non-positive test outcomes; QPLEX also fails despite representing unrestricted joint-action values.
  • 6.2.1 Robustness to increased exploration: With exploration annealed from 1 to 0.05 over 1 million timesteps, both Weighted QMIX variants significantly outperform all baselines on 3s5z and 5m_vs_6m.The schedule is longer than the 50k-timestep schedule used in prior work.
  • 6.2.1 Robustness to increased exploration: On bane_vs_bane, CW-QMIX and OW-QMIX solve the 24-agent task across exploration levels, whereas QMIX fails.The comparison uses two epsilon schedules.
  • 6.2.2 Necessity of increased exploration: On 6h_vs_8z, a larger exploration rate is required, and only Weighted QMIX successfully recovers a good policy.The result demonstrates a benefit in a challenging coordination problem.
  • 6.2.3 Limitations: On super-hard maps, extra exploration is unhelpful on 3s5z_vs_3s6z, while only extended-exploration QMIX achieves non-zero performance on corridor.Weighted QMIX’s poor corridor performance indicates that its additional complexity, particularly learning ˆQ∗, can sometimes harm performance.

7 Conclusions and Future Work

The conclusion presents Weighted QMIX as a weighted-projection extension of QMIX with theoretical policy-recovery guarantees and improved coordination and exploration robustness. It also identifies extra modeling complexity and the architecture of ˆQ∗ as practical limitations.

  • 7 Conclusions and Future Work: Weighted QMIX changes QMIX’s unweighted projection by emphasizing better joint actions without expanding the representable Qtot function class.The paper proves that two specific weightings recover the correct maximal joint action for any Q.
  • 7 Conclusions and Future Work: The method additionally learns an unrestricted joint-action ˆQ∗, which the paper proves converges to Q∗.The resulting weighted projection recovers an optimal policy in the idealised setting.
  • 7 Conclusions and Future Work: Deep Weighted QMIX improves coordination and robustness to increased exploration, but future work should examine more complicated weightings and limitations linked to ˆQ∗’s architecture.The conclusion notes that the weighting used in this paper is simplistic, with w taking values 1 or α.

Broader Impact

The paper situates Weighted QMIX within cooperative CTDE research and related methods that address joint-action value representation, decentralisation, or training stability. Its broader relevance is tied to coordination without communication, alongside added complexity and empirical trade-offs.

  • Broader Impact: Cooperative MARL considers agents acting independently without communication, with CTDE allowing shared information during training but only local action-observation histories during execution.This setting motivates methods that preserve decentralised execution while exploiting centralised training information.
  • Broader Impact: QMIX’s monotonic factorisation can fail when an agent’s best action depends on other agents’ actions, creating difficulties for same-timestep coordination.The paper frames Weighted QMIX as addressing this restriction rather than replacing decentralised execution.
  • Broader Impact: The method introduces extra complexity and can perform poorly in challenging domains, so the need for its additional modeling capacity should be assessed task by task.The paper presents this as a practical boundary on applicability rather than a universal failure.
  • Broader Impact: Weighted QMIX resembles an off-policy actor-critic view in which ˆQ∗ acts as critic and the policy implied by Qtot acts as actor.Its policy is trained indirectly through the weighted Qtot loss rather than through deterministic policy gradients.
  • Broader Impact: Related methods expand representable joint-action values or use alternative training objectives, including QPLEX, QOPT, QTRAN, and attention-based Qatten.QTRAN is empirically hard to scale to more complex tasks such as SMAC, while QOPT uses an unrestricted mixing network with non-negative-weight constraints removed.

B Proof of Theorems

The proofs establish that the proposed central and optimistic weightings preserve the maximal action under projection. Consequently, the weighted operator can recover optimal policies and Q∗ in the idealised setting.

  • Propositions: For any weighting w with positive weights, a maximising action of the weighted projection has a projected value at least as large as its original Q-value.If the projected maximiser is the true optimal action, its projected value equals the original value.
  • Proof strategy: The proof compares weighted projection losses after perturbing estimates of maximal actions, showing that an incorrect projected maximiser cannot remain optimal under the proposed weighting.The argument uses the action gap and bounds on Q-value differences to preserve the true maximal action.
  • Propositions: If the projected Qtot has a unique argmax, that action’s Qtot value exactly matches Q.This follows from the proposition guaranteeing at least one maximising action with equality.
  • Theorems 1 and 2: For both Idealised Central Weighting and Optimistic Weighting, some α > 0 makes argmax ΠwQ equal argmax Q for any Q.The proof uses the action gap to choose a sufficiently small positive weighting parameter.
  • Corollary 3: For either weighting, the unique fixed point of the weighted optimality operator is Q∗, and the weighted projection of Q∗ recovers an optimal policy.The maximum projected value also equals the maximum value of Q∗.
  • Implementation: The deep implementation uses Qtot, an unrestricted ˆQ∗ network, and a weighting function, with feed-forward architectures and selected α values for experiments.A modified hypernetwork-based architecture for ˆQ∗ is used in Figure 6’s corridor experiment.

C.2 SMAC Robustness to exploration

The experiments specify α settings for Weighted QMIX, its ablations, QTRAN, MASAC, and the Figure 6 comparison across SMAC tasks.

  • Weighted QMIX: Weighted QMIX evaluates α ∈ {0.01, 0.1, 0.5, 0.75}, selecting α = 0.75 for CW and α = 0.5 for OW.These settings are also stated for the SMAC experiments.
  • Ablations: Weighted QMIX ablations evaluate α ∈ {0.5, 0.75} and use α = 0.75.
  • Baselines: QTRAN uses λopt = 1, varies λnopt ∈ {1, 10}, and fixes the mixing-network dimension at 64.λnopt = 10 is used for 5m_vs_6m and λnopt = 1 for 3s5z.
  • Baselines: MASAC varies αent ∈ {0, 0.001, 0.01}, using 0 for 3s5z and 0.01 for 5m_vs_6m.
  • Figure 6: The Figure 6 experiment considers α ∈ {0.5, 0.75} and sets α = 0.75 for both methods.

D Ablations

The ablations separate the effects of improved bootstrap estimates from weighting, using predator-prey return and exploration-robustness comparisons.

  • Ablation design: The ablations add QMIX + ˆQ* and QMIX + CW/OW to isolate bootstrap and weighting effects.QMIX + ˆQ* removes weighting while using ˆQ*; QMIX + CW/OW adds CW/OW weighting without learning ˆQ*.
  • Predator Prey: Figure 7 compares median test return in predator prey among Weighted QMIX and three ablations.
  • Exploration robustness: Figure 8 compares median test win percentage under increased exploration for Weighted QMIX and three ablations.

E Results

Additional experiments report ablation results for Predator Prey and increased exploration, including the effects of weighting, bootstrap estimates, and α.

  • Additional experiments: The section presents additional experiments that did not fit in the main paper, including Predator Prey ablations and exploration-robustness tests.
  • Predator Prey: The Predator Prey ablations show that both ˆQ* and weighting are necessary to solve the task.QMIX + ˆQ* cannot solve the task, while uniform projection weighting prevents recovery of the optimal coordinated policy.
  • Exploration robustness: Under increased exploration, combining QMIX with weighting performs significantly worse in 5m_vs_6m and no better in 3s5z.These results support the need for both ˆQ* and weighting for consistent performance.
  • Effect of α: When α is too low, performance degrades considerably for CW-QMIX and OW-QMIX.
  • Effect of α: Figure 9 varies α in the weighting functions for CW-QMIX and OW-QMIX.
Loading 2006.10800v2…