PaperlayerView exact revision on arXiv

Source-linked AI summary

QPLEX: Duplex Dueling Multi-Agent Q-Learning

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, Chongjie Zhang

arXiv:2008.01062v3cs.LGcs.AIcs.MAstat.ML

TL;DR

Value-based MARL under CTDE must balance scalable value factorization with exact consistency between joint and local action choices. QPLEX addresses this with a duplex dueling architecture that encodes IGM, achieves a complete IGM function class, and significantly outperforms baselines in online and offline settings. Its offline-training capability supports high sample efficiency and use of multi-source offline datasets, although QTRAN’s relaxed IGM consistency can perform poorly in complex domains.

  • Problem

    Existing scalable MARL value factorizations limit representation expressiveness or relax IGM consistency, risking instability or poor performance in complex domains.

  • Method

    QPLEX factorizes joint action values with a duplex dueling network that encodes IGM consistency into the neural architecture.

  • Results

    QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings and achieves a complete IGM function class.

  • Takeaways & Limitations

    QPLEX supports offline training with high sample efficiency and opportunities to use multi-source offline datasets without additional online exploration.

  • Takeaways & Limitations

    QTRAN can perform well in didactic matrix games but suffers from relaxed IGM consistency in complex domains such as StarCraft II.

Abstract

from arXiv · show

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

1 INTRODUCTION

Cooperative MARL must address scalability and partial observability while preserving consistency between joint and individual action selection. QPLEX uses a duplex dueling architecture to guarantee IGM consistency, and experiments report strong performance in didactic and StarCraft II tasks.

  • Cooperative MARL faces scalability because the joint state-action space grows exponentially with the number of agents.
  • The IGM principle requires joint greedy action selection to match the collection of agents’ individual greedy actions for decentralized execution.
  • VDN and QMIX impose structural constraints that limit the joint action-value functions they can represent.
  • QPLEX factorizes joint action values through a duplex dueling architecture that encodes guaranteed IGM consistency.
  • QPLEX demonstrates full representation expressiveness in didactic problems and significantly outperforms multi-agent Q-learning baselines in online and offline StarCraft II settings.The reported offline-training ability is associated with stability, sample efficiency, and use of offline data without additional online exploration.

2 PRELIMINARIES

CTDE trains agents centrally but requires decentralized decisions from local histories under partial observability. Value-based methods use IGM-compatible factorizations, while existing additive, monotonic, or relaxed structures have expressiveness or consistency limitations.

  • A Dec-POMDP models cooperative agents acting in global states with partial observations, local action-observation histories, and a joint policy.
  • The joint action-value function Qπ(s, a) combines immediate reward with the discounted expected value of the next state.
  • Q-learning seeks the optimal joint action-value function using the maximum next-state action value, while deep Q-learning represents it with a neural network.
  • Multi-agent Q-learning uses replay memory and minimizes expected temporal-difference error with a periodically updated target network.
  • CTDE centralizes training but requires each agent to act from its local action-observation history during decentralized execution.
  • IGM requires consistency between greedy actions selected from the joint value and those selected from individual action values.
  • VDN uses additive factorization and QMIX uses monotonicity as sufficient IGM structures.
  • Qatten adds global information through multi-head attention, while QTRAN and WQMIX relax IGM-related constraints but may lose exact consistency or struggle in complex problems.

3 QPLEX: DUPLEX DUELING MULTI-AGENT Q-LEARNING

QPLEX realizes advantage-based IGM through a duplex dueling architecture that transforms individual values and mixes them into a joint value while preserving expressive factorization. Its attention-based importance weights and theoretical result support scalable, complete IGM value-function representation.

  • Advantage-based IGM: QPLEX reformulates IGM consistency as constraints on joint and individual advantage functions within a duplex dueling architecture.The architecture uses joint and individual decompositions Q = V + A, transferring consistency constraints from action values to advantages.
  • QPLEX architecture: The Transformation network incorporates centralized state or joint-history information into individual action-value functions using positive weights, preserving greedy-action consistency.For each agent, the positive transformation scales local advantages while adding a centralized bias.
  • QPLEX architecture: The Dueling Mixing network combines transformed individual values into joint value and advantage terms, then outputs Qtot(τ, a) = Vtot(τ) + Atot(τ, a).The joint value uses a sum structure, while the joint advantage is formed from individual advantages and importance weights.
  • QPLEX architecture: QPLEX uses scalable multi-head attention to learn positive importance weights λi from joint history and action, with sigmoid activation adding sparsity to credit assignment.The attention module supports different network capacities and scenarios while maintaining IGM consistency.
  • Empirical illustration: In the harder one-step game, QPLEX, QTRAN, and WQMIX reach optimal performance, whereas QMIX, VDN, and Qatten converge to local optima from miscoordination penalties.The figure also varies QPLEX attention-network size for learning importance weights λi.
  • Expressiveness: Proposition 2 states that, with universal neural-network approximation, QPLEX realizes exactly the action-value function class induced by IGM.This establishes complete IGM expressiveness rather than the restricted value classes of sufficient-condition decompositions.

4 EXPERIMENTS

QPLEX is evaluated on didactic coordination problems and 17 StarCraft II benchmark scenarios, where it demonstrates complete expressiveness, stable learning, and strong online performance. Its duplex dueling architecture also supports effective data collection and higher sample efficiency than the compared baselines.

  • 4.1 MATRIX GAMES: QPLEX, QTRAN, and WQMIX achieve optimal performance in harder matrix games, while QMIX, VDN, and Qatten fall into miscoordination-induced local optima.QTRAN performs well in matrix games but relaxes IGM consistency in complex domains.
  • 4.2 TWO-STATE MMDP: QPLEX’s complete IGM expressiveness is evaluated through a two-state MMDP designed to examine optimality and training stability.Two agents must coordinate on action A(1) at state s2 to obtain positive reward.
  • 4.3 STARCRAFT II: QPLEX is compared with QTRAN, QMIX, VDN, Qatten, and WQMIX on 17 StarCraft II scenarios using averaged median test win rate and scenario-level best-performance counts.Figure 4 summarizes performance across 14 SMAC tasks and three new super hard cooperative tasks.
  • 4.3.1 TRAINING WITH ONLINE DATA COLLECTION: At least 10% higher median test win rate averaged across all 17 scenarios is reported for QPLEX, which is best on up to eight tasks and underperforms on just two.After 0.8M timesteps, QPLEX is best on five tasks as other methods approach near-perfect performance on easier tasks.
  • 4.3.1 TRAINING WITH ONLINE DATA COLLECTION: On online StarCraft II learning curves, QPLEX significantly outperforms baselines with higher sample efficiency, including a test-win-rate gap exceeding 30% on 5s10z.The paper attributes QPLEX’s suitability for epsilon-greedy data collection to hard IGM constraints and effective individual greedy action selections.
  • 4.3.2 TRAINING WITH OFFLINE DATA COLLECTION: QPLEX is reported to outperform state-of-the-art baselines in both online and offline data-collection settings, while retaining theoretical advantages over QTRAN in offline centralized training.The supplied passages state that QPLEX can benefit from offline training and datasets without additional online exploration.

5 CONCLUSION

The paper introduces QPLEX as a centralized-training, decentralized-execution multi-agent Q-learning framework based on duplex dueling value factorization. It reports a complete IGM function class, superior online and offline benchmark performance, and strong support for offline training.

  • 5 CONCLUSION: QPLEX factorizes joint action values through a duplex dueling architecture that encodes IGM consistency for decentralized execution.The framework uses centralized end-to-end training.
  • 5 CONCLUSION: QPLEX achieves a complete IGM function class according to the paper’s theoretical analysis.This result concerns the action-value representations induced by the IGM principle.
  • 5 CONCLUSION: QPLEX significantly outperforms state-of-the-art baselines in both online and offline data-collection settings.The conclusion highlights strong ability to support offline training.
  • 5 CONCLUSION: QPLEX’s offline-training ability provides high sample efficiency and opportunities to use offline multi-source datasets.The paper identifies continuous-action offline MARL as a future direction.

A OMITTED PROOFS IN SECTION 3

The omitted proofs establish that advantage-based IGM is equivalent to IGM and that QPLEX can realize the full IGM-induced action-value function class under universal neural-network approximation.

  • Definitions: The duplex dueling formulation represents joint and individual action values as state values plus advantage functions.The joint form is Qtot = Vtot + Atot, while each individual form is Qi = Vi + Ai.
  • Proposition 1: Proposition 1 states that the advantage-based IGM and IGM function classes are equivalent.The proof uses the fact that state-value functions do not affect greedy action selection.
  • Fact 1: Advantage-based IGM constrains optimal joint advantages to zero and nonoptimal joint and individual advantages to nonpositive values.These value-range constraints preserve the greedy-action structure required by IGM.
  • Proposition 2: Proposition 2 states that, given universal function approximation, QPLEX realizes an action-value function class equivalent to the class induced by IGM.The construction uses positive transformation weights and a positive joint advantage function.
  • Proof of Proposition 2: The proof constructs transformed joint and individual functions satisfying the advantage-based IGM constraints, then derives equality between the QPLEX and target action-value functions.The conclusion follows from the equivalence established in Proposition 1.

B.2 IMPLEMENTATION DETAILS

This section presents implementation details for QPLEX and the experimental configurations used across benchmark and matrix-game studies.

  • Baseline implementation: The experiments evaluate QPLEX against QTRAN, QMIX, VDN, Qatten, and WQMIX using PyMARL implementations.Baseline hyperparameters follow SMAC settings and source-code defaults.
  • Network implementation: QPLEX uses a duplex dueling architecture with local advantage functions and gradient stopping for optimization stability.The joint action-value function is approximated using a variant of the local advantage function with stopped gradients.
  • Evaluation: Test win rate is the percentage of episodes in which agents defeat all enemies within the time limit.Evaluation uses 32 episodes with decentralized greedy action selection every 10k timesteps.
  • Online data collection: Online training uses 2 million timesteps, ε-greedy exploration, and a 5000-episode FIFO replay buffer.Exploration is annealed from 1.0 to 0.05 over 50k timesteps.
  • Offline data collection: Offline training uses datasets collected from QMIX or VDN behavior policies, with 20k or 50k experienced episodes.QPLEX and four baselines are evaluated over six random seeds across three datasets.

D EXPERIMENTS ON STARCRAFT II WITH ONLINE DATA COLLECTION

The online StarCraft II experiments compare learning performance across the remaining SMAC scenarios and summarize results over all 14 benchmark scenarios.

  • Scenario-level results: The remaining StarCraft II online-data learning curves are presented as deferred results.These curves complement the main online-data experiments on the other scenarios.
  • Benchmark-wide results: The averaged median test win rate is reported across all 14 SMAC scenarios.The figure also counts scenarios where an algorithm’s median test win rate is highest by at least 1/32.

E ABLATION STUDIES WITH ONLINE DATA COLLECTION

The ablation studies test whether QPLEX’s attention-based factorization and parameter count explain its online performance and examine learned strategies.

  • Attention ablation: The ablations compare QPLEX without multi-head attention against the full model using averaged median test win rate across StarCraft II tasks.Detailed learning curves are provided for the ablated model and the parameter-matched comparison.
  • Parameter-count control: QPLEX still significantly outperforms Large QMIX despite their similar parameter counts.The result supports the duplex dueling value-factorization structure as the main source of the difference rather than network size.
  • Learning-curve comparisons: The comparison includes learning curves for QPLEX, QPLEXwo-duel-atten, QMIX, Qatten, and Large QMIX.These curves separate attention effects from parameter-count effects across the online setting.
  • Visualized strategies: On the 5s10z map, QPLEX uses melee soldiers to form defensive lines, while QMIX fails to coordinate them effectively.The resulting strategy leaves ranged soldiers less exposed to direct enemy attacks under QPLEX.

G EXPERIMENTS ON STARCRAFT II WITH OFFLINE DATA COLLECTION

The offline StarCraft II experiments report deferred learning curves for the tested scenarios under offline data collection.

  • Offline-data results: Deferred learning curves show StarCraft II performance under offline data collection on the tested scenarios.The figure presents results without describing individual scenario values in the supplied passage.

H ABLATION STUDIES ABOUT QPLEX WITH DIFFERENT NETWORK CAPACITIES IN STARCRAFT II

QPLEX’s attention capacity trades off representational expressiveness against learning efficiency. In StarCraft II, one layer and four heads provide sufficient expressiveness, while deeper attention can reduce learning efficiency.

  • QPLEX trades off the expressiveness and learning efficiency of its multi-head attention module for estimating importance weights λ_i.More layers and heads increase representational capacity, but the added complexity can require more samples.
  • One layer and four heads are sufficient for QPLEX to handle complex StarCraft II micromanagement tasks.These tasks involve large state-action spaces and 2 to 27 agents.
  • Adding heads without adding layers does not change QPLEX’s performance in the StarCraft II ablation.The comparison is between QPLEX-1L4H and QPLEX-1L10H.
  • Adding layers may slightly reduce QPLEX’s learning efficiency in this complex domain.QPLEX-2L4H uses more parameters and requires more samples, which may offset the benefits of greater expressiveness.

I ABLATION STUDIES ABOUT QTRAN

The ablations examine whether QPLEX’s advantage over QTRAN comes from its architecture and modules. Results indicate that the duplex dueling architecture is the main contributor, while Transformation improves QPLEX but not QTRAN substantially; QPLEX also shows higher sample efficiency than QMIX in predator-prey.

  • QPLEX uses a duplex dueling architecture to impose IGM as a hard constraint, whereas QTRAN uses soft penalties to approximate it.QTRAN also lacks QPLEX’s Transformation module on individual Q-functions and cannot use multi-head attention in the same way.
  • QPLEX-wo-trans-atten significantly outperforms QTRAN and QTRAN-w-trans, indicating that QPLEX’s duplex dueling architecture largely explains its advantage over QTRAN.The comparison uses averaged median test win rate across all tested scenarios.
  • Adding the Transformation module does not significantly improve QTRAN’s performance.The QTRAN-w-trans variant adds Transformation to individual Q-functions.
  • The Transformation module is useful for QPLEX empirically and is necessary for QPLEX to realize IGM.The multi-head attention module is not necessary for QPLEX to handle the tested StarCraft II tasks.
  • QPLEX solves the predator-prey task with the introduced ε-greedy exploration strategy and achieves higher sample efficiency than QMIX using the same strategy.The strategy anneals ε from 1 to 0.05 over 1 million timesteps.
Loading 2008.01062v3…