Source-linked AI summary
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, Yung Yi
TL;DR
Value-factorization methods such as VDN and QMIX are limited by additive and monotonic structure, leaving many factorizable cooperative MARL tasks uncovered. QTRAN transforms the joint action-value function into an easily factorized form while preserving optimal actions, and experiments show substantial advantages over VDN and QMIX, especially under severe non-monotonicity.
Problem
VDN and QMIX cover only subsets of factorizable MARL tasks because they impose additivity and monotonicity constraints.
Method
QTRAN transforms the joint action-value function into an easily factorized one with the same optimal actions, using joint, individual, and state-value networks.
Results
QTRAN outperforms VDN and QMIX by a substantial margin, especially when games exhibit more severe non-monotonic characteristics.
Takeaways & Limitations
QTRAN provides a more general factorization method for cooperative MARL tasks while retaining centralized training and fully decentralized execution.
Takeaways & Limitations
QTRAN training can encounter instability because the joint action-value for sampled actions may be much smaller than for other actions.
Abstract
from arXiv · showhide
We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones for decentralized execution. VDN and QMIX address only a fraction of factorizable MARL tasks due to their structural constraint in factorization such as additivity and monotonicity. In this paper, we propose a new factorization method for MARL, QTRAN, which is free from such structural constraints and takes on a new approach to transforming the original joint action-value function into an easily factorizable one, with the same optimal actions. QTRAN guarantees more general factorization than VDN or QMIX, thus covering a much wider class of MARL tasks than does previous methods. Our experiments for the tasks of multi-domain Gaussian-squeeze and modified predator-prey demonstrate QTRAN's superior performance with especially larger margins in games whose payoffs penalize non-cooperative behavior more aggressively.
1. Introduction
Cooperative MARL requires scalable decentralized execution, but existing value-factorization methods cover only tasks satisfying additivity or monotonicity. QTRAN removes these structural constraints by transforming joint values while preserving optimal actions, and outperforms VDN and QMIX especially in strongly non-monotonic environments.
- Motivation: Cooperative MARL tasks remain difficult because their scale and distributed-execution constraints complicate learning effective joint policies.Deep learning has succeeded in single-agent domains, but robot swarm control and autonomous driving remain challenging cooperative multi-agent problems.
- Existing approaches: Value-function factorization handles joint action-value functions whose complexity grows exponentially with the number of agents.The approach supports decentralized execution by representing joint values through individual action-value functions.
- Existing approaches: VDN uses additive factorization, whereas QMIX represents joint values as a monotonic function of individual action values.QMIX covers a richer class of problems than VDN, but both retain structural restrictions.
- Limitations: These structural constraints can prevent VDN and QMIX from factorizing tasks whose optimal joint actions are otherwise factorable.The limitation arises from additive decomposability in VDN and monotonicity in QMIX.
- QTRAN: QTRAN transforms the original joint action-value function into an easily factorized function with the same optimal actions, using a learned state-value correction for partial observability.Its architecture includes joint action-value, individual action-value, and state-value networks.
- Evaluation: QTRAN successfully solves a non-additive, non-monotonic matrix game and shows increasing performance advantages over VDN and QMIX as environments become more non-monotonic.The evaluation covers a matrix game, modified predator-prey games, and multi-domain Gaussian squeeze tasks.
2. Background
The paper models cooperative MARL as a partially observable DEC-POMDP and uses CTDE to connect individual action selection with joint-value optimization. It defines factorization through IGM, while showing that VDN and QMIX cover only additive or monotonic cases.
- DEC-POMDP: A cooperative MARL task is modeled as a DEC-POMDP with shared rewards, state transitions, partial observations, and agent policies conditioned on action-observation histories.The model includes a joint action vector and a discount factor.
- Training and execution: CTDE: CTDE trains individual action-value functions with centralized information so decentralized execution can optimize the joint action-value function.At execution, agents select actions from individual value functions without referring to the joint function.
- IGM condition: IGM holds when individual action-value functions produce optimal actions equivalent to the optimal joint actions.A task is factorizable when this equivalence holds for every joint observation history.
- VDN and QMIX: Additivity and monotonicity are sufficient conditions for factorization, which VDN and QMIX respectively assume.Joint action-value functions that violate these conditions may not be well-factorized by either method.
3. QTRAN: Learning to Factorize with Transformation
QTRAN transforms the joint action-value function into one that preserves optimal joint actions while enabling more general factorization through individual action-values. Its architecture combines joint, individual, and state-value networks, with QTRAN-alt modifying non-optimal-action treatment to improve stability and convergence.
- Transformation: QTRAN transforms Qjt into Q′jt, preserving the optimal joint action while making the transformed function factorizable by individual action-values.The transformation targets factorizable tasks without imposing additivity or monotonicity on the original joint action-value function.
- Factorization conditions: Theorem 1 gives sufficient factorization conditions for individual functions [Qi], with affine transformations preserving those conditions because IGM is invariant to positive rescaling and additive shifts.The conditions relate local optimal actions and the transformed joint action-value function, while the affine transformation leaves the relevant argmax structure unchanged.
- Architecture: QTRAN uses separate individual action-value, joint action-value, and state-value networks trained centrally, while agents execute decentralized actions using their own Qi functions.The state-value network supplies flexibility for matching Qjt and Q′jt at the argmax and corrects discrepancies associated with partial observability.
- Training: Training combines TD, optimal-action, and non-optimal-action losses to estimate Qjt and enforce the factorization conditions for [Qi].The global objective is L = Ltd + λoptLopt + λnoptLnopt; Lopt checks optimal local actions, while Lnopt checks sampled actions.
- QTRAN variants: QTRAN-alt strengthens tracking for problematic non-optimal actions, improving its ability to distinguish optimal from non-optimal actions and supporting more stable, sample-efficient learning.The alternative condition makes Q′jt track Qjt even when a non-optimal action has a much smaller Qjt value than the optimum.
4. Experiment
Experiments in Multi-domain Gaussian Squeeze and modified predator-prey evaluate QTRAN against VDN and QMIX on non-monotonic cooperative tasks. QTRAN achieves higher rewards and more effective cooperation, with larger gaps as penalties or agent counts increase.
- Environments: QTRAN, VDN, and QMIX are evaluated in Multi-domain Gaussian Squeeze and modified predator-prey environments.The experiments include varying numbers of domains, predators, prey, penalties, and agents.
- Multi-domain Gaussian Squeeze: Under full exploration in GS, QTRAN learns approximately the same policy, while VDN and QMIX significantly deteriorate.The structural constraints of VDN and QMIX hinder accurate learning of non-monotonic action-value functions.
- Multi-domain Gaussian Squeeze: In MGS, VDN and QMIX learn sub-optimal policies, whereas QTRAN-base and QTRAN-alt achieve significantly higher rewards.QTRAN-alt is more stable because its alternative loss increases the gap between non-optimal and optimal actions.
- Modified predator-prey: In MPP, QTRAN outperforms VDN and QMIX across six settings, with larger performance gaps as penalty P increases.Higher P makes the environment less monotonic; QTRAN continues cooperating, while VDN and QMIX increasingly minimize penalty risk instead of catching prey.
- Modified predator-prey: With N = 4 agents, QTRAN-alt achieves higher scores than QTRAN-base, while QTRAN-base requires slower convergence.For N = 2, both variants ultimately train agents to cooperate well for every tested penalty value.
5. Conclusion
The paper presents QTRAN as a factorization method for a wide variety of MARL tasks under centralized training and decentralized execution. Its theoretical and simulation results indicate broader task coverage and substantial performance advantages over VDN and QMIX, especially in more severely non-monotonic games.
- Conclusion: QTRAN transforms and factorizes joint action-value functions into individual functions while supporting centralized training and decentralized execution.The method is designed for a wide variety of MARL tasks.
- Conclusion: QTRAN handles a richer class of tasks than its predecessors according to the paper’s theoretical analysis.
- Conclusion: QTRAN outperforms VDN and QMIX by a substantial margin, especially when games exhibit more severe non-monotonic characteristics.
A. QTRAN Training Algorithm
The training algorithm initializes QTRAN networks and replay memory, collects and samples transitions, and updates either QTRAN-base or QTRAN-alt with their respective losses. Target network parameters are periodically updated during training.
- Initialization: Algorithm 1 initializes replay memory, individual action-value networks, the joint action-value network, the state-value network, and target parameters.
- Interaction and replay: During each episode, agents observe histories, select actions with epsilon-greedy exploration, receive transitions, and store them in replay memory.
- Target computation: The algorithm samples transition minibatches and computes a DQN target using the target joint action-value network and locally greedy actions.
- Optimization: QTRAN-base and QTRAN-alt update parameters by minimizing their respective losses.
- Target updates: Target network parameters are periodically updated before the episode loop continues.
B.2. Necessity in Theorem 1 Under Affine-transformation
The section shows that affine transformations of individual action-value functions can preserve optimal actions while satisfying the factorization conditions required by Theorem 1. It then constructs transformed factors meeting a stronger condition for arbitrary numbers of agents.
- Necessity under affine transformation: An affine transformation φ(Q_i)=a_iiQ_i+b_i can preserve factorization while making the conditions of Theorem 1 hold.The transformation uses positive diagonal scaling and additive offsets.
- Fully observable case: For fully observable tasks, the state-value network is unnecessary because V_jt(τ) can be set to zero.The fully observable case is handled by Theorem 1a and is applied to the simple matrix game.
- Fully observable case: Theorem 1a links the factorization conditions to IGM by showing that the joint optimal action equals the collection of individual optimal actions.The proof uses conditions (9a) and (9b) to establish arg max_u Q_jt(τ,u)= [arg max_{u_i} Q_i(τ_i,u_i)].
- Stronger factorization condition: For the matrix-game illustration, replacing Q_1(B) and Q_1(C) produces a final table satisfying condition (7).The example reports β=0.23 for u_1=B before the replacements.
C.1. Environment
The experiments use a simple matrix game, Multi-domain Gaussian Squeeze, and modified predator-prey environments to test factorization under non-monotonicity and stronger cooperation requirements.
- Matrix game: The matrix game uses two agents with 21 actions each and includes distinct global and local reward maxima.The global maximum is at (u_1,u_2)=(5,15), while the local maximum is at (15,5).
- Multi-domain Gaussian Squeeze: Multi-domain Gaussian Squeeze uses ten agents with action spaces {0,1,...,9} and resource-dependent rewards across one or two Gaussian domains.The two-domain setting contains an easier lower-reward maximum and a harder higher-reward maximum.
- Modified predator-prey: Modified predator-prey uses grid worlds with two predators and one prey or four predators and two prey, requiring simultaneous captures for positive reward.The team receives 1 when at least two predators catch prey simultaneously and −P when only one predator catches it.
C.2. Experiment details
The experiments compare VDN, QMIX, and QTRAN using neural-network configurations and hyperparameters specified separately for the matrix game, Gaussian Squeeze, and predator-prey environments.
- Matrix game: In the matrix game, individual action-value networks use two hidden layers, while QMIX adds a monotone network and QTRAN adds a joint action-value network.The QTRAN joint action-value network also uses two hidden layers.
- Multi-domain Gaussian Squeeze: In Multi-domain Gaussian Squeeze, individual action-value networks use three hidden layers, with QMIX and QTRAN adding their respective mixing or joint-value components.The hidden-layer widths are 64.
- Modified predator-prey: In modified predator-prey, individual networks use three hidden layers, QMIX includes a monotone network, and QTRAN includes an additional joint action-value network.All hidden-layer widths are 64.
D. Additional results for matrix games
Additional matrix-game results show that QTRAN handles non-monotonic payoff structures more reliably than VDN and QMIX, while QTRAN variants recover the optimal action in the illustrated game.
- Random matrix games: QTRAN-base trains the optimal action in all 310 random 3×3 matrix games, whereas VDN and QMIX fail on more than half of non-monotonic games.The comparison uses final performance levels across 310 random payoff matrices.
- Structural constraints: QMIX outperforms VDN in 14 differing cases, while VDN outperforms QMIX in 5, supporting a stronger additivity constraint for VDN.The comparison concerns the 19 cases where their results differ.
- Value-function visualization: In the illustrated 20-action matrix game, VDN and QMIX learn the sub-optimal action (15,5), while QTRAN-base and QTRAN-alt learn the optimal action.QTRAN-alt more accurately distinguishes optimal from non-optimal actions in the transformed value function.
- Modified predator-prey: Figure 12 compares average reward per episode with 95% confidence intervals for VDN, QMIX, Dec-HDRQN, and QTRAN on modified predator-prey tasks.The caption identifies the compared methods and uncertainty intervals but does not state the numerical outcome.
E. Comparison with other value-based methods for modified predator-prey
Dec-HDRQN is competitive on relatively simple modified predator-prey tasks but performs worse as penalties and agent counts increase, while QTRAN outperforms it in the reported case.
- QTRAN outperforms Dec-HDRQN when modified predator-prey task complexity is relatively low.In this setting, Dec-HDRQN performs better than VDN and QMIX, but QTRAN performs better than Dec-HDRQN.
- Dec-HDRQN underperforms VDN and QMIX when the penalty and number of agents are larger.
- At N = 2 and P = 1.5, Dec-HDRQN scores an average of nearly 0.
- Dec-HDRQN has a limit because it is heuristic and does not perform centralized training.
- Dec-HDRQN shows slower convergence speed overall.