Source-linked AI summary
Value-Decomposition Multi-Agent Actor-Critics
Jianyu Su, Stephen Adams, Peter A. Beling
TL;DR
Multi-agent methods must support decentralized local-history policies, model-free learning, no communication, and parameter sharing. VDAC decomposes global state-values under a monotonicity constraint, and on StarCraft II tasks it reports strong performance, including an advantage over QMIX on a harder map.
Problem
The paper seeks multi-agent algorithms with local-history policies, unknown environment dynamics, no interagent communication, and parameter sharing.
Source: Methods · p. 3Method
VDAC is a credit-assignment actor-critic framework that enforces a monotonic relationship between global state-value and shaped local state-values.
Source: Conclusion · p. 7Results
VDAC-mix consistently achieves the best performance across tasks; on 3s5z its median test win rates at 2 million training steps exceed QMIX’s by 71%.
Source: Overall Results · pp. 6–7Takeaways & Limitations
On StarCraft micromanagement games, VDAC bridges the performance gap between multi-agent actor-critics and Q-learning while balancing training efficiency and performance.
Source: Conclusion · p. 7VDAC-sum omits state information during training and can represent only a limited class of centralized state-value functions.
Source: VDAC-sum · p. 4
Abstract
from arXiv · showhide
The exploitation of extra state information has been an active research area in multi-agent reinforcement learning (MARL). QMIX represents the joint action-value using a non-negative function approximator and achieves the best performance, by far, on multi-agent benchmarks, StarCraft II micromanagement tasks. However, our experiments show that, in some cases, QMIX is incompatible with A2C, a training paradigm that promotes algorithm training efficiency. To obtain a reasonable trade-off between training efficiency and algorithm performance, we extend value-decomposition to actor-critics that are compatible with A2C and propose a novel actor-critic framework, value-decomposition actor-critics (VDACs). We evaluate VDACs on the testbed of StarCraft II micromanagement tasks and demonstrate that the proposed framework improves median performance over other actor-critic methods. Furthermore, we use a set of ablation experiments to identify the key factors that contribute to the performance of VDACs.
Related Work
Prior MARL work ranges from decentralized learning to centralized and value-based methods. VDAC is positioned as a policy-based value-decomposition method, unlike QMIX and VDN, which decompose global action-values.
Decentralized and centralized approaches
- Decentralized learning can face non-stationarity as agents learn and explore simultaneously; centralized methods avoid that problem but sacrifice scalability.The cited work also describes approaches that stabilize decentralized learning and a paradigm alternating centralized training with decentralized training.Source: Related Work · p. 2
Value-based methods
- Value-based methods include VDN, which sums local action-values, QMIX, which uses a non-negative mixing network and state-conditioned hypernetworks, and QTRAN, a generalized factorization method.The literature also explores interagent communication, including CommNet, TarMAC, ATOC, MAAC, CCOMA, and BiCNet.Source: Related Work · p. 2
VDAC and related methods
- VDAC decomposes global state-values as a policy-based method, whereas QMIX and VDN decompose global action-values as Q-learning methods.The paper identifies COMA as its closest related work and describes it as a policy-gradient method inspired by difference rewards.Source: Related Work · p. 2
Background
The background formalizes cooperative partially observable multi-agent tasks and reviews policy-gradient and actor-critic tools. It then motivates centralized critics and counterfactual gradients for multi-agent learning.
Decentralized partially observable tasks
- In a Dec-POMDP, agents act simultaneously in a cooperative task, receive local observations, and condition stochastic policies on their own observation-action histories.The joint action is formed from the agents’ individual actions, and the environment supplies a global reward.Source: Background · p. 2
Single-agent policy gradients
- The joint value function is expected discounted return from a state, while the value-action function is expected return after a joint action and subsequent policy following.Policy gradients optimize expected reward by weighting log-policy gradients with action-values; actor-critics use a baseline to form an advantage, often estimated with TD error.Source: Background · p. 3
Multi-agent policy gradients
- Multi-agent policy gradients use decentralized policies and face high-variance estimates and credit assignment; central critics use extra state information to help address high variance.The cited example uses a central critic to estimate Q(s, joint action) for actor updates.Source: Background · p. 3
COMA
- COMA’s counterfactual policy gradients use an agent-specific advantage, and its authors argue that the gradients provide tailored credit assignment and reduce variance.The advantage compares the joint action-value with an expectation over that agent’s actions.Source: Background · p. 3
Methods
The methods specify a model-free, decentralized-policy setting and compare a naive central critic with value-decomposition actor-critics. VDAC decomposes global state-values into local values under a monotonicity constraint and establishes local convergence under stated assumptions.
Methods
- The framework targets local-history policies, unknown environment dynamics, no interagent communication, and parameter sharing across agents.These constraints define the intended general-purpose cooperative multi-agent setting.Source: Methods · p. 3
Naive Central Critic Method
- The naive central critic takes the state and previous joint action as input and outputs V(s), while actors use a TD-advantage policy gradient.Its action-value estimate is Q(s,u) = r + γV(s′), and the paper states that these gradients are unbiased estimates of the naive multi-agent policy gradients.Source: Naive Central Critic Method · p. 3
Value Decomposition Actor-Critic
- Inspired by difference rewards, VDAC decomposes global state-value into local state-values so improving an agent’s local value also improves global value when other agents remain at their local states.The paper studies VDAC-sum and VDAC-mix as two variants satisfying this relationship.Source: Value Decomposition Actor-Critic · p. 3
- VDAC-sum adds local state-values to form Vtot; actors share non-output layers with local critics that estimate those values.Its actor uses a simple TD advantage, but this variant omits training-time state information and represents only a limited class of centralized state-value functions.Source: VDAC-sum · p. 4
- VDAC-mix feeds local state-values into a network with non-negative weights to represent a broader class of monotonic global state-values.State-conditioned hypernetworks produce the mixing weights; biases are not constrained to be non-negative.Source: VDAC-mix · p. 4
- The distributed critics minimize a value loss, the central critic minimizes the corresponding loss against Vtot, and the policy network follows the same policy gradient as Equation 5.The mixing network is treated as part of the central critic and receives local state-values along with global state.Source: VDAC-mix · p. 4
- For a compatible TD(1) critic, the paper’s lemma states that the VDAC gradient approaches zero in limit inferior with probability one; the convergence argument yields a locally optimal policy.The argument derives a standard actor-critic policy gradient and invokes the cited convergence result subject to its assumptions; the paper also states that the naive critic converges locally.Source: Convergence of VDAC frameworks · p. 5
Experiments
On SMAC, the study compares actor-critics and VDAC variants under A2C, finding VDAC-mix strongest across six maps. Ablations test TD-advantage gradients, state-value factorization, A2C performance against QMIX, and nonlinear mixing.
Overall Results
- Across six SMAC maps, evaluation tracks median win percentage over 200k steps using 32 episodes per checkpoint and five training runs.Evaluation episodes disable exploratory behavior.Source: Overall Results · pp. 5–6
- VDAC-mix consistently achieves the best performance across all tasks; on harder 3s5z and 2s3z, only it matches or outperforms heuristic AI.On easier 8m, all algorithms generally perform well.Source: Overall Results · p. 6, Figure 3 · p. 6
- Under A2C, VDAC-mix outperforms QMIX on 2s vs 1sc; on harder 3s5z its median test win rate at 2 million steps exceeds QMIX's by 71%.On easier games, QMIX can be comparable to VDAC-mix.Source: Figure 6 · p. 7, Overall Results · p. 7
- The naive critic outperforms COMA across tasks, while COMA training is unstable on 2s vs 1sc.The authors suggest the instability might arise from inaccurate counterfactual action-value predictions for un-taken actions.Source: Figure 4 · p. 6, Overall Results · p. 7
- At 2 million training steps, VDAC-sum exceeds IAC's median win rates on every map, matches the naive critic on many, and is more stable during training.VDAC-sum and IAC have similar structure, but VDAC-sum enforces a simple state-value factorization.Source: Overall Results · p. 7, Figure 5 · p. 7
- Ablation 4 compares VDAC-mix (linear) with VDAC-sum to test extra-state access, and with VDAC-mix to test nonlinear value decomposition.The authors report that incorporating extra state information and assuming a nonlinear relationship between global and local state values are important.Source: Overall Results · p. 7, Figure 7 · p. 7
Conclusion
The paper proposes a credit-assignment actor-critic framework with a monotonic relationship between global and shaped local state-values. It reports local-optimal convergence, a StarCraft micromanagement performance-efficiency trade-off, and ablation-based identification of contributing factors.
Conclusion
- StarCraft micromanagement benchmarks show VDAC bridges the performance gap between multi-agent actor-critics and Q-learning while balancing training efficiency and performance.The authors also report convergence to a local optimum and use ablations to identify factors contributing to performance.Source: Conclusion · p. 7
- VDAC enforces a monotonic relationship between global state-value and shaped local state-value for credit assignment.The authors state that the proposed actor-critic method converges to a local optimum.Source: Conclusion · p. 7
- Ablation experiments identify factors that contribute to the performance of the proposed algorithms.Source: Conclusion · p. 7
Ethical Impact of Work
The section connects large-scale multi-agent control with societal challenges and identifies autonomous driving as one possible application of MARL. It also names healthcare, manufacturing, energy, and transportation as other areas of potential impact.
Ethical Impact of Work
- Effective autonomous driving is presented as a possible way to reduce the more than 300,000 annual highway-merging accidents cited in the section.Source: Ethical Impact of Work · p. 8
- The section identifies traffic management as one challenging societal problem for large-scale multi-agent control.Source: Ethical Impact of Work · p. 8
- Healthcare, smart manufacturing, smart grids, and transportation infrastructure are also named as areas of potential societal impact.Source: Ethical Impact of Work · p. 8
SMAC
The SMAC setup follows the default implementation described by Samvelyan et al. and uses partially observable agent observations alongside a global state available during centralized training.
SMAC
- SMAC experiments use default settings with game difficulty set to level 7, described as very difficult.Actions include directional movement, attacking an enemy, stopping, and no operation; attacks require an enemy within shooting range.Source: SMAC · p. 10
- Agents have partial observations: they see living units within sight range, while the global state containing all units is available only during centralized training.Source: SMAC · p. 10
- Observations include nearby-unit features, visible allies’ last actions, and terrain features; the state adds map-relative coordinates, unit features, energy or cooldown, and all agents’ last actions.Source: SMAC · p. 10
- Table 2 is titled “Map Descriptions.”The supplied passage does not specify the map descriptions themselves.Source: Table 2 · p. 10
Training Details and Hyperparameters
The experiments use recurrent agent networks and specified optimization, replay, critic, and mixing-network settings. Runs use RTX 2080 Ti cards and vary in duration by scenario.
Training Details and Hyperparameters
- Each independent run uses an Nvidia RTX 2080 Ti and takes 1 to 3 hours depending on the scenario.Each run uses a unique random seed initialized at the beginning.Source: Training Details and Hyperparameters · p. 10
- Most agent networks use a DRQN-style GRU with a 64-dimensional hidden state and fully connected layers before and after it.IAC, VDAC-sum, and VDAC-mix add a local-state-value output layer, and their policy networks output stochastic policies.Source: Training Details and Hyperparameters · p. 10
- Training uses RMSprop at 5 × 10^-4, with 8 independently initiated games, γ = 0.99, and λ = 0.8 when needed.Q-learning replay buffers hold 5,000 episodes per game, 40,000 total, and target networks update every 200 training steps when present.Source: Training Details and Hyperparameters · p. 10
- The COMA critic has two 128-unit hidden layers and an output layer of |U| units; the naive central critic differs by using one output unit.Source: Training Details and Hyperparameters · p. 10
- QMIX and VDAC-mix use the same mixing network: one 32-unit hidden layer with ELU, whose parameters come from 64-unit ReLU hypernetworks.Source: Training Details and Hyperparameters · p. 10
- For the listed critic methods, Q(s_t, u_t) is given by the referenced expression, with k varying by state and bounded above by T.Source: Training Details and Hyperparameters · p. 10
StarCraft II Results
The supplied figures frame the results as comparisons of VDACs with QMIX under A2C and of three VDAC variants, but provide no outcome values.
StarCraft II Results
- The VDAC-mix algorithm initializes a hypernetwork and actor network, then updates the hypernetwork during training.Source: StarCraft II Results · p. 12
- Figure 8 compares overall results for VDACs and QMIX under A2C.Its caption identifies the comparison but does not state which method performs better.Source: Figure 8 · p. 13
- Figure 9 compares overall results for VDAC-mix, VDAC-mix(linear), and VDAC-sum.Its caption identifies the variants but does not state their relative performance.Source: Figure 9 · p. 13