Source-linked AI summary
Value-Decomposition Networks For Cooperative Multi-Agent Learning
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, Thore Graepel
TL;DR
Cooperative MARL must learn from a single joint reward despite large joint spaces, partial observability, spurious rewards, and lazy-agent behavior. The paper trains agents with a learned additive value-decomposition network, combining local values into a team value during training while retaining independent action selection. Across complex partially observable tasks, value-decomposition outperforms naive centralized and independent approaches, especially with weight sharing and information channels.
Problem
Cooperative MARL requires multiple partially observing agents to optimize one joint reward despite large combined action and observation spaces and failures in naive centralized or independent learning.
Method
The paper learns an additive decomposition of the team Q-function into local agent-wise value functions from the joint reward, using centralized training and independent deployment.
Results
Value-decomposition networks show much better performance across the benchmark tasks than centralized and independent learners, with further gains from weight sharing and information channels.
Takeaways & Limitations
Value-decomposition provides a way to turn complex cooperative learning into local, more readily learnable sub-problems while preserving independently deployable agents.
Takeaways & Limitations
The paper leaves scaling to growing team sizes and non-linear value aggregation for future work.
Abstract
from arXiv · showhide
We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In the fully centralized and decentralized approaches, we find the problem of spurious rewards and a phenomenon we call the "lazy agent" problem, which arises due to partial observability. We address these problems by training individual agents with a novel value decomposition network architecture, which learns to decompose the team value function into agent-wise value functions. We perform an experimental evaluation across a range of partially-observable multi-agent domains and show that learning such value-decompositions leads to superior results, in particular when combined with weight sharing, role information and information channels.
1 Introduction
Cooperative MARL must coordinate agents through a single team reward despite partial observability and large joint spaces. The paper proposes learning an additive value decomposition and reports stronger performance than centralized and independent alternatives.
- 1 Introduction: Centralized learning can fail by producing inefficient policies in which one agent remains “lazy” while another acts.The second agent may avoid exploration because its actions temporarily reduce team reward.
- 1 Introduction: Independent learners face non-stationary dynamics and spurious rewards caused by teammates’ changing, unobserved behavior.These signals make it difficult for an agent to explain its own observed rewards.
- 1 Introduction: The paper avoids hand-designed reward shaping by learning a team-value decomposition from the single team reward.The approach represents individual component values using local observations and backpropagates the total Q gradient through them.
- 1 Introduction: Value-Decomposition, Independent Learners, and Centralized architectures are benchmarked across new two-player coordination domains, with weight sharing, role information, and information channels also evaluated.The study compares the main architectures and combinations of additional techniques.
- 1 Introduction: Value-Decomposition performs much better than centralized and fully independent approaches, especially when combined with the evaluated enhancements.The reported advantage is consistent across the benchmarked domains.
2 Background
The background frames MARL as sequential decision-making under partial observability, where agents act simultaneously while receiving a joint reward. It introduces recurrent deep Q-learning architectures, including independent and value-decomposition networks.
- 2.1 Reinforcement Learning: Reinforcement learning models an agent that sequentially acts, observes outcomes, receives rewards, and maximizes cumulative reward.The framework is typically formalized as a Markov decision process.
- 2.1 Reinforcement Learning: In partially observed environments, policies use observation history rather than the full state, and recurrent neural networks provide a practical parameterization.The history includes prior actions, observations, and rewards.
- 2.2 Deep Q-Learning: Deep Q-learning uses target networks, experience replay, recurrent networks, dueling architectures, and multi-step updates for partially observed tasks.The paper defines Q-functions over agent observation histories and trains recurrent networks with truncated BPTT.
- 2.3 Multi-Agent Reinforcement Learning: The independent architecture processes local observations through linear, recurrent, and dueling layers to produce individual Q-values.The value-decomposition architecture instead sums individual values into a joint Q-function for training while producing actions independently.
- 2.3 Multi-Agent Reinforcement Learning: In MARL, agents choose individual actions simultaneously, receive new observations after transitions, and share a joint reward.This setting is consistent with the Dec-POMDP framework.
- 2.3 Multi-Agent Reinforcement Learning: The considered joint policies have independent action-distribution components, except for the combinatorial centralized joint-action learner.Agents’ observations and actions are represented across d agents.
3 A Deep-RL Architecture for Coop-MARL
The architecture learns an additive decomposition of the joint action-value function into agent-wise values based on local observations, while supporting centralized learning and independent deployment. It also incorporates recurrent histories, communication, weight sharing, and role information to address partial observability and role-specific behavior.
- The main assumption is that the joint action-value function can be additively decomposed into value functions across agents.
- Each agent-wise value function depends only on that agent’s local observations and is learned implicitly by backpropagating gradients from the team-reward Q-learning rule.The method does not impose that an agent-wise value function correspond to a specific individual reward.
- Although learning requires some centralization, agents can be deployed independently because greedy local actions are equivalent to centralized maximization of the summed values.
- The architecture can represent additive rewards across agents’ observations and actions, including settings where agents observe their own goals but not necessarily teammates’ goals.For two agents, the illustrative reward decomposition is r(s, a) = r1(o1, a1) + r2(o2, a2).
- When local observations are insufficient, agents can retain historical observations in an LSTM or receive teammate information through a communication channel.
- Weight sharing reduces learnable parameters and supports agent invariance, while role information allows shared networks to represent specialized roles.Role information is supplied as a one-hot identity encoding concatenated with each observation at the first layer.
4 Experiments
The experiments compare value-decomposition, independent-learning, centralized, and enhanced architectures across partially observable two-player coordination domains. Value-decomposition performs much better overall, while sharing, role information, and communication have task-dependent effects.
- Experimental design: The study evaluates eight architectures on seven tasks using ten runs with different environment spawn points and network initializations.Performance is measured over 50,000 episodes using normalized area under the curve and normalized final performance.
- Environments: The environments use short-sighted 3 × 5 × 5 RGB observations and include Switch, Fetch, and Checkers coordination tasks.Agents act in 2D grid worlds with eight available actions and task-specific maps and objectives.
- Results: Value-decomposition architectures perform much better than centralized approaches and individual learners across the evaluated tasks.A centralized agent with value-decomposition outperforms combinatorially centralized and individual-learning agents, but underperforms more individual agents with value-decomposition.
- Results: Weight sharing is especially beneficial for the difficult one-corridor Fetch task, where sharing weights with role information enables perfect learning.Without sharing, the individual value-decomposition agent suffers from the lazy-agent problem; on Switch, non-identical agents can coordinate more easily in one-corridor settings.
- Results: Shared weights are problematic in Checkers because one agent’s reward and value-function magnitudes are ten times larger than the other’s.This task assigns asymmetric rewards: 10 and −10 to the sensitive agent versus 1 and −1 to the less sensitive agent for apples and lemons.
- Results: Information channels increase learning complexity, but low-level communication learned faster than high-level centralized communication and was useful in Checkers.Low-level channels combine observations before the LSTM, whereas high-level communication combines outputs after the LSTM.
5 Conclusions
The paper finds that value-decomposition networks outperform naive independent and centralized approaches across complex cooperative tasks, especially when combined with weight sharing and information channels. It identifies scaling to larger teams and nonlinear value aggregation as future directions.
- Value-decomposition networks outperform naive individual-agent and fully centralized approaches across a range of complex cooperative tasks.
- Combining value decomposition with weight sharing and information channels produces agents that consistently optimally solve the benchmark challenges.
- Scaling value decomposition to growing team sizes remains future work because individual learners become more confused and centralized learners become less practical.
- Future work will investigate nonlinear value aggregations beyond the current decomposition approach.
Appendix A: Plots
The appendix plots report average reward with 90% confidence intervals across ten runs of nine architectures in Fetch, Switch, and Checkers configurations. The figures vary map structure for Fetch and Switch and use the Checkers domain as a separate comparison.
- Fetch: Fetch plots compare nine architectures using average reward and 90% confidence intervals across ten runs on open-map, one-corridor, and two-corridor configurations.
- Switch: Switch plots compare nine architectures using average reward and 90% confidence intervals across ten runs on open-map, one-corridor, and two-corridor configurations.
- Checkers: The Checkers plot reports average reward with 90% confidence intervals across ten runs for nine architectures.
Appendix B: Diagrams
The appendix diagrams present the architectures evaluated in the paper, spanning independent, value-decomposition, centralized, and communication-based designs.
- The diagrams include Independent Agents and Value-Decomposition Individual architectures.
- The communication architectures include Low-level communication, High-level communication, and High+Low-level communication designs.
- The set also includes a second Independent Agents diagram and a Combinatorially Centralized Architecture.