Source-linked AI summary
Actor-Attention-Critic for Multi-Agent Reinforcement Learning
Shariq Iqbal, Fei Sha
TL;DR
Multi-agent reinforcement learning must handle environments made nonstationary by other learning agents while avoiding the scalability and communication costs of centralized joint-action policies. The paper introduces decentralized actors with centrally trained, attention-based critics that select relevant agents dynamically. The approach reports improved performance in complex interactions and supports cooperative, competitive, and mixed reward settings across varied action spaces, while future work targets more complicated large-group environments.
Problem
Multi-agent learning must account for other agents’ changing policies, while fully centralized joint-action policies scale poorly and require substantial execution-time communication.
Method
The algorithm trains decentralized policies using centralized critics with attention that selectively incorporates relevant information from other agents.
Results
The attention critic dynamically selects relevant agents during training and improves performance in multi-agent domains with complex interactions.
Takeaways & Limitations
The approach applies across cooperative, competitive, and mixed environments, supports different agent action spaces, and scales linearly with the number of agents’ inputs.
Takeaways & Limitations
The authors identify more complicated environments with large groups of agents requiring selective attention as future work.
Abstract
from arXiv · showhide
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.
1. Introduction
Multi-agent reinforcement learning is difficult because other learning agents make each agent’s environment nonstationary, while centralized joint-action policies become unscalable. The proposed approach centrally trains attention-based critics while retaining decentralized policies, dynamically selecting relevant agents during learning.
- Motivation: Other learning agents make each individual agent’s environment dynamic and nonstationary, invalidating standard stationary Markov decision-process algorithms.Independent training treats other agents as part of the environment, whose changing policies violate stationarity and Markovian assumptions.
- Motivation: Modeling all agents as one centralized policy enables coordination but causes exponential joint-action growth and requires substantial communication during execution.The central policy must collect observations and distribute actions, which can be problematic in real-world settings.
- Prior Approaches: Prior centralized-critic methods allow decentralized execution, avoiding the need for other agents’ actions during testing while addressing non-Markovian and nonstationary learning environments.These methods centrally learn critics from information across agents while each actor receives only its own information.
- Proposed Approach: The proposed method learns a centralized critic with attention that dynamically selects which agents each critic should consider at each timestep.The motivating example is a soccer defender attending to nearby attackers and the ball carrier rather than every opposing player.
- Scope and Evaluation: The approach scales linearly with the number of agents, applies to cooperative, competitive, and mixed environments, and was validated on three simulated environments and tasks.This extends beyond prior work focused only on cooperative settings.
2. Related Work
Prior MARL research spans cooperative communication and competitive play, but deep multi-agent learning faces nonstationarity and centralized approaches impose scalability or execution constraints. The proposed attention-based centralized-critic framework broadens flexibility across reward structures and agent action spaces.
- MARL Background: MARL encompasses cooperative communication and competitive optimal play, although earlier work largely used simple gridworlds with tabular learning.Deep learning later enabled multi-agent learning in high-dimensional and continuous state spaces.
- MARL Background: Naively applying deep reinforcement learning to MARL encounters nonstationarity from each individual agent’s perspective.Other agents’ changing policies alter the environment faced by each learner.
- Centralized Critics: Centralized-training and decentralized-execution actor-critic methods address nonstationarity, but prior critics may use all other agents’ actions and observations as inputs.Related methods include separate centralized critics for mixed rewards and counterfactual baselines for cooperative credit assignment.
- Attention in MARL: Attention has been used to select contextual information in vision, language, and reinforcement learning, including prior MARL work that centralizes policies while keeping critics decentralized.The prior attention-based actor-critic approach is complementary because it learns information sharing between policies.
- Proposed Positioning: The proposed algorithm supports any reward setup, different action spaces, and centralized critics that dynamically attend to relevant information for each agent.The authors characterize this design as more scalable in agent number and more broadly applicable than the aforementioned approaches.
3. Our Approach
The approach trains centralized critics that selectively attend to other agents’ information while policies execute using decentralized information. It combines attention-based value estimation with multi-agent baselines and shared critic parameters.
- Attentive critics: Centralized critics selectively attend to other agents’ observations and actions to estimate each agent’s value function.Each agent encodes its observation and action, while the attention mechanism returns a weighted sum of other agents’ encodings.
- Attentive critics: The attention mechanism queries other agents’ information without assuming temporal or spatial locality.Attention weights compare agent embeddings through a query-key mapping, and multiple heads can focus on different mixtures of agents.
- Shared critics: Shared selector, key, and value parameters encourage a common embedding space and permit critic sharing across agents with different rewards.The critics are updated together through a joint regression loss enabled by parameter sharing.
- Policy updates: Policy gradients use the multi-agent baseline, while current policies supply all agents’ sampled actions for gradient estimation.This differs from sampling other agents’ actions from the replay buffer, which the paper associates with possible coordination overgeneralization.
- Multi-agent advantage: The multi-agent advantage baseline compares an agent’s action value with its expected value while holding other agents’ actions fixed.This comparison addresses credit assignment by distinguishing an agent’s contribution from reward changes attributable to other agents.
- Multi-agent advantage: The attention-based baseline does not require identical action spaces or global rewards and dynamically attends to other agents.It uses the same attention decomposition as the Q-function, supporting a more flexible multi-agent baseline.
4. Experiments
The experiments evaluate MAAC across environments testing scalability, changing relevance of agent information, and general multi-agent effectiveness. MAAC is competitive across tasks, scales better as agents are added, and learns to attend to reward-relevant agents.
- Experimental setup: The experiments test scalability, dynamic attention to relevant information, and general effectiveness across Cooperative Treasure Collection, Rover-Tower, and Cooperative Navigation.Cooperative Treasure Collection varies agent count, Rover-Tower tests changing relevance and coordination, and Cooperative Navigation provides a benchmark comparison.
- Baseline comparison: MAAC is competitive with other methods on Cooperative Treasure Collection, Rover-Tower, and Cooperative Navigation.The reported comparisons include average rewards on the two introduced environments and Cooperative Navigation results.
- Attention and rewards: Uniform attention is competitive in Cooperative Treasure Collection and Cooperative Navigation but not Rover-Tower, where rewards depend on another specific agent’s observations.Shared rewards and informative local observations make selective attention less necessary in the first two tasks, while Rover-Tower requires identifying a relevant partner.
- Baseline comparison: COMA variants perform poorly in Cooperative Treasure Collection and Rover-Tower, while COMA+SAC performs reasonably well in Cooperative Navigation.The paper relates this pattern to differences in rewards, agent roles, and action spaces across environments.
- Scalability: The improvement of MAAC over MADDPG+SAC in Cooperative Treasure Collection grows with the number of agents.The comparison uses average rewards normalized by the environment’s reward range because changing agent counts changes the task’s reward nature.
- Scalability: MAAC’s attention mechanism learns to select relevant agents and compress their information into a constant-sized vector, unlike critics that use all information non-selectively.This design supports better scaling as the number of agents increases.
- Scalability: MAAC performance does not deteriorate as agents are added in Rover-Tower, whereas MADDPG performance does not scale despite performing well in the eight-agent task.Rewards can be compared directly because each rover-tower pair has the same reward scale regardless of other agents.
- Attention visualization: In Rover-Tower, rovers learn to attend strongly to their paired towers without explicit supervision, indicating that the model identifies reward-relevant agents.The attention visualization fixes a rover’s paired tower and finds attention to other rovers near zero.
5. Conclusion
The paper proposes decentralized policy training with attention-based critics and evaluates the approach across agent counts, reward configurations, and relevant observational information.
- The proposed algorithm trains decentralized policies in multi-agent settings.
- Attention selects relevant information for estimating centralized critics.
- The evaluation considers different numbers of agents, reward configurations, and spans of relevant observational information.
6. Appendix
The appendix details the attention-actor-critic training procedure, discrete-action adaptations, and analyses of how attention changes during training and across scenarios.
- Training Procedure: Training uses parallel environments, replay-buffer sampling, critic updates, policy updates, and target-parameter updates.
- Training Procedure: The implementation uses Soft Actor-Critic with target networks, discount factor γ of 0.99, hidden dimension 128, and 4 attention heads.
- Reparametrization of DDPG/MADDPG for Discrete Action Spaces: For discrete-action comparisons, DDPG and MADDPG use differentiable samples from a Gumbel-Softmax distribution.
- Visualizing Attention: Lower attention entropy indicates more focused attention, with entropy 0 representing attention focused on one agent.
- Visualizing Attention: In Collective Treasure Collection, all agents use attention heads similarly, while Rover-Tower exhibits agent-specific combinations of heads.
- Visualizing Attention: Figure 8 shows attention weights for Rover 1 under multiple Tower pairings in Rover-Tower.