Source-linked AI summary
Learning Attentional Communication for Multi-Agent Cooperation
Jiechuan Jiang, Zongqing Lu
TL;DR
Large-scale MARL cooperation is hindered by globally shared information that agents cannot selectively use and by fixed communication structures that restrict interaction. The paper proposes ATOC, which learns when communication is needed and dynamically forms local groups using an attention unit and bi-directional LSTM channel. Across cooperative scenarios, ATOC outperforms existing methods and produces more coordinated strategies, under assumptions including shared policies and fixed communication bandwidth.
Problem
Large-scale MARL communication can be ineffective or costly when information is globally shared, while predefined architectures restrict potentially useful cooperation.
Method
ATOC learns whether an agent should communicate, selects collaborators in its observable field, and integrates their information through a bi-directional LSTM communication channel.
Results
ATOC outperforms existing methods across a variety of cooperative multi-agent environments and enables more coordinated and sophisticated strategies.
Takeaways & Limitations
Selective, dynamically formed communication can support effective and efficient cooperation in large-scale multi-agent environments.
Takeaways & Limitations
ATOC assumes agents share a policy network and uses fixed communication bandwidth; shared policies may produce homogeneous strategies.
Abstract
from arXiv · showhide
Communication could potentially be an effective way for multi-agent cooperation. However, information sharing among all agents or in predefined communication architectures that existing methods adopt can be problematic. When there is a large number of agents, agents cannot differentiate valuable information that helps cooperative decision making from globally shared information. Therefore, communication barely helps, and could even impair the learning of multi-agent cooperation. Predefined communication architectures, on the other hand, restrict communication among agents and thus restrain potential cooperation. To tackle these difficulties, in this paper, we propose an attentional communication model that learns when communication is needed and how to integrate shared information for cooperative decision making. Our model leads to efficient and effective communication for large-scale multi-agent cooperation. Empirically, we show the strength of our model in a variety of cooperative scenarios, where agents are able to develop more coordinated and sophisticated strategies than existing methods.
1 Introduction
MARL cooperation is difficult because independently learned policies face changing, uncertain teammates, while existing communication can overwhelm agents or restrict useful interactions. ATOC addresses these issues by learning when and with whom agents communicate, using selective information integration for large-scale cooperation.
- Independent reinforcement learning makes collaboration difficult because other agents’ changing strategies create an unstable environment and can cause policy overfitting.
- Existing communication methods can overwhelm large agent populations by globally sharing information that agents cannot distinguish by cooperative value.
- Predefined communication architectures restrict interactions among specific agents, limiting potential cooperation; unrestricted communication also incurs bandwidth, delay, and computational costs.
- ATOC learns whether communication is needed and dynamically selects collaborators within an initiator’s observable field to form temporary communication groups.
- ATOC extends actor-critic learning with shared policy, value, attention, and communication parameters, and is evaluated across three cooperative reward settings.
2 Related Work
Prior MARL communication methods range from differentiable message passing and broadcast networks to fixed architectures and centralized critics. These approaches differ in scalability, information preservation, observability assumptions, and whether policies are shared or independently trained.
- DIAL enables learnable communication through backpropagation by passing an agent’s message as another agent’s next-timestep input.
- CommNet broadcasts agents’ hidden states through an averaged message, but its single large network does not easily scale to many agents.
- BiCNet connects individual agents’ policy and value networks with recurrent networks, while Master-Slave combines slave and master contributions in each slave’s action.
- MADDPG and COMA use centralized critics over all agents’ observations and actions, whereas their independent policy networks can overfit to the number of agents.
- Mean Field uses neighboring agents’ mean actions, which removes differences among neighbors’ actions and observations that may support cooperative decisions.
3 Background
The background introduces value-based and policy-gradient reinforcement learning, including DQN and deterministic policy gradients, and describes DDPG as their actor-critic combination. Recurrent attention models provide the attention mechanism’s conceptual basis.
- DQN learns an action-value function from state, action, reward, and successor-state transitions, selecting greedy or exploratory actions using ε-greedy behavior.
- DQN maximizes expected discounted reward, with γ controlling the contribution of future rewards.
- Policy-gradient methods directly adjust policy parameters to maximize expected return along the policy-gradient direction.
- Deterministic policy gradients require a continuous action space so that the action-gradient of the value function exists.
- DDPG combines deterministic policy gradients with actor-critic networks: the actor infers actions and the critic estimates value and supplies the gradient.
- Recurrent Attention Models process partial observations through an RNN-based attention mechanism rather than the entire perception field.
4 Methods
ATOC enables agents to learn when communication is needed, which nearby agents should collaborate, and how shared information should guide coordinated actions. Its attention, communication, and training components are integrated in an end-to-end actor-critic extension designed for large-scale partially observable environments.
- Training: ATOC extends actor-critic training with shared policy, Q-network, attention, and communication parameters, using experience replay and gradient updates.The attention unit is trained as a binary classifier whose target is based on the normalized Q-value improvement from communication.
- Attention and communication groups: ATOC uses an attention unit to determine whether an agent should communicate and, when needed, select nearby collaborators for a dynamically formed group.The decision is based on encoded local observation and action intention; the group remains fixed for T timesteps.
- Attention and communication groups: The integrated communication representation is merged with each agent’s local thought before the policy network produces its action.Sharing encoded observation and action intention can provide a more global perception and help agents infer one another’s intent.
- Attention and communication groups: ATOC permits communication only when necessary, avoiding the full connectivity used by methods such as CommNet and BiCNet.The selective design addresses communication costs and reduces exposure to potentially useless shared information.
- Communication channel: Agents selected by multiple initiators participate in each group, bridging information gaps and strategy divisions between groups.The paper identifies this bridging role as especially important when all agents collaborate on a single task.
- Communication channel: A bidirectional LSTM communication channel selectively retains information that promotes cooperation and forgets information that impedes it.This differs from arithmetic-mean and weighted-mean integration in CommNet and BiCNet.
5 Experiments
Experiments evaluate ATOC across large-scale navigation, pushball, and predator-prey scenarios, showing coordinated strategies, higher rewards, and scalability than baselines. Analyses associate these gains with selective communication and attention to useful nearby information.
- Experimental Setup: Experiments use a modified multi-agent particle environment with local observations across scenarios involving local rewards, shared global rewards, and competitive rewards.Agents act independently and cooperatively in a two-dimensional continuous-space, discrete-time world.
- Cooperative Navigation: With 50 agents and 50 landmarks, ATOC converges to higher mean reward and outperforms baselines in navigation.The evaluation measures average mean reward, collisions, and occupied-landmark percentage over 30 test games; ATOC agents choose alternative vacant landmarks when nearby targets are likely occupied.
- Scalability: ATOC remains better than baselines across all navigation metrics when trained models scale from 50 to 100 agents and landmarks.Collision counts increase for every method, while BiCNet shows substantially degraded performance across metrics.
- Cooperative Navigation: ATOC dynamically communicates mainly in dense, complex regions, then reduces communication as agents occupy more landmarks.Communication helps agents spread across different landmarks rather than target the same landmark and risk collisions.
- Cooperative Pushball: In cooperative pushball, ATOC converges to much higher normalized mean reward and develops role-divided strategies for controlling the ball.Agents strike the ball’s center or sides, reduce its speed near the target, and split into two groups to strike from opposite directions; communication ablation lowers performance.
- Predator-Prey: In predator-prey, cross-comparisons show ATOC predators outperform baseline predators against ATOC and baseline prey, supporting stronger cooperation in competition.The comparison uses 0-1 normalized mean predator scores from 30 test runs per game.
6 Conclusions
The paper proposes ATOC, an attentional communication model for large-scale multi-agent environments. ATOC dynamically determines when communication is needed and empirically outperforms existing methods across cooperative environments.
- ATOC uses an attention unit to dynamically determine whether communication is needed for cooperation.It also uses a bidirectional LSTM communication channel to interpret encoded information from other agents.
- ATOC is designed to exploit communication effectively and efficiently for cooperative decisions.
- ATOC outperforms existing methods in a variety of cooperative multi-agent environments.
Attentional Communication Algorithm
The attentional communication algorithm alternates between communication decisions, action execution, experience collection, and end-to-end parameter updates. It trains the attention classifier using normalized communication benefits measured from critic values.
- Initialization and rollout: The algorithm initializes actor, critic, communication, attention, target-network, replay-buffer, and queue components before training across episodes and timesteps.
- Communication decisions: Each agent uses its thought representation to decide whether to initiate communication every T timesteps; initiators form groups and exchange encoded thoughts.
- Action execution: Agents select actions with or without communication, execute them, and obtain new observations.
- Attention learning: The attention classifier is trained with log loss after each communication-benefit difference is min-max normalized to [0, 1].
- Parameter updates: The algorithm stores transitions and updates the critic, actor, communication channel, and target networks using sampled experience.
Video
The supplementary video shows trained-agent behavior in cooperative navigation and cooperative pushball, while omitting predator-prey because methods are not visually distinguishable there.
- The video shows ATOC, DDPG, and ComNet agents in cooperative navigation, and ATOC and DDPG agents in cooperative pushball.
- Predator-prey is omitted because the performance of different methods cannot be visually differentiated.