Source-linked AI summary
TarMAC: Targeted Multi-Agent Communication
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, Joelle Pineau
TL;DR
TarMAC tackles the challenge of effective communication in partially observable multi-agent tasks, where broadcast messages may not suit agents with different goals. It learns targeted continuous communication and extends it to multiple rounds, achieving improved performance across diverse environments while producing interpretable attention strategies.
Problem
Broadcast communication can be inefficient when agents have different roles and goals, motivating learned recipient-specific communication in partially observable cooperative tasks.
Method
TarMAC uses sender signatures and receiver queries for soft-attention targeting, recurrent policies for multiple communication rounds, and task reward as the sole supervision.
Results
Across four diverse environments, TarMAC learns intuitive attention behavior and improves performance, including 97.1±1.6% success on hard Traffic Junction with two rounds versus 79.1% for CommNet.
Takeaways & Limitations
Targeted communication can support agent-goal-specific information exchange, adaptation to variable team sizes, interpretability, and extension to mixed or competitive settings.
Takeaways & Limitations
The evaluation leaves more challenging 3D navigation tasks for future benchmarking, particularly those involving many agents and high-dimensional observations.
Abstract
from arXiv · showhide
We propose a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments. This targeting behavior is learnt solely from downstream task-specific reward without any communication supervision. We additionally augment this with a multi-round communication approach where agents coordinate via multiple rounds of communication before taking actions in the environment. We evaluate our approach on a diverse set of cooperative multi-agent tasks, of varying difficulties, with varying number of agents, in a variety of environments ranging from 2D grid layouts of shapes and simulated traffic junctions to 3D indoor environments, and demonstrate the benefits of targeted and multi-round communication. Moreover, we show that the targeted communication strategies learned by agents are interpretable and intuitive. Finally, we show that our architecture can be easily extended to mixed and competitive environments, leading to improved performance and sample complexity over recent state-of-the-art approaches.
1. Introduction
TarMAC addresses the limits of broadcast communication by learning which messages to send to which agents, using task reward alone. It extends this targeted mechanism to multiple communication rounds and evaluates it across diverse tasks, environments, and team sizes.
- Targeted communication: TarMAC lets each agent select which other agents should receive its messages through signature-based soft attention.Sender signatures encode intended-recipient properties, while receivers use query vectors to assess message relevance.
- Targeted communication: The targeting behavior is learned end-to-end from task reward without attention supervision.
- Benefits: Soft attention supports agent-goal-specific messages, variable team sizes, and interpretable sender-recipient attention probabilities.
- Multi-round communication: TarMAC adds multiple rounds of targeted interaction so recurrent policies can persist and exchange relevant information before acting.
- Communication protocol: Machine-only teams use continuous-vector communication because discrete tokens introduce additional training challenges.
- Evaluation: The paper evaluates TarMAC across cooperative tasks spanning SHAPES, traffic junctions, House3D, and mixed or competitive environments.
2. Related Work
The related work situates TarMAC within multi-agent reinforcement learning, centralized training with decentralized execution, and learned communication protocols. It emphasizes decentralization and communication as responses to centralized controllers' scalability limits.
- Multi-Agent Reinforcement Learning: TarMAC builds on recurrent policies, multi-agent training stabilization, and coordination tasks in domains including 3D navigation.
- Centralized Training & Decentralized Execution: Earlier centralized frameworks process all agents’ observations and produce joint-action distributions during both training and testing.
- Centralized Training & Decentralized Execution: Centralized controllers become intractable with many agents and high-dimensional observations, motivating centralized learning with decentralized execution and communication.
- Emergent Communication Protocols: Emergent-communication research learns protocols end to end from perceptual input through messages to actions, using discrete or continuous symbols.
- Comparison: Table 1 summarizes comparison axes between TarMAC and prior collaborative communication methods using continuous vectors.
3. Technical Background
The technical background formalizes decentralized partially observable multi-agent decision-making and actor-critic optimization. It also describes centralized critics that guide decentralized agent policies using joint action information.
- Dec-POMDPs: A Dec-POMDP defines global states, rewards, transitions, agent actions, local observations, and observation functions for decentralized teams.
- Dec-POMDPs: Agents choose actions from local observations while the team maximizes discounted global return over the episode horizon.
- Actor-Critic Algorithms: Policy gradients update policy parameters toward higher expected return using action-value estimates.
- Actor-Critic Algorithms: The critic approximates the unknown action-value function, while the actor is the policy being optimized.
- Multi-Agent Actor-Critic: Multi-agent actor-critic methods use decentralized policies conditioned on local observations and a centralized critic conditioned on all agents’ actions.
4. TarMAC: Targeted Multi-Agent Communication
TarMAC augments recurrent decentralized policies with centralized training and targeted, multi-round communication. Agents broadcast signatures and values, receive attention-weighted messages, and update internal states before selecting actions and messages.
- Policies and Decentralized Execution: Each agent’s recurrent policy uses local observations and aggregated messages to update a hidden state encoding its message-action-observation history.
- Policies and Decentralized Execution: The policy outputs both a categorical environment-action distribution and an outgoing continuous message vector.
- Centralized Critic: During training, a centralized critic estimates joint action-values from all agents’ predicted actions and internal states.
- Targeted Communication: Each message contains a signature encoding intended-recipient properties and a value containing the actual message.
- Centralized Critic: At test time, the critic is removed and each policy executes using local observations and communication messages.
- Multi-Round Communication: Multiple rounds repeatedly update hidden states and recompute signatures, queries, values, and aggregated messages before the next timestep.
- Targeted Communication: Receivers compute query-signature dot products and apply softmax to obtain attention weights over incoming messages, including self-attention.
- Multi-Round Communication: The differentiable communication architecture learns message vectors through backpropagation, with the number of rounds treated as a hyperparameter.
5. Experiments
Experiments evaluate TarMAC across cooperative navigation, traffic junction, House3D, and competitive settings. Results show benefits from targeted and multi-round communication, interpretable attention behavior, and adaptation to changing team sizes.
- Experimental Setup: TarMAC is evaluated on cooperative navigation tasks in SHAPES, traffic junction, and House3D, with varying environments, difficulties, and team configurations.The experiments use shared-parameter policies, recurrent hidden states, continuous messages, and results averaged across independent seeds.
- Targeted Communication: Agents learn targeted communication because attention depends jointly on sender signatures and receiver queries, rather than receiver-side message selection alone.The mechanism lets senders influence which receivers attend to each message.
- Traffic Junction: 97.1±1.6% success on hard traffic junction with 2-round communication is an 18% absolute improvement over CommNet.One-round TarMAC reaches 84.6±3.2%, while the no-communication baseline reaches 74.1±3.9% on hard.
- Traffic Junction: Multiple communication rounds outperform simply increasing message size on hard traffic junction, while a single-scalar message performs almost as well as a 64-d message.The study reports no further gains beyond two rounds in this environment.
- Traffic Junction: TarMAC attention is interpretable: cars brake near junctions, attend to sensitive locations, and dynamically track team size with Spearman’s σ=0.49.With a relative time shift of 3, the rank correlation increases to 0.53; the behavior emerges from task rewards without hand-coding.
- House3D: In House3D, TarMAC achieves 68.9±1.1% fireplace-navigation success and reaches targets in 82.5 steps on average, outperforming no-attention and no-communication policies.The corresponding success rates are 64.3±2.3% and 62.1±5.3%, with average steps of 101.3 and 186.5.
6. Conclusions and Future Work
TarMAC combines targeted continuous communication with multiple rounds of collaborative reasoning and improves performance across four diverse environments, including noncooperative settings. Future work will benchmark it more extensively on challenging 3D navigation tasks.
- TarMAC supports targeted continuous communication through sender-receiver soft attention and multiple rounds of collaborative reasoning.
- Across four diverse environments, TarMAC learns intuitive attention behavior and improves performance, including in noncooperative settings, using task reward alone.
- Future work will exhaustively benchmark TarMAC on more challenging 3D navigation tasks, where decentralized targeted communication may support many agents with high-dimensional observations.