Source-linked AI summary
Learning Multiagent Communication with Backpropagation
Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus
TL;DR
The paper addresses the limitation that multiagent communication is typically manually specified, despite its importance for cooperation under partial visibility. It introduces CommNet, which learns continuous communication alongside agent policies, and evaluates it across diverse cooperative tasks. Across four tasks, CommNet outperforms models without communication, fully connected models, and discrete-communication models, while learning a sparse, meaningful protocol in the traffic task.
Problem
Multiagent systems need coordination under limited visibility, but communication formats are usually predetermined rather than learned.
Method
CommNet is a neural controller in which cooperating agents exchange learned continuous communication vectors and train the communication policy through backpropagation, reinforcement learning, or supervision.
Results
Across four diverse tasks, CommNet outperforms models without communication, fully connected models, and models using discrete communication.
Takeaways & Limitations
The traffic task shows that CommNet can learn a sparse communication protocol conveying meaningful information between agents.
Takeaways & Limitations
The model’s ability to handle heterogeneous agent types was not fully exploited, and scaling to large agent populations remains future work.
Abstract
from arXiv · showhide
Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNet, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand.
1 Introduction
CommNet addresses cooperative multiagent tasks by learning continuous communication jointly with agent policies, replacing manually specified protocols in partially observed settings.
- Communication helps agents coordinate as a group when each has limited capabilities or visibility of the environment.
- CommNet gives each cooperating agent a continuous communication channel whose transmitted vector is learned rather than specified in advance.Agents receive the summed transmissions of other agents.
- Continuous communication allows the model to train through backpropagation and combine with standard reinforcement learning or supervised learning.
- The model is designed for tasks with partial visibility and supports runtime variation in the number and type of agents.
- The controller is evaluated under full cooperation, where all agents maximize a shared reward independently of individual contribution.
2 Communication Model
CommNet is a modular neural controller that encodes each agent’s state, repeatedly exchanges continuous messages, and decodes the resulting hidden states into actions. Its communication can be broadcast, locally connected, or recurrent over time.
- Controller Structure: CommNet maps the concatenated state views of all agents to their actions through a single controller encompassing individual policies and communication.
- Controller Structure: Each agent module processes its hidden state and communication vector, while shared parameters make the architecture applicable across agents.
- Controller Structure: Communication is normalized by J −1 and pooled in a permutation-invariant way, supporting dynamically varying agent sets without dependence on agent order.
- Controller Structure: The model encodes agent observations, iterates hidden states and communication vectors for K steps, then samples actions from decoded output distributions.
- Model Extensions: With local connectivity, each agent communicates only with agents in its current neighborhood, which can change as agents enter, leave, or move.
- Model Extensions: Temporal recurrence reuses the communication module across time, allowing agents to join or leave at any timestep and permitting RNN or LSTM modules.
3 Related Work
The paper positions CommNet within multiagent reinforcement learning and learned-communication research, distinguishing it from approaches that assume broad visibility, fixed protocols, or discrete messages.
- Prior cooperative MARL methods either avoid communication through strong visibility assumptions or use communication protocols specified in advance.
- Related approaches learn communication under partial visibility using tabular reinforcement learning, evolutionary algorithms, or factored message passing.
- Compared with concurrent learned-communication work, CommNet uses multiple continuous communication cycles rather than discrete communication through actions.
- CommNet differs from Neural GPU and recurrent graph models through global pooling, unstructured inputs, and a non-recurrent feed-forward formulation that need not reach steady state.
4 Experiments
The experiments compare CommNet with independent, fully connected, and discrete-communication baselines across cooperative lever, traffic, combat, and bAbI tasks. CommNet generally improves coordination, especially when agents have limited visibility, while learned communication can encode task-relevant strategies.
- Experimental setup: CommNet is compared with independent, fully connected, and discrete-communication controllers across several cooperative tasks.The baselines differ in whether agents communicate, whether communication is continuous or discrete, and whether the controller is fully connected.
- Lever pulling task: Communication enables agents to coordinate lever choices, allowing the lever game to be solved more successfully than with an independent controller.The experiment samples five agents from a pool of 500 and rewards distinct levers pulled.
- Traffic junction: CommNet significantly reduces traffic-junction failure rates across module types, with the LSTM module achieving the best performance.On the harder game variant, local connectivity performs at least as well as broadcasting, whereas discrete communication helps more on the easy variant.
- Traffic junction: 90% success with zero visibility shows that CommNet can coordinate traffic agents even when cars cannot observe one another.The communication advantage increases as each agent’s vision range decreases.
- Communication analysis: Communication vectors are usually near zero but form distinct clusters that correlate with specific car locations and braking events relevant to collision avoidance.The clustered communication pathway and the more diverse hidden-state pathway therefore carry different information.
- Combat task: CommNet consistently improves combat win rates across module choices, team sizes, and visibility settings, including teams of 10 agents.The LSTM performs best among the tested modules, while the fully connected baseline performs worse than the independent controller.
- bAbI tasks: CommNet outperforms the LSTM baseline on the 10K bAbI tasks and solves most tasks, but remains below MemN2N.The model also solves tasks requiring information sharing between two or more agents.
5 Discussion and Future Work
CommNet learns continuous communication among dynamically changing agents and outperforms several non-communicative and discrete-communication baselines. The authors leave heterogeneous agent types and scaling to larger agent populations for future work.
- CommNet learns continuous communication between a dynamically changing set of agents.
- Across four diverse tasks, CommNet outperforms models without communication, fully connected models, and models using discrete communication.
- Traffic-task analysis reveals a sparse communication protocol that conveys meaningful information between agents.
- The model’s ability to handle heterogeneous agent types was not fully exploited and is left for future work.
- Scaling to large numbers of agents may require more sophisticated connectivity structures and is also left for future work.
A Reinforcement Training
The reinforcement-training procedure uses policy gradients with a state-specific baseline to provide learning signals for episodic agent behavior. The baseline objective is balanced against reward maximization during parameter updates.
- Policy gradients deliver the training gradient using a state-specific baseline computed by an extra model head.
- The states and actions across an episode are used to update model parameters after the episode finishes.
- The hyperparameter α balances reward maximization and baseline objectives, with α set to 0.03 in all experiments.
B Lever Pulling Task Analysis
In the supervised lever-pulling task, hidden-state visualizations show a smooth ordering by agent identity. This ordering suggests that agents communicate their IDs to solve the task.
- 3D PCA plots visualize the hidden states of agents in the lever-pulling analysis.
- Color intensity in the plot represents agent identity.
- The smooth ordering of hidden states suggests that agents communicate their IDs, enabling them to solve the task.
C Details of Traffic Junction
The traffic-junction experiments compare increasingly difficult road layouts and use curriculum learning to ease training. The hard version contains four connected two-way junctions on an 18 × 18 grid with eight arrival points and seven routes per point.
- Curriculum learning gradually increases traffic parameters across successive training phases.The learning schedule changes arrival probability and, for the easy task, the total number of cars.
- The easy task is a junction of two one-way roads on a 7 × 7 grid with two arrival points and two possible routes per point.
- The hard task has four connected junctions of two-way roads in an 18 × 18 grid, with 8 arrival points and 7 routes for each point.
- For the hard task, Ntotal = 20 and parrive increases from 0.02 to 0.05 during curriculum.
D Traffic Junction Analysis
Figure 6 visualizes communication-vector norms and brake locations across a 14 × 14 spatial grid, revealing directional communication patterns and asymmetric yielding behavior.
- Figure 6 maps average communication-vector norms and brake locations across the 14 × 14 spatial grid.The left panel shows communication-vector norms, while the right panel shows brake locations.
- Each incoming direction contains one location where the communication signal is stronger.
- Cars coming from the left never yield to other directions in the observed brake pattern.
E bAbI Tasks Details
The bAbI setup represents story sentences, a question, and the correct answer for supervised answer prediction, while incorporating question broadcasting and temporal ordering. The section also describes the network, training procedure, and reported task breakdown.
- The bAbI input consists of story sentences, a question sentence, and the correct answer word.Multiple-word answers are concatenated into a single word.
- Initial communication broadcasts the question to all agents, and each story sentence receives a temporal word when sentence order matters.The temporal word is defined as “t = J −j” for sentence sj.
- The f module uses a 2 layer network with a skip connection.
- After K = 2 communication steps, the model outputs an answer word.
- Training uses supervised cross-entropy learning for 100 epochs with Adam, a learning rate of 0.003, and mini-batches of 32.The hidden layer size is 100, and 10% of the training data is used for validation-based hyperparameter selection.
- Table 5 reports experimental results on bAbI tasks, showing only tasks with high errors.