Source-linked AI summary
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, Shimon Whiteson
TL;DR
The paper asks how multiple partially observant agents can learn communication protocols needed to maximize shared utility. It introduces RIAL and DIAL, using deep Q-learning and gradient-based communication, respectively, and shows that these methods solve complex benchmark tasks while learning protocols. The study also contributes environments and engineering techniques for protocol learning.
Problem
Agents in cooperative, partially observable environments need to learn communication protocols, but discovering coordinated sending and interpretation is difficult because useful rewards are sparse.
Method
The paper proposes RIAL, based on recurrent deep Q-learning, and DIAL, which passes gradients through real-valued communication during centralised learning before discretised decentralised execution.
Results
The methods solve the benchmark tasks and learn communication protocols in complex environments involving sequences and raw input images.
Takeaways & Limitations
The study provides benchmark environments, comparative analyses, and engineering innovations for learning communication protocols with deep networks.
Takeaways & Limitations
The paper is a first attempt, while understanding communication and language in their full scope—including compositionality, concept lifting, and conversational agents—remains ahead.
Abstract
from arXiv · showhide
We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.
1 Introduction
The paper studies how deep learning can help agents discover communication protocols for cooperative, partially observable tasks. It introduces benchmark tasks and two learning approaches, RIAL and DIAL, then analyzes their successes and failures.
- Research program: The paper proposes multi-agent benchmark tasks requiring communication, learning algorithms for them, and analyses of learned communication protocols.The programmatic approach covers task design, algorithm formulation, and protocol analysis.
- Problem setting: The tasks are cooperative, partially observable, sequential decision problems in which agents share a discounted-reward objective and communicate through limited-bandwidth discrete channels.Agents receive private observations correlated with the hidden Markov state and must act as well as communicate.
- Problem setting: The paper assumes centralised learning but decentralised execution, so agents may interact freely during training but use only the limited channel at execution.The authors note this setting applies to many scenarios, such as training robots in a simulator.
- Methods: RIAL applies recurrent deep Q-learning, with either independently learned agent networks or a shared network whose execution remains decentralised.Different observations lead agents using shared parameters to produce different behavior during execution.
- Methods: DIAL passes real-valued messages and error gradients through communication channels during centralised learning, then discretises messages for decentralised execution.This makes the system end-to-end trainable across agents rather than only within each agent.
- Findings: The methods solve the benchmark tasks and often discover communication protocols, including protocols learned with deep networks in complex sequential and raw-image environments.The study also reports engineering innovations essential for success in its proposed benchmarks.
2 Related Work
Prior cooperative communication work often used predefined protocols, small tasks, heuristics, or handcrafted rules. This paper instead emphasizes deep representations and gradient-based optimization for scalable protocol learning.
- Research landscape: AI communication research spans learned versus predefined protocols, planning versus learning, evolution versus reinforcement learning, and cooperative versus competitive settings.The paper positions its contribution within these major research axes.
- Prior protocol learning: Most prior multi-agent reinforcement-learning studies with communication assumed predefined protocols rather than learning them.Exceptions included tabular Q-learning and evolutionary approaches on predator-prey tasks.
- Prior protocol learning: Earlier open-ended communication studies commonly relied on heuristics, handcrafted rules, or small tasks suitable for global optimization methods.The paper contrasts these constraints with its use of deep representations and gradient-based optimization.
- Paper's departure: Deep representations and gradient-based optimization are presented as an important departure that is essential for scalability and further progress.The paper relates this rationale to another end-to-end differentiable reinforcement-learning example.
- Discrete communication: The paper uses discrete communication channels and binarises signals during decentralised execution, connecting this design to work on binary neural representations and codes.The cited related work addresses memory- and computation-limited devices and binary document codes.
3 Background
The paper builds on DQN, independent Q-learning, and recurrent Q-networks to handle cooperative multi-agent reinforcement learning under partial observability. These foundations motivate recurrent agents and expose independent-learning limitations.
- DQN: DQN represents the action-value function with a neural network and optimizes it using a Bellman-target loss.The underlying objective is the expected discounted return, with actions commonly selected by an epsilon-greedy policy.
- DQN: DQN training uses a frozen target network and experience replay, sampling mini-batches from stored episodic experiences.The target network remains fixed for a number of iterations while the online network is updated.
- Independent Q-learning: Independent Q-learning extends DQN to cooperative multi-agent settings by having each agent learn its own Q-function while sharing team rewards.Each agent independently and simultaneously learns its parameters.
- Independent Q-learning: Independent Q-learning can create convergence problems because one agent's learning makes the environment appear non-stationary to the others.Despite this issue, the approach has a strong empirical track record and was applied successfully to two-player Pong.
- Partial observability: Deep recurrent Q-networks replace feed-forward Q-function approximation with recurrent networks for partially observable environments.Partial observability hides the Markov state and provides only an observation correlated with it.
4 Setting
The paper studies cooperative multi-agent reinforcement-learning problems where agents have private, incomplete observations and must learn how to communicate. Protocol discovery is difficult because useful communication requires coordinated sending and interpretation under sparse rewards.
- Setting: Agents share a discounted-return objective, receive private observations correlated with a hidden state, and choose both environment and communication actions.Communication actions affect other agents' observations but have no direct impact on the environment or reward.
- Setting: Communication is needed because multiple agents and partial observability coexist, while no communication protocol is specified in advance.Agents must develop and agree on a protocol for solving the task.
- Protocol discovery: The protocol space is extremely high-dimensional because protocols map action-observation histories to message sequences.Learning also requires agents to coordinate both sending messages and interpreting them.
- Protocol discovery: Positive rewards are sparse because a useful message helps only when the receiving agent interprets it correctly and acts on it.Failed coordination can discourage the sender from repeating a potentially useful message, making random exploration difficult.
- Execution constraint: The setting uses centralised learning but restricts execution to communication through the limited-bandwidth channel.The paper notes that this paradigm applies to many problems, including robot training in simulation, but not all real-world problems.
5 Methods
The methods compare RIAL, which learns action and communication choices with independent deep Q-learning, with DIAL, which enables gradient flow across agents during centralised learning. DIAL uses real-valued messages during training and processes them for discrete decentralised execution.
- RIAL: RIAL combines DRQN with independent Q-learning to select both environment and communication actions.Each agent learns its own network parameters, treating the other agents as part of the environment.
- RIAL: RIAL separates environment and communication outputs, requiring |U| + |M| outputs instead of |U||M|.The action selector independently chooses environment and communication actions using an ϵ-greedy policy.
- RIAL: RIAL disables experience replay and feeds previous environment and communication actions into the next time-step to address non-stationarity and partial observability.During learning, agents are treated as independent networks, so execution and learning use the same decentralised setup.
- DIAL: DIAL sends richer inter-agent training signals because incoming gradients can adjust messages using downstream DQN loss rather than only the selected-message error.The incoming gradient is an |m|-dimensional vector and reduces reliance on trial-and-error exploration.
- DIAL: DIAL replaces communication actions with direct network connections, allowing real-valued messages and end-to-end backpropagation across agents during centralised learning.The messages are processed before being passed to the next agent, while decentralised execution remains constrained by the communication channel.
- DIAL: DIAL also supports continuous protocols and scales to large discrete message spaces through binary encodings with |m| = O(log(|M|).The analysis focuses on discrete messages, although continuous message spaces are used during centralised learning.
6 Experiments
The experiments evaluate RIAL and DIAL, with and without parameter sharing, on the Switch riddle and MNIST communication games. DIAL generally learns effective protocols more reliably, while channel noise and architecture choices materially affect training.
- Experimental setup: The evaluation compares RIAL and DIAL with and without parameter sharing against NoComm across two cooperative multi-agent problems.Rewards are averaged across runs and normalized by the best average reward available with access to the true state.
- Switch riddle: DIAL with parameter sharing reaches optimal Switch performance faster for n = 3 and outperforms the other methods for n = 4.For n = 3, all four methods learn an optimal policy in 5k episodes; for n = 4, RIAL without parameter sharing does not beat NoComm.
- Switch riddle: Parameter sharing is crucial for learning the Switch communication protocol, while DIAL-NS performs similarly to RIAL.The results indicate that independently learning the same protocol is difficult and that gradients provide a richer source of information than trial-and-error learning in this setting.
- MNIST games: Multi-step MNIST requires agents to integrate four 1-bit messages across time because reward is provided only at the final step.The protocol must also keep discretisation error low when continuous training messages are converted to discrete encodings.
- MNIST games: DIAL substantially outperforms the other methods on both MNIST games, whereas RIAL fails on multi-step MNIST and fluctuates around local minima on colour-digit MNIST.The NoComm baseline remains stagnant at zero, and parameter sharing is crucial for reaching the optimal protocol.
- MNIST games: DIAL learns a binary encoding and decoding of digits in multi-step MNIST, with gradients optimizing message content for rewards occurring many time-steps later.The extracted coding scheme is shown as the communication bit sent by agent 1 at time-step t as a function of its input digit.
- Channel noise: Adding noise is essential for successful DIAL training because it forces communication activations into two modes rather than centered values.The reported effect is shown for the Switch riddle by comparing learned activations with and without noise in the communication channel.
7 Conclusions
The paper contributes new environments and techniques for learning communication protocols with deep networks, alongside a comparative analysis of key design factors. It presents this work as an initial step toward the broader problem of learning communication and language.
- Contributions: The paper introduces novel environments and successful techniques for learning communication protocols with deep networks.Its analysis covers differentiable communication, architecture design, channel noise, tied parameters, and other methodological factors.
- Scope and outlook: The authors characterize the work as a first attempt, leaving compositionality, concept lifting, conversational agents, and other broader language problems for future research.The conclusion expresses optimism that the proposed approaches can contribute to those challenges without claiming to solve them here.
A DIAL Details
DIAL trains agents with differentiable communication while preserving discrete messages during decentralised execution. Its procedure combines recurrent action selection, message-gradient backpropagation, target-Q updates, and channel regularization.
- Forward pass: DIAL selects actions ε-greedily from Q-functions while producing outgoing messages at each time-step for every agent.The algorithm runs until a terminal state or the time horizon, then performs a backward pass over the episode.
- Differentiable communication: During the backward pass, DIAL regresses target Q-values and accumulates gradients that include downstream effects transmitted through communication messages.Message gradients propagate through later agents’ outgoing messages and their associated message gradients.
- Optimization: DIAL updates network parameters using accumulated gradients and periodically synchronizes target parameters.The parameter update first moves θ_i along the accumulated gradient and then copies it to θ−_i every C steps.
- Execution: Continuous channel activations are mapped to binary vectors during decentralised execution so the task exchanges discrete messages.The mapping uses 1{m^a_t > 0}, ensuring that the execution channel remains discrete.
- Channel regularization: During centralised learning, Gaussian noise and a logistic function regularize the bottleneck and reduce discretisation error.The noisy mapping is ˆm = Logistic(N(m, σ)), while execution uses the binary mapping.
B MNIST Games: Further Analysis
DIAL handles stochastic rewards more effectively than RIAL because its inter-agent gradients remain informative when sender TD errors cancel across outcomes.
- DIAL deals more effectively with stochastic rewards in the colour-digit MNIST game than RIAL.
- In a two-agent antisymmetric-reward problem, the sender’s TD error is zero for every message before a protocol is learned.Averaging over the sender’s observation makes the receiver’s reward equally likely to be +1 or −1.
- DIAL avoids this cancellation because its message gradient depends on the receiving agent’s observation and action.Consequently, gradient updates across +1/−1 outcomes no longer cancel.
C Effect of Noise: Further Analysis
Noise regularisation shapes the communication channel’s capacity and discretisation, with its effectiveness depending on noise level and how many bits the task requires.
- With σ = 2, only two bits can be encoded reliably within the logistic function’s decodable range.This estimate requires distinct activations to be separated by at least six standard deviations.
- At σ > 2, only two message options can be reliably encoded, so the channel effectively transmits one bit.Two message values are distinguishable when their outgoing distributions have little overlap.
- Figure 8 plots P(ˆm|m) across noise levels, shading probabilities above 0.1 and marking intervals whose outputs have little overlap.These intervals indicate message values that can be decoded reliably.
- Small noise is sufficient when sending more bits does not increase reward, because the network can push activations toward sigmoid tails.This reduces noise effects and encourages discretisation.
- Figure 9 compares final evaluation performance normalised by training performance across σ ∈ {0, 0.5, 1, 1.5, 2} and 2–5 steps.
- For σ = 0.5, normalised performance is around 0.7 with 2 steps and above 1 with 5 steps.Two steps provide 1 bit, whereas five steps provide 4 bits; four bits are required to encode all 10 digits.