Source-linked AI summary
Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning
Sheng Li, Yutai Zhou, Ross Allen, Mykel J. Kochenderfer
TL;DR
Continuous messages can be difficult for humans to interpret despite their representational capacity. The paper introduces end-to-end learned discrete communication using broadcast-and-listen with self-attention, and reports comparable or better performance in some domains with smaller vocabularies. It also proposes human interaction through discrete messages.
Problem
Continuous messages have high representational capacity but can come at the expense of interpretability to human designers or other agents.
Source: §1 · p. 1Method
Agents learn discrete messages through end-to-end broadcast-and-listen communication that aggregates other agents’ messages with self-attention.
Source: §1 · p. 1Results
In some domains, discrete messaging yields return performance and positive listening and signaling on par with or exceeding continuous messaging.
Source: §5 · p. 8Takeaways & Limitations
The authors propose human-agent interaction in which humans send discrete messages to agents.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Communication is a important factor that enables agents work cooperatively in multi-agent reinforcement learning (MARL). Most previous work uses continuous message communication whose high representational capacity comes at the expense of interpretability. Allowing agents to learn their own discrete message communication protocol emerged from a variety of domains can increase the interpretability for human designers and other agents.This paper proposes a method to generate discrete messages analogous to human languages, and achieve communication by a broadcast-and-listen mechanism based on self-attention. We show that discrete message communication has performance comparable to continuous message communication but with much a much smaller vocabulary size.Furthermore, we propose an approach that allows humans to interactively send discrete messages to agents.
1 INTRODUCTION
The paper replaces continuous communication, whose representational capacity can come at the expense of interpretability, with learned discrete protocols. It uses broadcast-and-listen communication and reports comparable performance with smaller vocabularies, alongside human-agent interaction.
1 INTRODUCTION
- Prior approaches include centralized aggregation in CommNet and IC3Net, attention in TarMAC, recurrent channels in BiCNet [Peng et al., 2017] and ATOC, and graph convolution in DICG.Source: §1 · p. 1
- Continuous messages offer representational capacity but can be less interpretable to human designers or other agents.Source: §1 · p. 1
- The model learns discrete protocols using argmax message selection and end-to-end broadcast-and-listen communication with self-attention aggregation.The authors use argmax rather than sampling for greater stability.Source: §1 · p. 1
- Discrete communication has performance comparable to continuous communication across varied domains with a much smaller vocabulary.Source: §1 · p. 1
- The paper proposes human-agent interaction through discrete messages and reports demonstrating their interpretability.Source: §1 · p. 1
2 BACKGROUND
The paper frames cooperative multi-agent learning as a Dec-POMDP and uses policy optimization for training. It also introduces self-attention as a way to learn which agents’ messages matter to each recipient.
2 BACKGROUND
- A Dec-POMDP describes agents, global states, actions, observations, transitions, rewards, and policies; communication messages are binary vectors in a bandwidth-b vocabulary.Source: §2 · p. 2
2.1 POLICY OPTIMIZATION
- Policy optimization maximizes expected discounted return, with generalized advantage estimation and clipped PPO used for stable updates and entropy encouraging exploration.Source: §2.1 · p. 2
2.2 SELF-ATTENTION
- Self-attention relates positions within one sequence, and prior MARL work uses attention to learn agent relations and communication patterns.Source: §2.2 · p. 2
- Here, learned attention weights distinguish the importance of messages in the public communication channel for each agent.Source: §2.2 · p. 2
3 APPROACH
The approach lets agents broadcast discrete messages in a shared channel and selectively aggregate them using learned attention. Encoded messages and attention information join agent-specific representations to inform action selection.
3 APPROACH
- Agents share a public chat room, weight other agents’ messages with attention, aggregate them, and combine the result with agent-specific vectors for action selection.The information flow can be vectorized for any number of agents.Source: §3 · p. 2
- A shared observation encoder maps each agent’s observation to an embedding, which shared attention and message encoders use to produce attention logits and message logits.Source: §3 · p. 2
- Multiplicative attention scores are softmax-normalized into weights for each agent pair, including an agent’s weight toward itself.Source: §3 · p. 3
- Argmax extracts discrete messages from logits using one-hot or bit-string encodings; their vocabulary sizes are b and 2^b, respectively.The paper gives five-bit examples for both encodings.Source: §3 · p. 3
- The augmented representation concatenates each agent’s observation embedding, attention weights, message logits, and aggregated message, with skip connections compensating for argmax gradient cutoff.Source: §3 · p. 3
- Continuous communication is obtained by bypassing argmax and broadcasting message logits; the proposed model uses one communication round, making protocol representational capacity especially important.Source: §3 · pp. 3–4
4 EXPERIMENTS
The experiments examine discrete communication across three cooperative environments and assess vocabulary and bandwidth, self-attention, communication metrics, and human interaction. They use PPO for policy optimization.
4 EXPERIMENTS
- The experiments analyze bandwidth and vocabulary effects, compare discrete with continuous communication, ablate self-attention, assess positive listening and signaling, and test human interaction.Source: §4 · p. 4
- Evaluation uses Pulling Levers, Predator-Prey, and Multi-Walker, cooperative tasks in which local information alone cannot achieve high returns.Source: §4 · p. 4
- The authors use PPO for policy optimization and a global-state MLP value-function baseline to reduce variance in advantage estimation with GAE.Source: §4 · p. 4
4.1 ANALYSIS FOR BANDWIDTH AND VOCABULARY SIZE
The section compares communication protocols across three cooperative tasks, varying bandwidth and vocabulary size and using continuous messaging as a baseline. Discrete protocols can approach continuous-message performance, but their effectiveness depends on task and vocabulary size; overly large bandwidths can be harmful.
4.1 ANALYSIS FOR BANDWIDTH AND VOCABULARY SIZE
- The experiments compare average returns across communication protocols with varied bandwidth and vocabulary size, using continuous messaging as the baseline.The paper describes average return as an indirect measure of communication's effects on behavior, including positive listening.Source: §4.1 · p. 4
4.1.1 Pulling Levers
- In Pulling Levers, continuous messaging performs best; discrete performance generally rises with vocabulary size, but one-hot performance drops beyond bandwidth 128.The 32-bit-string performs best among discrete protocols, while no communication is close to random action.Source: §4.1.1 · pp. 4–5
- In Pulling Levers with 100 participants, the reported performance trend is consistent with the 20-participant setting.Figure 3 reports normalized average returns for five levers and 100 total participants.Source: Figure 3 · p. 4, §4.1.1 · p. 5
4.1.2 Predatory-Prey
- In Predator-Prey, all discrete communication protocols outperform no communication by a large margin; continuous messaging performs best, while one-hot slightly exceeds bit-string at equal vocabulary size.Bit-string messaging matches continuous messaging when its vocabulary reaches 2^32.Source: §4.1.2 · p. 5
4.1.3 Multi-Walker
- Multi-Walker tests communication in a continuous observation-and-action task where agents coordinate locomotion to carry a package forward.The engineered observations omit LIDAR and neighboring-walker information, making information about other agents accessible only through communication.Source: §4.1.3 · pp. 5–6
- In Multi-Walker, communicative methods outperform the original observation setting, and one-hot messaging is similar to continuous messaging, with o32 performing best.Performance falls below the no_F setting as one-hot bandwidth increases to 256; overly large bandwidths are harmful.Source: §4.1.3 · p. 6
4.2 ABLATION WITH SELF-ATTENTION
The ablation compares learned self-attention with uniform attention in four-agent Predator-Prey. Uniform attention harms discrete messaging more than continuous messaging, consistent with discrete messages being less able to compensate for lost attention weighting.
4.2 ABLATION WITH SELF-ATTENTION
- The ablation assigns equal attention weight 1/n to each agent instead of using the multiplicative attention operation.Results are averaged over five random seeds in four-agent Predator-Prey.Source: §4.2 · p. 6
- Uniform attention has a more significant negative performance impact on discrete messaging than on continuous messaging.The authors suggest continuous message content can compensate for differing attention weights, whereas bit-string messages can have drastically different meanings.Source: §4.2 · p. 6
- Discrete messages’ limited representational power cannot replace the contribution of learned self-attention weights.Source: §4.2 · pp. 6–7
4.3 POSITIVE LISTENING AND SIGNALING OF COMMUNICATION
The section measures communication through supervised prediction of agents’ actions: received messages assess positive listening, while an agent’s own message assesses positive signaling. Higher bandwidth or larger vocabulary generally strengthens listening, while signaling comparisons vary with message type and bandwidth.
4.3.1 Positive Listening
- Positive listening tests whether masked messages from other agents predict the receiving agent’s actions.The receiver’s own contribution is removed from the aggregated message before the supervised prediction task.Source: §4.3.1 · p. 7
- Higher bandwidth or larger vocabulary generally strengthens positive listening across communication protocols.The results are from five random Predator-Prey runs; continuous messaging leads by bandwidth, closely followed by bit-string messaging.Source: §4.3.1 · p. 7
- One-hot messaging has higher action-prediction accuracy than bit-string messaging at vocabulary sizes |V| ≤32.The broadcast-and-listen architecture allows agents to learn others’ behaviors and intentions through their messages.Source: §4.3.1 · p. 7
4.3.2 Positive Signaling
- Positive signaling tests whether an agent’s sent message predicts its own action; continuous messaging is strongest and one-hot messaging weakest in the reported comparison.Bit-string messaging trails continuous messaging at low bandwidth, but outperforms it at bandwidth 32; the figure analyzes four-agent Predator-Prey.Source: §4.3.2 · p. 7, Figure 10 · p. 7
4.4 HUMAN INTERPRETATION AND INTERACTION
The proposed human-agent protocol uses clustered observations labeled with discrete messages as a reference for humans selecting messages. In four-agent Predator-Prey with 4-bit-string messaging, human selection performs slightly worse than agent selection but better than random selection.
4.4 HUMAN INTERPRETATION AND INTERACTION
- Humans can project new observations onto reference clusters and select the most probable discrete messages to send to agents.The reference is built by clustering observations with t-SNE and labeling them with the trained agents’ messages; Figure 11 depicts the clustering and a projected observation.Source: §4.4 · p. 8
- Human message selection performs slightly worse than agent selection but outperforms random selection in four-agent Predator-Prey with 4-bit-string messaging.The authors attribute the gap from agent selection to empirical prediction error when humans select messages from clustering and projection estimates; Figure 13 presents the comparison.Source: §4.4 · p. 8, Figure 13 · p. 8
- The protocol demonstrates interpretability of discrete messaging and a way to integrate human and AI agents.Source: §4.4 · p. 8
5 CONCLUSIONS
The conclusion presents an end-to-end broadcast-and-listen model for emergent discrete communication and reports that performance depends on bandwidth and vocabulary size. In some domains, discrete communication matches or exceeds continuous communication on returns, positive listening, and positive signaling.
5 CONCLUSIONS
- The paper presents a broadcast-and-listen model for end-to-end learning of emergent discrete message communication.Source: §5 · p. 8
- Discrete messaging’s performance depends on its bandwidth and vocabulary size.Source: §5 · p. 8
- In some domains, discrete communication matches or exceeds continuous communication in returns, positive listening, and positive signaling.Source: §5 · p. 8
- The paper proposes a human-agent interaction protocol for sending discrete messages, and identifies multi-headed attention and more direct communication metrics as future work.Source: §5 · p. 8