Source-linked AI summary
On the Pitfalls of Measuring Emergent Communication
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, Yann Dauphin
TL;DR
Emergent communication is often assessed through reward improvements and message-related metrics, but these provide incomplete evidence about whether agents actually use messages. The paper evaluates such metrics with deep-RL agents in Matrix Communication Games, finding that positive signaling can arise without positive listening and advocating causal-intervention measures.
Problem
Existing emergent-communication evaluations lack a formal assessment of how well their metrics determine whether messages influence other agents.
Method
The paper surveys signaling and listening metrics, tests them with deep-RL agents in Matrix Communication Games, and proposes causal influence of communication as an intervention-based measure.
Results
Agents can show positive signaling without positive listening, including strong apparent communication when messages are scrambled before observation.
Takeaways & Limitations
Assessing emergent communication should include interventions that test the causal impact of messages on other agents’ actions, not only reward or signaling measures.
Takeaways & Limitations
The analysis may depend on shared feature layers between action and communication outputs and may therefore occur only in some emergent-communication architectures.
Abstract
from arXiv · showhide
How do we know if communication is emerging in a multi-agent system? The vast majority of recent papers on emergent communication show that adding a communication channel leads to an increase in reward or task success. This is a useful indicator, but provides only a coarse measure of the agent's learned communication abilities. As we move towards more complex environments, it becomes imperative to have a set of finer tools that allow qualitative and quantitative insights into the emergence of communication. This may be especially useful to allow humans to monitor agents' behaviour, whether for fault detection, assessing performance, or even building trust. In this paper, we examine a few intuitive existing metrics for measuring communication, and show that they can be misleading. Specifically, by training deep reinforcement learning agents to play simple matrix games augmented with a communication channel, we find a scenario where agents appear to communicate (their messages provide information about their subsequent action), and yet the messages do not impact the environment or other agent in any way. We explain this phenomenon using ablation studies and by visualizing the representations of the learned policies. We also survey some commonly used metrics for measuring emergent communication, and provide recommendations as to when these metrics should be used.
1 INTRODUCTION
The paper argues that reward gains and qualitative inspection provide only coarse evidence of emergent communication, motivating finer evaluation tools. It analyzes signaling and listening metrics in Matrix Communication Games and shows that signaling can occur without useful listening.
- 1 INTRODUCTION: Adding a communication channel and inspecting message distributions or policies are useful but coarse ways to assess emergent communication.The authors seek finer measurements to support human monitoring, fault detection, performance assessment, and trust.
- 1 INTRODUCTION: The study analyzes intuitive communication metrics with deep reinforcement learning agents playing interpretable Matrix Communication Games.These games use bidirectional communication channels and are designed to support efficient learning and policy interpretation.
- 1 INTRODUCTION: Agents can exhibit positive signaling without positive listening, so messages may correlate with actions without influencing another agent’s behavior.The paper reports that this challenges the assumption that signaling necessarily serves information sharing to improve both agents’ rewards.
- 1 INTRODUCTION: The paper surveys existing metrics, proposes causal influence of communication, and recommends interventions for evaluating whether messages affect behavior.Its contributions include distinguishing metrics of positive signaling from metrics of positive listening.
2 BACKGROUND
The paper models multi-agent interaction as a partially observable Markov game and separates environment actions from communication actions. It uses policy-gradient reinforcement learning, specifically REINFORCE, to optimize agent policies.
- 2 BACKGROUND: Each agent observes a private state-dependent observation, selects actions through a stochastic policy, and receives rewards determined by the state and joint actions.The Markov-game formulation includes states, agent-specific action and observation spaces, transitions, rewards, and an initial-state distribution.
- 2 BACKGROUND: Communication actions are discrete symbol sequences observed by other agents at the next step, while environment actions affect dynamics and rewards.The communication channel is modeled in a cheap-talk setting with zero communication cost.
- 2 BACKGROUND: Policy-gradient learning adjusts policy parameters to maximize expected cumulative reward using gradients involving action log-probabilities and expected action values.The paper uses REINFORCE with sampled returns because it is simple and common in deep-RL emergent-communication work.
3 EMERGENT COMMUNICATION METRICS
The paper distinguishes positive signaling—messages related to observations or actions—from positive listening, where messages alter another agent’s behavior, and evaluates metrics against these prerequisites. Its review shows that common measures often capture signaling without establishing that communication is used, while causal influence is designed to measure message-driven behavioral change.
- A categorization of metrics: Positive signaling requires messages to correlate with an agent’s observations or intended actions, whereas positive listening requires another agent to update beliefs or behavior after receiving them.These criteria provide the paper’s pragmatic basis for categorizing emergent-communication metrics.
- What metrics are being used now?: The reviewed literature predominantly measures positive signaling, while only instantaneous coordination is explicitly designed to measure positive listening.The survey covers prevalent metrics in deep reinforcement-learning research on emergent communication.
- What metrics are being used now?: Reward gains and qualitative policy visualizations provide only loose or potentially misleading evidence about communication, especially when improved actions rather than better communication explain performance.In situated environments, higher reward implies some channel use but does not distinguish communication quality across algorithms.
- What metrics are being used now?: Message entropy is not a reliable communication measure because low entropy can occur with constant messages and it captures neither signaling nor listening.Entropy measures consistency for a given input but not whether different inputs receive different messages or whether another agent responds.
- What metrics are being used now?: Instantaneous coordination can miss genuine listening when messages change actions conditionally on the game input but average to no message–action association across inputs.In the matrix-game example, truthful signaling supports input-dependent best responses, yet random payoff matrices make the averaged instantaneous coordination zero in expectation.
- Causal influence of communication: Causal influence of communication measures positive listening by estimating how intervening on another agent’s message changes the recipient’s action distribution within each game.The method normalizes probabilities over games and requires conditioning on variables that can affect the recipient’s action to avoid back-door paths.
4 EXPERIMENTAL SETUP
The experiments use Matrix Communication Games with turn-based discrete messaging, simultaneous actions, and rewards determined by payoff matrices. Agents are trained with REINFORCE-style policy objectives, and behavior is examined through metrics and learned-policy visualizations.
- 4.1 Matrix Communication Games: Matrix Communication Games augment matrix games with discrete, costless messages exchanged before agents act.They are designed to study communication alongside environmental action in a simple, interpretable setting.
- 4.1 Matrix Communication Games: Each round randomly selects a first speaker, allows a reply, then both agents act simultaneously and receive a payoff.The experiments use non-iterated rounds, so previous-round actions and messages are not observed.
- 4.1 Matrix Communication Games: The study varies general-sum payoff matrices from 2x2 to 8x8, with agents having partially aligned rather than strictly cooperative or competitive objectives.The communication channel is slightly larger than the action space, according to the experimental description.
- 4.2 Model and learning algorithm: Agents use two-layer feedforward policies with separate output heads for actions, communication, and a learned value baseline.The action and communication policies are optimized with REINFORCE-style objectives, while the value function reduces policy-gradient variance.
- 4.2 Model and learning algorithm: The training objective combines action and communication policy losses, entropy bonuses, and a value-function loss.Entropy terms encourage exploration, and the coefficients are tuned separately.
- 4.2 Model and learning algorithm: Figure 2 visualizes learned policies for two agents in 2x2 games, averaged over 1,000 test games, including an agent that signals its next action.The visualization is used as a qualitative examination of learned communication behavior.
5 RESULTS
In randomized matrix communication games, agents developed strong signaling indicators, but their messages generally did not influence opponents’ actions. Shared policy representations explained why speaker consistency could arise even when communication was unused.
- 5.1 Positive signaling with random payoffs: With fixed payoffs, agents generally failed to learn communication, instead collapsing to one action in partially common-interest games or cycling in zero-sum games.Direct adaptation to opponents’ actions was easier than learning a communication protocol in this setting.
- 5.1 Positive signaling with random payoffs: Agents showed positive speaker consistency in randomized-payoff games, yet causal influence was near the minimum where changing messages had no effect on action distributions.This pattern held across matrix sizes in the causal-influence evaluation.
- 5.2 Positive signaling̸ =⇒positive listening: Speaker consistency remained positive when received messages were randomly scrambled, showing that action–message correlation did not require useful communication.The authors interpret this correlation as a byproduct of optimization rather than evidence that messages helped agents coordinate.
- 5.1 Positive signaling with random payoffs: Shared hidden representations separated inputs by intended action, allowing even untrained communication outputs to correlate with actions.Speaker consistency disappeared when action and communication used separate networks.
- 5.2 Positive signaling̸ =⇒positive listening: Removing the opponent’s message did not significantly reduce action-prediction accuracy, indicating that payoff matrices already contained the relevant predictive information.This held for probes trained from the input and for probes reading the policy network’s last layer.
- 5.2 Positive signaling̸ =⇒positive listening: The same general pattern persisted in iterated MCGs using A2C agents with five-round memory, including positive speaker consistency in the randomized setting.The iterated experiment extended the evaluation beyond the non-iterated case.
6 DISCUSSION
The discussion distinguishes positive signaling from positive listening and argues that no single metric reliably characterizes emergent communication. It recommends causal and task-specific evaluations, while noting limits for long-horizon compositional communication.
- 6.1 How general is this analysis?: Positive signaling can occur without positive listening, especially when action and communication outputs share feature-learning layers.The authors present this as a possible failure mode beyond the specific MCG experiments, not as a claim that it is frequent.
- 6 DISCUSSION: The paper recommends combining several metrics because no single test captures the whole story of agents’ communication behavior.The proposed evaluation should illuminate both message production and message effects on behavior.
- 6.2 Recommendations: Reward gains or test-time message removal are insufficient by themselves to establish useful communication, so the authors recommend causal influence metrics such as CIC.Reward can improve through learned conventions, while removing messages can create an input-distribution shift that harms neural policies regardless of message usefulness.
- 6.2 Recommendations: Speaker consistency measures one-step positive signaling but can reflect spurious relationships and miss communication effects that occur later or through compositional messages.Longer-horizon settings require metrics that track effects on future behavior.
- 6.2 Recommendations: The message input norm can conclusively show absent positive listening when it is zero, but a nonzero norm does not establish that messages affect behavior.The norm concerns the message-input portion of the first policy-network layer.
- 6.2 Recommendations: Communication-quality metrics should match the environment’s objective, such as compositionality measures for compositional protocols or long-term causal influence for behavioral impact.The paper recommends developing or selecting metrics case by case.
7 CONCLUSION
The paper argues that positive signaling does not necessarily imply positive listening, and recommends intervention-based metrics to assess whether messages affect another agent’s next action. Its algorithms operationalize matrix-game steps and measurements such as speaker consistency and instantaneous coordination.
- 7 CONCLUSION: Positive signaling metrics do not necessarily establish positive listening, so communication evaluation should investigate causal effects of messages on other environment variables.The paper identifies intervention-based measures, including causal influence of communication, as a way to evaluate message impact.
- 7 CONCLUSION: The MCG environment generates states, observations, messages, simultaneous actions, and payoffs through turn-based communication.One agent is selected to communicate first, the second conditions its message on the first, and both then act using observations and messages.
- 7 CONCLUSION: Speaker consistency tracks message–own-action co-occurrences, whereas instantaneous coordination tracks each agent’s message against the other agent’s action.The algorithms update separate co-occurrence matrices before calculating mutual information.
B.1 Fixed matrix SC results
For fixed payoff matrices, positive signaling does not emerge according to speaker consistency, including in coordination settings where communication could serve as a coordination device.
- B.1 Fixed matrix SC results: Fixed-payoff games show no positive signaling under speaker consistency, even when communication could coordinate agents in games such as Battle of the Sexes.The reported measurements cover popular MCGs and 20 random fixed payoff matrices.
- B.1 Fixed matrix SC results: Agents can instead adapt to the opponent’s action and converge to a deterministic equilibrium, or cycle through actions in competitive games.
B.2 Reward and SC results on iterated MCGs
The iterated experiments extend the environment with memory and use A2C to evaluate reward and speaker consistency over repeated rounds.
- B.2 Reward and SC results on iterated MCGs: Iterated MCG experiments give each agent the previous five rounds of actions and messages and increase the discount factor γ to 0.9.The agents retain the same policy architectures while concatenating this history to each round’s input.
- B.2 Reward and SC results on iterated MCGs: The iterated setting replaces REINFORCE with a learned baseline by using the A2C algorithm.
B.3 Full metric results for randomized R
The randomized-payoff results report speaker consistency, policy entropy, and context independence, while the ablation results report reward and speaker consistency during late training.
- B.3 Full metric results for randomized R: Table 6 reports speaker consistency, policy entropy, and context independence for the randomized R setting.
- B.3 Full metric results for randomized R: Table 7 reports reward and speaker consistency over the last 10,000 training games for the paper’s ablation studies.
B.4 Result plots
The result plots compare reward and speaker consistency across matrix sizes and training settings. They also distinguish generated from random messages and report evaluation protocols for the plotted results.
- Test-game metrics are generally averaged over 10 random seeds and 1,000 games per seed, while causal influence uses five seeds.
- The experimental setup augments matrix games with a discrete one-hot communication channel and commonly uses Gaussian-randomized payoff matrices.
- Reward and speaker consistency are plotted for 2x2, 4x4, and 8x8 payoff matrices in the iterated A2C setting.
- The reported tables compare reward and speaker consistency across training variants, including scrambled messages, separate communication networks, and no communication training.
- Generated messages provide information about subsequent actions across matrix sizes, unlike randomly generated messages.