Source-linked AI summary
Emergent Communication through Negotiation
Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, Stephen Clark
TL;DR
The paper asks how communication can emerge when agents coordinate in negotiation and studies this question with multi-agent reinforcement learning. It compares grounded proposals with ungrounded cheap talk under selfish and prosocial rewards, finding that grounded communication supports fair selfish negotiation while cheap talk supports effective negotiation for prosocial agents. It also examines diverse populations, where agent identification aids negotiation.
Problem
The paper investigates how communication emerges in coordination tasks beyond cooperative referential games, particularly when agents negotiate with hidden utilities and potentially conflicting interests.
Method
The authors train selfish and prosocial agents with task-success rewards in a negotiation environment comparing grounded proposal communication, cheap talk, combined communication, and no communication.
Results
Selfish agents negotiate fairly through grounded communication but fail to use cheap talk effectively, whereas prosocial agents use cheap talk for effective negotiation and identifiable diverse populations improve negotiation success.
Takeaways & Limitations
The findings support the paper’s conclusion that cooperation is necessary for cheap talk to emerge as effective language in negotiation.
Takeaways & Limitations
The experiments use paired agents without explicit memory of previous interactions, and the authors suggest iterated games may make cheap talk more verifiable.
Abstract
from arXiv · showhide
Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve specific problems. In this paper, we study the emergence of communication in the negotiation environment, a semi-cooperative model of agent interaction. We introduce two communication protocols -- one grounded in the semantics of the game, and one which is \textit{a priori} ungrounded and is a form of cheap talk. We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly, but are unable to effectively use the ungrounded channel. However, prosocial agents do learn to use cheap talk to find an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge. We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identifiability can aid negotiation.
1 INTRODUCTION
The paper studies how communication emerges when agents must coordinate in negotiation, extending emergent-communication research beyond fully cooperative referential games. It finds that selfish agents use grounded proposals to negotiate fairly, while cheap talk requires prosociality and agent identification can aid negotiation in diverse populations.
- Negotiation requires agents to exchange strategic information about hidden preferences while establishing a mutually acceptable division of shared items.
- The paper examines communication in a semi-cooperative negotiation game rather than only in cooperative referential or signaling games.
- Self-interested agents learn fair divisions through task-specific communication with inherent semantics but fail to negotiate effectively through ungrounded cheap talk.
- Prosocial agents can use cheap talk for effective negotiation, providing experimental evidence that cooperation is necessary for language emergence.
- In populations with varied prosociality, identifying and modeling other agents’ beliefs aids negotiation success.
2 GAME SETTING
The game gives agents hidden utilities over sampled item pools and lets them negotiate through grounded proposals, arbitrary-symbol messages, both channels, or no communication. Agents learn policies from task rewards, with selfish and prosocial reward schemes defining different negotiation objectives.
- 2.1 NEGOTIATION ENVIRONMENT: Each round samples quantities of three item types and gives each agent a hidden utility vector over those items.Item quantities range from 0 to 5, while utilities range from 0 to 10 with at least one non-zero utility.
- 2.1 NEGOTIATION ENVIRONMENT: Agents alternate proposal and message actions for a bounded number of turns, and either agent can terminate by accepting the other agent’s latest proposal.Invalid accepted proposals and negotiations reaching the turn limit yield no reward.
- 2.1 NEGOTIATION ENVIRONMENT: The turn limit is sampled from 4 to 10 using a truncated Poisson distribution with mean 7 to avoid a predictable first-mover advantage.
- 2.2 COMMUNICATION CHANNELS: The proposal channel transmits binding, task-specific divisions, whereas the linguistic channel transmits arbitrary, non-binding, unverifiable symbols as cheap talk.
- 2.2 COMMUNICATION CHANNELS: Experiments compare proposal-only, linguistic-only, combined, and no-communication configurations while replacing closed channels with fixed dummy inputs.
- 2.3 AGENT SOCIALITY AND REWARD SCHEMES: Selfish agents optimize their own reward, while prosocial agents receive the sum of both agents’ rewards, incentivizing communication about hidden values for joint allocation.
- 2.4 AGENT ARCHITECTURE AND LEARNING: Each agent encodes item context, prior utterances, and prior proposals with separate embeddings and LSTMs before producing termination, utterance, and proposal policies.
- 2.4 AGENT ARCHITECTURE AND LEARNING: Training independently maximizes expected trajectory reward plus entropy regularization using REINFORCE with an exponentially smoothed mean baseline.
3 EXPERIMENT 1: CAN SELF-INTERESTED AGENTS LEARN TO NEGOTIATE?
Self-interested agents learn fair negotiation through the grounded proposal channel, but fail to use cheap talk effectively. The authors relate this failure to divergent interests and suggest non-iterated interaction as a possible contributing factor.
- Grounded communication: Self-interested agents learn to divide items fairly when exchanging proposals directly.Each agent receives roughly equal utility, generally above 50% of total utility.
- Grounded communication: Proposal-based negotiation shows compromise, with initial over-optimistic demands decreasing over time and joint reward increasing.The reported proposal-to-final-reward drops are 0.24 for agent 1 and 0.18 for agent 2.
- Cheap talk: With cheap talk, self-interested agents do not negotiate optimally and instead oscillate between taking all items.Their messages predominantly consist of a single symbol, indicating little meaningful information exchange.
- Cheap talk: The findings align with prior cheap-talk analysis predicting no communication when agent interests diverge by a finite amount.The paper also connects this pattern to the hypothesis that cooperation is necessary for language emergence.
- Interpretation: The authors conjecture that the non-iterated setting contributes to cheap talk failure because agents lack explicit memory of previous interactions.Repeated interaction could make utterances more verifiable through credible punishment threats.
4 EXPERIMENT 2 - CAN PROSOCIAL AGENTS LEARN TO COORDINATE?
Prosocial agents use communication to improve coordination toward the optimal joint allocation. They perform especially well with cheap talk and can repurpose the proposal channel to transmit information when given sufficient turns.
- Experiment design: Prosocial rewards align negotiation with the optimal joint allocation, where each item goes to the agent valuing it more.Joint reward optimality is measured against the maximum possible reward for the item context.
- Cheap talk: The linguistic channel gives prosocial agents much better task success than the other communication schemes.Its bandwidth is theoretically unlimited, unlike the proposal channel, whose bandwidth is restricted by item types and quantities.
- Cheap talk: During linguistic negotiation, agent A adjusts its hidden proposal in response to agent B’s utterances.The decoded proposal is not revealed to the other agent, so the adjustment reflects information extracted from the utterances.
- Proposal channel: With all 10 turns available, prosocial agents using the proposal channel outperform the no-communication baseline.Under random termination, the same channel performs worse than the baseline, suggesting that sufficient time supports information transmission.
- Proposal channel: Prosocial agents appear to repurpose task-specific proposals as messages and learn a codebook without prior agreement.The paper relates this behavior to reusing task-specific actions for communication, as in bridge bidding.
5 ANALYSIS OF LINGUISTIC COMMUNICATION
Prosocial agents developed informative linguistic communication, while selfish agents mostly repeated a fixed symbol without grounding messages to item context. Probe classifiers further showed that prosocial messages encoded hidden utilities and accepted proposals.
- Symbol usage: Prosocial agents used diverse symbols and long-tailed bigram distributions, indicating structured linguistic communication.Agent B produced diverse messages, whereas agent A transmitted no information through the linguistic channel.
- Symbol usage: Selfish agents mostly sent strings of one fixed symbol regardless of item context, so no information was exchanged.Their messages showed no evidence of grounded symbol usage.
- Message content: Probe classifiers predicted hidden utilities and accepted proposals from message transcripts combined with item-pool representations.The method encoded message transcripts and item pools with separate LSTMs before classification.
- Message content: The proposal-prediction analysis used all-zero transcripts and item pools as random and item-pool-only baselines to measure transcript information.The baselines isolate how much additional proposal information is contained in the message transcript.
- Message content: Prosocial messages contained semantic information about agent B’s hidden utilities and the proposal that was made.This indicates that prosocial agents gave meaning to symbols and used them to transmit information.
6 EXPERIMENT 3: A SOCIETY OF AGENTS
In a community with mixed prosociality, fixed agents negotiated with sampled opponents while varying communication, identification, and reward conditions. Agent IDs improved selfish-agent performance, but prosocial communication and identification produced mixed outcomes, including one emergent community language.
- Community experiment: The experiment trained fixed agents against a 10-agent community containing varying proportions of selfish and prosocial agents.Episodes sampled a community opponent, played negotiation games, and updated both agents.
- Community experiment: Testing focused on whether selfish agents exploited prosocial agents and whether prosocial agents cooperated with other prosocial agents.Performance was averaged across repeated batches and all fixed-agent/community-prosocial pairs.
- Identifying agents: Providing agent IDs uniformly improved selfish fixed-agent performance, while effects for cooperative fixed agents were mixed.For a fixed prosocial agent A using the linguistic channel, IDs harmed performance; the only prosocial agents beating the 0.95 no-communication baseline lacked IDs.
- Identifying agents: Opponent embeddings clustered by the other agent’s reward scheme, showing that agents could distinguish reward schemes from observed negotiating behaviour.This pattern appeared even when agent IDs did not aid negotiation.
- Community linguistic phenomena: A prosocial community developed a language and achieved better negotiation success in one starred Table 5 condition.The starred result identifies the community that used the linguistic channel to transmit information.
- Analysis caveat: A proposal-prediction caveat is that high accuracy can arise because agents propose taking all items on every turn, which correlates with the item pool.Thus, proposal accuracy does not necessarily indicate information carried by messages.
- Community linguistic phenomena: The only emergent-language condition lacked ID information, possibly because ID-based discrimination created a poor local optimum.Prosocial agents could use IDs to distinguish prosocial from selfish agents and ignore linguistic utterances.
- Community linguistic phenomena: Different responder pairs developed distinct bigram protocols despite sharing the same fixed agent A.Their Spearman correlations ranged from -0.22 to 0.27.
7 DISCUSSION
The discussion concludes that binding, verifiable communication supports fair negotiation by self-interested agents, whereas cheap talk supports negotiation for prosocial but not self-interested agents. The paper presents interaction-based learning as a bottom-up alternative for communicative behaviour and identifies discovering cheap talk among self-interested agents as future work.
- Main findings: Self-interested agents learned fair negotiation through a verifiable, binding channel using only task success as the reward signal.The channel grounds communication in proposals that commit the sender to an action.
- Main findings: Cheap talk facilitated negotiation for prosocial agents but not self-interested agents, consistent with cited game-theoretic results.The discussion identifies emergence of cheap talk among self-interested agents as an open research direction.
- Implications: The paper demonstrates a bottom-up alternative to learning communicative behaviour directly from interaction with peers rather than dialogue data.The authors connect this approach to the possibility of learning domain-specific reasoning through interaction with a general-purpose language layer.
A ADDITIONAL FIGURES AND TABLES
Additional figures and tables document symbol usage, community bigram patterns, paired-agent negotiation outcomes, and correlations between responders’ bigram protocols.
- Additional figures: Figure 5 reports unigram statistics by turn and utterance position, plus sorted bigram counts, for selfish agents using the linguistic channel.These distributions provide the corresponding symbol-usage analysis for selfish agents.
- Additional figures: Figure 6 reports bigram usage across all interaction pairs between one fixed prosocial agent A and five prosocial agents B.It is the only case where average joint optimality reward exceeds the no-communication baseline.
- Additional tables: Table 6 reports correlations between bigram usage among different responders in the community.The table summarizes protocol similarity across responder pairs.
- Additional tables: Table 7 reports joint reward success and average negotiation turns for paired agents across reward schemes and communication channels.Results are averaged across 20 seeds with 128 games per seed and include standard deviations and quartiles.
B HYPERPARAMETER DETAILS
The experiments used 100-dimensional embeddings and hidden states, separate ADAM optimizers, policy-specific entropy regularization, and a small bounded utterance vocabulary.
- Embeddings and all neural-network hidden states had dimension 100.
- Each agent used a separate ADAM optimizer with default parameter settings.
- Entropy regularization weights were λ = 0.05 for πterm and πprop, and λ = 0.001 for πutt.
- Agents generated utterances from an 11-symbol vocabulary with a maximum length of 6.