Source-linked AI summary
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
Serhii Havrylov, Ivan Titov
TL;DR
The paper asks whether agents can develop communicative protocols through interaction without relying on explicit supervision, using language-like sequences of discrete symbols. It studies a referential game, compares reinforcement learning with a straight-through Gumbel-softmax approach, and finds faster convergence, more effective protocols, and emergent compositionality and variability, while also exploring natural-language grounding.
Problem
The paper addresses how agents can learn to communicate through interaction when supervised learning and prior approaches using atomic categories do not provide language-like sequences of symbols.
Method
Two neural agents play a referential image-guessing game with discrete symbol-sequence messages, using straight-through Gumbel-softmax training and natural-language grounding methods.
Results
Straight-through estimator relaxations converge faster and produce more effective protocols than reinforcement learning, while the induced protocols show hierarchical encoding and multiple paraphrases.
Takeaways & Limitations
Optimizing communication success can induce structured, variable protocols with properties associated with natural language, and the protocol can additionally be shaped using natural-language information.
Takeaways & Limitations
Using discrete tokens at test time after continuous-token training can create a train-test performance gap of up to 20% in communication success.
Abstract
from arXiv · showhide
Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.
1 Introduction
The paper studies referential games in which agents invent variable-length symbol sequences from scratch, addressing communication beyond atomic categories and explicit supervision. It compares learning approaches and analyzes whether the resulting protocols acquire properties associated with natural language.
- Motivation: The work targets communication protocols that agents can invent through interaction, motivated by limitations of supervised learning and the desire to understand language emergence.The broader motivation includes communication between intelligent agents and with humans, as well as principles underlying natural-language evolution.
- Setting: Unlike prior game-based approaches using atomic categories, the agents exchange variable-length strings of discrete symbols.The sender observes the target image and produces a message sequence; the receiver uses that message and candidate images to identify the target.
- Setting: The referential game requires a sender to describe a target image and a receiver to select it from among distracting images.The sender does not access the distracting images, making the setup more challenging from the learning perspective.
- Contributions: The study demonstrates that structured protocols can be induced from scratch by optimizing reward in collaborative tasks.This contribution specifically concerns protocols represented as strings of symbols rather than single atomic messages.
- Contributions: Straight-through estimator relaxations are reported as more effective than reinforcement learning for this task.The comparison addresses the difficulty of learning over a message space whose size grows proportionally to |V|^L, producing high-variance naïve Monte Carlo gradients.
- Contributions: The induced protocol exhibits hierarchical encoding and multiple paraphrases for the same semantic content.These properties correspond to compositionality and variability, while symbol sequences also permit supervision that can move the protocol toward natural language.
2 Model
The model uses LSTM sender and receiver agents to communicate through variable-length discrete symbol sequences, with natural-language grounding and gradient-estimation methods addressing learning challenges.
- 2.1 Agents’ architectures: The sender encodes a target image into a sampled symbol sequence, while the receiver uses that message and candidate images to identify the target.Messages terminate at a maximum length or a special token; the sender lacks access to distracting images.
- 2.1 Agents’ architectures: The sender and receiver are implemented as LSTM networks, with image features and the receiver’s final message representation mapped through affine transformations.The sender generates tokens sequentially from categorical distributions, and the receiver scores candidate images from the interpreted message.
- 2.2 Grounding in Natural Language: Natural-language grounding uses KL regularization or image-captioning supervision to encourage language-like statistics or preserve more direct links between words and image meanings.KL regularization is weak grounding because identical words need not preserve their natural-language meanings; the VAE interpretation treats messages as discrete latent variables.
- 2.3.1 REINFORCE: REINFORCE estimates sender gradients but faces high variance from the enormous space of possible messages and the receiver’s non-stationary adaptation.A moving-average, input-dependent baseline and reward-standard-deviation learning-rate adaptation are used to stabilize training.
- 2.3.2 Gumbel-softmax estimator: Gumbel-softmax replaces sampled one-hot symbols with a continuous relaxation, making the communication game differentiable and trainable by backpropagation.The temperature controls approximation accuracy; as it approaches zero, relaxed samples become one-hot and match categorical sampling.
- 2.3.2 Gumbel-softmax estimator: Continuous relaxation enables efficient optimization but permits messages to encode extra real-valued information and can cause a training–testing communication-success drop of up to 20%.The gap arises because recurrent receivers may behave differently with discrete test-time tokens than with continuous training-time tokens.
- 2.3.3 Straight-through Gumbel-softmax estimator: Straight-through Gumbel-softmax discretizes messages in the forward pass while using the continuous relaxation for backward gradients, aligning training and testing message usage.The estimator is biased, so its direction is evaluated as a possible pseudogradient using stochastic perturbation estimates.
3 Experiments
Experiments compare communication-learning methods, analyze the structure of emergent protocols, and test grounding artificial languages in natural-language statistics and captions.
- More than 74k MSCOCO images were used for training and more than 8k for validation, while evaluation used the MSCOCO 2014 validation set with more than 40k images.
- 3.1 Tabula rasa communication: Argmax encoding slightly but consistently outperformed sampling, while REINFORCE beat GS-ST when the maximum message length was L = 1.
- 3.1 Tabula rasa communication: GS-ST required fewer updates as maximum message length increased, whereas REINFORCE usually required five-fold more updates and showed no clear dependence on message length.
- 3.1 Tabula rasa communication: Perplexity increased with sentence length for GS-ST, indicating redundant encodings in which multiple paraphrases express the same semantic content.
- 3.1 Tabula rasa communication: After 400 updates, the estimated GS-ST direction formed an acute angle with the true gradient in almost always (> 99%) of cases.This suggests that GS-ST can serve as a pseudogradient for the referential-game problem.
- 3.2 Qualitative analysis of the learned language: Qualitative analysis found position-sensitive and hierarchical coding: prefixes identified animal images, while repeated symbols distinguished a particular type of bears.
- 3.3 Indirect grounding of artificial language in natural language: Indirectly grounded protocols achieved communication success rates similar to natural language but had slightly lower omission scores.The omission score measures the change in target-image probability after removing the most important word.
- 3.4 Direct grounding of artificial language in natural language: With direct captioning supervision, systems had comparable image-captioning performance across metrics, while communication loss produced longer captions.The setup used 25% of the dataset for captioning and 75% for the referential game, combining losses as L = Lcaption + λLgame.
4 Related work
Related work spans longstanding studies of language emergence, recent deep-learning communication systems, and applications involving human-machine dialogue and discrete latent sequences.
- Recent multi-agent studies use deep learning to induce communication protocols for cooperative tasks.Examples include one-bit messaging for puzzles and concurrent work using Gumbel-softmax or natural-language communication in cooperative games.
- Earlier dialogue systems for collaborative human-machine activities commonly combined reinforcement learning with supervised learning.
- Concurrent research applied Gumbel-softmax to multi-agent communication in a continuous simulated environment, where the induced language showed compositional and interpretable structure.
- Natural-language communication has also been studied in cooperative image-guessing games using policy gradients.
- Sequences of discrete latent variables and variational objectives have been explored for text summarization and semantic parsing, motivating related learning strategies here.
5 Conclusion
The paper shows that neural-network agents can invent languages made of discrete token sequences and proposes efficient learning and natural-language grounding strategies. It also identifies practical dialogue modeling as future work.
- Neural-network agents can successfully invent a language consisting of sequences of discrete tokens.
- The proposed straight-through Gumbel-softmax estimator provides an efficient strategy for training these models.
- The study analyzes the learned language and learning dynamics while testing two ways to inject prior natural-language knowledge.
- Future work will extend the approach to practical dialogues and use supervised losses such as captions to support human interpretability.The authors hope this integration will reduce the amount of necessary human supervision.