Source-linked AI summary
Multi-Agent Cooperation and the Emergence of (Natural) Language
Angeliki Lazaridou, Alexander Peysakhovich, Marco Baroni
TL;DR
Passive text training does not capture the interactive demands of communication. The paper trains agents through cooperative referential games, where senders and receivers develop a code to identify target images. Simple networks learn to coordinate, and combining gameplay with supervised naming increases symbol grounding in human labels, although keeping communication close to natural language remains future work.
Problem
Supervised training on canned conversations does not provide experience with interactive communication, motivating language-learning methods for agents that must cooperate with others.
Method
The paper trains blank-slate sender and receiver networks through reinforcement learning in image-based referential games, then mixes the game with supervised object naming.
Results
The agents reach full coordination after 10k training trials, use 88 symbols, achieve 70% symbol purity, and produce the correct supervised-label symbol in 47% of eligible cases.
Takeaways & Limitations
Multi-agent games can bootstrap communication about real images, while supervised naming provides a path toward grounding emergent codes in natural language.
Takeaways & Limitations
Ensuring that emergent communication stays close to human natural language remains future work, alongside learning structural and function-driven aspects of language.
Abstract
from arXiv · showhide
The current mainstream approach to train natural language systems is to expose them to large amounts of text. This passive learning is problematic if we are interested in developing interactive machines, such as conversational agents. We propose a framework for language learning that relies on multi-agent communication. We study this learning in the context of referential games. In these games, a sender and a receiver see a pair of images. The sender is told one of them is the target and is allowed to send a message from a fixed, arbitrary vocabulary to the receiver. The receiver must rely on this message to identify the target. Thus, the agents develop their own language interactively out of the need to communicate. We show that two networks with simple configurations are able to learn to coordinate in the referential game. We further explore how to make changes to the game environment to cause the "word meanings" induced in the game to better reflect intuitive semantic properties of the images. In addition, we present a simple strategy for grounding the agents' code into natural language. Both of these are necessary steps towards developing machines that are able to communicate with humans productively.
1 INTRODUCTION
The paper argues that interactive communication is central to cooperative AI, while passive text learning and human-in-the-loop systems have important limitations. It introduces multi-agent coordination games to study whether agents can bootstrap communicative codes and make them more interpretable.
- Cooperation among agents requires communication, making natural-language communication important for AI systems intended to interact with humans.
- Training on canned conversations does not expose learners to the interactive aspects of communication.
- The paper introduces multi-agent coordination games in which blank-slate agents develop language by playing together and bootstrapping knowledge from one another.
- The central design question is how to foster language that transfers to new situations and communication partners, especially humans.
- The study asks whether tabula rasa agents can communicate and which environmental features produce codes resembling human language.
- The paper emphasizes simpler multi-agent environments that let agents bootstrap learning, while avoiding continual human intervention or heavy scripted-environment engineering.
2 GENERAL FRAMEWORK
The framework models communication as a cooperative referential game between a sender and receiver. Success depends on the receiver using one discrete symbol to identify the target image.
- The framework consists of K players, tasks or games, a communication protocol V, and payoffs determined by a defined goal.
- In the referential game, two images are sampled and one is designated as the target.
- A sender and receiver see the images, with the sender additionally informed which image is the target.
- The sender selects one symbol from a vocabulary V and transmits it to the receiver.
- The receiver combines the two images with the sender’s symbol to guess which image is the target.
- Both players receive a payoff of 1 when the receiver identifies the target and 0 otherwise.
3 EXPERIMENTAL SETUP
The experiments use simple neural sender and receiver networks trained by reinforcement learning on image-based referential games. Sender architectures differ in whether they have an inductive bias toward combining the two image representations dimension-by-dimension.
- Data and task: The image set contains 463 concrete concepts across 20 broad categories, with 100 ImageNet images sampled for each concept.
- Agent players: Both sender architectures receive target and distractor representations in a fixed order, while the receiver sees the images in random order.
- Sender architectures: The agnostic sender maps image vectors into a learned game-specific embedding space and produces vocabulary scores from their concatenation.
- Sender architectures: The informed sender applies dimension-wise convolutions to the two image embeddings, adding an inductive bias toward combining corresponding dimensions.
- Communication and receiver: A discrete communication bottleneck converts vocabulary activations into a Gibbs distribution and samples one symbol.
- Communication and receiver: The receiver embeds images and the symbol, compares them with dot products, and samples an image from the resulting Gibbs distribution.
- Training: Training uses reinforcement learning with batch size 32 for 50k games, followed by evaluation on 10k test games.
4 LEARNING TO COMMUNICATE
Agents learn to coordinate successfully in referential games, while sender architecture affects convergence speed and symbol usage. Their emergent symbols show above-chance alignment with broad object categories, with stronger semantic purity under image-level targets.
- Communication success: Agents almost perfectly coordinate after 10k training games across every architecture and parameter choice.Evaluation used the 1k rounds following training.
- Communication success: The informed sender reaches higher communication levels more quickly than the agnostic sender.Figure 2 compares communication success over the first 5,000 training rounds.
- Symbol usage: The informed sender uses more vocabulary symbols, whereas the agnostic sender consistently uses a compact 2-symbol vocabulary.The informed sender’s usage requires multiple dimensions to summarize; cross-validated SVD suggests 50 dimensions, despite some redundancy.
- Emergent semantics: Symbol-based object clusters have purity significantly above chance in all cases, and the informed sender produces more semantically natural symbols than the agnostic sender.Purity measures agreement between cluster labels and majority category labels.
- Emergent semantics: Even the 2-symbol system shows above-chance purity, qualitatively characterizing objects as living versus non-living.The paper relates this distinction to a basic division recognized in the human semantic system.
- Object-level reference: With image-level targets, agents still coordinate, while symbol purity increases because coordination is restricted to general object-class properties.Figure 3 visualizes the relationship between visual embeddings and symbols in this setting.
5 GROUNDING AGENTS’ COMMUNICATION IN HUMAN LANGUAGE
The paper grounds an interactively learned communication protocol in human language by combining referential-game play with supervised image labeling. This preserves coordination while making symbols more interpretable to people, including when targets fall outside the supervised categories.
- The authors combine interactive referential-game communication with supervised image labeling to ground emergent symbols in natural language.The sender alternates between game playing and image classification, with shared game-embedding layers connecting the two objectives.
- The supervised objective does not reduce coordination: agents reach full coordination after 10k training trials, including 5k reference-game trials.
- 88 symbols were used after training, while symbol purity rose to 70% and the obs-chance purity difference to 37%.
- On 632 image pairs with supervised target labels, the sender produced the corresponding supervised label in 47% of cases versus 1% chance.
- Human participants selected the correct image in 68% of cases, indicating partial grounding beyond conventional word denotations.The result was statistically significant, with z = 16.75 and p < 0.0001.
- Human interpretation also succeeded through metonymic links, such as using dolphin for sea and fence for land images.In the reported examples, 10/10 subjects guessed the dolphin target and 8/10 guessed the fence target.
6 DISCUSSION
The discussion concludes that simple neural agents can coordinate over many real images and develop symbols reflecting general conceptual properties. It presents mixed supervised and interactive learning as a path toward human-language grounding, while identifying closeness to natural language as an unresolved goal.
- Simple neural-network agents learn to coordinate in referential games involving many real pictures.
- The emergent symbol meanings appear to capture general conceptual properties of depicted objects rather than low-level visual properties.
- Mixing the referential game with supervised learning provides a path toward grounding agent communication in natural language.
- Ensuring that emergent communication remains close to human natural language remains a subject for future work.The authors propose retaining predictive learning for structural language properties while using interactive games for function-driven communication.