Source-linked AI summary
Emergence of Communication in an Interactive World with Consistent Speakers
Ben Bogin, Mor Geva, Jonathan Berant
TL;DR
Prior emergent-communication work focused on simpler settings, while interactive multi-task worlds require raw-pixel communication and long action sequences. This paper proposes CCO, which structures learned representations to stabilize speaker utterances, and reports improved task performance and context-independence. Its Euclidean-distance choice is not guaranteed to reflect actual goal quality.
Problem
Prior work mainly studied simple referential games, leaving communication in interactive settings with multiple tasks and long action sequences as a more challenging problem.
Source: §1 · p. 1, §5.2 · p. 7Method
CCO structures the learned representation space so similar contexts produce more consistent utterances, and the paper introduces an alignment-based metric for context-independence.
Source: §1 · p. 2Results
CCO substantially improves performance over policy gradient and produces more context-independent communication than policy gradient and competitive baselines.
Source: §6 · p. 9Takeaways & Limitations
Under a mild assumption, perfectly context-independent communication implies compositionality, making context-independence a useful property to evaluate.
Source: §5.2 · p. 7Euclidean distance between representations is not guaranteed to track actual goal quality, although the authors report that it works well empirically.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Training agents to communicate with one another given task-based supervision only has attracted considerable attention recently, due to the growing interest in developing models for human-agent interaction. Prior work on the topic focused on simple environments, where training using policy gradient was feasible despite the non-stationarity of the agents during training. In this paper, we present a more challenging environment for testing the emergence of communication from raw pixels, where training using policy gradient fails. We propose a new model and training algorithm, that utilizes the structure of a learned representation space to produce more consistent speakers at the initial phases of training, which stabilizes learning. We empirically show that our algorithm substantially improves performance compared to policy gradient. We also propose a new alignment-based metric for measuring context-independence in emerged communication and find our method increases context-independence compared to policy gradient and other competitive baselines.
1 Introduction
The paper moves beyond simple referential games to test communication in a multi-task, interactive world, then introduces training and evaluation methods aimed at improving learning and measuring context-independence.
1 Introduction
- Prior work largely used simple referential games, unlike real-world tasks involving multiple tasks, long action sequences, and delayed feedback.These simpler settings made policy-gradient training feasible despite agents changing during training.Source: §1 · p. 1
- The proposed environment has a speaker and listener interact in a 2D world from raw pixels, across tasks requiring multi-symbol messages and long action sequences.For example, a task can require collecting exactly two yellow blocks and bringing them to a drop-zone.Source: §1 · pp. 1–2
- The proposed training algorithm substantially improves task-solving performance over policy gradient, especially as tasks become more complex.The authors attribute the improvement to structuring the learned representation space so similar contexts elicit more consistent utterances.Source: §1 · p. 2
- The alignment-based metric finds that the proposed method produces more context-independent communication than policy gradient and other competitive baselines.The metric aligns concepts with symbols; context-independence concerns whether symbols retain their semantics across contexts.Source: §1 · p. 2
2 Related work
Related work studies communication emergence mainly in referential games and examines its properties through qualitative analyses, generalization tests, or alignment; this paper instead considers interactive action sequences and proposes a multi-symbol alignment metric.
2 Related work
- Referential-game studies ask a listener to identify a speaker’s referent among distractors, whereas this paper requires agents to complete long action sequences in an interactive environment.The paper notes that the large action-sequence space and sparse positive feedback make optimization challenging.Source: §2 · p. 2
- Mordatch and Abbeel (2017) studied communication in an interactive 2D world but assumed known, differentiable dynamics and structured world-and-goal inputs rather than raw pixels.This limits the applicability of their method, according to the paper.Source: §2 · p. 2
- Earlier evaluations used qualitative analyses, generalization tests, or symbol-concept alignment for single-symbol utterances; this paper proposes a metric based on multi-symbol alignment algorithms.Lazaridou et al. (2018b) studied multi-symbol communication without explicitly testing symbol-concept alignment.Source: §2 · pp. 2–3
3 A Multi-task Grid World
The multi-task grid world gives a speaker visual access to both the world and goal, while a listener acts from the world and a one-shot utterance to complete one of three missions.
3 A Multi-task Grid World
- The 5x5 world randomizes colored objects, and the speaker sees both the initial world and a goal inventory specifying the mission, object colors, and counts.The listener sees the world and its collected-object inventory.Source: §3 · p. 3
- The speaker emits a discrete-symbol utterance once at the initial state, and the listener uses it while selecting actions over a sequence of world states.The listener can move in four directions, PAINT, or STOP; the game ends on task completion, failure, or the step limit.Source: §3 · p. 3
- The environment supports COLLECT, PAINT, and BRING missions, which overlap because each requires understanding COLLECT.PAINT adds painting collected objects; BRING requires taking collected objects to a drop-zone.Source: §3 · p. 3
- The speaker receives reward only after the full utterance when the listener succeeds; the listener receives success reward, step penalties, and object-collection rewards.The listener’s intermediate returns encourage short action sequences and navigation toward objects.Source: §3 · p. 3
- A single-agent mode lets an agent observe both goal and world and act without communication, supporting pre-training for action in the environment.In the communication game, the speaker must convey the task in one shot, while the listener must execute the actions; random success is therefore unlikely.Source: §3 · p. 3
4 Method
The method contrasts policy-gradient training with CCO, which scores utterances using learned representations and updates their geometry from reward; the paper also describes network and pre-training choices.
4.1 Policy gradient training
- Policy gradient treats each agent as acting independently while incorporating the other agent into its environment, and optimizes expected reward with sampled trajectories and critic baselines.The speaker state is the initial world-goal pair; the listener state is the current world-utterance pair.Source: §4.1 · p. 4
- High-entropy utterances and action sequences, together with non-stationarity between agents, make policy-gradient optimization fail in this environment.Similar speaker-observed worlds can yield different utterances, hindering the listener’s learning of a symbol-to-action mapping.Source: §4.1 · p. 4
4.2 Context-Consistent Obverter
- CCO replaces direct utterance generation with a scoring function and greedy token-by-token decoding, aiming for consistent utterances in similar contexts.It encodes world-goal and world-utterance pairs as vectors and scores an utterance by their negative Euclidean distance; reward-based updates move utterance representations closer or farther from the goal representation.Source: §4.2 · pp. 4–5
- CCO uses Euclidean distance even though a closer representation is not guaranteed to correspond to a better match with the actual goal.The authors report that it works well empirically and leave learning the distance function for future work.Source: §4.2 · p. 5
4.3 Neural network details
- The listener combines visual and utterance encodings to choose actions, while the speaker encodes world-goal input and, in CCO, separately encodes candidate utterances.The architectures are described as high-level network overviews; during pre-training the listener learns utterance representations from scratch, while the speaker updates only parameters representing utterances.Source: Figure 3 · p. 6, §4.3 · p. 6, §4.4 · p. 6
5 Experimental Evaluation
The task-completion experiments compare algorithms across increasingly difficult tasks and show that CCO performs well where policy-gradient methods degrade. The language evaluation uses an alignment-based context-independence metric and finds differences across algorithms, but no improvement as the number of missions increases.
5.1 Evaluation of Task Completion
- CCO performs well across tasks as task difficulty increases, while PG performance substantially decreases and PGLOWENT fails on complex tasks.Table 1 reports task-completion test results across setups with increasing numbers of colors, numbers, and missions.Source: Table 1 · p. 7, §5.1 · p. 7
- CCO and FIXEDRAND solve the 8C/1N/1M task faster than PG methods, with a slight advantage for CCO.Figure 4 plots success rate against games played for vocabulary size 15 and maximum sentence length 20.Source: Figure 4 · p. 7, §5.1 · p. 7
5.2 Evaluation of Language
- The context-independence metric scores one-to-one alignment between task concepts and utterance symbols using probabilities estimated by IBM model 1 in both directions.The score averages per-concept products of alignment probabilities; fair comparisons require matching vocabulary size and number of concepts.Source: §5.2 · pp. 7–8
- The CCO-BOW example language has a context-independence score of 0.74, with some concept-symbol alignments perfect and others asymmetric.For example, symbol 8 always refers to BLUE, but BLUE is not always described by 8.Source: Table 3 · p. 8, §5.2 · p. 8
- CCO is more context-independent than OBVERTER and PG in the referential game, especially with BOW; replacing CCO’s GRU with bag-of-words substantially improves its score.PG has very low context-independence in the referential game and on COLLECT with varying colors; its messages are much longer than CCO’s.Source: §5.2 · p. 8
- Increasing the number of missions in the multi-task setup did not improve context-independence, measured using only color and number concepts.The comparison used three colors and three numbers while varying the number of missions.Source: §5.2 · p. 8
6 Conclusions
The paper presents an interactive environment for studying emergent communication and a training approach that improves on policy gradient. CCO also produces more context-independent communication than competitive baselines.
6 Conclusions
- The paper presents a new environment for testing emergent communication in an interactive world.Source: §6 · p. 8
- CCO substantially improves performance over policy gradient and produces more context-independent communication than competitive baselines.Source: §6 · p. 9
- CCO uses structure in the learned representation space to develop a more consistent speaker, stabilizing learning.Source: §6 · p. 9