Source-linked AI summary
Emergence of Compositional Language with Deep Generational Transmission
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, Dhruv Batra
TL;DR
Goal-driven multi-agent tasks can produce languages that enable coordination but are usually noncompositional and difficult to interpret, despite compositionality's relevance to generalization. The paper periodically reinitializes some agents so new agents learn language from surviving agents, creating an implicit teaching setting in a multi-agent goal-driven task. Periodic agent replacement effectively induces cultural transmission and yields more compositionally generalizable language in our setting.
Problem
Goal-driven multi-agent tasks can produce languages that enable coordination but are usually noncompositional and difficult to interpret, despite compositionality's relevance to generalization.
Source: §1 · p. 1Method
The paper periodically reinitializes some agents so new agents learn language from surviving agents, creating an implicit teaching setting in a multi-agent goal-driven task.
Source: §1 · p. 2Results
Periodic agent replacement effectively induces cultural transmission and yields more compositionally generalizable language in our setting.
Source: §1 · p. 2Takeaways & Limitations
In this setting, implicit cultural transmission through agent replacement is an effective way to obtain more compositionally generalizable language.
Source: §1 · p. 2, §7 · p. 6The study uses implicit cultural transmission; the effects of explicit cultural transmission on language structure remain for future work.
Source: §7 · p. 6
Abstract
from arXiv · showhide
Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate to solve a task. Of particular interest are the factors that cause language to be compositional -- i.e., express meaning by combining words which themselves have meaning. Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute significantly to the emergence of compositionality. In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language. We show that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization.
1 Introduction
The introduction motivates compositional language as a route to generalization and frames generational cultural transmission as a missing dimension in multi-agent language emergence. It proposes periodic agent replacement to induce implicit teaching and reports improved compositional generalization.
1 Introduction
- Compositionality lets finite meaningful elements express infinitely many concepts and supports generalization to unseen combinations from limited examples [Lake and Baroni, 2018].The introduction gives the example of understanding purple squares without directly observing them.Source: §1 · p. 1
- Evolutionary linguistics accounts for compositionality emerging over generations from competing expressivity and compressibility pressures, studied in simulations and with human subjects.Those studies aim to understand humans, whereas this paper studies artificial neural networks.Source: §1 · p. 1
- Goal-driven multi-agent tasks produce languages that enable coordination but are usually noncompositional and hard to interpret; prior compositionality methods studied fixed populations [Mordatch and Abbeel, 2018; Kottur et al., 2017].The cited fixed-population settings examine language within a single generation.Source: §1 · pp. 1–2
- The paper bridges these lines by periodically reinitializing some agents, so new agents learn grounded language from survivors and the population faces pressures for expressivity and learnability.The resulting knowledge gap creates an implicit teaching setting analogous to iterative-learning models.Source: §1 · p. 2
- Through experiments and analysis, the authors find that periodic agent replacement effectively induces cultural transmission and yields more compositionally generalizable language in their setting.Source: §1 · p. 2
2 Task & Talk: A Testbed for Compositional
Task & Talk tests whether cooperating agents develop grounded languages that support generalization to unseen attribute combinations. The section also describes stricter evaluation and population replacement as a means of introducing implicit cultural transmission.
Task & Talk
- The Task & Talk game pairs an object-observing agent with a task-aware partner, who exchanges two rounds of single-token messages before predicting requested attributes.Both agents are rewarded when the prediction is correct.Source: §2 · p. 2
Evaluation
- Random-instance splits can reward paired-attribute shortcuts; novel-pair splits withhold all instances for selected attribute pairs.The paper reports means and variances over four seeds and four-way cross-validation, totaling 16 experiments.Source: §2 · pp. 2–3
Cultural Transmission
- Periodic replacement creates knowledge gaps between agents and is introduced as implicit cultural transmission without explicit teaching.The reference game already encourages expressivity; replacement adds pressure for compressibility.Source: §2 · p. 3
Populations of Agents
- Sampling Q-bot–A-bot pairs from populations encourages a unified language, while languages easier for newly initialized agents to learn can earn greater population reward.The paper also notes that populations may add compressibility pressure even without replacement.Source: §2 · p. 3
Replacement
- The study compares uniform-random, epsilon-greedy, and oldest-agent replacement strategies, reinitializing selected agents and their optimizers every E epochs.Epsilon-greedy replacement uses ε = 0.2 in the experiments.Source: §2 · p. 3
4 Experimental Setting
The experiments compare multiple agent and vocabulary settings on novel-pair and novel-instance datasets. They report accuracy requiring both requested attributes to be correct, with means and variances across repeated trials.
Experimental Setting
- Accuracy requires Q-bot to predict both requested attributes correctly, and results are means and variances across 16 trials.The 16 trials comprise four random seeds with four-way cross-validation.Source: §4 · p. 3
- The experiments compare memory and vocabulary configurations, including Minimal Vocab, Memoryless + Minimal Vocab, Overcomplete, and Memoryless + Overcomplete.The settings vary A-bot memory and the vocabulary sizes of both bots.Source: §4 · p. 3
- Memoryless + Minimal Vocabulary yields the best compositional generalization, but is described as an extreme setting; the paper focuses mainly on Overcomplete.The extreme setting assumes a minimum number of groundable symbols and removes A-bot's ability to remember its previous utterance.Source: §4 · p. 3
Training and Baselines
- Multi-agent experiments use five Q-bots and five A-bots, stop after eight generations (199,000 epochs), and compare with single-agent populations and a no-replacement condition.Single-agent runs stop after 39,000 epochs; the no-replacement condition trains for the full 199,000 (39,000) epochs.Source: §4 · p. 4
5 Results and Analysis
Periodic agent replacement improves compositional generalization, and language-similarity measurements and conversation examples provide evidence of cultural transmission and compositional structure.
5.1 Impact of Cultural Transmission on Compositional Generalization
- On the harder dataset, cultural-transmission approaches outperform baselines without transmission, with significant differences in every paired comparison (p ≤0.05).Figure 2 reports test-set accuracies with standard deviations.Source: §5.1 · p. 4, Figure 2 · p. 5
- Multi-agent populations can show some compositionality without replacement, while replacement strategy variations perform similarly across model settings.Memory effects are small and mixed: removing memory can help Small Vocab models and hurt Overcomplete models in multi-agent settings.Source: §5.1 · p. 4
5.2 Is Generational Transmission Occurring?
- Lower language-dissimilarity scores indicate greater similarity; replacement strategies increase similarity among multi-agent bots, unlike independently trained single-agent groups.The metric averages KL divergence between token distributions over contexts and bot pairs; the independently trained groups speak similarly only by chance.Source: §5.2 · pp. 4–5
5.3 Visualizing Emergent Languages
- In the shape-color task, utterance positions map systematically to shape and color, while style is ignored, demonstrating compositional language.The first token identifies the grid row/shape and the second the column/color; blue circles, for example, receive the same utterance despite differing styles.Source: §5.3 · p. 5
6 Related work
Related work connects cultural transmission to the emergence of linguistic structure and identifies several deep-learning priors for compositionality. This paper brings generational language evolution into deep learning, alongside related work on transmission and replacement.
Language Evolution Causes Structure
- Evolutionary-linguistics accounts link cultural transmission and iterated learning to the emergence of linguistic structure in computational and human studies.The cited account treats cultural transmission as one contributor alongside innate cognitive capacity and language co-evolution.Source: §6 · p. 5
Language Emergence in Deep Learning
- Generational language evolution had not been considered in the cited deep-learning work on compositionality, and this approach is presented as viable and complementary.The related work also notes other factors that encourage compositionality.Source: §6 · p. 6
- Deep-learning studies show that limiting vocabulary can encourage compositionality, while other work examines input structure or multi-agent interaction.Multi-agent interaction has also improved emergent translation, though that work did not measure compositionality.Source: §6 · p. 6
Cultural Evolution and Neural Nets
- Prior neural-network work explored cultural ideas through direct environmental supervision, unlike this paper’s indirect, agent-mediated supervision.The cited experiments used intermediate-representation supervision for a toy task.Source: §6 · p. 6
- Concurrent studies use replacement to encourage teachable languages or bottlenecked transmission, and report increased compositionality or language efficiency.One also reports that co-evolving agents with language amplifies the effect.Source: §6 · p. 6
7 Conclusion
The paper applies implicit cultural transmission to deep neural dialog agents and reports that it produces cultural transmission and compositionality. The authors identify explicit transmission and non-language representations as directions for future work.
7 Conclusion
- Implicit, goal-driven transmission in deep neural dialog agents is reported to produce cultural transmission and compositionality.The paper contrasts its implicit approach with explicit cultural-transmission models.Source: §7 · p. 6
- The study uses implicit transmission; the effects of explicit cultural transmission on language structure remain a future-work question..Source: §7 · p. 6
- Cultural transmission may also provide a suitable prior for neural representations of non-language information.Source: §7 · p. 7
A.1 Results on Novel Instance Dataset of [Kottur et al., 2017]
The proposed approach continues to outperform models without replacement and without multiple agents on the novel-instance dataset. Results there are higher overall than on the paper’s novel-pair dataset, which the authors describe as more difficult.
A.1 Results on Novel Instance Dataset of [Kottur et al., 2017]
- The appendix evaluates the models on the novel-instance dataset from [Kottur et al., 2017] to compare against the paper’s harder novel-pair split.The stated aims are to show improvement in this setting and demonstrate that the new dataset is more difficult.Source: §A.1 · p. 9
- The proposed approach still outperforms models without replacement and without multiple agents on the novel-instance dataset.The results use four random seeds and do not use cross-validation.Source: §A.1 · p. 9
- Performance is much lower on the novel-pair dataset than on the novel-instance dataset, indicating that the novel-instance split is easier.The authors report the comparison between Figure 2 and Figure 5.Source: §A.1 · p. 9
A.2 Replacement Strategies
The appendix specifies replacement-strategy inputs and distinguishes single-agent from multi-agent replacement. It lists random, alternate, uniform-random, epsilon-greedy, and oldest strategies.
A.2 Replacement Strategies
- Replacement strategies can use the current epoch, replacement period, validation accuracy, and agent age as inputs.For each agent type, validation accuracy is averaged over possible partners.Source: §A.2 · p. 9
- Single-agent strategies replace one agent, whereas multi-agent strategies replace one Q-bot and one A-bot.The appendix points to separate algorithms for each strategy family.Source: §A.2 · p. 9
A.3 Visualization for Language Comparison at Dififerent Training Stages
This subsection introduces visualizations of agents’ learned languages at different training stages to reinforce earlier conclusions and build intuition.
A.3 Visualization for Language Comparison at Dififerent Training Stages
- The subsection visualizes language learned by agents at different stages of training.The stated purpose is to reinforce the preceding conclusions and build intuition.Source: §A.3 · p. 9
1 Input: e
The visual comparison suggests that multi-agent replacement is associated with more compositional language and with similar languages across generations. The appendix also documents replacement variants and the paired tests used to compare models and strategies.
1 Input: e
- The replacement algorithms include single-agent return choices and multi-agent strategies that return both an A-bot and a Q-bot.The listed multi-agent strategies include uniform random, epsilon greedy, and oldest replacement.Source: §1 · p. 9
- Figure 6 compares conversations between an old Q-bot and a recently reinitialized A-bot with conversations involving the same Q-bot and an older A-bot.The bots are from the eighth and final generation of a Multi Oldest run.Source: §1 · p. 9
- Figure 6 contrasts a single-agent no-replacement language with a multi-agent replacement language, which is described as somewhat compositional despite similar training time.The old and newly reinitialized A-bots’ languages are extremely similar and make the same mistakes, which the authors say suggests language transmission.Source: §1 · pp. 9–10, Figure 6 · p. 11
A.4 Detailed Results
- The appendix uses dependent paired t-tests across random seeds and cross-validation folds to compare models and replacement strategies.Model comparisons are reported in Figure 7; replacement-strategy comparisons are reported in Figures 8 and 9.Source: §A.4 · p. 10