Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning

Introduction

• The “what/how” factorization separates utility-driven content planning from surface realization, enabling integration of multi-agent communication with data-driven language learning.The authors frame content planning as functional learning and surface realization as structural learning, in line with traditional natural language generation.Source: §1 · pp. 1–2
• Text-corpus learning captures statistical structure but omits communicative function, while emergent protocols can solve tasks yet lack natural-language properties (Chaabouni et al., 2019; Kottur et al., 2017; Lazaridou et al., 2018).The cited work reports that emergent protocols do not follow Zipf’s Law of Abbreviation or natural-language compositionality patterns, and can be sensitive to experimental conditions.Source: §1 · pp. 1–2
• The paper combines generic language-model structure for “how” to say something with task-reward-driven multi-agent learning of “what” to say.Generic language data provides a structural prior, while multi-agent interactions supply task-specific rewards for functional learning.Source: §1 · pp. 1–2
• The study applies this program to a speaker-listener referential game, using image captioning for data-driven learning and agent self-play for functional learning, then evaluates reward-learned rerankers in a human study.The rerankers combine the two learning signals.Source: §1 · p. 2
• The paper also introduces automatic measures for detecting language drift and examines whether oracle rewards can support progress toward learning language use from human rewards.Source: §1 · p. 2

2 Research framing

• The research framing pairs a functional communication task with generic natural-language data and uses multi-agent language-game rewards to judge whether an utterance prompts the listener’s correct behavior.The generic language examples are not grounded in the functional task; the reward supplies task-specific feedback.Source: §2 · p. 2

2.1 Experimental setup

• In the visual referential game, a speaker describes a target among image distractors, and a listener uses the utterance to identify it; both receive positive reward for a correct choice and negative reward otherwise.Utterances are sequences of word-like units.Source: §2.1 · p. 2
• The setup uses Abstract Scenes, a dataset of 10k synthetic images with descriptive captions, and divides data 80/10/10 for train, validation, and test; each test split has 1,000 target–distractor pairs.The easy and difficult referential splits are based on target–distractor similarity.Source: §2.1 · pp. 2–3
• The task serves as a placeholder for communication: the model has no explicit task-specific bias, and task information comes only through reward.Using Abstract Scenes leaves some visual challenges outside the study’s scope while focusing on language use for referential communication.Source: §2.1 · pp. 2–3
• A human speaker choosing a discriminative caption outperformed one choosing randomly, showing room to improve on structural-only captioning and that captions can support human communication in this task.The random-caption condition gives the speaker no distractor information; the discriminative-caption condition gives access to both the target captions and distractor.Source: §2.1 · p. 3

3.1 Speaker

• The speaker embeds images with a pre-trained ResNet and one-layer MLP, then generates messages with a one-layer LSTM conditioned on the image embedding.Section 4 specifies different speaker models using these modules.Source: §3.1 · p. 3
• Two weight-free oracle speakers use ground-truth captions: one chooses randomly, while the task-aware speaker selects the caption with least word overlap with distractor captions.Random-caption performance tests whether grounded language alone suffices; the overlap score is normalized by caption length excluding stop words.Source: §3.1 · p. 3

3.2 Listener

• The listener estimates functional-task performance for evaluation or training rewards; evaluation uses a human, while training uses a learned listener that embeds utterances with an LSTM and selects by dot-product similarity.It uses the same pretrained ResNet as the speaker, then a one-layer MLP for image embeddings; cross-entropy trains listener weights, and rewards are 1 for correct choices and -1 otherwise.Source: §3.2 · pp. 3–4
• Experiments compare a joint listener trained with the speaker against a fixed listener pretrained for best response to the oracle discriminative-caption speaker; the fixed listener also evaluates all speakers.The fixed setup is expected to be less prone to language drift because it is grounded in language data, potentially improving communication with human listeners.Source: §3.2 · p. 4

4.1 Functional-only learning

• Functional-only learning uses listener rewards and REINFORCE to learn utterances that maximize task reward, but its symbols are not grounded in natural language.The speaker learns a communication policy over messages conditioned on target and distractor embeddings and the target index.Source: §4.1 · p. 4

4.2 Structural-only learning

• Structural-only learning trains an image-captioning model on paired visual embeddings and captions to produce fluent, grammatical, target-related utterances, while ignoring communication function.The model conditions only on the target image; decoding uses either greedy selection or the highest-probability message among 20 stochastic samples.Source: §4.2 · p. 4

4.3 Structural and functional learning

• The combined-learning framework starts from an image-captioning model and adds REINFORCE optimization of listener accuracy, varying how task reward updates the base model.The approaches differ in how they parameterize the speaker policy and whether reward updates the captioner’s weights.Source: §4.3 · p. 4

4.3.1 Reward finetuning

• Reward finetuning first trains an image captioner, then optimizes it with listener reward while retaining target-only conditioning, allowing distractor information to affect the policy through reward.Reward gradients can cause catastrophic forgetting of language knowledge, so the method uses a KL-divergence regularizer between pre-trained and fine-tuned language distributions.Source: §4.3.1 · p. 5

4.3.2 Multi-task learning

• Multi-task learning jointly optimizes captioning and functional objectives, using λf = 1 and variants with λs = 0.1 or λs = 1 to vary the language component’s importance.Reward gradients also flow into the base captioning model, creating a trade-off between task success and language quality.Source: §4.3.2 · p. 5

4.3.3 Reward-learned rerankers

• Reward-learned rerankers freeze a captioning model and use REINFORCE to select among its sampled sentences, extending target-only captions to task-conditional messages that account for distractors.They sample 20 candidates and apply a task-conditional reranking policy; structural and functional signals update separate submodules.Source: §4.3.3 · p. 5
• The product-of-experts reranker combines task-specific scores conditioned on target and distractor with caption-model probabilities, with variants setting λs to 0 or 1.It represents samples with transformed bag-of-words and computes similarity to a combined target-distractor embedding.Source: §4.3.3 · p. 5
• The noisy-channel reranker factors the policy into a target-given-message listener model and a target-conditioned caption prior, learning the speaker’s listener model from reward.Unlike fixed-listener RSA models, it learns the speaker’s approximation of listener behavior end-to-end; the speaker-side model is distinct from the listener’s own model.Source: §4.3.3 · p. 6

5.1 Referential success of joint listeners

• In the difficult split, reward finetuning performs worst among reward-optimizing models, multi-task does better, and emergent communication performs near perfectly in both splits.The authors attribute reward finetuning's weaker performance to its large vocabulary action space, while concurrent captioning may aid multi-task learning; emergent communication is least constrained by natural-language regularization.Source: §5.1 · p. 6

5.2 Referential success of human listeners

• Joint-listener performance does not consistently predict human performance: emergent communication leads against its joint listener but leaves humans at chance because its protocols are maximally uninterpretable.The authors identify this mismatch as pragmatic drift.Source: §5.2 · p. 7
• Reward-learned rerankers, noisy channel and PoE, achieve the best human performance, outperforming image-captioning baselines and approaching the ground-truth-caption discriminative oracle.With ground-truth target captions, noisy channel and PoE surpass the oracles; the authors expect stronger base language models to improve rerankers further.Source: §5.2 · p. 7
• The fixed grounded listener significantly predicts human performance (p < 0.005, t-test), supporting its use to provide rewards to speaker models.The paper later uses this listener as a fixed reward model.Source: §5.2 · p. 7

6.1 Structural drift: Definition and measures

• Structural drift concerns fluency and grammaticality, measured by a generated message’s log probability under a pretrained unconditional language model.Reward finetuning without a KL term produces ungrammatical sentences, while multi-task training appears less affected, with occasional repeated consecutive words.Source: §6.1 · p. 7

6.2 Semantic drift: Definition and measures

• Semantic drift concerns whether messages are grounded in the target object and is measured by their target-image-conditional log probability.When captions for target images are available, unigram and 3-gram overlap with ground-truth captions can serve as proxies; these measures do not account for distractors or task-specific information.Source: §6.2 · pp. 7–8

6.3 Structural and semantic drift results

• Reward finetuning and multi-task models show the highest structural drift, while PoE and noisy-channel rerankers perform best on semantic drift and well on other measures.The paper reports that model-specific language regularizers effectively limit both structural and semantic drift.Source: §6.3 · p. 8

6.4 Pragmatic drift

• Pragmatic drift is the difference between a human’s interpretation and the speaker’s assumed interpretation; its proxy is the difference between agent- and human-listener referential success.It is difficult to measure automatically because it is task-specific and requires access to the human’s interpretation; emergent communication illustrates an extreme case, with near-perfect joint success but human performance at chance.Source: §6.4 · p. 8
• As more components are updated, the gap between joint-listener and human referential success increases, despite fluent, target-grounded captions.Speaker and listener can co-adapt around conventions that differ from human interpretations; for example, “Mike has a hat” could signal an image property beyond the phrase’s literal meaning.Source: §6.4 · pp. 8–9, Table 5 · p. 9

7 Speakers trained using fixed listener

• The small gap between rerankers trained with fixed versus learned joint rewards suggests the joint-reward approach remains promising despite different language drift.The fixed listener is pre-trained and remains fixed while providing speaker-training rewards.Source: §7 · p. 9
• Grounded rewards from the fixed listener improve referential success for weaker speaker models.Table 6 reports referential success against both fixed and human listeners.Source: Table 6 · p. 9, §7 · p. 9
• The language-learning models may be used with fixed reward models and potentially trained directly from human rewards (Ziegler et al., 2019).The fixed oracle listener was found in Section 5 to predict human referential success.Source: §7 · p. 9

• The method combines learning signals from multi-agent communication and traditional data-driven natural-language learning, building on efforts to blend emergent communication with natural language (Lowe et al., 2020; Lu et al., 2020).Source: §8 · p. 9
• Concurrent speaker-listener self-play can produce pair-specific policies that deviate from human-learned policies, with pragmatic drift the most severe form; this pathology also occurs beyond language (Carroll et al., 2019).Source: §8 · p. 9
• The reward-learned reranker alleviates language drift and achieves the highest human performance by constraining functional learning to utterances generated by a pre-trained language model.The functional signal does not currently influence sampling from the language model.Source: §8 · p. 9
• With weaker conditioning, general language models such as GPT-2 may produce samples that do not fit the functional context, a shortcoming to address for dialogue user simulation (Schatzmann et al., 2006; Shah et al., 2008).Source: §8 · p. 9

Hyperparameters

• Hyperparameters were selected on validation data: Table 7 gives settings shared across experiments, while Table 8 gives settings for particular speakers.Table 7 settings were selected for image captioning; Table 8 settings were selected for the referential task.Source: A Appendices · p. 11, Table 7 · p. 11, Table 8 · p. 11

ResNet module

• The ImageNet-pretrained ResNet-50 receives gradients for captioning and caption-model-based methods, but is frozen during reranker reward optimization and joint listener learning to limit drift.The fixed listener’s ResNet is updated during pretraining, grounded through the discriminative caption speaker; the appendix cites He et al. (2016) for ResNet-50.Source: A Appendices · p. 11