Source-linked AI summary
Reasoning About Pragmatics with Neural Listeners and Speakers
Jacob Andreas, Dan Klein
TL;DR
The paper addresses how to generate pragmatic, context-sensitive descriptions from ordinary non-pragmatic captions rather than task-specific pragmatic supervision. It combines neural listener and speaker models with higher-order reasoning about listener behavior, achieving 17% more correct interpretation than a recent learned baseline in human referring-expression games.
Problem
Generating a description that uniquely identifies a referent in context is difficult because isolated captions are not guaranteed to work as referring expressions for particular distractors.
Method
The model combines neural listener and speaker modules with a higher-level reasoning model that selects descriptions according to expected listener behavior.
Results
17% more often was the improvement in correct interpretation over a recent learned baseline in human referring-expression games.
Takeaways & Limitations
The approach obtains pragmatic behavior without hand-engineered domain knowledge or targeted pragmatic training data.
Takeaways & Limitations
Direct models require pragmatics-specific training data, which is scarce in practice, while the experiments use vector indicator features for abstract-scene referents.
Abstract
from arXiv · showhide
We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics. Like previous learned approaches to language generation, our model uses a simple feature-driven architecture (here a pair of neural "listener" and "speaker" models) to ground language in the world. Like inference-driven approaches to pragmatics, our model actively reasons about listener behavior when selecting utterances. For training, our approach requires only ordinary captions, annotated _without_ demonstration of the pragmatic behavior the model ultimately exhibits. In human evaluations on a referring expression game, our approach succeeds 81% of the time, compared to a 69% success rate using existing techniques.
1 Introduction
The paper combines learned neural language and image models with pragmatic reasoning about listener behavior to generate contextually appropriate descriptions. It targets referring-expression games using ordinary captions and improves human interpretation over a learned baseline.
- 1 Introduction: The model combines neural language and image-retrieval modules with probabilistic pragmatic reasoning about listener behavior.It learns grounded generation while embedding base models in a higher-order model that reasons about listener responses.
- 1 Introduction: In the reference game, a speaker describes a secretly selected target among two revealed candidates, and both players win when the listener selects that target.The model must produce descriptions that are informative, fluent, concise, and sensitive to listener behavior.
- 1 Introduction: Figure 1 illustrates contrastive description: the model mentions a tree distinguishing the target image from its distractor and situates it in the scene.The figure contrasts a target image, a distractor image, and the generated description.
- 1 Introduction: Direct approaches learn pragmatic behavior from task-specific annotations without listener representations, while derived approaches reason about listeners but depend on hand-engineered grammars and small domains.The paper positions its method between these lines of work.
- 1 Introduction: The proposed approach aims to scale derived pragmatic modeling to real-world datasets without domain engineering or targeted pragmatic training data.It combines learned grounded generation with higher-order reasoning and claims robust behavior across contexts.
- 1 Introduction: 17% more often was the improvement in correct interpretation over a recent learned baseline in human referring-expression games.The evaluation used a large dataset of abstract scenes, with humans as listeners and the system as speaker.
2 Related Work
Related work spans direct models that learn referring expressions from pragmatic annotations and derived models that combine language generation with probabilistic or game-theoretic reasoning. The paper also connects computational pragmatics to broader language-and-world representation tools and tasks.
- Direct pragmatics: Direct pragmatics models learn distributions over referring expressions from human-generated or task-specific pragmatic data.FitzGerald et al. learn a maximum-entropy distribution over logical expressions denoting a target set.
- Derived pragmatics: Derived pragmatics models combine template-driven generation with probabilistic or game-theoretic reasoning about speakers and listeners.These approaches are also known as rational speech acts models and have been used to explain human behavior.
- Other applications: Computational pragmatics has also been applied to instruction following and discourse analysis beyond referring-expression games.
- Representing language and the world: The paper's approach relies on multimodal tools including image retrieval models and neural conditional language models.These tools connect textual descriptions with image or other unstructured representations.
3 Approach
The approach combines learned neural speaker and listener models with inference over listener behavior, using ordinary non-contrastive captions to generate contextually appropriate descriptions. Its modular architecture maps referents and descriptions into shared embeddings, supports literal models, and samples candidate descriptions for pragmatic reasoning.
- Motivation: The model addresses referring-expression generation from isolated captions, where training descriptions may not uniquely identify a referent in context.Given a target and distractor, it must produce a description that uniquely identifies the target.
- Modules: Its modular architecture includes referent and description encoders, a choice ranker, and a referent describer.The encoders create common embeddings; the ranker predicts referent choices, while the describer generates strings.
- Base models: The base models comprise a literal listener L0 that predicts referents from descriptions and candidates, and a literal speaker S0 that generates descriptions from referents.L0 is trained contrastively against random distractors, while S0 is trained by maximizing caption likelihood.
- Reasoning model: The reasoning speaker reweights candidate descriptions using both literal-speaker plausibility and literal-listener interpretation.S0 supplies samples likely to be plausible descriptions, avoiding exhaustive enumeration of the utterance space.
- Design rationale: The approach combines learned grounded generation with higher-order reasoning about listener responses, reducing reliance on hand-written grammars and pragmatically annotated training data.This combines the data-driven acquisition of direct approaches with the listener representation used by derived pragmatic models.
4 Evaluation
The evaluation tests pragmatic captioning on human-created abstract scenes, measuring fluency, referring-game accuracy, sampling efficiency, and whether reasoning can be compiled into a direct speaker. The reasoning model achieves strong accuracy and fluency, while the compiled approximation does not match its overall performance.
- Base-model tradeoff: Varying λ changes caption naturalness and specificity: listener-heavy outputs are discriminative but disfluent, whereas speaker-heavy outputs are fluent but generic.Increasing λ makes captions more generic.
- Base-model tradeoff: A small speaker weight, λ = 0.02, achieves near-perfect fluency without substantially reducing listener-based accuracy.On development data, average fluency is 4.8, matching the literal speaker’s 4.8.
- Sampling efficiency: Sampling gains continue up to 100 samples, motivating the use of 100 samples for final predictions.The final evaluation fixes λ = 0.02 and uses 100 samples.
- Is reasoning necessary?: The compiled speaker improves over S0 on scenes with 2–3 differences, but its overall mean-score gain is negligible and it significantly underperforms the reasoning model.The overall difference from the base model is not significant.
- Final evaluation: The reasoning model improves RG success by 17% on All pairs and 15% on Hard pairs over the compared baselines.The baselines are Literal, a standard captioning model, and Contrastive, a reimplementation of prior work.
5 Conclusion
The paper learns pragmatic descriptions from ordinary non-pragmatic captions by combining neural listener and speaker models with higher-order reasoning. In a referring expression game, the resulting descriptions produce correct human interpretation more often than existing baselines.
- Contribution: The approach learns pragmatic descriptions without training data collected in a pragmatic context.It combines simple neural listener and speaker base models with a high-level model that reasons over their outputs.
- Contribution: By synthesizing direct and derived pragmatics, the model obtains pragmatic behavior without hand-engineered domain knowledge or targeted training data.The authors suggest this reasoning strategy may apply more broadly to contextual behavior from neural decoding models.