Source-linked AI summary
Pragmatically Informative Text Generation
Sheng Shen, Daniel Fried, Jacob Andreas, Dan Klein
TL;DR
Pragmatic reasoning has been used less in standard language-generation tasks, despite underinformativeness in generated text and the need to preserve input information. The method selects highest-scoring outputs under a rationality-controlled objective from beam-search candidates and can construct or select a single distractor contrasting with the input. Both pragmatic methods improve structured meaning-representation generation and abstractive summarization, including 0.2–0.5 ROUGE points on CNN/Daily Mail and 2 BLEU points on E2E, with new state-of-the-art results.
Problem
Pragmatic reasoning has been used less in standard language-generation tasks, despite underinformativeness in generated text and the need to preserve input information.
Method
The method selects highest-scoring outputs under a rationality-controlled objective from beam-search candidates and can construct or select a single distractor contrasting with the input.
Results
Both pragmatic methods improve structured meaning-representation generation and abstractive summarization, including 0.2–0.5 ROUGE points on CNN/Daily Mail and 2 BLEU points on E2E, with new state-of-the-art results.
Takeaways & Limitations
Explicit pragmatic modeling improves strong conditional generation systems by encouraging outputs that preserve information needed to identify their inputs.
Takeaways & Limitations
The coverage analysis is a lower bound on informativeness because it does not credit paraphrases, and distractor-based pragmatics requires constructing or selecting a single contrasting input.
Abstract
from arXiv · showhide
We improve the informativeness of models for conditional text generation using techniques from computational pragmatics. These techniques formulate language production as a game between speakers and listeners, in which a speaker should generate output text that a listener can use to correctly identify the original input that the text describes. While such approaches are widely used in cognitive science and grounded language learning, they have received less attention for more standard language generation tasks. We consider two pragmatic modeling methods for text generation: one where pragmatics is imposed by information preservation, and another where pragmatics is imposed by explicit modeling of distractors. We find that these methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations.
1 Introduction
The paper extends computational pragmatics from grounded language learning to structured generation and summarization, where outputs should preserve enough information for listeners to identify their inputs. Two pragmatic systems improve informativeness and task performance over a strong base generator.
- Pragmatic language generation treats production and interpretation as game-theoretic or Bayesian procedures, with speakers selecting descriptions that help listeners identify intended inputs.
- Figure 1 shows the base sequence-to-sequence system omitting input attributes, while both pragmatic systems and the human reference describe all attributes.
- The paper addresses limited NLP application of Rational Speech Acts models by applying them to structured meaning representations and natural-language texts.
- Experiments cover meaning-representation generation and abstractive summarization using reconstructor-based and distractor-based pragmatic models.
- Both pragmatic methods improve both tasks, raising ROUGE by 0.2–0.5 points on CNN/Daily Mail and BLEU by 2 points on E2E, with new state-of-the-art results.
2 Tasks
The paper evaluates pragmatic generation in two conditional text-generation settings: restaurant descriptions from structured meaning representations and multi-sentence abstractive summarization. Both tasks require substantial preservation of input information in the output.
- The framework models conditional generation as producing an output token sequence from an input, using a learned base speaker S0 to assign output probabilities.
- Meaning Representations: Meaning-representation generation maps attribute-value pairs describing restaurants to textual descriptions, building on a publicly released neural system with performance comparable to leading published results.
- Abstractive Summarization: Abstractive summarization generates multi-sentence summaries from documents, building on a system that extracts salient sentences before rewriting and concatenating them.
- Abstractive Summarization: For summarization, pragmatics is applied to the sentence-level abstractive step while the fixed RNN-EXT model supplies extracted input sentences and ABS serves as S0.
3 Pragmatic Models
The paper extends a learned base speaker with listeners that reward outputs informative about their inputs. It compares reconstructor-based selection over candidate sequences with distractor-based incremental generation against contrasting inputs.
- Both pragmatic approaches derive a speaker S1 from a base speaker S0 and listener model L, favoring outputs that help identify the correct input.
- 3.1 Reconstructor-Based Pragmatics: Reconstructor-based pragmatics uses an independently trained listener LR and selects the highest-scoring beam-search candidate under LR(i | o)^λ · S0(o | i)^(1−λ).λ controls the weight assigned to discriminative output quality.
- 3.1 Reconstructor-Based Pragmatics: For meaning representations, LR predicts each attribute value and combines those predictions into the joint probability of the input MR given the output.
- 3.1 Reconstructor-Based Pragmatics: For summarization, LR reverses the abstractive mapping by predicting a source sentence from a summary sentence using aligned training data.
- 3.2 Distractor-Based Pragmatics: Distractor-based pragmatics contrasts the true input with a task-specific distractor and makes pragmatic decisions at each word rather than across complete candidates.
- 3.2 Distractor-Based Pragmatics: For meaning representations, distractors mask present attributes and replace absent attributes with frequent values; for summarization, the preceding extracted sentence serves as the distractor.
4 Experiments
The pragmatic models are evaluated on E2E meaning-representation generation and CNN/Daily Mail abstractive summarization against strong baselines and prior systems. Both improve over the base speaker, with gains reaching new state-of-the-art results on E2E and competitive performance in summarization.
- Experimental Setup: The experiments use standard benchmark datasets, automatic evaluation against human references, and development-set tuning of beam size, α, and λ.The E2E evaluation reports five automatic metrics, while the summarization evaluation uses ROUGE and METEOR.
- Meaning Representations: 2.1 BLEU, 0.2 NIST, 0.8 METEOR, 1.5 ROUGE-L, and 0.1 CIDEr are the largest improvements from the SR1 model over S0 on E2E.SR1 outperforms the previous best E2E challenge results on BLEU, NIST, and CIDEr, with comparable METEOR and ROUGE-L.
- Meaning Representations: Both pragmatic systems improve over S0 on all five E2E metrics, while SR1 achieves new state-of-the-art results on three metrics.The evaluation reports BLEU, NIST, METEOR, ROUGE-L, and CIDEr.
- Abstractive Summarization: 0.2–0.5 ROUGE and 0.2–1.8 METEOR improvements over S0 are obtained on CNN/Daily Mail, with SD1 outperforming SR1.SD1 is strong across all metrics and competitive with the best previous abstractive systems.
5 Analysis
The analysis examines whether pragmatic decoding preserves input content in E2E outputs using attribute-level coverage ratios. Reconstruction-based pragmatics consistently increase coverage, while distractor-based effects depend on the masked attributes and the metric’s exact-match limitation.
- Coverage Analysis: SR1 increases coverage relative to S0 across all attributes, whereas SD1 increases coverage in four of six categories and usually by less.SR1 explicitly selects outputs using reconstruction scores, while SD1 optimizes less directly for attribute mentions.
- Distractor Analysis: The highest SD1 coverage usually occurs when the corresponding attribute is masked in the distractor, especially for FAMILYFRIENDLY, FOOD, PRICERANGE, and AREA.These are typically the diagonal entries in the single-attribute masking analysis.
- Measure: Coverage ratio is an exact-match measure and therefore provides roughly a lower bound on actual informativeness because paraphrases receive no credit.It is computed as the fraction of specified attribute values whose exact text appears in the generated output.
- Distractor Analysis: Masking one attribute can decrease coverage, while masking other attributes can produce substantial increases, including masking FAMILYFRIENDLY or CUSTOMERRATING.The analysis also reports an equal CUSTOMERRATING coverage increase for CR and another condition, potentially reflecting correlations from their small value sets.
6 Conclusion
Explicit pragmatic modeling improves generation by encouraging outputs that preserve enough information to identify their inputs. The paper also points toward finer-grained communication-cost modeling and earlier content selection as future directions.
- Both pragmatic methods improve generation by encouraging outputs that help identify their inputs through reconstruction or distractor discrimination.These methods produce similar improvements because both promote input-identifying outputs.
- S0 models are strong but still imperfectly capture how people generate text, leaving room for explicit pragmatic modeling to improve results.
- Future work could model under- and over-informativity more finely with learned communication costs or apply pragmatics earlier during content selection.
A.1 Reconstructor Model Details
The reconstructor model handles sparse MR attributes with delexicalized placeholders and predicts attribute information from attention-weighted sentence representations. Its task-specific outputs produce probability vectors for the MR attributes.
- A.1 Reconstructor Model Details: Sparse NAME and NEAR values are replaced with delexicalized placeholders, so the reconstructor predicts their presence as boolean variables.Other MR attributes are represented with corresponding categorical variables.
- A.1 Reconstructor Model Details: A bidirectional GRU encodes each word, and attention aggregates word information separately for each MR-attribute prediction task.The attention mechanism reflects that words contribute unequally to predicting different attributes.
- A.1 Reconstructor Model Details: Task-specific sentence representations feed softmax layers that return a probability vector for each MR attribute.
A.2 Hyperparameters
The experiments use different beam sizes and pragmatic-model weights for structured generation and abstractive summarization. Hyperparameters are tuned on development-set objectives specific to each task.
- A.2 Hyperparameters: Structured generation uses beam size 10, λ = 0.4, and α = 0.2, tuned on the normalized average of five development metrics.
- A.2 Hyperparameters: Abstractive summarization uses beam size 20, λ = 0.9, and α = 1.0, tuned to maximize development-set ROUGE-L.