Source-linked AI summary
Learning to Write with Cooperative Discriminators
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, Yejin Choi
TL;DR
RNNs often generate generic, repetitive, and self-contradictory text because their objective does not capture communicative goals well enough. The paper complements an RNN generator with a learned mixture of discriminators inspired by communication principles. Human evaluations preferred the resulting text over competitive baselines by a large margin, while revealing remaining coherence and subtle consistency limitations.
Problem
RNN language models often generate repetitive, overly generic, and self-contradictory long-form text, motivating objectives that better capture communicative qualities.
Method
L2W combines an RNN language-model probability with discriminatively trained communication-model scores weighted by learned mixture coefficients.
Results
Human evaluation preferred L2W over competitive baselines by a large margin in two domains, with gains in coherence, style, and information content.
Takeaways & Limitations
A learned combination of linguistically motivated sub-models can produce a stronger decoding objective for long-form text generation.
Takeaways & Limitations
L2W still produces mostly disconnected sentences in TripAdvisor generations and does not address subtle linguistic consistency issues such as switching pronouns midgeneration.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models, but when used to generate natural language their output tends to be overly generic, repetitive, and self-contradictory. We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the notion of communicative goals described by linguistic principles such as Grice's Maxims. We propose learning a mixture of multiple discriminative models that can be used to complement the RNN generator and guide the decoding process. Human evaluation demonstrates that text generated by our system is preferred over that of baselines by a large margin and significantly enhances the overall coherence, style, and information content of the generated text.
1 Introduction
RNNs often generate repetitive, self-contradictory, and overly generic long-form text. L2W addresses these problems by combining cooperative discriminators that guide decoding toward stronger communicative qualities, with human evaluations showing substantial gains over baselines.
- RNN language models often produce long-form text that is repetitive, self-contradictory, and overly generic.
- L2W constructs a stronger generator by composing discriminators, each specializing in a different principle of communication.The framework starts with an RNN language model and learns to combine discriminative models that address limitations of the base generator.
- The framework learns discriminator weights to form a stronger, more global decoding objective.The discriminators complement one another and the base language model during decoding.
- The discriminators are inspired by Grice’s maxims and learn communicative qualities through specialized training data.Examples include distinguishing true continuations from random samples or language-model-generated continuations.
- Human evaluation preferred L2W over competitive baselines by a large margin in two domains, enhancing coherence, style, and information content.Automatic evaluation also found L2W less repetitive and more diverse than baselines.
2 Background
RNNs struggle with long-term context and tend toward generic or repetitive continuations. L2W responds by learning a better decoding objective that unifies several modifications through discriminative guidance.
- RNNs assign high probability to repetitive and overly generic sentences, while gated variants also struggle to incorporate long-term context.The passage attributes these difficulties to explaining-away effects, diminishing gradients, and limited inductive bias.
- Existing methods improve generic generation through probability normalization, future-cost estimation, or diversity-oriented objectives.
- Hard trigram-recurrence constraints reduce repetition but cannot prevent paraphrased repetition and may block intentional repetition.
- L2W learns a better decoding objective that addresses multiple long-form generation challenges and generalizes existing objective modifications.
3 The Learning Framework
The framework combines an RNN language model with discriminatively trained communication models and learned mixture weights to guide beam-search generation. Its models target repetition, entailment, relevance, and lexical style using prefix-aware scoring during inference.
- The decoding objective combines RNN language-model probability with learned weighted discriminator scores for conditional generation.When discriminator scores are log probabilities, the combination forms a Product of Experts model.
- Beam search scores incomplete prefixes while allowing discriminator scores to capture non-factorized long-range context.Prefix scores are estimated at each time step, and scorers are trained on prefixes of matching lengths to approximate inference.
- The framework trains multiple discriminators to distinguish good and bad generations, varying model parameterization and examples to target different Gricean communication principles.Discriminator outputs are treated as classification probabilities and interpolated as log probabilities.
- Repetition Model: The repetition discriminator biases generation against excessive recurrence while preserving natural repetition by modeling similarity among words within a fixed previous-word window.It compares RNN-generated and gold continuations and uses a recurrent model over similarity scores.
- Entailment Model: The entailment discriminator uses neutral-class probability to discourage contradiction and paraphrasing, scoring completed sentences against preceding context and continuation sentences.The score is carried from the last complete sentence until the next sentence ends, because sentence endings can change entailment.
- Relevance and Lexical Style Models: The relevance model distinguishes true continuations from random human-written endings, while the lexical-style model targets word-choice distributions associated with writing manner.The relevance model represents context and continuation sequences with convolution and maxpooling before scoring their similarity.
4 Experiments
The experiments evaluate long-form continuation generation on TripAdvisor and BookCorpus against language-model, seq2seq, and GAN baselines. Human evaluation compares repetition, contradiction, relevance, and clarity, while automatic evaluation includes standard and custom measures.
- 4.1 Corpora: Evaluation uses TripAdvisor hotel reviews and BookCorpus novels, with each example split into five context sentences and five continuation sentences.TripAdvisor contains 330 million words and BookCorpus contains 980 million words.
- 4.2 Baselines: The baselines include AdaptiveLM, CacheLM, an attention-based seq2seq model, and SeqGAN, using beam search except for SeqGAN’s greedy sampling.AdaptiveLM uses a 100k vocabulary, CacheLM and seq2seq use 50k, and SeqGAN uses 25k.
- 4.3 Evaluation Setup: Models generate five-sentence continuations from initial contexts because the open-ended task does not require a fixed continuation length.The five-sentence target matches the construction of the discriminator and seq2seq datasets.
- 4.3 Evaluation Setup: BLEU and Meteor are reported alongside custom metrics, although prior work found them unreliable for long or creative generation with many acceptable outputs.For open-ended generation, the paper identifies human evaluation as the only reliable measure.
- 4.3 Evaluation Setup: Human evaluation presents two possible endings and scores repetition, contradiction, relevance, and clarity, criteria closely inspired by Grice’s Maxims.Each criterion is compared on a 5-point scale.
- 4.3 Evaluation Setup: The human evaluation samples 100 test examples per corpus for each generator pair, using three Amazon Mechanical Turk annotators per example.Likert results are averaged by criterion, while overall-quality judgments aggregate votes across examples.
5 Results and Analysis
L2W generally outperformed competitive baselines in automatic and human evaluations, especially for repetition, relevance, coherence, and style, while retaining important limitations in discourse connectivity and subtle self-repetition.
- Quantitative Results: L2W was superior or competitive with all baselines on BLEU and Meteor, diversity, and generation length, although absolute BLEU and Meteor performance remained low.For generation length, L2W exceeded ADAPTIVELM on BookCorpus, while ADAPTIVELM was better on TripAdvisor.
- Quantitative Results: On BookCorpus, crowd-sourced evaluators consistently favored L2W over baselines across all comparison dimensions, particularly for repetition, clarity, and relevance.ADAPTIVELM was the strongest competing baseline; SEQGAN was diverse but produced shorter sentences, while CACHELM and SEQ2SEQ had broader quality problems.
- Quantitative Results: On TripAdvisor, L2W ranked higher in overall quality and most individual metrics, with strongest gains in repetition and relevance but no improvement over ADAPTIVELM in contradiction or clarity.ADAPTIVELM remained the most competitive baseline.
- Ablation Analysis: Ablation results indicate that each discriminator helps its intended purpose, while the repetition module provides the largest boost and the entailment module is weakest.The entailment module’s weakness is hypothesized to reflect training-test mismatch and a nonsmooth entailment score.
- Qualitative Analysis: Qualitative examples show that L2W produces more topical, stylistically coherent continuations and avoids common generic sentences through semantically varied phrasing.Baselines instead repeat over-used or generic sentences, with SEQ2SEQ failing to align context and continuation and SEQGAN eventually reverting to generic repetition.
- Limitations: L2W does not eliminate all RNN degeneration: TripAdvisor outputs can be meaningful but disconnected, subtly paraphrase earlier content, and switch pronouns midgeneration.The authors state that none of their models addresses these subtle linguistic issues.
6 Related Work
Prior work improves decoding objectives, models pragmatic communication, explores GANs, or targets domain-specific long-context generation. This paper instead presents a general framework for learning a stronger generator through discriminative decoding objectives.
- Alternative Decoding Objectives: Alternative decoding objectives promote diversity, incorporate reverse conditional probabilities, or impose anti-repetition constraints, but hard trigram constraints can miss paraphrased repetition.These approaches modify decoding rather than constructing the paper’s unified learned objective.
- Pragmatic Communication Models: Pragmatic communication models apply Grice-inspired reasoning to referring expressions and neural generation, often combining generative speakers with ranking losses.The cited models provide precedent for modeling communicative goals beyond likelihood.
- Generative Adversarial Networks: GAN-based language generation avoids maximum likelihood estimation but faces difficulties from discrete-sequence backpropagation and unstable training, without demonstrated high-quality long-form text.The passage contrasts these challenges with the present work’s reported results.
- Generation with Long-term Context: Prior long-context generation studies develop domain-specific architectures for discourse patterns, whereas this work proposes a general framework for learning a generator beyond maximum likelihood.The comparison spans image captions, reviews, sports reports, and recipes.
7 Conclusion
The paper concludes that learned combinations of linguistically motivated sub-models provide a unified decoding framework for more coherent long-form text. Human evaluation found that its outputs exceeded competitive baselines by a large margin.
- Conclusion: The framework targets long, coherent text by overcoming some common limitations of RNN-based generation.The conclusion states this as the paper’s central scope.
- Conclusion: It learns a generation-suitable decoding objective by combining sub-models that represent linguistically motivated qualities of good writing.The conclusion characterizes the contribution at the framework level.
- Conclusion: Human evaluation showed that the model’s generated text exceeded competitive baselines by a large margin in quality.This is the principal author-reported outcome in the conclusion.
Entailment Model
The passage describes implementation and evaluation details for the entailment component, including its MLP configuration, optimization settings, and validation accuracy on SNLI and MultiNLI.
- Model Configuration: The entailment model projects word embeddings to hidden size 200 and uses two-layer MLPs with dropout rate 0.2.These settings follow the cited hyperparameters of Parikh et al. (2016).
- Evaluation: The entailment classifier achieves 82% accuracy on SNLI validation and 68% on MultiNLI validation.These are the reported validation accuracies for the classifier.
- Model Configuration: Its convolutional layer uses one-dimensional filters of size 3 with stride 1, with padding preserving input and output sequence lengths.The padding keeps the sequence dimensions equal across the convolution.
B Baseline Details
The baselines use distinct decoding and modeling settings, with memory and tractability constraints shaping vocabulary, sequence length, rollout, and update choices.
- CACHELM: CACHELM uses a 50k-word vocabulary and beam search with beam size 5 because of memory constraints.
- SEQGAN: SEQGAN restricts vocabulary to 25k words, sequence length to 250, and Monte Carlo rollouts to length 4 to make training tractable.Its discriminator is updated once every 10 generator steps, and greedy decoding with temperature 0.7 outperformed beam search in development.
- SEQ2SEQ: SEQ2SEQ uses a 50k-word vocabulary and beam search with beam size 5 because of memory constraints.
C Corpora
The study trains and evaluates language models and discriminators on ten-sentence segments from TripAdvisor and BookCorpus, using held-out data for discriminator training and evaluation.
- The language model and discriminators use a 100,000-word vocabulary, with larger vocabularies empirically producing better generation quality.
- Each segment contains ten sentences: the first five provide context and the second five provide the reference continuation.
- TripAdvisor segments use the first ten sentences of reviews at least ten sentences long, while BookCorpus is split into ten-sentence segments.
- Twenty percent of each corpus is held out, with the remainder used for language-model training.
D Evaluation Setup
The evaluation materials are presented as separate first- and second-half forms for BookCorpus and TripAdvisor human evaluations.
- Table 5 presents the first half of the BookCorpus human-evaluation form.
- Table 6 presents the second half of the BookCorpus human-evaluation form.
- Table 7 presents the first half of the TripAdvisor human-evaluation form.
- Table 8 presents the second half of the TripAdvisor human-evaluation form.