PaperlayerView exact revision on arXiv

Source-linked AI summary

Language GANs Falling Short

Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, Laurent Charlin

arXiv:1811.02549v6cs.CLcs.LG

TL;DR

The paper addresses how to evaluate and improve text generation when quality and diversity trade off, especially amid claims that GANs solve MLE exposure bias. It proposes temperature sweeps to compare models across that spectrum and finds that MLE models outperform considered GAN variants, while noting important computational and distributional limitations.

  • Problem

    NLG evaluation must account for both generated-text quality and diversity, while MLE models were criticized for exposure bias and GAN evaluations often used incomplete comparisons.

  • Method

    The paper varies softmax temperature to trace models’ quality-diversity trade-offs, comparing MLE baselines with text GANs and discussing alternative decoding controls.

  • Results

    MLE models reliably outperform the considered GAN variants across the quality-diversity space on synthetic and long-text generation experiments.

  • Takeaways & Limitations

    Temperature tuning provides a computationally efficient way to characterize NLG performance and establishes a boundary that future GAN approaches must surpass.

  • Takeaways & Limitations

    The sweep changes conditional rather than joint-distribution entropy, biasing samples toward lower entropy at sentence beginnings; joint-entropy control can be intractable.

Abstract

from arXiv · show

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort

1 INTRODUCTION

The paper argues that evaluating NLG requires balancing quality and diversity, then proposes temperature sweeps to compare models across that trade-off. Using this framework, it concludes that MLE models outperform GAN variants across the quality-diversity space.

  • Motivation: Exposure bias motivated text GANs because their adversarial objective directly targets generated-sample quality.MLE models use teacher forcing, whereas GANs were proposed as an alternative that avoids exposure bias during training.
  • Evaluation problem: Quality-only evaluation can favor models that generate likely but repetitive sequences, because it ignores sample diversity.Evaluating quality and diversity jointly is also ambiguous when neither model dominates on both dimensions.
  • Proposed evaluation: A temperature sweep characterizes each model’s quality-diversity trade-off by varying the softmax temperature during generation.Lower temperatures yield less diverse, higher-quality samples, while higher temperatures yield more diverse, lower-quality samples.
  • Main conclusion: MLE models still dominate GANs under the paper’s temperature-sweep evaluation across the studied quality and diversity metrics.The framework is presented as a clearer and computationally efficient way to assess NLG progress.

2 ADVERSARIAL TEXT GENERATION

The section reviews why GANs were adapted for discrete sequential text and why evaluating them requires both quality and diversity measures. Existing evaluation frameworks often leave model preferences unresolved.

  • GAN formulation: GANs train a generator and discriminator in a minimax competition, with the discriminator distinguishing real from generated samples.The generator defines an implicit distribution over samples, while the discriminator supplies the adversarial signal.
  • Discrete text generation: Text GANs require alternative optimization because discrete sequential outputs prevent directly differentiating through the original GAN objective.Some approaches estimate generator gradients with REINFORCE policy gradients.
  • Evaluation limitations: Likelihood-only evaluation is blind to diversity and can reward a model that repeatedly generates one highly likely sequence.This limitation affected early synthetic evaluation frameworks for text GANs.
  • Evaluation limitations: Quality and Self-BLEU jointly measure quality and diversity, but scatter plots remain inconclusive when no model dominates on both metrics.This ambiguity motivated evaluating models over their quality-diversity trade-off.
  • GAN variants: Non-RL GAN variants were introduced partly because reinforcement learning can be difficult to optimize, unstable, and hyperparameter-sensitive.Cooperative training and FM-GAN are cited as fully differentiable alternatives.

3 TEMPERATURE SWEEP: TOWARDS ROBUST NLG EVALUATION

The paper uses softmax temperature to control conditional-distribution entropy and proposes sweeping temperatures to compare models across quality and diversity. It also discusses alternative decoding controls and the resulting computational trade-offs.

  • Temperature mechanism: Decreasing the Boltzmann temperature α below 1.0 lowers the entropy of the generator’s conditional token distribution.The conditional distribution is defined as Gθ(x_t | x_1:t−1) = softmax(o_t · W/α).
  • Temperature mechanism: At α = 0.7, MLE-generated samples achieve both high quality and sufficient diversity, while α > 1.0 degrades syntax and α = 0.0 produces one sequence.At α = 1.0, samples are syntactically correct but often lack global coherence.
  • Evaluation framework: A temperature sweep generates samples at multiple temperatures and traces each model’s quality-diversity curve.This enables comparisons at desired quality or diversity levels and can support heuristic-based early stopping.
  • Scope and bias: Changing conditional temperatures is straightforward but differs from changing the joint-distribution entropy, introducing a bias toward lower entropy at sentence beginnings.Joint-distribution temperature changes become intractable with vocabulary size or sequence length.
  • Alternative methods: Stochastic beam search and generator rejection sampling offer less-biased ways to navigate quality and diversity, but they are computationally expensive.Temperature tuning is retained for experiments because it provides smooth and computationally efficient control.

4 RELATED WORK

Related work found no evidence that GAN-trained models outperform language models on improved global metrics. This paper extends that comparison by evaluating models across temperature sweeps.

  • Prior evaluations: Prior empirical work found no evidence that GAN-trained models outperform language models on improved global quality metrics.The paper’s temperature-sweep analysis further examines this comparison across the quality-diversity space.
  • Temperature-related work: The paper disputes the claim that increasing training-time temperature necessarily produces more diverse test samples.It argues that the model can adapt to the temperature change, lowering test-time entropy instead.

5 EMPIRICAL RESULTS

Across synthetic and long-text experiments, MLE models consistently outperform GANs across the quality-diversity space. Temperature tuning offers an efficient evaluation and decoding baseline, while less biased alternatives provide limited gains at substantially higher computational cost.

  • Empirical results: MLE-trained models reliably outperform textual GANs across synthetic and long-text generation in the quality-diversity space.The experiments compare MLE against several RL and non-RL GAN variants using temperature tuning.
  • Experimental details: 300 hyperparameter-search trials were used for the authors’ language GAN, while the matched MLE model required only one-sixth as many trials.The study also reports that official implementations were often prone to extreme mode collapse, complicating reproduction.
  • Synthetic data experiment: On the synthetic task, reduced-temperature MLE achieved the highest quality according to NLLoracle, although quality-only evaluation can reward repeatedly producing a single sequence.This illustrates why the synthetic results are interpreted jointly through quality and diversity rather than quality alone.
  • Long-text generation: On EMNLP News 2017, MLE under a temperature sweep achieved a better quality-diversity trade-off than all considered RL GANs and FM-GAN across local and global metrics.The conclusion held for Negative BLEU, Self-BLEU, Language Model score, and Reverse Language Model score.
  • Empirical remarks: Less biased decoding methods offered small trade-off gains, but beam search was discrete and generator rejection sampling became exponentially slower as its threshold increased.Temperature tuning remained useful for efficient assessment because it provides smooth control over the quality-diversity trade-off.
  • Empirical remarks: Adversarial training caused a drastic entropy reduction immediately after GAN updates began.The paper identifies this entropy change as a potential reason GAN outputs may appear high-quality despite poorer quality-diversity performance.

6 DISCUSSION

The paper argues that well-adjusted language models remain strong baselines and that temperature sweeping clarifies quality-diversity comparisons, while GAN-based methods face computational and optimization challenges.

  • Temperature sweeping characterizes model performance across quality and diversity, analogous to a receiver operating curve.A single scalar can be obtained from the curve’s area, with lower values preferred for the considered metrics.
  • Well-adjusted language models outperform the considered GAN variants on both local and global quality-diversity metrics.
  • GANs face difficulties on discrete data, and the considered tasks suggest GAN optimization issues were more severe than exposure bias.
  • Stochastic beam search modulates quality-diversity through beam size but cannot smoothly trade quality for diversity like temperature tuning.Infinite beam size samples the most likely sentences, whereas temperature 0 corresponds to greedy decoding.
  • Generator rejection sampling controls quality-diversity with a likelihood threshold but becomes computationally expensive as the threshold increases.

C QUALITATIVE ANALYSIS OF THE SAMPLES

Qualitative samples from SeqGAN, LeakGAN, and MLE appear similar on Image COCO, while News samples show examples of generation beyond observed training combinations.

  • Samples from SeqGAN, LeakGAN, and MLE appear similar on the Image COCO captioning task.The paper attributes this similarity to caption generation being relatively easier.
  • News samples include “post Brexit strategy,” an n-gram absent from the training set.
  • A News sample combines “leak” with “Freedom of Information request,” although that pairing never appears in training data.These examples are presented as evidence of some generalization beyond the training set.

D FULL BLEU AND SELF-BLEU RESULTS

The section reports that complete BLEU and Self-BLEU results are provided in Table 4.

  • Full BLEU and Self-BLEU results are shown in Table 4.

E THE LIMITATIONS OF BLEU

The paper identifies a limitation of BLEU as a quality proxy: matching the training data’s score can still yield text lacking semantic or global coherence.

  • BLEU-matched MLE samples can be grammatical yet lack semantic and global coherence.
  • Consistently strong samples required reducing temperature until BLEU-5 was twice the training data’s value.
  • Sample comparisons include Image COCO and EMNLP2017 WMT News, with MLE temperatures reduced to match reported BLEU scores.
  • Tables report BLEU and Self-BLEU results for EMNLPNEWS 2017 and Image COCO, with higher BLEU and lower Self-BLEU preferred.

F ISSUES OF VARYING TEMPERATURE DURING TRAINING

Training-time temperature changes can reduce diversity because models adapt to the altered temperature, while inference-time controls and noise-variance adjustments provide ways to navigate quality and diversity. Additional experiments examine discriminator-based rejection sampling and decoding mechanisms.

  • Temperature during training: Training-time temperature adaptation produced a colder inference temperature and reduced diversity, so the authors recommend adjusting temperature only at inference.The authors suggest using other techniques to facilitate exploration during training.
  • Temperature during training: Figure 7 compares negative BLEU-5 with SBLEU5 across training temperatures and reports mode collapse when training temperature is increased.The figure uses dot redness to indicate α, representing greater pressure to increase entropy.
  • Quality-diversity modulation: The experiments also compare discriminator rejection sampling, quality-diversity modulation strategies in FM-GAN, and MLE against RL-GAN under different decoding mechanisms.Discriminator rejection sampling accepts or rejects generator samples using a discriminator-derived realness score and threshold.
  • Quality-diversity modulation: FM-GAN retains stochasticity at temperature 0 because it samples noise before generating the first word and concatenates that noise to every token.The sampled noise is z ∼N(0, 1).
  • Quality-diversity modulation: Reducing noise variance was additionally explored as a quality-modulating tool because temperature tuning did not cover enough of FM-GAN’s quality-diversity space.

J EMNLP 2017 NEWS SAMPLES

The EMNLP 2017 News samples section presents examples from the authors’ model and samples from SeqGAN and LeakGAN for comparison. It also includes MLE samples selected to match BLEU scores reported by Guo et al. (2017).

  • The section provides additional samples for the EMNLP 2017 News dataset.
  • Samples from SeqGAN are included using examples taken from Guo et al. (2017).
  • Samples from LeakGAN are included using examples taken from Guo et al. (2017).
  • The authors provide MLE samples with temperature chosen to match BLEU scores reported in Guo et al. (2017).
Loading 1811.02549v6…