PaperlayerView exact revision on arXiv

Source-linked AI summary

On Accurate Evaluation of GANs for Language Generation

Stanislau Semeniuta, Aliaksei Severyn, Sylvain Gelly

arXiv:1806.04936v3cs.CL

TL;DR

Text GAN evaluations often rely on n-gram metrics and single best runs, although these may miss quality changes and model sensitivity. The paper proposes alternative metrics and a systematic comparison protocol, then finds that properly tuned conventional language models outperform the considered GAN systems.

  • Problem

    Text-GAN evaluations commonly use n-gram metrics and single best scores, providing limited evidence about model quality and sensitivity to seeds or hyperparameters.

    Source: §2 · p. 2, §1 · p. 2

  • Method

    The paper evaluates text GANs with alternative metrics for quality and diversity, a multi-run and hyperparameter comparison protocol, and comparisons with conventional language models.

    Source: §1 · p. 2

  • Results

    Properly tuned conventional language models yield better results than any considered GAN-based system, while FD detects deteriorations that BLEU misses.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that BLEU alone and single-best reporting are insufficient for evaluating text GANs, and that systematic metrics and comparisons give a better picture of model behavior.

    Source: §6 · p. 10

    The experiments are not exhaustive, and the authors note that a well-behaved language GAN may not have been included in their search.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.

1 Introduction

The introduction argues that evaluating text GANs requires metrics and comparisons that reflect more than a single best score. It motivates systematic evaluation and reports that carefully tuned language models perform at least as well as the tested GANs.

1 Introduction

  • The introduction identifies exposure bias and mismatch between next-word NLL training and task-specific metrics as problems for neural text generation.Exposure bias arises because training uses ground-truth histories, whereas inference conditions on the model's own predictions.Source: §1 · p. 1
  • Previously used n-gram metrics such as BLEU can be insufficient or misleading, while single best-run reporting obscures sensitivity to random seeds and hyperparameters.The authors say these reporting choices introduce bias that can prevent informed model choices.Source: §1 · p. 2
  • The paper aims to compare text generators fairly with meaningful metrics, multiple runs and hyperparameter choices, and experiments across GANs and conventional language models.The authors note that GANs are difficult to compare with language models because perplexity disadvantages GANs and n-gram statistics can mislead.Source: §1 · p. 2
  • A conventional neural Language Model performs at least as well as any tested GAN, and hyperparameter search consistently finds adversarial learning hurts performance.The authors argue that reporting multiple metrics is important to support this comparison.Source: §1 · p. 2

2 Related Work

Prior evaluations of text GANs rely heavily on n-gram metrics and single best results, limiting what can be concluded from comparisons. This paper introduces alternative metrics and a more systematic evaluation approach.

2 Related Work

  • Prior text-GAN studies use n-gram metrics such as BLEU and self-BLEU for quality and diversity, and often report only one best score despite sensitivity to seeds and hyperparameters.The paper argues that inadequate metrics and tuning can make comparisons difficult to interpret.Source: §2 · p. 2
  • A recent comparison of text GANs [21] also used n-gram metrics and a single best result, making convincing conclusions difficult.The authors contrast this with an image-GAN study [22] that found careful tuning and computational budget can substantially affect comparisons.Source: §2 · pp. 2–3
  • This work proposes alternatives to BLEU and self-BLEU that detect failure modes those metrics miss, alongside extensive tuning and an initial comparison of recent text GANs.The paper presents this as its response to limitations in the accepted evaluation approach.Source: §2 · p. 3

3 Models

The paper benchmarks core continuous and discrete GAN techniques for text on equal footing by separating architectural choices from other training dimensions. Its model set includes continuous generators and discrete reinforcement-learning variants.

3 Models

  • The benchmark decouples core GAN techniques from their original models to compare them on equal footing, grouping them as continuous or discrete.The authors describe the models as differing substantially, motivating a component-focused benchmark.Source: §3 · p. 3

3.1 Continuous models

  • Continuous models treat token sequences as one-dimensional signals and backpropagate discriminator gradients into the generator; the paper considers convolutional and RNN-based generators.For the RNN generator, the full softmax output becomes the next input so gradients remain differentiable through the sequence.Source: §3.1 · p. 3

3.2 Discrete models

  • Discrete generators sample or take argmax from next-token distributions, blocking discriminator gradients, so reinforcement learning uses discriminator scores as rewards.The paper notes additional RL challenges including sparse rewards, credit assignment, and large action spaces.Source: §3.2 · p. 3
  • SeqGAN variants address credit assignment through full-sequence REINFORCE, per-step discriminator rewards, or rollout-based rewards for incomplete sequences.The per-step variant uses a discounted return; the rollout variant continues partial sequences to completion before scoring them.Source: §3.2 · p. 4
  • The LeakGAN variants differ in which hidden states the generator receives, and the simplified setup freezes discriminator weights during generator updates.Leak, noleak, and mixed expose discriminator state, generator state, or both; the simplifications isolate architectural effects from complicated RL and interleaved NLL training.Source: §3.2 · p. 4, Figure 1 · p. 4

4 Methodology

The section evaluates automatic metrics for unsupervised text generation and sets out a repeated-run tuning protocol to measure GAN sensitivity.

4.1 Metrics

  • Figure 2 indicates that neither BLEU nor self-BLEU captures semantic deterioration; BLEU is higher-is-better, while the other metrics are lower-is-better.FD values computed with UniSent embeddings are scaled by 10 for visualization.Source: Figure 2 · p. 5
  • Human ratings assess grammaticality and understandability, but one-sample-at-a-time judgments measure precision rather than recall and are expensive for tuning.The study sends 200 samples per model to three raters each, using a 1–5 scale.Source: §4.1 · pp. 5–6
  • The authors seek a fast metric that captures precision and recall or distribution distance, and assess BLEU, text FD, reverse-LM scores, and pretrained-LM scores.Source: §4.1 · p. 5
  • BLEU and unique n-gram counts offer some insight but miss semantic variation and detect only relatively simple syntax problems.The study uses BLEU4, having found similar results across n-gram sizes.Source: §4.1 · pp. 4–5
  • A pretrained Language Model’s likelihood for generated samples serves as a quality-only metric and proxy for precision, but rewards repeatedly generating a few likely sentences.Source: §4.1 · p. 5
  • A reverse Language Model is trained on generated samples and evaluated on held-out real text, but its score is biased by model imperfections and finite training data.Source: §4.1 · p. 5
  • Frechet Distance (FD) adapts the feature-distribution distance to text using InferSent embeddings, whose quality may affect the metric.The authors note that sequence-embedding research is ongoing.Source: §4.1 · p. 6

4.2 Parameter optimization procedure

  • With a 100-trial random search, the authors retrain the selected configuration seven times and report each metric’s mean and standard deviation.They note that a fortunate random seed often produces the best result in a run.Source: §4.2 · p. 6

5 Experiments

The experiments test whether FD, Language Model, and BLEU scores capture sample variation, then compare GAN text models with a conventional Language Model. Across the reported metrics, no GAN model is convincingly better than the Language Model.

5 Experiments

  • Experiments use SNLI for model comparison and MultiNLI for metric evaluation, with MultiNLI topics enabling mode-collapse emulation.The SNLI training data comprises 600,000 unique sentences after labels are discarded; preprocessing uses a 4k-vocabulary SentencePiece model.Source: §5 · p. 6

5.1 Metric Evaluation

  • When topics are removed from samples, BLEU fails to capture semantic variation, whereas FD and reversed LM scores rise as more topics are removed.FD results were comparable across three sequence-embedding models; self-BLEU did not detect this form of mode collapse.Source: §5.1 · p. 7
  • Under word dropout, FD worsens substantially while BLEU is not very sensitive; under word swapping, BLEU reacts more than FD.The authors attribute FD's difficulty with word swapping to possible position invariance from max-pooling in the bidirectional LSTM embedding model.Source: §5.1 · p. 7
  • FD and reverse LM scores track the ordering of three LSTM Language Models of different sizes, although neither detects a model that memorizes the training set.Reverse LM is generally more sensitive but prohibitively expensive for tuning, so the authors use FD for hyperparameter searches.Source: §5.1 · p. 7

5.2 GAN model comparison

  • No GAN model is convincingly better than the Language Model, and metric preferences diverge: FD favors the LM while LM scores and human evaluations favor SeqGAN-reinforce.The table reports BLEU4 scores only; the prose notes that SeqGAN-reinforce has higher reverse LM scores than the LM.Source: Table 2 · p. 8, §5.2 · p. 9
  • Hyperparameter search generally favors low generator learning rates, which the authors suggest may keep generator weights close to their Language Model initialization.Improved BLEU scores are interpreted as suggesting higher precision for GAN models.Source: §5.2 · pp. 9–10
  • BLEU scores differ by less than one point for Conv-LSTM GAN and Language Model samples, despite qualitative differences in spelling, syntax, and coherence.Human evaluation supports FD and reverse LM scores and assigns better scores to the Language Model.Source: §5.2 · p. 10

6 Conclusions

BLEU is insufficient for evaluating textual GANs, while FD and reverse LM scores detect deteriorations that BLEU misses. A systematic evaluation finds tuned conventional LMs outperform the considered GAN systems, but the comparison is not exhaustive.

6 Conclusions

  • FD and reverse LM scores detect textual deteriorations that BLEU scores are not sensitive to.The proposed protocol also provides a better picture of model behavior than reporting only the single best result.Source: §6 · p. 10
  • Properly tuned conventional Language Models yield better results than any of the considered GAN-based systems.With FD, the best results occur at the lowest generator learning rate after pre-training, corresponding to no GAN training.Source: §6 · p. 10
  • The experiments are not exhaustive, and a well-behaved language GAN may not have been included in the search.Future work includes a larger array of GAN models and comparisons for conditional tasks such as image captioning and machine translation.Source: §6 · p. 10

Related papers

Loading 1806.04936v3…