PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskGAN: Better Text Generation via Filling in the______

William Fedus, Ian Goodfellow, Andrew M. Dai

arXiv:1801.07736v3stat.MLcs.AIcs.LG

TL;DR

Teacher-forced and maximum-likelihood text generators can produce poor free-running samples, while discrete text makes standard GAN training difficult. MaskGAN uses an actor-critic conditional GAN to infill missing text from surrounding context, producing higher-quality samples in human evaluation despite higher ground-truth perplexity.

  • Problem

    Teacher forcing and validation perplexity do not reliably ensure high-quality generated text, especially when generation conditions on unseen sequences.

  • Method

    MaskGAN is an actor-critic conditional GAN that fills missing text while conditioning on true surrounding context.

  • Results

    MaskGAN samples were significantly better than a tuned MaskMLE model in human evaluation, despite much higher perplexity on the ground-truth test set.

  • Takeaways & Limitations

    Generated-sample quality can improve through GAN training even when validation perplexity is substantially worse.

  • Takeaways & Limitations

    GAN training had not achieved state-of-the-art word-level validation perplexity on Penn Treebank, and hyperparameter search did not improve that result.

Abstract

from arXiv · show

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of the quality of the generated text. Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing. These methods are well-suited to optimizing perplexity but can result in poor sample quality since generating text requires conditioning on sequences of words that may have never been observed at training time. We propose to improve sample quality using Generative Adversarial Networks (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them. We claim that validation perplexity alone is not indicative of the quality of text generated by a model. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic conditional and unconditional text samples compared to a maximum likelihood trained model.

1 INTRODUCTION

The paper addresses poor free-running text generation by introducing MaskGAN, an adversarial model that fills missing text using surrounding context. It also highlights evaluation beyond validation perplexity and an actor-critic design for discrete language generation.

  • Teacher forcing trains RNNs on ground-truth prefixes, but generation can encounter unseen prefixes that cause unpredictable hidden-state dynamics.
  • Discrete text makes standard GAN training difficult because discriminator gradients cannot be propagated through sampling operations.
  • Autoregressive text GANs also face mode dropping and training instability, with discriminator feedback arriving only after complete sentences.
  • MaskGAN trains by deleting text portions and infilling them while conditioning on the true surrounding context.
  • Token-level discriminator evaluation provides the generator with fine-grained supervision for individual generated words.
  • The paper introduces MaskGAN, considers actor-critic architectures for extremely large action spaces, and calls for evaluation metrics beyond perplexity.

2 RELATED WORK

Prior work adapts GANs and reinforcement learning to discrete language generation through policy gradients, alternative objectives, continuous relaxations, and task-specific rewards. MaskGAN instead combines actor-critic training with text infilling to provide per-step rewards and address mode collapse and gradient variance.

  • Discrete language GANs use policy gradients, continuous reformulations, or reinforcement learning because sampling prevents ordinary discriminator backpropagation.
  • SeqGAN uses policy gradients and Monte Carlo rollouts to train a language-model generator against a CNN-based discriminator.
  • Professor Forcing discriminates hidden states from real- and synthetic-conditioned runs so inference-time dynamics follow training-time dynamics.
  • Other approaches report dialogue improvements with REINFORCE, replace discrete sampling with gradient approximators, or operate on continuous generator outputs.
  • MaskGAN differs by combining actor-critic training with an infilling task that supplies rewards at every time step.

3 MASKGAN

MaskGAN uses a seq2seq generator to infill masked tokens from surrounding context and an actor-critic GAN objective to train realistic completions. The discriminator evaluates token realism given the true masked context, while policy gradients train the discrete generator.

  • 3.2 ARCHITECTURE: The generator encodes a masked sequence and autoregressively decodes missing tokens conditioned on the masked text and previously filled tokens.The encoder supplies future context through its hidden states, enabling infilling rather than left-to-right generation alone.
  • 3.2 ARCHITECTURE: The discriminator outputs a realness probability for each filled token while receiving the original known context.Providing the true context prevents ambiguity about which token is fake and supplies a more accurate learning signal.
  • 3.3 TRAINING: The critic estimates the discounted total return of the filled sequence as an additional head on the discriminator.The discriminator’s logarithmic estimates serve as rewards for generator training.
  • 3.3 TRAINING: Because sampling makes the generator nondifferentiable, MaskGAN trains it with policy gradients using the critic’s value estimate as a variance-reducing baseline.The generator follows an actor-critic formulation in which token choices are actions and the baseline is the critic.
  • 3.3 TRAINING: Per-time-step rewards provide fine-grained credit assignment, while each token’s gradient depends on discounted future rewards.The full-sequence gradient sums contributions from all generated tokens; dynamic sequence lengths and distribution-level rewards address longer sequences and large vocabularies.
  • 3.3 TRAINING: Training begins by pretraining a maximum-likelihood language model and then a seq2seq model on the infilling task.The pretrained weights initialize the seq2seq encoder and decoder, and masked-task validation perplexity selects the model.

4 EVALUATION

The evaluation considers heuristic measures intended to correlate with human judgments, while questioning validation perplexity as a sufficient indicator of generated-text quality. Preliminary GAN configurations slightly reduced perplexity but did not substantially improve samples, so the authors prioritized sample quality instead.

  • 4 EVALUATION: Evaluation uses heuristic metrics intended to correlate with human evaluation, including validation-corpus coverage of unique generated n-grams.The metric is motivated by BLEU and examines small n-grams produced by the generator.
  • 4 EVALUATION: Small decreases in validation perplexity on O(1)−point did not yield considerable improvements in sample quality for the tested GAN configurations.The authors therefore abandoned attempts to optimize validation perplexity further.

5 EXPERIMENTS

Experiments evaluate MaskGAN against MaskMLE on PTB and IMDB using conditional and unconditional samples, perplexity, diversity, and human judgments. MaskGAN is more robust to free-running sampling errors and receives stronger human preference on IMDB, although it shows some mode collapse and does not achieve state-of-the-art PTB perplexity.

  • Experimental setup: MaskGAN generators are initialized from pretrained language models and further pretrained on text infilling with half the text masked.The PTB and IMDB pretraining stages reach validation perplexities of 55.3 and 87.1, respectively, on the masked task.
  • Perplexity of generated samples: GAN training still does not achieve state-of-the-art word-level validation perplexity on PTB, where maximum-likelihood models remain strongest.The experiments therefore emphasize sample quality rather than reducing validation perplexity.
  • Perplexity of generated samples: MaskGAN samples are more likely under the initializing language model than MaskMLE samples, whose inference errors compound under teacher-forced training.The authors interpret MaskGAN's free-running training as supporting robustness to sampling perturbations.
  • Mode collapse: MaskGAN shows some mode collapse through fewer unique quadgrams, although every complete generated sample remains unique.Sample quality remained relatively consistent even as ground-truth validation perplexity increased to 400 during reinforcement-learning training.
  • Human evaluation: Human evaluators judge MaskGAN's IMDB samples more human-looking than MaskMLE's, while the PTB comparison is closer and MaskGAN beats SeqGAN.The blind Mechanical Turk evaluation compares grammaticality, topicality, and overall quality; models have the same inference-time parameter count.

6 DISCUSSION

The discussion argues that matching training and inference through contiguous in-filling improves textual GAN samples, while attention and learned critics support conditioning and training. MaskGAN samples outperform MaskMLE in human evaluation despite higher test perplexity.

  • 6 DISCUSSION: The authors conclude that matching training and inference procedures supports higher-quality language samples.MaskGAN directly implements this matching through GAN training.
  • 6 DISCUSSION: Contiguous masking generally produced better samples than random masking by allowing longer free-running sequences during generation.The explanation is presented as a conjecture; random masks usually create shorter blanks to fill.
  • 6 DISCUSSION: Policy-gradient methods were effective when paired with a learned critic, while newer discrete-node methods might provide more stable training.This identifies a possible avenue for improving training stability rather than reporting a demonstrated result.
  • 6 DISCUSSION: Attention was important for conditioning infilled words on surrounding context; without it, locally reasonable subsequences became contextually implausible.The authors suggest attention-only GAN models as a further direction.
  • 6 DISCUSSION: MaskGAN improved generated samples over a tuned MaskMLE model on IMDB reviews according to human evaluation.The authors report this comparison on a larger dataset and note that MaskGAN had much higher ground-truth test perplexity.

A TRAINING DETAILS

The training details specify shared generator, discriminator, and critic representations, plus recurrent dropout and tuned optimization settings. Additional PTB samples are provided for inspection.

  • A TRAINING DETAILS: The model uses two 650-unit LSTM layers, 650-dimensional word embeddings, variational dropout, and Adam optimization.Variational dropout and learning rates were tuned with Bayesian hyperparameter optimization.
  • A TRAINING DETAILS: The generator and discriminator use shared embeddings, while the critic shares discriminator parameters except for its value-estimation output head.The generator also shares its embedding and softmax weights.
  • A TRAINING DETAILS: Additional samples on PTB are presented.The passage identifies the dataset but does not state a comparative outcome.

B.1.1 CONDITIONAL SAMPLES

The conditional-sample appendix presents ground-truth continuations alongside MaskGAN and MaskMLE outputs for contiguous and non-contiguous in-filling conditions.

  • B.1.1 CONDITIONAL SAMPLES: The contiguous conditional examples place Ground Truth, MaskGAN, and MaskMLE outputs alongside the same surrounding text.The PTB examples begin with the context “the next day’s show” and compare each model’s infilled continuation.
  • B.1.1 CONDITIONAL SAMPLES: The appendix also considers filling in non-contiguous masks.Examples include ground-truth text and outputs from MaskGAN and MaskMLE.

B.1.2 LANGUAGE MODEL (UNCONDITIONAL) SAMPLES

The appendix supplements the paper with unconditional PTB language-model samples, comparing MaskGAN with SeqGAN under matched generator architecture.

  • B.1.2 LANGUAGE MODEL (UNCONDITIONAL) SAMPLES: SeqGAN was modified to train and generate PTB samples using the same generator architecture size as MaskGAN.The appendix presents samples from both models for comparison.
  • B.1.2 LANGUAGE MODEL (UNCONDITIONAL) SAMPLES: Additional unconditional samples from MaskGAN and SeqGAN are presented for PTB.The passage lists model outputs but does not state a comparative evaluation outcome.
  • B.1.2 LANGUAGE MODEL (UNCONDITIONAL) SAMPLES: Additional samples on IMDB are provided.No comparison or evaluation result is stated in the supplied passage.

B.2.1 CONDITIONAL SAMPLES

The conditional and unconditional samples expose several MaskGAN failure modes, including mode collapse, syntactic boundary errors, and loss of global coherence.

  • C.1 MODE COLLAPSE: MaskGAN samples can repeat grammatical but inanely repetitive phrases across n-gram levels, indicating mode collapse.
  • C.1 MODE COLLAPSE: The generator may move between common text modes frequently, even when the discriminator identifies out-of-distribution samples.
  • C.2 MATCHING SYNTAX AT BOUNDARIES: MaskGAN often struggles to produce syntactically correct sequences at hard boundaries where filled text must end.
  • C.2 MATCHING SYNTAX AT BOUNDARIES: The intersection between filled and present text can be ungrammatical at the boundary.
  • C FAILURE MODES: Produced samples can lose global coherence despite being locally sensible, although larger model capacity may mitigate some issues.

C.4 n-GRAM METRICS MAY BE MISLEADING PROXIES

The paper argues that n-gram statistics are crude proxies for generated-text quality: improving a metric can coincide with low diversity, while language complexity remains insufficiently captured.

  • n-gram language statistics are monitored because training lacks a global scalar objective, but they are crude proxies for sample quality.
  • Figure 2 shows a failure mode that succeeds at optimizing a 4-gram metric while sacrificing validation perplexity.
  • A MaskGAN model can improve a particular n-gram metric at the extreme expense of validation perplexity while devolving to very low sample diversity.
  • Capturing the complexities of natural language with n-gram metrics alone is insufficient.
Loading 1801.07736v3…