PaperlayerView exact revision on arXiv

Source-linked AI summary

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu

arXiv:1609.05473v6cs.LGcs.AI

TL;DR

GANs work well for real-valued data but face gradient and credit-assignment difficulties when generating discrete-token sequences. SeqGAN treats the generator as a stochastic reinforcement-learning policy, uses discriminator rewards with Monte Carlo rollouts, and reports significant improvements over strong baselines on synthetic and real-world tasks. Its experiments also show that training stability depends on strategy parameters, while the reported experiments use fixed-length generated sequences.

  • Problem

    GANs have difficulty training discrete-token generators because discriminator gradients do not pass usefully through discrete outputs, and complete-sequence rewards are hard to assign to partial sequences.

  • Method

    SeqGAN models sequence generation as reinforcement-learning policy optimization, using a discriminator to score complete sequences and Monte Carlo rollouts to estimate intermediate state-action values.

  • Results

    SeqGAN significantly outperforms maximum-likelihood, scheduled-sampling, and PG-BLEU methods on synthetic data and compared baselines across poem, speech-language, and music generation tasks.

  • Takeaways & Limitations

    Adversarial training with policy gradients is presented as a viable approach for discrete sequence generation, with discriminator feedback reported as more general and effective than predefined BLEU guidance in synthetic experiments.

  • Takeaways & Limitations

    The generated sequences in the reported work have fixed length T, although the discriminator architecture can also support variable-length sequences with max-over-time pooling.

Abstract

from arXiv · show

As a new way of training generative models, Generative Adversarial Nets (GAN) that uses a discriminative model to guide the training of the generative model has enjoyed considerable success in generating real-valued data. However, it has limitations when the goal is for generating sequences of discrete tokens. A major reason lies in that the discrete outputs from the generative model make it difficult to pass the gradient update from the discriminative model to the generative model. Also, the discriminative model can only assess a complete sequence, while for a partially generated sequence, it is non-trivial to balance its current score and the future one once the entire sequence has been generated. In this paper, we propose a sequence generation framework, called SeqGAN, to solve the problems. Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the intermediate state-action steps using Monte Carlo search. Extensive experiments on synthetic data and real-world tasks demonstrate significant improvements over strong baselines.

Introduction

Sequence generation methods face a training–inference discrepancy and GAN-specific obstacles with discrete tokens and partial sequences. SeqGAN addresses these issues by treating generation as reinforcement-learning policy optimization and reports gains over strong baselines across synthetic and real-world tasks.

  • Motivation: Maximum-likelihood RNN training can suffer exposure bias because inference conditions on previously predicted tokens that may never appear in training data.The discrepancy can accumulate as sequence length increases.
  • Motivation: Scheduled sampling partially replaces true prefixes with synthetic data during training, but it has been shown to be inconsistent and insufficient to fundamentally solve the discrepancy.
  • GAN limitations: GANs are difficult to apply to discrete sequences because discriminator gradients provide no meaningful slight-change guidance when outputs are tokens from a limited dictionary.
  • GAN limitations: A discriminator scores only complete sequences, making it non-trivial to balance the quality of a partial sequence against its future completed-sequence score.
  • SeqGAN approach: SeqGAN models generation as sequential decision making, uses discriminator evaluation as the reward, and applies Monte Carlo policy gradients to train the stochastic generator directly.This bypasses differentiation through discrete outputs and propagates complete-sequence feedback to intermediate actions.
  • Reported results: SeqGAN significantly outperforms maximum likelihood, scheduled sampling, and PG-BLEU on synthetic data and compared baselines across poem, speech-language, and music generation tasks.The real-world comparisons include various metrics, including human expert judgment.

Related Work

Prior work established GANs for generative modeling and RNNs for structured sequences, but GAN progress on discrete sequence generation remained limited. Reinforcement-learning formulations supplied a route to optimize sequence generators when rewards are available for complete sequences.

  • Generative modeling: Deep generative-model research includes contrastive divergence for deep belief nets and denoising autoencoders that learn low-dimensional representations before decoding.
  • Adversarial training: GANs introduced minimax training between generative and discriminative models and achieved striking success in natural-image generation.
  • Open problem: GANs remained difficult for discrete sequence generation because their generators are designed to adjust outputs continuously.
  • Sequence generation: RNNs produce structured token sequences in applications such as machine translation, commonly using token-level likelihood maximization.
  • Reinforcement-learning formulation: Sequence generation can be formulated as sequential decision making, allowing policy-gradient optimization when an implicit reward guides next-token selection.For many tasks, the meaningful reward is available only for the entire sequence.

Sequence Generative Adversarial Nets

SeqGAN formulates sequence generation as reinforcement learning, using a discriminator's complete-sequence reward to train a stochastic generator through policy gradients and Monte Carlo rollouts. The framework alternates generator and discriminator updates, with recurrent architectures providing the generator and discriminator components.

  • Policy formulation: SeqGAN models each generated prefix as a reinforcement-learning state and the next token as an action under stochastic policy Gθ.State transitions are deterministic after token selection.
  • Adversarial training: The discriminator Dφ distinguishes real sequences from generator-produced sequences and supplies the reward used to improve Gθ.Dφ(Y1:T) indicates how likely a complete sequence is to come from real data.
  • Policy optimization: Policy gradients directly update the generator to maximize expected end reward, using likelihood-ratio estimation over sampled intermediate states.The objective assumes zero intermediate rewards and deterministic state transitions.
  • Monte Carlo search: Because Dφ rewards only finished sequences, SeqGAN estimates intermediate action values by rolling out the remaining tokens with policy Gβ and averaging N sampled completions.The rollout policy can match the generator or use a simplified version when speed is prioritized.
  • Training procedure: SeqGAN pre-trains Gθ with maximum likelihood, initializes a rollout policy, trains Dφ on real and generated samples, then alternates generator and discriminator updates.Discriminator retraining maintains balance by using equal numbers of positive and negative examples per update step.
  • Model architecture: The implementation uses recurrent neural networks, commonly LSTM-based, as generators and a CNN as the discriminator, while other RNN variants can also serve as generators.The generator output distribution is produced with a softmax layer over hidden states.

Synthetic Data Experiments

The synthetic-data experiment uses a randomly initialized LSTM as an oracle language model to generate the real data distribution, with fixed-length sequences as the experimental setting.

  • Synthetic data setup: Synthetic experiments model the real data distribution with a randomly initialized LSTM oracle that generates sequences token by token.The oracle defines p(x_t|x_1, . . . , x_{t−1}) for the experiments.
  • Synthetic data setup: The generated sequences have a fixed length T, although the CNN discriminator can also handle variable-length sequences with max-over-time pooling.The variable-length capability is noted as an architectural scope boundary.

2 Experiment code: https://github.com/LantaoYu/SeqGAN

The synthetic-data experiments evaluate SeqGAN against several generative baselines using oracle likelihood, convergence curves, and training-strategy analyses. SeqGAN improves over the baselines, but its stability depends strongly on how generator and discriminator updates are scheduled.

  • Evaluation: 100,000 generated test sequences are evaluated with NLLoracle, whose average score measures performance against the oracle generator.Statistical significance tests compare SeqGAN with the baselines.
  • Synthetic-data results: SeqGAN significantly outperforms maximum likelihood, scheduled sampling, and PG-BLEU on NLLoracle.A significance T-test reports improvement over all compared models.
  • Synthetic-data results: After about 150 epochs, maximum likelihood and scheduled sampling converge to relatively high NLLoracle scores, while SeqGAN improves the generator limit.The learning curves also show SeqGAN outperforming PG-BLEU, whose predefined BLEU signal is less general than the discriminator signal.
  • Training stability: SeqGAN stability depends on g-steps, d-steps, and k, which substantially affect convergence and performance.Training the generator too often can leave the discriminator undertrained and produce misleading signals.
  • Training stability: Training the generator for one epoch and updating the discriminator before it is fooled produces stable learning.Using additional negative-example sets can further improve performance through bootstrapping.

Real-world Scenarios

SeqGAN is evaluated on poem, political-speech, and music generation tasks using corpus-based and human-oriented comparisons. It significantly outperforms MLE in text generation and in both reported music metrics, while matching human poem data comparably.

  • Tasks: SeqGAN is tested on Chinese poem composition, Obama political speech generation, and music composition.The real-world experiments cover text and music sequence generation.
  • Text Generation: 16,394 Chinese quatrains and 11,092 Obama speech paragraphs provide the text-generation corpora.The poem corpus contains four-line quatrains, while BLEU measures similarity for generated speeches.
  • Text Generation: SeqGAN significantly outperforms MLE in text generation and performs comparably to real human data for poem composition.The results are reported in Tables 2 and 3.
  • Music Generation: The music task uses 695 Nottingham folk tunes represented as length-32 sequences over 88 piano pitches.Piano-key patterns are evaluated with BLEU, and continuous pitch patterns with MSE.
  • Music Generation: SeqGAN significantly outperforms MLE on both BLEU and MSE in music generation.This result is reported in Table 4.

Conclusion

SeqGAN trains generative adversarial networks for structured sequence generation through policy gradients and performs strongly across synthetic and real-world settings.

  • SeqGAN trains generative adversarial networks for structured sequence generation via policy gradient.
  • Synthetic experiments used oracle evaluation to demonstrate SeqGAN’s superiority over strong baselines.
  • SeqGAN showed excellent performance in poem, speech-language, and music generation, with robustness and stability also investigated.

Appendix

The appendix provides derivations, model implementation details, and an ablation study related to the synthetic-data experiments.

  • Appendix: Section 1 presents a step-by-step derivation of Eq. (6).
  • Appendix: Section 2 discusses detailed generative and discriminative model realizations, including parameter settings.
  • Appendix: Section 3 provides an ablation study supplementing the synthetic-data experiments.

Proof for Eq. (6)

The proof derives the generator objective gradient by exploiting deterministic sequence-state transitions and recursively expanding state and action values.

  • The derivation follows reinforcement-learning results to obtain the gradient of the objective with respect to generator parameters.
  • After an action is chosen, the sequence state transitions deterministically to the state containing the newly generated token.
  • The start-state value is the objective function J(θ), whose gradient is identified with the result in Eq. (6).

Model Implementations

SeqGAN implements sequence generation with recurrent neural networks and sequence discrimination with convolutional models, using token distributions, pooling, highway layers, and task-specific structures.

  • Generative model: The generator uses recurrent neural networks, with standard LSTM selected while GRU and soft-attention variants remain compatible.
  • Generative model: A softmax output layer maps hidden states to the next-token distribution p(y_t|x_1, . . . , x_t).
  • Generative model: LSTM gating addresses vanishing and exploding gradients associated with learning long-term dependencies in recurrent networks.
  • Discriminative model: The discriminator represents tokens as embeddings, applies convolutional kernels over word windows, and extracts features with nonlinear transformations.
  • Discriminative model: Max-over-time pooling and a fully connected softmax layer produce the probability that a sequence is real.
  • Discriminative model: Discriminator kernel window sizes and numbers are selected empirically, and task-specific structures are recommended.

More Ablation Study

The ablation examines how the amount of supervised MLE pre-training affects SeqGAN before adversarial training. Insufficient pre-training makes adversarial improvement slow and unstable, indicating that sufficient pre-training is necessary.

  • More Ablation Study: 20 pre-training epochs, far from convergence, lead to slow and unstable improvement during adversarial training.Figure 4 evaluates negative log-likelihood across different pre-training durations before adversarial training.
  • More Ablation Study: When the generator acts almost randomly, the discriminator assigns nearly every action a low, unified reward.This reward provides little guidance for improving the generator.
  • More Ablation Study: The results indicate that sufficient supervised pre-training is necessary before applying adversarial training to sequence generative models.
Loading 1609.05473v6…