Source-linked AI summary
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, Yoshua Bengio
TL;DR
GANs have limited success on discrete data because standard back-propagation cannot traverse discrete variables and discriminator-based reinforcement learning is unstable. MaliGAN augments maximum likelihood with a normalized, importance-sampling-based objective derived from discriminator output. The paper reports theoretical consistency, lower-variance optimization, and successful, stable modeling of discrete sequences across experiments.
Problem
GANs have not achieved equivalent success on discrete domains because discrete credit assignment and unstable reinforcement-learning algorithms make discriminator information difficult to pass to the generator.
Method
MaliGAN starts from KL(p_d||pθ), combines importance sampling with discriminator output, and uses normalization and variance-reduction techniques to optimize the resulting objective.
Results
MaliGAN is theoretically consistent and successfully and stably models discrete data sequences, with an objective that has much less variance than vanilla discriminator-reward reinforcement learning.
Takeaways & Limitations
The normalized target lets the model maximize good behaviors while minimizing bad behaviors using discriminator knowledge, and the approach is reported as robust to overfitting.
Takeaways & Limitations
The moving-reward reinforcement-learning approach remains unstable and does not work on large-scale datasets, even with dedicated pre-training and variance reduction.
Abstract
from arXiv · showhide
Despite the successes in capturing continuous distributions, the application of generative adversarial networks (GANs) to discrete settings, like natural language tasks, is rather restricted. The fundamental reason is the difficulty of back-propagation through discrete random variables combined with the inherent instability of the GAN training objective. To address these problems, we propose Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. Instead of directly optimizing the GAN objective, we derive a novel and low-variance objective using the discriminator's output that follows corresponds to the log-likelihood. Compared with the original, the new objective is proved to be consistent in theory and beneficial in practice. The experimental results on various discrete datasets demonstrate the effectiveness of the proposed approach.
1. Introduction
Discrete GANs face two linked obstacles: standard back-propagation cannot pass through discrete variables, while reinforcement-learning training with discriminator rewards is unstable. MaliGAN addresses these issues with a normalized, importance-sampling-based objective that augments maximum likelihood.
- Background: Teacher forcing exposes autoregressive models to ground-truth histories rather than their own predictions, causing exposure bias.This limits the model’s experience of its generated sequences during training.
- Alternative: GANs offer an alternative by letting a discriminator distinguish real from generated sequences and provide training information to the generator.This setup can expose autoregressive models to their own predictions and provide an overall view of generated sequences.
- Problem: Discrete sequence generation prevents standard back-propagation from directly updating the generator.Reinforcement learning can use discriminator output as a reward, but the resulting policy may receive positive and stable signals only with difficulty.
- MaliGAN: MaliGAN replaces direct discriminator-reward optimization with a normalized maximum-likelihood target optimized using importance sampling and variance reduction.The objective is designed to sidestep instability associated with using discriminator output directly as a reinforcement-learning reward.
- MaliGAN: The proposed target is theoretically consistent and supports both increasing the likelihood of good behaviors and decreasing the likelihood of bad behaviors.The discriminator supplies information for excluding probable harmful behaviors alongside maximum-likelihood improvement.
2. Preliminaries and Overview
Discrete GAN training aims to match generated and real distributions, but discontinuity forces the generator to rely on unstable reinforcement-learning signals. The overview therefore combines discriminator information with maximum likelihood through importance sampling and normalization.
- Framework: The discrete sequence framework fits an autoregressive model pθ to data {x_i}^N_{i=1} from an underlying distribution p_d.The generator is trained together with a discriminator under the GAN setting.
- Framework: GAN training seeks to match the generated distribution p_g to the real distribution p_d using a generator and a discriminator.The generator transforms noise into samples, while the discriminator distinguishes real from generated data.
- Training challenge: Discrete discontinuity prevents standard back-propagation, so common alternatives use REINFORCE-like algorithms with D or log D as rewards.Additional variance-reduction techniques may be used in these reinforcement-learning approaches.
- Traditional objective: The traditional generator objective maximizes E_x∼pθ[log D(x)], while its equivalent KL formulation uses a normalized distribution q′ that moves with D.The moving target is undesirable for stability and convergence.
- Traditional objective: On large-scale datasets, moving-reward reinforcement learning remains unstable because initially poor generators rarely produce good sequences and therefore receive positive rewards infrequently.Pre-training and variance-reduction mechanisms help but do not eliminate this problem.
- Proposed objective: The proposed objective adds discriminator information to maximum likelihood and uses importance sampling to make the target trainable with lower variance than vanilla discriminator-reward reinforcement learning.The detailed variance analysis is presented in Section 3.2.
3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
MaliGAN replaces the unstable direct GAN reward for discrete generation with a normalized maximum-likelihood objective optimized through importance sampling and variance reduction. The objective is theoretically sound and supports more stable training signals, including refinements for long sequences and mode missing.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: MaliGAN replaces the standard GAN generator objective with an importance-sampling-based objective closer to maximum-likelihood training, reducing gradient variance.The discriminator remains standard, while the generator uses a novel objective designed for more stable optimization.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: The target distribution q is approximately the data-generating distribution p_d when the discriminator is sufficiently trained, making it effectively fixed during training.The generator gradient is estimated with importance sampling from the current or delayed generator.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: The objective approximately optimizes KL(q(x)||p_θ(x)) when the discriminator is close to optimal and remains meaningful when it is well trained but not optimal.The theory also provides conditions under which the generator is pushed in a descent direction despite discriminator suboptimality.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: Mini-batch normalization adjusts generated paths by relative discriminator quality rather than absolute reward, helping provide a learning signal when discriminator rewards are otherwise sparse.With baseline b = 1, the generated-path weights sum to zero; the normalization is also argued to reduce practical variance.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: Monte Carlo tree search assigns different weights across generation steps and stabilizes partition-function estimation, reducing variance for long sequence generation.The full procedure combines the renormalized objective with additional variance-reduction techniques.
- 3. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks: Single real-data-based renormalization trains on simpler conditional distributions and provides each real prefix with a moderately strong signal, improving stability and robustness to mode missing.For a real prefix S≤N, the conditional completion distribution may contain only one or a few modes, while batch-wise normalization can let high-weight completions dominate.
4. Experiments
Across discrete image generation, poetry generation, and sentence-level language modeling, MaliGAN produced stable or stronger results than comparison models and showed robustness to overfitting.
- MaliGAN achieved promising results across three discrete sequence-generation tasks, including the challenging sentence-level language-modeling task.The experiments also examined robustness to overfitting.
- 4.1. Discrete MNIST: MaliGAN samples had much better visual quality and more closely resembled the training data than REINFORCE-like samples on Discrete MNIST.The comparison used generators with the same network architecture, differing in how discriminator output was used.
- 4.2. Poem Generation: MaliGAN-full obtained the best BLEU-2 scores on both poetry tasks, followed by MaliGAN-basic, while MLE lagged far behind.Perplexity produced qualitatively similar results.
- 4.2. Poem Generation: Both MaliGAN variants achieved lower perplexity on Poem-7 and prevented overfitting there when MLE overfit.MaliGAN-full also had less variance than MaliGAN-basic, indicating the advanced variance-reduction techniques stabilized training.
- 4.3. Sentence-Level Language Modeling: Both MaliGAN-basic and MaliGAN-full obtained notably lower sentence-level perplexity than MLE on the PTB dataset.The authors report that MaliGAN was more robust to overfitting despite PTB’s relative data scarcity.
- 4.3. Sentence-Level Language Modeling: MaliGAN-full trained more stably and could achieve lower perplexity than MaliGAN-basic when advanced variance-reduction techniques were included.This result was discussed alongside the perplexity curves and ablation tests.
5. Related Work
Related work addresses exposure bias, loss-evaluation mismatch, and discrete GAN training, while MaliGAN’s approach is distinguished by its treatment of sequence-level objectives through importance sampling.
- Exposure bias arises because autoregressive models are not exposed to their own predictions during training.The training algorithm instead uses demonstrated behaviors from real data samples.
- Loss-evaluation mismatch occurs when word-level training probabilities differ from sequence-level test metrics such as BLEU.One direct response is to incorporate evaluation metrics into the training objective.
- GANs can expose generative models to their own predictions through discriminator feedback, but discrete discontinuity and GAN instability make training difficult.Prior work addressed discriminator rewards using techniques including Monte Carlo tree search.
- Bornschein and Bengio proposed mini-batch reweighting for discrete latent-variable models, but their inference-network requirement is infeasible in the GAN setting.
- Unlike conditional-generation metrics that decompose across time steps, sequence-level GAN objectives cannot directly sample augmented distributions and therefore require importance sampling.This distinction connects MaliGAN to the objective studied by Norouzi et al.
6. Discussions and Future Work
The paper reports that MaliGAN reduces reinforcement-learning variance and is robust to overfitting, while leaving larger datasets and conditional generation for future work.
- MaliGAN successfully reduces the variance of standard reinforcement-learning algorithms.
- MaliGAN is reported to be surprisingly robust to overfitting.Teacher forcing can fit noise in training data, whereas the discriminator can capture differences when generated samples do not look good.
- Future work will evaluate MaliGAN on Google’s one billion words dataset and on conditional generation tasks such as dialogue generation.