Source-linked AI summary
How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
Ferenc Huszár
TL;DR
The paper examines which objectives should train and evaluate generative models, critiquing scheduled sampling and questioning maximum likelihood for realistic sample generation. It proposes KL[Q∥P] as an ideal objective and generalised Jensen-Shannon divergence as a tractable interpolation between it and maximum likelihood.
Problem
It remains unclear which objective functions should train and evaluate generative models, especially when evaluation emphasizes the perceptual quality of generated samples.
Method
The paper analyzes scheduled sampling and contrasts maximum likelihood's KL[P∥Q] objective with KL[Q∥P], then introduces generalized adversarial training based on Jensen-Shannon divergence.
Results
Maximum likelihood can overgeneralize and generate implausible samples, scheduled sampling is inconsistent, and generalized Jensen-Shannon divergence can interpolate between maximum likelihood and KL[Q∥P]-minimization.
Takeaways & Limitations
For realistic sample generation, the paper identifies KL[Q∥P] as an idealized objective and adversarial training as a more tractable approach toward it.
Takeaways & Limitations
Adversarial training is inefficient in high-dimensional spaces and remains unclear for discrete probabilistic models because sampling is not differentiable.
Abstract
from arXiv · showhide
Modern applications and progress in deep learning research have created renewed interest for generative models of text and of images. However, even today it is unclear what objective functions one should use to train and evaluate these models. In this paper we present two contributions. Firstly, we present a critique of scheduled sampling, a state-of-the-art training method that contributed to the winning entry to the MSCOCO image captioning benchmark in 2015. Here we show that despite this impressive empirical performance, the objective function underlying scheduled sampling is improper and leads to an inconsistent learning algorithm. Secondly, we revisit the problems that scheduled sampling was meant to address, and present an alternative interpretation. We argue that maximum likelihood is an inappropriate training objective when the end-goal is to generate natural-looking samples. We go on to derive an ideal objective function to use in this situation instead. We introduce a generalisation of adversarial training, and show how such method can interpolate between maximum likelihood training and our ideal training objective. To our knowledge this is the first theoretical analysis that explains why adversarial training tends to produce samples with higher perceived quality.
1 INTRODUCTION
The paper examines which objectives suit generative models intended to produce realistic samples. It critiques scheduled sampling and proposes adversarially approximated objectives that connect maximum likelihood with a perceptual alternative.
- Generative modeling has advanced, but many applications remain far from producing fully realistic samples, leaving training and evaluation objectives unresolved.
- The paper focuses on models whose purpose is generating realistic-looking samples, including captioning, texture generation, translation, and dialogue.
- Scheduled sampling is improper: Scheduled sampling is reformulated using Kullback-Leibler divergences and shown to be an improper, inconsistent training objective.
- KL-divergence as a model of perceptual loss: Maximum likelihood is argued to be inappropriate for realistic-sample generation because it minimizes KL[P∥Q] rather than the reverse-direction KL[Q∥P].
- Generalised adversarial training: Generalised Jensen-Shannon divergence can interpolate between maximum likelihood and KL[Q∥P], while generalized adversarial training can approximately minimize it.
2 AUTOREGRESSIVE MODELS FOR SEQUENCE GENERATION
Autoregressive generative models define sequence distributions recursively through tractable conditional components. Their tractable likelihood makes maximum-likelihood training feasible even for complex deep models.
- Autoregressive models explicitly define a sequence’s joint distribution recursively through conditional distributions over each next symbol.
- The component conditional distributions are assumed tractable and easy to compute, making model likelihood typically tractable.
- Tractable likelihood supports maximum-likelihood estimation for complicated deep architectures such as stacked LSTMs.
3 THE SYMPTOMS
Autoregressive models are trained on prefixes from real sequences but generate prefixes recursively, creating a train–generation mismatch. An unlikely generated prefix can trigger progressively poor-quality output.
- Training minimizes each next-symbol predictive likelihood conditioned on the preceding real prefix, a factorized form of maximum likelihood.
- At generation time, the model samples each next symbol from a conditional distribution and appends it to the recursively generated prefix.
- The RNN sees only real prefixes during training but may encounter unseen prefixes during generation, creating a train–generation mismatch.
- Once an unlikely prefix is generated, the model often struggles to recover, producing seemingly random symbols and samples with poor perceptual quality.
4 SYMPTOMATIC TREATMENT: SCHEDULED SAMPLING
Scheduled sampling replaces some real prefixes with model-generated prefixes during training, but its limiting objective is inconsistent and can favor position-based symbol distributions over sequence content.
- Scheduled sampling procedure: Scheduled sampling randomly replaces training-prefix symbols with model samples, then predicts the real next symbol using the possibly modified prefix.The replacement probability is controlled by ϵ, which is annealed from 1 to 0 during training.
- Objective formulation: The scheduled-sampling objective is a convex combination of the maximum-likelihood objective and an alternative divergence, evaluated in the infinite-data limit.When the first symbol remains original, the objective matches maximum likelihood; when it is replaced, the conditional prediction changes.
- Consistency analysis: Scheduled sampling is not a consistent estimation strategy because its divergence is not necessarily minimized when Q = P as ϵ approaches 0.Strictly proper divergences are minimized at the true distribution and guarantee consistency under sufficient data and model flexibility.
- Consequences: The analysis suggests scheduled sampling pushes models toward memorizing symbol distributions conditioned on position rather than preceding symbols.In an RNN, the corresponding optimal behavior uses hidden states primarily as a sequence-position counter and ignores prefix content.
- Consequences: Although scheduled sampling may make models more likely to recover from mistakes, the authors argue that it does not solve the limitations of maximum-likelihood training.The paper speculates that state-of-the-art image-captioning performance could result from incomplete optimization and interactions among training choices.
5 THE DIAGNOSIS
The paper diagnoses a mismatch between likelihood-based training and perceptual evaluation, then considers KL[Q∥P] as an ideal perceptual objective while identifying its practical support limitation.
- The diagnosis: The symptoms of maximum-likelihood-trained generative models arise from a mismatch between the likelihood training loss and perceptual sample-quality evaluation.The proposed remedy is an objective more closely aligned with the perceptual metric while retaining a consistent estimation framework.
- Perceptual objective: Perceptual evaluation asks whether samples appear naturally plausible to human observers, including responses that fit a conditioning context.In practice, researchers draw model samples and visually assess them, or present conditional outputs to human observers.
- Perceptual objective: The paper proposes KL[Q∥P] as an idealized objective for no-reference perceptual quality assessment, under assumptions about human subjective priors and natural-stimulus modeling.This reverses the roles of model and data distributions relative to the usual likelihood-based objective.
- Perceptual objective: Maximizing the perceptual objective alone would favor deterministic selection of the most likely stimulus, so entropy is added to enforce diversity.The resulting objective combines sample plausibility with diversity of the model distribution.
- Practical limitation: KL[Q∥P] is consistent when P = Q but is not a viable practical training objective when P is an empirical sample distribution and Q is smooth.The divergence requires P to be positive and bounded across the full support of Q.
- Divergence behavior: KL[P∥Q] tends to cover all modes while potentially generating unseen behaviors, whereas KL[Q∥P] and KL[P∥Q] differ under severe model misspecification, especially in high dimensions.The paper connects this behavior to maximum-likelihood recurrent models producing sequences never observed in real data.
6 GENERALISED ADVERSARIAL TRAINING
The paper introduces generalised Jensen-Shannon divergence as a tractable objective that interpolates between maximum-likelihood and mode-seeking behaviour, and shows how adversarial training can approximate it. Its parameter π controls this interpolation and changes the discriminator’s sampling balance.
- Objective motivation: KL[Q∥P] may better reflect perceptual sample quality than likelihood, but it is impractical as a training objective.The paper therefore seeks a more tractable alternative for realistic sample generation.
- Jensen-Shannon divergence: Adversarial training approximates Jensen-Shannon divergence, whose behaviour is more similar to KL[Q∥P] than KL[P∥Q] under model misspecification.The paper connects this mode-seeking behaviour with the appealing samples reported for adversarial methods.
- Generalised divergence: JSπ[P∥Q] combines KL divergences against the mixture πP + (1 − π)Q, providing a parameterised generalisation of Jensen-Shannon divergence.For non-trivial π, this generalisation is no longer symmetric in P and Q.
- Illustration: π = 0.1 produces moment matching, π = 0.5 recovers standard adversarial JS divergence, and π = 0.99 approaches mode-seeking behaviour.The figure uses a single isotropic Gaussian to approximate a multivariate Gaussian distribution.
- Interpolation: For small π, JSπ has the same optima as KL[P∥Q], while values near 1 recover the optima of KL[Q∥P].Thus, varying π interpolates between maximum-likelihood and mode-seeking behaviour.
- Adversarial implementation: Adversarial approximation interprets π as the ratio of labelled samples from Q and P received by the discriminator.π = 1/2 gives balanced classification; values below or above 1/2 overrepresent P or Q, respectively.
- Caveat: The proposed adversarial modification may adversely affect convergence properties, which the paper does not investigate.The claim concerns the flexibility of the adversarial procedure rather than an established convergence result.
7 CONCLUSIONS
The conclusions reject maximum likelihood and scheduled sampling for realistic sample generation, propose KL[Q∥P] as an ideal but impractical objective, and recommend generalised adversarial training as a more tractable alternative. The paper also identifies important practical limitations for adversarial methods.
- Conclusions: Maximum likelihood should not train models whose end goal is realistic sampling because it tends to overgeneralise and generate implausible samples.This conclusion is restricted to models created for drawing samples, excluding uses such as semi-supervised feature learning.
- Conclusions: Scheduled sampling fails to solve maximum likelihood’s fundamental problems and is an inconsistent training strategy.The conclusion concerns scheduled sampling as a training method for generative models.
- Conclusions: KL[Q∥P] is theorised as an ideal objective for no-reference perceptual quality assessment but is impractical in practice.The paper presents it as a conceptual target rather than a directly viable training procedure.
- Conclusions: Generalised Jensen-Shannon divergence is proposed as a more tractable objective interpolating between maximum likelihood and KL[Q∥P] minimisation.This interpolation is the paper’s proposed response to the gap between likelihood training and perceptual sample quality.
- Conclusions: The analysis suggests adversarial training strategies are the best choice for generative modelling and motivates a more flexible generalised-JS algorithm.This recommendation is presented as a theoretical conclusion of the paper’s analysis.
- Limitations: Adversarial training remains practically limited by inefficient high-dimensional sampling, hyperparameter sensitivity, and unclear applicability to discrete probabilistic models.The paper identifies addressing these limitations as future work.