PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Quantitative Analysis of Decoder-Based Generative Models

Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, Roger Grosse

arXiv:1611.04273v2cs.LG

TL;DR

Decoder-based generative models are difficult to evaluate because likelihood estimation is often intractable and samples can be misleading. The paper applies AIS and validates it with BDMC, finding substantially more accurate estimates than KDE and using them to compare models and study their behavior. These analyses show differences in likelihood, overfitting, and mode coverage that existing estimators and visual inspection miss.

  • Problem

    Rigorous evaluation is difficult because decoder-based models often have intractable log-likelihoods, while sample inspection and common estimators can be misleading.

  • Method

    The paper uses annealed importance sampling to estimate log-likelihoods and obtain approximate posterior samples, validating the estimates with bidirectional Monte Carlo.

  • Results

    AIS is two orders of magnitude more accurate than KDE for most evaluated models and reveals that VAEs score several hundred nats higher than GANs and GMMNs, while GANs miss important data modes.

  • Takeaways & Limitations

    Accurate likelihood estimation provides insight into decoder-based model behavior that KDE, importance-weighted bounds, and sample inspection cannot provide.

  • Takeaways & Limitations

    The GAN generative distribution uses a deterministic decoder with a Dirac delta observation model, which can make pointwise data likelihood ill-defined.

Abstract

from arXiv · show

The past several years have seen remarkable progress in generative models which produce convincing samples of images and other modalities. A shared component of many powerful generative models is a decoder network, a parametric deep neural net that defines a generative distribution. Examples include variational autoencoders, generative adversarial networks, and generative moment matching networks. Unfortunately, it can be difficult to quantify the performance of these models because of the intractability of log-likelihood estimation, and inspecting samples can be misleading. We propose to use Annealed Importance Sampling for evaluating log-likelihoods for decoder-based models and validate its accuracy using bidirectional Monte Carlo. The evaluation code is provided at https://github.com/tonywu95/eval_gen. Using this technique, we analyze the performance of decoder-based models, the effectiveness of existing log-likelihood estimators, the degree of overfitting, and the degree to which these models miss important modes of the data distribution.

1 INTRODUCTION

Decoder-based models can generate convincing samples, but evaluating them rigorously is difficult because likelihood estimation is often intractable and visual inspection can mislead. The paper uses AIS validated by BDMC to compare models and investigate estimator accuracy, overfitting, and missed modes.

  • Decoder-based models generate data by mapping latent variables from a simple fixed distribution through a learned deterministic decoder network.Examples include VAEs, GANs, GMMNs, and nonlinear independent components estimation.
  • Log-likelihood remains an important evaluation measure, but estimation is typically intractable unless the decoder is reversible.Visual sample inspection is labor-intensive and potentially misleading, while VAE encoders provide only a tractable lower bound.
  • KDE is widely used for GANs and GMMNs, yet high-dimensional density estimation can be inaccurate and produce misleading model comparisons.The KDE error can exceed meaningful log-likelihood differences between models.
  • AIS, validated with BDMC, estimates decoder-based model log-likelihoods accurately enough for fine-grained comparisons and is two orders of magnitude more accurate than KDE for most evaluated models.For VAEs, the recognition network can initialize AIS and make repeated estimation during training feasible.
  • VAEs achieve log-likelihoods several hundred nats higher than GANs and GMMNs, while GANs and GMMNs show smaller relative train-test gaps and are not simply memorizing training data.AIS posterior visualizations also reveal that GANs miss important modes represented in the training data.
  • Accurate likelihood estimation reveals model behavior that KDE, importance-weighted bounds, and sample inspection cannot measure.The authors argue that these measurements provide insight into decoder-based models and suggest directions for improving them.

2 BACKGROUND

Decoder-based models transform simple latent samples into data, but their likelihoods can be difficult or ill-defined to evaluate. AIS and BDMC provide a framework for estimating likelihoods and validating posterior approximations.

  • 2.1 Decoder-Based Generative Models: Decoder-based models generate data by transforming samples from a simple latent distribution through a learned decoder network.The paper considers VAEs, GANs, and GMMNs.
  • 2.1.1 Variational Autoencoder (VAE): VAEs define a joint distribution p(x,z)=p(x|z)p(z), with a Gaussian prior and decoder-parameterized observation model.The reparametrization trick reduces gradient-estimation variance.
  • 2.1.2 Generative Adversarial Network (GAN): GANs use a discriminator-based training game, but their deterministic decoder makes pointwise likelihoods ill-defined without additional assumptions.The resulting conditional distribution is a Dirac delta, so densities can be infinite or zero.
  • 2.1.3 Generative Moment Matching Network (GMMN): GMMNs train with maximum mean discrepancy and, like GANs, have undefined log-likelihoods.Kernel mean embeddings avoid unnecessary distributional assumptions in the training criterion.
  • 2.2 Annealed Importance Sampling: Likelihood weighting uses prior samples to estimate p(x), but fails when the broad prior differs substantially from the concentrated posterior.AIS instead bridges distributions through a sequence of similar intermediate targets and uses MCMC transitions that preserve each target.
  • 2.2 Annealed Importance Sampling: AIS variance tends to zero as the number of intermediate distributions increases, while BDMC combines forward and reverse bounds to assess likelihood accuracy and posterior KL divergence.The forward logarithmic estimate is a stochastic lower bound; reverse AIS supplies an upper bound when initialized from an exact posterior sample.

3 METHODOLOGY

The methodology evaluates test-example likelihoods under explicit generative distributions. For GANs and GMMNs, a fixed-variance Gaussian observation model makes likelihoods well-defined and enables AIS-based estimation.

  • 3 Methodology: The evaluation task is to measure log p(x_test) for test examples under a specified joint generative distribution.The joint is written as p(x,z)=p(z)p(x|z).
  • 3 Methodology: For VAEs, the observation model is Gaussian with mean and variance predicted by the decoder from the latent code.The prior is standard normal.
  • 3 Methodology: For GANs and GMMNs, the paper assumes a Gaussian observation model with fixed variance σ^2, denoted p_σ, to define likelihoods on the observable space.Without this assumption, their delta-function observation distributions cover only a submanifold.
  • 3 Methodology: KDE is equivalent to likelihood weighting for p_σ, so its prior-based estimate can be inaccurate when the evidence has low prior probability.This motivates using AIS for more accurate likelihood estimation under p_σ.
  • 3 Methodology: AIS is initialized mainly from the model prior, or from an encoder approximation q(z|x) when an encoder is available.For continuous data, the target is defined using the joint generative distribution.

4 RELATED WORK

Related work established important evaluation pitfalls and alternative measures for generative models. Prior studies also motivated examining mode proportions and human-aligned sample quality alongside likelihood.

  • 4 Related Work: AIS had previously been used to evaluate normalizing constants in deep belief networks, but this paper applies it to directed decoder-based models.The earlier models were undirected graphical models with a well-defined density formulation.
  • 4 Related Work: Prior analysis showed that a model can receive high likelihood while failing to model the distribution’s mode proportions.This motivates examining whether decoder-based models miss important modes.
  • 4 Related Work: Salimans et al. proposed an image-quality measure based on feeding generated samples to an Inception model for comparison with human visual judgment.The passage states that the measure was highly correlated with human visual judgment.

5 EXPERIMENTS

Experiments on MNIST evaluate AIS against existing likelihood estimators and compare decoder-based models under controlled architectures and observation models. AIS is validated with BDMC, while KDE and IWAE bounds show important accuracy limitations.

  • Experimental setup: Experiments use continuous and binarized MNIST, with decoder architectures of 10 or 50 latent dimensions and VAE, GAN, and GMMN training objectives.Models use a standard Normal prior; most evaluations use continuous inputs, while an additional IWAE experiment uses binarized MNIST.
  • Experimental setup: All models use a spherical Gaussian observation model with fixed variance, constraining VAEs relative to more flexible observation models.A more flexible diagonal Gaussian model achieved at least 2200 nats on continuous MNIST, so reported values should not be compared directly with models using that flexibility.
  • Estimator validation: For five of six networks, the BDMC gap was less than 1 nat, supporting the accuracy of AIS estimates on simulated data.Forward AIS supplies a lower bound and reverse AIS an upper bound; their gap bounds AIS error for model-simulated examples.
  • Estimator comparison: AIS estimates become more accurate than KDE within similar evaluation time, while KDE levels off even after orders of magnitude more samples.The comparison varies AIS intermediate distributions and KDE sample counts, using GMMN-10 as an illustration.
  • Estimator comparison: KDE accuracy declines sharply for small observation noise σ, biasing likelihood-based estimates toward larger σ values.The comparison uses 1 million simulated KDE samples and evaluates GAN-50 on MNIST training and validation examples.
  • Estimator comparison: On continuous MNIST, the IWAE bound underestimated true log-likelihoods by at least 33.2 nats on training data and 187.4 nats on test data.The result is more accurate than KDE but remains significant and suggests recognition-network overfitting.

5.4 SCIENTIFIC FINDINGS

AIS-based evaluation reveals substantial differences in likelihood, overfitting, and mode coverage that KDE, importance-weighted bounds, and visual inspection can miss.

  • Model likelihood comparison: AIS found VAEs substantially higher log-likelihoods than GANs and GMMNs, while KDE treated the three model families as roughly equivalent.The evaluation used AIS and KDE on 1000 MNIST test examples.
  • Model likelihood comparison: KDE errors matched model differences and misordered candidates, estimating only 15 nats between GMMN-10 and GMMN-50 versus AIS’s 247 nats.KDE also estimated VAE-50 below VAE-10 despite an almost 300-nat true advantage for VAE-50.
  • Model likelihood comparison: GAN-10 and GAN-50 looked visually indistinguishable despite nearly 300-nat log-likelihood differences on both training and test sets.This illustrates why sample inspection alone cannot reliably compare models.
  • Measuring the degree of overfitting: GAN-50’s training and test likelihoods stayed nearly identical, while GAN-50 and GMMN-50 overfit less than VAE-50.The training curves were measured with AIS, KDE, and, where applicable, the IWAE lower bound; all plotted estimates were lower bounds.
  • Measuring the degree of overfitting: For VAE-50, the IWAE bound declined after 200 epochs while AIS stayed steady, suggesting recognition-network rather than generative-network overfitting.For GMMN-50, AIS-based training and validation error continued improving at 10,000 epochs despite KDE indicating a plateau.
  • How appropriate is the observation model?: Accounting for non-Gaussianity substantially improved every model, but the effect remained insufficient to explain the VAE–GAN/GMMN likelihood gap.The observation-model analysis treated non-Gaussianity as a possible source of the measured difference.
  • Are the networks missing modes?: Posterior samples from VAEs nearly matched observations, whereas GANs and GMMNs failed to reconstruct some training and validation examples, suggesting missed distributional modes.AIS was used to approximately sample latent variables conditioned on each image before decoding.

A.1 MODELS ON CONTINUOUS MNIST

The continuous-MNIST experiments used fully connected decoder architectures with separate smaller and larger configurations, plus model-specific optimization choices.

  • Architectures: The smaller decoder used 10–64–256–256–1024–784 units, while the larger used 50–1024–1024–1024–784.All decoder layers were fully connected.
  • GAN: GAN-10 and GAN-50 used discriminator architectures 784-512-256-1 and 784-4096-4096-4096-4096-1, respectively.Hidden layers used dropout and tanh activations, with logistic output layers.
  • VAE: The larger VAE encoder was 784-1024-1024-1024-100, while the smaller encoder was 784-256-64-20.Encoder dropout was 0.2; generator and encoder activations differed as specified.
  • GMMN: GMMN hidden layers used ReLU activations and a sigmoid output unit.The reported implementation description specifies the output activation as sigmoid.
  • Optimization: GAN/VAE training used Adam with learning-rate search over {0.001, 0.0001, 0.00001}, whereas GMMN used momentum SGD over {0.1, 0.5, 1, 2}.GMMN momentum was 0.9.

A.2 MODELS ON BINARIZED MNIST

The binarized-MNIST IWAE model used a 50-200-200-784 decoder and a symmetric encoder, with training based on 50 importance samples.

  • Architecture: The IWAE decoder architecture was 50-200-200-784 with tanh hidden layers and a sigmoid output layer.The encoder was symmetric and used a linear output layer.
  • Training: The IWAE bound was computed with 50 samples during training, using the implementation’s hyperparameter choices.The reported implementation was taken from the cited IWAE repository.

B HOW PROBLEMATIC IS THE GAUSSIAN OBSERVATION MODEL?

The Gaussian observation model was tested by allowing per-example variance optimization, which improved likelihoods but did not remove the VAE–GAN/GMMN gap.

  • Motivation: The analysis asks whether Gaussian noise, known to be a poor fit, unfairly affects GAN and GMMN likelihood comparisons.VAE training uses an observation model, whereas GAN and GMMN objectives do not build in a particular observation model.
  • Comparison: Table 3 compares fixed observation variance with variance tuned independently for each example.The per-example tuning is not a proper generative model and serves as an upper bound for a heavy-tailed, radially symmetric noise model.
  • Results: 30–100 nats: independently tuning variance improved log-likelihoods for all networks.The largest jumps occurred for images of 1s and 2s, which preferred smaller and larger variance, respectively.
  • Results: The variance-tuning effect was smaller than the VAE–GAN/GMMN likelihood differences, so it did not explain the observed gap.The authors conclude that a better observation model could nevertheless significantly improve likelihood scores.

C POSTERIOR VISUALIZATION OF DIGIT “2"

Digit “2” is hardest to model: posterior reconstructions reveal that GAN-10 misses many training instances, GAN-50 captures the class with stylistic differences, and VAE-10 reconstructs all samples.

  • Digit “2” is the hardest digit for modelling according to log-likelihood evaluation.
  • GAN-10 fails to capture many training instances of digit “2”, often generating digits “1”, “7”, “9”, “4”, or “8” instead.
  • GAN-50 reconstructs digit “2” samples correctly, with only some stylistic differences from the true data.
  • VAE-10 perfectly reconstructs all sampled digit “2” instances.
  • Posterior samples expose model differences that are not visible when directly sampling from the models.
Loading 1611.04273v2…