PaperlayerView exact revision on arXiv

Source-linked AI summary

A note on the evaluation of generative models

Lucas Theis, Aäron van den Oord, Matthias Bethge

arXiv:1511.01844v3stat.MLcs.LG

TL;DR

Generative models serve diverse applications, but heterogeneous formulations and evaluation criteria make direct comparison difficult. The paper reviews these criteria and their relationships, showing that likelihood, sample quality, and Parzen estimates can diverge, so models should be evaluated for their intended applications.

  • Problem

    Generative models are used across diverse applications, yet heterogeneous formulations, training procedures, and evaluation criteria make direct comparison difficult.

  • Method

    The paper reviews and analyzes the implications of training and evaluating image generative models with log-likelihood, sample quality, and Parzen window estimates.

  • Results

    Log-likelihood, visual sample fidelity, and Parzen window estimates are largely independent in high-dimensional settings, and k-means can outperform the true distribution under Parzen evaluation.

  • Takeaways & Limitations

    Models should be evaluated directly for their intended applications, and Parzen window estimates should generally be avoided unless the application specifically requires that loss.

  • Takeaways & Limitations

    Visual samples are unreliable proxies for density-model quality and for performance in applications such as classification or inpainting.

Abstract

from arXiv · show

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

1 INTRODUCTION

Generative models are evaluated with heterogeneous criteria because applications and computational constraints differ. The paper examines how training and evaluation measures relate, finding that performance on one criterion does not reliably predict performance on others.

  • Motivation: Log-likelihood is a standard criterion, but many generative models make it computationally difficult to calculate.Unnormalized energy-based models have difficult normalization constants, while latent-variable models may require complex integrals.
  • Motivation: Researchers therefore also compare models using samples, parameter interpretations, Parzen estimates, or surrogate tasks.These alternatives are often more accessible than likelihood, even for density-estimation tasks.
  • Contribution: The paper investigates how training and evaluation criteria shape the resulting models and the conclusions drawn from them.
  • Contribution: Performance across log-likelihood, sample quality, classification, and Parzen estimates need not agree.The paper reports that good or bad performance on one metric does not guarantee corresponding performance on the others.

2 TRAINING OF GENERATIVE MODELS

Different objectives agree when the model is correct and data is effectively unlimited, but can produce different optima under model mismatch. A toy Gaussian-mixture example shows that MMD and JSD favor different trade-offs from log-likelihood.

  • Training objectives: With a correct model and infinite data, many consistent training procedures converge to the same result.This agreement does not generally hold when the model family mismatches the data distribution.
  • Training objectives: Under model-data mismatch, different objective functions can lead to very different fitted models.
  • Toy example: MMD and JSD produce isotropic Gaussians that fit one mixture mode well while ignoring other data regions.The example connects MMD to generative moment matching networks and JSD to the objective used by GANs.
  • Toy example: Log-likelihood, equivalently minimizing KLD, avoids extremely small probability on data points but assigns substantial mass to non-data regions.
  • Implementation: JSD optimization was performed directly using the data density, although practical applications generally provide only samples from the data distribution.The passage notes that GANs may approximately optimize JSD, while their practical objective can differ substantially.

3 EVALUATION OF GENERATIVE MODELS

Evaluation metrics for generative models capture different properties and can support conflicting conclusions. Log-likelihood requires careful treatment of discrete image data, while samples provide useful but limited diagnostic evidence.

  • Choosing an evaluation metric should reflect the intended application and the conclusions it is meant to support.
  • Model samples can reveal why a model fails, but sample-based analyses may misrepresent density-estimation performance and other applications.
  • Average log-likelihood can become arbitrarily high when discrete integer-valued images are modeled with continuous densities without dequantization.
  • Adding real-valued noise to integer pixels makes continuous-model likelihood closely related to discrete-model likelihood on the original data.
  • The expected log-likelihood on dequantized data corresponds to the discrete model’s log-likelihood, whose negative equals the average lossless-compression bits under base-2 logarithms.

SEMI-SUPERVISED LEARNING

Log-likelihood motivates semi-supervised learning because it supports probabilistic modeling of images and labels. In practice, model mismatch, intractable inference, and MAP-induced overfitting can limit this approach.

  • Bayesian prediction for an unseen image requires integrating the distribution over model parameters.
  • With sufficient data and certain assumptions, Bayesian prediction approaches prediction using parameters at the maximum-a-posteriori solution.
  • The maximum-a-posteriori parameters combine prior, image-data, and labeled-data log probabilities in the optimization objective.
  • In the unsupervised setting with a uniform parameter prior, optimizing log p(X | θ) is a natural objective.
  • The approach may fail because of model-data mismatch, inability to solve the parameter integral, or overfitting from the MAP approximation.

3.2 SAMPLES AND LOG-LIKELIHOOD

In high-dimensional image settings, sample quality and average log-likelihood need not track one another. Either metric can be poor while the other is good.

  • A model can have poor log-likelihood while producing great samples, or great log-likelihood while producing poor samples.

POOR LOG-LIKELIHOOD AND GREAT SAMPLES

Convincing samples can coexist with poor likelihood when a model concentrates narrowly around training or plausible images. Earlier correlations between likelihood and sample quality depended on restricted experimental conditions.

  • A lookup table containing enough training images can generate convincing images but perform poorly on unseen test-data likelihood.
  • With sufficiently small Gaussian noise, mixtures centered on training or plausible transformed images produce great samples despite poor log-likelihood.
  • A reported correlation between log-likelihood and human judgments of sample realism used small image patches and models optimized to minimize KLD.

GREAT LOG-LIKELIHOOD AND POOR SAMPLES

A model can achieve high average log-likelihood while generating poor samples, because high dimensionality makes small likelihood penalties negligible. Conversely, realistic samples can be obtained with little change to log-likelihood, showing the criteria are largely independent.

  • A mixture can generate samples from a poor model 99% of the time while retaining high average log-likelihood.The mixture combines a strong model p with a poor model q.
  • For high-dimensional data, log p(x) grows with dimensionality while the mixture penalty log 100 remains constant.For 32 by 32 CIFAR-10 images, model log-likelihood differences can be thousands of nats, whereas log(100) is about 4.61 nats.
  • Thus, large average log-likelihood does not ensure visually plausible samples.
  • Choosing q to reproduce training examples makes samples realistic 99% of the time while changing log-likelihood by at most 4.61 nats.
  • Log-likelihood and visual sample quality are therefore largely independent criteria.

3.3 SAMPLES AND APPLICATIONS

Sample quality need not predict application performance: a model whose samples are mostly poor can still retain useful posterior predictions. In high-dimensional settings, application performance can therefore be dominated by the good component rather than by the samples viewers see.

  • A mixture generating poor samples 99% of the time can still perform well in inpainting, denoising, or classification.The good component p is more likely a posteriori for predicting labels or missing pixels, even when q dominates prior samples.
  • Figure 2 evaluates nearest-neighbor stability after shifting image windows by one to four pixels, using 1,000 queries and 90% confidence intervals.
  • For high-dimensional x, α ≈ 1, so mixing with the poor model q barely changes the posterior over y.Here σ is the sigmoidal logistic function.
  • Classification performance is dominated by p even though generated samples are dominated by q.
  • High visual fidelity of samples is generally not necessary for good performance in these applications.

3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS

Sample-based evaluation and nearest-neighbor comparisons can mischaracterize generative models, especially when detecting overfitting or judging density quality. Euclidean proximity may not reflect perceptual similarity, while limited samples cannot reveal failures across the vast image space.

  • 3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS: Sample-based qualitative assessment can favor models that overfit, and nearest-neighbor comparisons detect only the starkest forms of overfitting.The approach is therefore unfit as a general overfitting detector.
  • 3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS: Perceptually small image changes can produce large Euclidean-distance changes, weakening nearest-neighbor comparisons as evidence of visual similarity.The paper illustrates this by comparing CIFAR-10 image patches with versions shifted one pixel down and right.
  • 3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS: An overfitting model may generate neither plausible images nor more than a small fraction of plausible images, without reproducing close copies of training data.In such cases, nearest-neighbor distances will not expose the model’s limited coverage.
  • 3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS: Even on 6 by 6 CIFAR-10 patches with many samples, a Parzen estimate for a Gaussian can remain far from the true log-likelihood.This example illustrates why likelihood proxies can be unreliable even in relatively small image spaces.
  • 3.4 EVALUATION BASED ON SAMPLES AND NEAREST NEIGHBORS: On MNIST, Parzen window estimates rank samples from the true distribution below samples from a simple k-means-trained model.This comparison shows that the estimate can prefer a less desirable model over the data distribution.

3.5 EVALUATION BASED ON PARZEN WINDOW ESTIMATES

Parzen window estimates approximate a generative model’s log-likelihood by fitting a tractable density estimator to generated samples. However, these estimates can be inaccurate and can produce misleading model rankings.

  • 3.5 EVALUATION BASED ON PARZEN WINDOW ESTIMATES: Parzen window evaluation fits a Gaussian-kernel density estimator to samples generated by the model, then evaluates test log-likelihood under that estimator.The resulting value is used as a proxy for the model’s true log-likelihood.
  • 3.5 EVALUATION BASED ON PARZEN WINDOW ESTIMATES: Parzen estimates can produce rankings different from other likelihood estimates across generative models.This undermines the hope that inaccurate estimates might nevertheless preserve a useful ordering.
  • 3.5 EVALUATION BASED ON PARZEN WINDOW ESTIMATES: A GMMN+AE can receive a higher Parzen score than images from the training set, which are samples from the true distribution.The ranking can therefore favor a learned model over samples from the data distribution itself.

4 CONCLUSION

Generative models should be trained and evaluated according to their intended applications because different metrics favor different trade-offs. The paper specifically cautions against treating Parzen estimates as a general evaluation method.

  • 4 CONCLUSION: Different metrics and evaluation procedures favor different models, so training and evaluation should match the target application.Good performance in one application should not be treated as evidence of good performance in another.
  • 4 CONCLUSION: Sample-based evaluation favors overfitting and is a poor indicator of density quality, whereas high likelihood does not guarantee visually pleasing samples.Samples can have arbitrary form while being only a few bits from the likelihood optimum.
  • 4 CONCLUSION: Visual-fidelity assessment remains appropriate for image synthesis because the data distribution performs well on that task.The paper contrasts this with Parzen estimates, under which the data distribution can perform worse than less desirable models.
  • 4 CONCLUSION: Parzen window estimates should generally be avoided for evaluating generative models unless the application specifically requires such estimates.Their use is cautioned against because they can rank the data distribution below less desirable models.
Loading 1511.01844v3…