PaperlayerView exact revision on arXiv

Source-linked AI summary

Classification Accuracy Score for Conditional Generative Models

Suman Ravuri, Oriol Vinyals

arXiv:1905.10887v2cs.LGstat.ML

TL;DR

Evaluating whether generative models support downstream tasks remains difficult because conventional sample-quality metrics may not reflect task performance. The paper introduces CAS by training classifiers on synthetic images and testing them on real images, finding that CAS exposes model deficiencies and that IS and FID are not predictive of it.

  • Problem

    Evaluating generative models remains an open problem, particularly whether models that score well on sample-quality metrics perform effectively in downstream tasks.

  • Method

    CAS trains an image classifier using synthetic data and evaluates its Top-1 and Top-5 accuracy on real data for conditional image models.

  • Results

    Across the benchmark, CAS reveals model deficiencies: BigGAN-deep accuracy decreases by 27.9% for Top-1 and 41.6% for Top-5 versus real data, while VQ-VAE-2 and HAM outperform BigGAN-deep; IS and FID are not predictive of CAS.

  • Takeaways & Limitations

    CAS is practically useful for uncovering particular classes where conditional generative models fail to capture the data distribution, supporting better diagnosis of generative models.

  • Takeaways & Limitations

    CAS can indicate good inference even when the underlying generative model is poor, such as when synthetic samples mix true-distribution and disjoint noise samples.

Abstract

from arXiv · show

Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes---variational autoencoders, autoregressive models, and generative adversarial networks (GANs)---to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.

1 Introduction

The paper evaluates whether conditional generative models support downstream image classification by training classifiers on synthetic data and testing them on real data. CAS exposes failures and model-quality differences that traditional metrics miss.

  • 1 Introduction: CAS trains an image classifier on synthetic data and evaluates its Top-1 and Top-5 accuracy on real data.The performance gap between classifiers trained on real and synthetic data diagnoses deficiencies in the generative model.
  • 1 Introduction: 27.9% and 41.6%: BigGAN-deep’s Top-1 and Top-5 accuracies decrease by these amounts compared to using real data.The result uses a state-of-the-art GAN with an off-the-shelf ResNet-50 inference classifier.
  • 1 Introduction: VQ-VAE-2 and HAM outperform BigGAN-deep on CAS despite relatively poor Inception Scores and Frechet Inception Distances.Because these models produce visually appealing samples, the result suggests IS and FID are poor measures of non-GAN performance.
  • 1 Introduction: CAS identifies particular classes that BigGAN-deep and VQ-VAE-2 fail to model and that were previously unknown in the literature.Figure 1 shows four such classes for BigGAN-deep.
  • 1 Introduction: Neither IS nor FID, alone or in combination, predicts CAS, motivating metrics that better measure downstream task performance.The study applies FID, IS, and CAS to state-of-the-art generative models on ImageNet and CIFAR-10.
  • 1 Introduction: 0.2%: augmenting ImageNet with low-diversity BigGAN-deep images improves Top-5 accuracy by this amount, whereas other synthetic images degrade performance.NAS trains the classifier on both real and synthetic images and shows improvement only in limited circumstances.

2 Metrics for Generative Models

The paper reviews task-agnostic metrics for generative models and argues for evaluating model distributions through downstream-task performance. It introduces CAS as an approximate-inference framework while documenting limitations of likelihood, IS, and FID.

  • 2 Metrics for Generative Models: Evaluating a model distribution without a known downstream task requires assessing how closely pθ(x) matches pdata(x), a broad and difficult goal.Different metrics weight plausibility of samples and coverage of possible data examples differently.
  • 2 Metrics for Generative Models: Likelihood models provide scaled KL-divergence estimates and implicit diversity measures, but likelihood can score out-of-domain data highly and poorly measure sample quality.These observations apply to variational, autoregressive, bijective, and energy-based likelihood models.
  • 2 Metrics for Generative Models: IS measures classifier confidence and class distribution, but it does not penalize intra-class diversity and can reward certain out-of-distribution samples.Some out-of-distribution samples obtain Inception Scores three times higher than the data.
  • 2 Metrics for Generative Models: FID has high bias, and its pool3 feature layer may not correlate well with human judgments of sample quality.The paper also finds that non-GAN models can have poor IS and FID despite visually appealing samples.
  • 2 Metrics for Generative Models: The proposed alternative trains an inference network on model samples and evaluates it on a downstream task, analogous to evaluating speech generation through speech recognition.For conditional image models, image classification serves as the task, with CAS defined by Top-1 and Top-5 accuracy.
  • 2 Metrics for Generative Models: The work extends prior GAN evaluation by emphasizing large-scale and non-GAN models, identifying failed classes, and open-sourcing an ImageNet metric.The paper also describes limitations of classification as approximate inference and verifies the approximation later.

3 Classification Accuracy Score

CAS evaluates conditional generative models by training classifiers on synthetic data and testing them on real data, revealing distribution-matching failures that conventional metrics may miss.

  • CAS definition: CAS trains an image classifier on synthetic data and evaluates its Top-1 and Top-5 accuracy on real data.The classifier is trained from model-generated samples for conditional image models, where labels are image classes.
  • Interpretation: A lower CAS than the original data indicates that the generative model fails to capture the data distribution, despite theoretical failure modes.CAS close to the real-data score is not sufficient to establish accurate distribution modeling.
  • Metric implications: IS and FID are not predictive of CAS, while likelihood-based models can perform well on CAS despite relatively poor values for those metrics.The metric requires classifier training, which is more expensive than calculating IS or FID, but the authors open-source it for use.
  • Diagnostics: CAS can expose particular classes that BigGAN-deep and VQ-VAE-2 fail to model, including classes previously unknown as failure cases.Per-class accuracy comparisons are shown for BigGAN-deep, VQ-VAE-2, and HAM.

4 Experiments

Experiments compare CAS with traditional metrics across ImageNet and CIFAR-10, examine class-level failures, and test synthetic-data augmentation. CAS distinguishes model performance and deficiencies that IS and FID often do not capture.

  • 4.1 Model Comparison on ImageNet: At 256×256, BigGAN-deep reaches CAS Top-5 65.92%, VQ-VAE-2 reaches 77.59%, and the original dataset reaches 91.47%.VQ-VAE-2 outperforms BigGAN-deep on CAS despite poorer FID and IS.
  • 4.1 Model Comparison on ImageNet: At 128×128, HAM and VQ-VAE-2 outperform BigGAN-deep on CAS, while all models underperform the original dataset.BigGAN-deep has similar CAS at 128×128 and 256×256 despite substantially different IS and FID results.
  • 4.2 Uncovering Model Deficiencies: BigGAN-deep shows validation accuracy of 0% for balloon, paddlewheel, pencil sharpener, and spatula.Nearly every BigGAN-deep class declines relative to the original dataset, with only six classes showing marginal improvement.
  • 4.2 Uncovering Model Deficiencies: VQ-VAE-2 exceeds the original data on 31 classes, generalizes better than BigGAN-deep on 87.6% of classes, and has no class with 0% accuracy.HAM's per-class results broadly mirror VQ-VAE-2's results.
  • 4.3 A Note on FID and a Second Note on IS: CAS has little correlation with IS and FID; for BigGAN-deep truncation, Top-1 accuracy correlates 0.16 with FID and -0.86 with IS.The best-performing truncation values, 1.5 and 2.0, have relatively poor IS and FID.

5 Conclusion

The paper proposes CAS for diagnosing deficiencies in conditional image generative models and concludes that conventional metrics can misrepresent non-GAN performance. It also identifies generalization beyond the training set as an unresolved question, especially for downstream deployment.

  • CAS was practically useful for uncovering deficiencies in conditional image generative models.
  • GAN models of ImageNet underperformed likelihood-based models despite high sample quality.
  • IS and FID unfairly penalized non-GAN models.
  • The extent to which these models generalize beyond the training set remains an open question.The paper notes that current results suggest even state-of-the-art models underfit, while underfitting may be temporary.

A.1 ImageNet

The ImageNet experiments use a ResNet-50 classifier with single-crop evaluation and a large-batch TPU training setup. BigGAN-deep is evaluated across truncation parameters to examine its quality–diversity trade-off.

  • ImageNet evaluation uses a ResNet-50 classifier with single-crop evaluation.
  • The classifier is trained for 90 epochs with warmup, scheduled learning-rate reductions, and an 8,192-sample batch setup.Training uses 128 TPU chips and takes roughly 45 minutes.
  • BigGAN-deep experiments sweep truncation parameters from 0.2 to 2.0.The sweep examines a truncation trick described as trading off sample quality for diversity.

A.2 CIFAR-10

The CIFAR-10 experiments use a ResNet-56 classifier trained on fixed training and validation splits with a scheduled learning rate and batch size 128. The setup broadly follows the cited reference and TPU tutorial conventions.

  • CIFAR-10 evaluation uses a ResNet-56 classifier with 45,000 training samples and 5,000 validation samples.
  • The classifier is trained for 182 epochs with learning-rate reductions at epochs 91 and 136.
  • The CIFAR-10 setup uses batch size 128 and mirrors the setup in reference [37].
  • The implementation broadly follows the Google Cloud TPU ResNet tutorial, with some changes for the metric.
  • The workflow includes creating a cloud project, enabling billing, and creating a us-central storage bucket.

1. Launch google cloud shell (https://cloud.google.com/shell/)

The launch workflow configures a preemptible Cloud TPU virtual machine, identifies the zone suffix convention, starts persistent SSH with tmux, and changes to the TPU ResNet directory.

  • The TPU is launched with an n1-standard-8 machine, a v2-8 TPU, preemptibility, and a us-central zone.
  • The zone placeholder uses a or b for paying customers and f for TFRC participants.
  • After entering the virtual machine, the workflow runs tmux for persistent SSH and changes to the official ResNet directory.

6. Set

The setup resamples extreme noise-vector dimensions using a truncation range, then references conversion of synthetic data to TFRecords.

  • Dimensions of noise vector z outside the range −2τ to 2τ are resampled.τ is the truncation parameter.
  • Lower τ values produce less diverse datasets.
  • The setup references converting synthetic data to TFRecords.

7. Set

The instructions launch a Google Cloud TPU environment, enter the ResNet directory, and use synthetic and validation data in TFRecords for training.

  • The workflow begins by launching Google Cloud Shell and provisioning a preemptible v2-128 TPU.The command uses an n1-standard-8 machine in a us-central-<x> zone.
  • After entering the virtual machine, run tmux and change to the official ResNet directory.
  • Training uses TFRecords of synthetic data and validation data, followed by a modified 10-hour training procedure.The instructions state that steps 6 and 9 are exceptions.
  • The TPU can be turned off by deleting it with ctpu delete --zone <ZONE>.
Loading 1905.10887v2…