PaperlayerView exact revision on arXiv

Source-linked AI summary

How good is my GAN?

Konstantin Shmelkov, Cordelia Schmid, Karteek Alahari

arXiv:1807.09499v1cs.CVcs.LG

TL;DR

The paper addresses the difficulty of evaluating and comparing GAN-generated images, where subjective inspection and existing quantitative measures provide limited or incomplete evidence. It introduces classification-based GAN-train and GAN-test measures, which distinguish generated-image diversity from quality. Experiments show that GAN-image quality decreases significantly as dataset complexity increases, while the measures reveal performance differences among models.

  • Problem

    Evaluating and comparing GANs is difficult because subjective inspection is imprecise and existing measures have important limitations.

  • Method

    The paper uses classifiers trained on generated or real images to define GAN-train and GAN-test as measures related to diversity and image quality, respectively.

  • Results

    GAN-image quality decreases significantly as dataset complexity increases, and GAN-train and GAN-test highlight performance differences among evaluated models.

  • Takeaways & Limitations

    The proposed measures complement existing scores and can distinguish whether generated images are less diverse or have worse image quality.

  • Takeaways & Limitations

    SWD may correlate poorly with other metrics because it approximates Wasserstein distance between image patches, which aligns with Wasserstein GAN objectives but not necessarily other models.

Abstract

from arXiv · show

Generative adversarial networks (GANs) are one of the most popular methods for generating images today. While impressive results have been validated by visual inspection, a number of quantitative criteria have emerged only recently. We argue here that the existing ones are insufficient and need to be in adequation with the task at hand. In this paper we introduce two measures based on image classification---GAN-train and GAN-test, which approximate the recall (diversity) and precision (quality of the image) of GANs respectively. We evaluate a number of recent GAN approaches based on these two measures and demonstrate a clear difference in performance. Furthermore, we observe that the increasing difficulty of the dataset, from CIFAR10 over CIFAR100 to ImageNet, shows an inverse correlation with the quality of the GANs, as clearly evident from our measures.

1 Introduction

Evaluating and comparing GAN-generated images is difficult because subjective inspection is imprecise and existing quantitative measures are limited. The paper introduces classification-based GAN-train and GAN-test measures and reports that GAN quality declines as dataset complexity increases.

  • Evaluation challenge: Evaluating GANs is challenging because subjective visual inspection cannot precisely judge generated-image quality.The lack of an explicit likelihood measure has encouraged reliance on visual evaluation.
  • Limits of prior measures: Existing measures such as Inception score and FID do not compare samples with a target distribution or separate image quality from diversity.Precision-and-recall approaches are also difficult to compute on natural images because their manifold is unknown.
  • Proposed measures: GAN-train trains a classifier on generated images and evaluates it on real images to measure similarity between generated and target distributions.Its accuracy can reflect mode dropping, insufficient realism, or class confusion.
  • Proposed measures: GAN-test evaluates a classifier trained on real images using generated images, with higher accuracy indicating a more realistic approximation of natural-image distributions.The paper also studies generated images for training-data augmentation as a measure of diversity.
  • Experimental findings: Experiments on MNIST, CIFAR10, CIFAR100, and ImageNet show that the proposed measures distinguish performance among recent GANs and other generative models.The reported image quality decreases significantly as dataset complexity increases.

2 Related work

Existing GAN evaluation includes Inception-based scores, distribution distances, precision/recall, and data augmentation, but each has important limitations. GAN-train and GAN-test are presented as complementary classification-based measures that separate image diversity from quality.

  • Inception score: Inception score uses an ImageNet-pretrained Inception network to evaluate generated images without comparing them to real data.It measures the KL-divergence between conditional and marginal class distributions.
  • Inception score: Inception score is limited to generated-image diversity and cannot measure how well the generator approximates the real distribution.It is also sensitive to small changes in Inception weights and has large score variance.
  • Fréchet Inception distance: FID compares Gaussian models of real and generated Inception-embedding distributions using their means and covariances.Its distance is inversely correlated with Inception score and suffers from the same issues discussed for that measure.
  • Limitations and proposed direction: Inception-based measures cannot distinguish poor image quality from low diversity, whereas GAN-train and GAN-test can separate these failure modes.The same low score may result from unrealistic images or overly similar generated images.
  • Other evaluation measures: Sliced Wasserstein distance measures multi-scale statistical similarity between local patches from real and generated images, but is limited across GANs with varied resolutions.It uses randomized projections of Wasserstein-1 distance on Laplacian-pyramid patches.
  • Other evaluation measures: Synthetic-manifold precision and recall assess proximity to and coverage of a known manifold, but are impractical for natural images whose manifold is unknown.Their reported use is limited to synthetic data such as grayscale triangles.
  • Data augmentation: GANs can augment training data with generated samples, making classification performance on augmented data another evaluation perspective.The paper analyzes training image classification networks with data augmentation.
  • Proposed direction: The paper introduces GAN-train and GAN-test as performance-based measures intended to complement existing quantitative evaluation schemes.The approach uses image-classification networks and is motivated by the difficulty of evaluating GANs subjectively.

3 GAN-train and GAN-test

GAN-train evaluates classifiers trained on generated images against real validation data, while GAN-test evaluates classifiers trained on real data against generated images. Together, they diagnose diversity and quality differently, with GAN-test also revealing overfitting or poor approximation of the target distribution.

  • GAN-train: GAN-train measures the accuracy of a classifier trained on generated images and tested on real validation images.Its accuracy can fall because of mode dropping, unrealistic samples, or class mixing.
  • GAN-train: GAN-train near the real-data validation accuracy indicates generated images are high quality and as diverse as the training set.The measure is affected by both generated-sample diversity and whether samples contain relevant features.
  • GAN-test: GAN-test measures the accuracy of a classifier trained on real images and tested on generated images.Ideally, its accuracy should be close to validation accuracy when both sets share the same distribution.
  • GAN-test: A GAN-test score significantly above validation accuracy indicates overfitting, whereas a significantly lower score indicates poor target-distribution capture and image quality.The measure is related to precision because it quantifies closeness to the data manifold.
  • Complementary diagnosis: GAN-test does not measure sample diversity because memorizing one training image perfectly can still produce a high score.Diversity is instead investigated by measuring GAN-train across generated sets of different sizes.

4 Datasets and Methods

The evaluation uses multiple image-classification datasets and compares several generative models, primarily conditioned WGAN-GP, SNGAN, DCGAN, and PixelCNN++.

  • Datasets: The datasets progress from MNIST and CIFAR10 to CIFAR100 and ImageNet1k, increasing the number of labels from 10 to 1,000.CIFAR10 and CIFAR100 each contain 50k training and 10k validation images; ImageNet1k contains 1.3M training and 50k validation images.
  • Datasets: Validation images are excluded from GAN training so test accuracy can be evaluated independently.This differs from some GAN studies and may explain minor differences in reported IS and FID scores.
  • WGAN-GP: WGAN-GP is selected for analysis because it is widely accepted and replaces the discriminator with a Wasserstein-1 distance critic.Its gradient-penalty regularizer enforces the critic’s K-Lipschitz constraint instead of weight clipping.
  • WGAN-GP: The experiments use a class-conditioned WGAN-GP that concatenates noise with class labels in the generator and predicts sources and labels in the discriminator.The setup follows ACGAN-style label conditioning.
  • Other evaluated methods: The comparison includes class-conditioned SNGAN, a class-conditioned DCGAN, and PixelCNN++, a tractable-likelihood autoregressive model.SNGAN uses spectral normalization, while DCGAN uses convolutional layers and batch normalization; PixelCNN++ predicts pixels sequentially with masked convolutions.

5 Experiments

Experiments compare GAN-train, GAN-test, and existing metrics across datasets and models, showing that the proposed measures distinguish image quality from diversity more clearly. Results also examine human evaluation, dataset difficulty, generated-set size, and mixing generated with real data.

  • MNIST: 99.0% GAN-train and 99.2% GAN-test accuracies show that SNGAN nearly reproduces MNIST's simple distribution, versus 99.3% real-data classifier accuracy.These scores indicate high generated-image quality and diversity on MNIST.
  • Metric comparison: 91% versus 80% GAN-train accuracy distinguishes the diversity of noisy CIFAR10 training data from its random 5k subset, while GAN-test remains 95% for both.FID changes only from 27.1 to 29.6, so it does not isolate diversity from quality in this experiment.
  • CIFAR100: SNGAN and WGAN-GP (10M) have similar IS and FID on CIFAR100 but very different GAN-train and GAN-test accuracies, separating quality and diversity.The measures identify SNGAN as better on both dimensions, while WGAN-GP (2.5M) has reasonable IS and FID but very low GAN-test accuracy.
  • ImageNet: 9.3% and 21.9% GAN-train accuracy for SNGAN at ImageNet 128 × 128 resolution indicate good quality but much lower diversity than the original data.WGAN-GP performs dramatically poorer in comparison.
  • GAN image diversity: GAN-train saturates around 15-20k SNGAN-generated CIFAR10 images and shows no CIFAR100 increase beyond 25k images for any model.On CIFAR100, 5k SNGAN-generated images have diversity comparable to 5k real images, whereas WGAN-GP (10M) has very low diversity beyond 5k generated images.
  • GAN data augmentation: Adding 2.5k or 5k real images to 50k generated images improves classifier accuracy, whereas adding 50k real images provides no noticeable improvement and slightly reduces CIFAR100 performance.The reduction is potentially due to lack of image diversity; in a low-data regime, combining reduced real and generated sets also causes accuracy to drop slightly.

6 Summary

The paper introduces GAN-train and GAN-test as quantitative measures for evaluating and comparing images generated by GANs. Experiments show that these measures reveal performance differences and complement existing scores.

  • GAN-train and GAN-test are new quantitative measures for evaluating and comparing images generated by GANs.
  • The measures are motivated by precision and recall scores used to evaluate discriminative models.
  • Extensive experiments evaluate recent GAN approaches and other popular generative models with these measures.
  • GAN-train and GAN-test highlight performance differences between methods and complement existing evaluation scores.

Appendix A: Implementation details

The appendix documents model implementations, training protocols, and classifier setups used across CIFAR10, CIFAR100, MNIST, and ImageNet experiments.

  • The study reimplemented WGAN-GP, SNGAN, DCGAN, and validated the implementations on CIFAR10 against published results.PixelCNN++ used reference implementations for training and accelerated inference.
  • CIFAR experiments train SNGAN and WGAN-GP for 100k iterations with Adam, a decaying learning rate, five critic steps per generator step, and batch size 64.
  • ImageNet experiments use ResNet protocols with 250k iterations at 64 × 64 resolution and 450k iterations at 128 × 128 resolution.
  • CIFAR10 and CIFAR100 GAN-train and GAN-test scores use a preactivation ResNet-32 classifier with the original training schedule and standard data augmentation.
  • MNIST uses a CIFAR10-adjusted GAN architecture and a four-layer convolutional baseline classifier.

Appendix B: Size comparison

The appendix compares generator and dataset sizes, showing that GAN compression is relatively easier for CIFAR10/100 than for ImageNet.

  • SNGAN has 8.3M parameters, while WGAN-GP has 2.5M or 10M parameters in the CIFAR experiments.These correspond to 33Mb, 10Mb, and 40Mb respectively.
  • CIFAR10 and CIFAR100 each contain 50k training images totaling 150Mb of uncompressed data.
  • SNGAN for ImageNet has 42M parameters and WGAN-GP has 48M parameters, versus 16Gb or 64Gb for ImageNet training data.The dataset sizes correspond to 64 × 64 and 128 × 128 resolutions.
  • The size difference may partially explain why compressing CIFAR10/100 into a GAN is relatively easier than compressing ImageNet.

Appendix C: t-SNE Embeddings of Datasets

The appendix visualizes CIFAR100 train, test, and generated images in the baseline classifier’s feature space using t-SNE. The strongest GAN-test models lie near training images, while weaker models form unaligned point clouds.

  • The t-SNE embedding includes 500 images each from CIFAR100 train and test splits and four GAN models across five classes.The classes are apple, aquarium fish, baby, bear, and bicycle.
  • Images are embedded in the feature space of the baseline CIFAR100 classifier used to compute GAN-test accuracy.
  • SNGAN and WGAN-GP (10M) generate images lying close to training images and achieve the two best GAN-test accuracies.
  • WGAN-GP (2.5M) and DCGAN form point clouds that do not correspond to any of the image clusters.
Loading 1807.09499v1…