PaperlayerView exact revision on arXiv

Source-linked AI summary

Pros and Cons of GAN Evaluation Measures

Ali Borji

arXiv:1802.03446v5cs.CV

TL;DR

Grounded quantitative and qualitative ways to assess GANs remain missing, while existing measures have differing strengths and limitations. The paper critically reviews the strengths and limitations of 24 quantitative and 5 qualitative measures for evaluating GANs. Agreement between evaluation measures and human perceptual judgments is unclear, and several highly regarded measures require substantial samples and computation.

  • Problem

    Grounded quantitative and qualitative ways to assess GANs remain missing, while existing measures have differing strengths and limitations.

    Source: §1 · p. 2

  • Method

    The paper critically reviews the strengths and limitations of 24 quantitative and 5 qualitative measures for evaluating GANs.

    Source: §4 · p. 42

  • Results

    Agreement between evaluation measures and human perceptual judgments is unclear, and several highly regarded measures require substantial samples and computation.

    Source: §3.3 · p. 41

  • Takeaways & Limitations

    Selecting appropriate GAN evaluation measures remains an important open problem for fair comparison and for understanding, improving, and developing generative models.

    Source: §4 · p. 42

    The Inception score cannot reliably detect mode collapse because models that memorize one ImageNet example per class can achieve high scores.

    Source: §2.2 · p. 7

Abstract

from arXiv · show

Generative models, in particular generative adversarial networks (GANs), have received significant attention recently. A number of GAN variants have been proposed and have been utilized in many applications. Despite large strides in terms of theoretical progress, evaluating and comparing GANs remains a daunting task. While several measures have been introduced, as of yet, there is no consensus as to which measure best captures strengths and limitations of models and should be used for fair model comparison. As in other areas of computer vision and machine learning, it is critical to settle on one or few good measures to steer the progress in this field. In this paper, I review and critically discuss more than 24 quantitative and 5 qualitative measures for evaluating generative models with a particular emphasis on GAN-derived models. I also provide a set of 7 desiderata followed by an evaluation of whether a given measure or a family of measures is compatible with them.

1. Introduction

GANs have advanced rapidly across generative modeling applications, but grounded quantitative and qualitative evaluation remains underdeveloped. This paper critically reviews existing GAN measures and proposes suggestions for more efficient, fair evaluation and comparison.

Motivation

  • GANs have enabled substantial theoretical and application progress, yet relatively little effort has addressed their evaluation.GANs generate realistic high-resolution images and support applications including representation learning, domain adaptation, synthesis, restoration, translation, and video generation.Source: §1 · p. 1

Research gap

  • Grounded ways to quantitatively and qualitatively assess generative models remain missing despite the emergence of multiple evaluation measures.Existing benchmarking efforts, including [23] [24], are helpful, but further research is needed to understand measures and their strengths and limitations [22] [27] [28] [29] [30].Source: §1 · p. 2

Evaluation challenges

  • Evaluation is difficult because qualitative tests can reward memorization or low diversity, while quantitative measures may not match human perception.Additional challenges include varied probability criteria and the lack of perceptually meaningful image-similarity measures [22].Source: §1 · p. 2

Paper contribution

  • The paper critically reviews available GAN measures to help researchers assess them objectively and suggests more efficient measures for fair GAN evaluation and comparison.The review covers both quantitative and qualitative approaches and their strengths and limitations.Source: §1 · p. 2

2. GAN Evaluation Measures

This section surveys quantitative and qualitative measures for evaluating GANs, organizing them by the aspects of quality, diversity, structure, and distribution they capture. It also considers their differing assumptions, interpretability, computational demands, dependencies, and practical limitations.

2.1. Desiderata

  • The seven desiderata require measures to reward fidelity, diversity, disentanglement, boundedness, transformation sensitivity, human agreement, and low sample and computational complexity.They should detect overfitting, mode collapse, and mode dropping while remaining invariant to semantically irrelevant image transformations.Source: §2.1 · p. 3

2.2. Quantitative Measures

  • Log-likelihood can rank models inconsistently with sample quality: models may produce great samples with poor likelihood or poor samples with high likelihood.Parzen estimates also become unreliable in high-dimensional spaces and can require very large sample sets, making memorization and missing modes difficult to assess.Source: §2.2 · pp. 4, 6
  • Inception Score rewards classifiable and diverse samples but can favor memorization, overlook mode collapse, ignore real-data support, depend on ImageNet classes and resolution, and be deceived by unrelated natural images.Its theoretical minimum is 1.0 when all generated samples collapse to one point, yet class-conditional memorization can still achieve high scores.Source: §2.2 · pp. 6–7
  • FID compares Gaussian feature distributions for real and generated data; lower FID means closer distributions, and it detects artifacts and intra-class mode dropping better than Inception Score.FID is reported as consistent with human judgments and more robust to noise than IS, although its Gaussian feature assumption is not guaranteed.Source: §2.2 · p. 10
  • MMD measures dissimilarity between real and generated distributions, while classifier two-sample tests assess equality using a discriminator trained and evaluated on held-out data.Kernel MMD can work well in pretrained-CNN feature space with low sample and computational complexity.Source: §2.2 · pp. 10, 11, 13
  • Wasserstein distance can detect memorization and mode collapse without saturating, but its high sample and time complexity limits practical use; birthday testing instead detects near-duplicate samples and mode collapse.Arora and Zhang [27] found with probability ≥50% that batches of about 400 CelebA samples contained duplicates for both DCGAN and MIX+DCGAN.Source: §2.2 · p. 12
  • Classifier two-sample tests distinguish real and generated distributions through classification accuracy, while classification-performance measures assess generated images or representations on surrogate tasks but depend heavily on classifier choice.C2ST accuracy near chance suggests P = Q; 1-NN ranges from 0% for memorization to 100% for complete separability, with 50% as chance.Source: §2.2 · pp. 13–15
  • Boundary distortion detects diversity loss as covariate shift through synthetic-versus-real classifier generalization, whereas NDB compares statistically different bin proportions directly between distributions.NDB is domain agnostic and avoids fixed deep representations, but pixel-space L2 similarity may be unsuitable.Source: §2.2 · pp. 15–17
  • Image retrieval measures evaluate generated-image quality by nearest-neighbor distances to held-out images, using statistical comparisons between generators and deviation from an ideal real-data distance distribution.The relative increase in mean nearest-neighbor distance is interpretable as the percentage above distances for samples from the real distribution.Source: §2.2 · pp. 17–18
  • GAM and tournament-based measures compare generators through adversarial matches, supporting relative model comparisons and training-progress monitoring but requiring calibration, pairwise computation, or tournament-specific interpretation.Tournament win rate and skill rating can incorporate learned features or human judges, yet provide relative rather than absolute scores and can be expensive to reproduce.Source: §2.2 · pp. 18–20
  • NRDS ranks models by the discriminator-output curve area during training: samples closer to real data decay more slowly, and higher NRDS indicates better performance.In the toy example, fake-close has area 145.4955 versus 71.1057 for fake-far, so fake-close is relatively better.Source: §2.2 · pp. 20, 22, Figure 9 · p. 22
  • Adversarial accuracy and divergence compare human-annotated conditional classifiers, while geometry score compares real and generated manifold topology but is technically difficult to understand and compute.Adversarial divergence has lower bound zero when Pg(y|x) = Pr(y|x), but these measures require substantial human labeling unless class-conditional generators provide labels.Source: §2.2 · pp. 22–23
  • MRLT compares topological distributions without auxiliary networks; WGAN-GP performs better than WGAN on almost all MNIST classes, although visual quality may require FID.MRLT averages relative living times over random subsets and compares datasets using L2 distance; both models are close to ground truth, but WGAN-GP is usually better.Source: §2.2 · pp. 23–24, Figure 10 · p. 25
  • MS-SSIM evaluates image similarity and diversity, but its Inception-score correlation is r2 = −0.16 and it misses smaller color or texture variation losses.Across ImageNet, 847 classes had sample mean MS-SSIM below the maximum training-data MS-SSIM; the measure reliably detects large-scale mode collapse but does not directly assess training-set similarity.Source: §2.2 · p. 27, Figure 11 · p. 28
  • Low-level statistics reveal that DCGAN [1], WGAN [39], and VAE [19] resemble natural scenes yet retain significant differences, including non-scale-invariant power spectra.Evaluated statistics include mean power spectrum, connected components, random filter responses, and contrast distribution; multiscale patch distances separately reflect large-scale structure and pixel-level attributes.Source: §2.2 · p. 29
  • Precision measures generated-sample quality and recall measures reference-distribution coverage, but manifold-based scores are impractical for real images where the manifold is unknown.The approach estimates latent codes by gradient-descent inversion and uses distance thresholds; FID captures both precision and recall, whereas IS captures only precision according to Lucic et al. [23].Source: §2.2 · pp. 29–30

2.3. Qualitative Measures

  • Human visual evaluation is intuitive but expensive, biased, difficult to reproduce, variable across inspectors, and unable to reveal mode dropping or density quality.Sample-based judgments can favor overfitting and models concentrating on common modes, because mode dropping may improve apparent visual quality.Source: §2.3 · pp. 30–31
  • Nearest-neighbor checks can be misleading because Euclidean distance is perceptually sensitive and memory GANs can trivially pass; perceptual distances and multiple neighbors help.Small image shifts can produce large Euclidean-distance changes, while transformed training-image storage can evade the test [22].Source: §2.3 · p. 31, Figure 14 · p. 32
  • Rapid scene categorization and preference judgments provide intuitive human comparisons, but crowd conditions, cost, overfitting, and weak diversity assessment limit their reliability.Denton et al. found that a class-conditional LAPGAN fooled about 40% of subjects versus ≤10% for the standard GAN, compared with >90% for real images.Source: §2.3 · p. 33, Figure 15 · p. 33
  • Mode collapse maps different latent inputs to one output, whereas mode drop ignores difficult data modes; synthetic or labeled datasets enable measurement, but real-data evaluation remains difficult.Known-mode methods count high-quality samples near mode centers or compare induced mode distributions using reverse KL divergence, yet nearest-mode reasoning is non-trivial in high dimensions.Source: §2.3 · pp. 34–35, Figure 18 · p. 36
  • Disentanglement tests assess whether latent axes correspond to semantic factors whose controlled variation produces predictable visual changes, including known factors in simulated datasets.Examples include facial hair or pose changes and evaluating whether latent spaces recover ground-truth variation factors.Source: §2.3 · p. 36
  • Latent walks and vector arithmetic probe semantic representations, memorization, hierarchical collapse, and whether learned codes support meaningful visual concepts.Sharp transitions may signal memorization, whereas semantic changes and operations such as combining exemplar codes can indicate useful representations.Source: §2.3 · pp. 36–37
  • Latent-space interpolations evaluate continuity and the ability to generate novel, visually plausible images between realistic samples rather than merely memorizing training examples.Spherical linear interpolation can avoid departing from the prior distribution and produce sharper samples; Figure 20 illustrates interpolation examples.Source: §2.3 · p. 37, Figure 20 · p. 38
  • Visualizing generator and discriminator features reveals learned hierarchies and semantically responsive internal representations, with guided backpropagation and t-SNE used to inspect them.For DCGAN, discriminator features responded to scene elements such as beds and windows, while t-SNE projected learned latent spaces into two dimensions.Source: §2.3 · p. 37

3. Discussion

The discussion surveys task-specific GAN evaluation measures, defines sample and computational efficiency, and finds no universally best score because measures capture different aspects and have important limitations. It emphasizes jointly evaluating fidelity, diversity, and controllable sampling while adapting measures to applications and model types.

3.1. Other Evaluation Measures

  • Task-specific measures evaluate semantic correctness, attribute coverage, and compositionality [127], while diversity can use deep-feature pairwise distances and four divergences yield consistent rankings.Zhu et al. compare average pairwise cosine distance for generated and real images; Im et al. report robust agreement across Jensen-Shannon, constrained Pearson χ2, Maximum Mean Discrepancy, and Wasserstein measures.Source: §3.1 · p. 39

3.2. Sample and Computational Efficiencies

  • Sample efficiency measures how few samples distinguish generated from real data, while computational efficiency measures evaluation time per number of samples for training monitoring and diagnosis.The reference real set is disjoint from the training comparison set, all evaluated sets have equal size n, and efficiency supports early stopping.Source: §3.2 · p. 39

3.3. What is the Best GAN Evaluation Measure?

  • No consensus identifies a best GAN score: measures differ in overfitting, disentanglement, bounds, perceptual agreement, efficiency, and distortion sensitivity.Table 2 summarizes these relative properties, while the discussion highlights that only two measures explicitly address overfitting and most do not assess disentanglement.Source: Table 2 · p. 40, §3.3 · p. 41
  • Application-specific evaluation is preferable because likelihood, psychophysics, user ratings, generic-GAN measures, and conditional-GAN measures suit different tasks and available correspondences.Theis et al. [22] argue against task-independent image-generation evaluation; likelihood suits compression, psychophysics and user ratings suit reconstruction and synthesis, and FCN score suits conditional generation with correspondences.Source: §3.3 · pp. 41–42
  • A good measure should jointly capture fidelity, diversity, and controllable sampling, with defined bounds and sensitivity to image distortions and transformations.Fidelity-only measures miss diversity, diversity-only measures miss fidelity, and some practical measures handle about 2000 images while Wasserstein distance does not scale to large samples.Source: §3.3 · pp. 41–42

4. Summary and Future Work

The section reviews 24 quantitative and 5 qualitative GAN evaluation measures, emphasizing that selecting appropriate measures remains an open problem for fair comparison and model development. It proposes shared repositories, rigorous comparisons, and controlled multi-measure benchmarking as priorities for future work.

Summary

  • The review covers the strengths and limitations of 24 quantitative and 5 qualitative measures for evaluating GANs.Appropriate evaluation remains important for fair model comparison and for understanding, improving, and developing generative models.Source: §4 · p. 42
  • Evaluation measures should remain constant under semantics-preserving rotations and translations, although some are more sensitive to pixel-space than convolutional-space changes.This robustness analysis concerns mixtures of real and transformed real samples, where the transformations do not alter image semantics.Source: Figure 21 · p. 43
  • Lucic et al. [23] found no empirical evidence that GAN models claiming superiority outperformed the original GAN.The lack of a universal powerful measure may hinder progress, motivating ideas from natural scene statistics and cognitive vision, including human perception of symmetry and image clutter.Source: §4 · p. 43

Future Work

  • Future work should create a code repository, conduct detailed comparative empirical and analytical studies, and benchmark models under matched conditions using multiple measures.Matched conditions include architectures, optimization, hyperparameters, and computational budget.Source: §4 · p. 43

Related papers

Loading 1802.03446v5…