Source-linked AI summary
Pros and Cons of GAN Evaluation Measures
Ali Borji
TL;DR
GAN evaluation lacks grounded, consensual measures that fairly capture model strengths and limitations. This paper critically reviews existing measures and concludes that selecting appropriate measures remains an important open problem for fair comparison and for understanding, improving, and developing generative models.
Problem
Grounded quantitative and qualitative ways to evaluate and compare GANs are still missing, while existing approaches have distinct strengths and limitations.
Method
The paper critically reviews 24 quantitative and 5 qualitative measures for evaluating GANs and offers suggestions for more efficient, fair evaluation and comparison.
Results
The review finds that choosing appropriate GAN evaluation measures remains an important open problem for fair model comparison and for understanding, improving, and developing generative models.
Takeaways & Limitations
Researchers need objective evaluation measures that support fair GAN assessment and comparison.
Takeaways & Limitations
The Inception score may assign high values to class-conditional models that memorize one example per ImageNet class, so it cannot reliably reveal mode collapse.
Abstract
from arXiv · showhide
Generative models, in particular generative adversarial networks (GANs), have received significant attention recently. A number of GAN variants have been proposed and have been utilized in many applications. Despite large strides in terms of theoretical progress, evaluating and comparing GANs remains a daunting task. While several measures have been introduced, as of yet, there is no consensus as to which measure best captures strengths and limitations of models and should be used for fair model comparison. As in other areas of computer vision and machine learning, it is critical to settle on one or few good measures to steer the progress in this field. In this paper, I review and critically discuss more than 24 quantitative and 5 qualitative measures for evaluating generative models with a particular emphasis on GAN-derived models. I also provide a set of 7 desiderata followed by an evaluation of whether a given measure or a family of measures is compatible with them.
1. Introduction
GANs have advanced rapidly across theory and applications, but their evaluation remains underdeveloped. Existing qualitative and quantitative measures have complementary limitations, motivating a critical review and guidance for fairer comparison.
- GANs have attracted substantial attention, with many variants and applications following their introduction in 2014.They are used for tasks including representation learning, domain adaptation, synthesis, restoration, translation, and video generation.
- Grounded quantitative and qualitative ways to assess generative models are still missing.
- Qualitative judgments can favor models that overfit or memorize limited data, while quantitative measures may not match human perception.These approaches therefore have different strengths and limitations rather than providing an unqualified evaluation standard.
- The paper critically reviews available GAN measures and aims to help researchers assess models objectively.
2. GAN Evaluation Measures
GAN evaluation measures span diverse quantitative and qualitative approaches, each capturing different aspects of generated distributions and carrying distinct limitations. The paper reviews these measures through their behavior, assumptions, and practical trade-offs.
- Inception Score: Inception Score can correlate with image quality and diversity, but may fail to reliably detect mode collapse and is affected by classifier choice and image resolution.A class-conditional model memorizing one example per ImageNet class can still achieve a high IS, while different classifiers produce different entropy terms.
- Fréchet Inception Distance: FID measures distance between generated and real distributions and can detect intra-class mode dropping and image artifacts that Inception Score may miss.FID performs well for discriminability, robustness, and efficiency, but assumes Gaussian feature distributions and uses only first- and second-order moments.
- Coverage and Diversity Measures: Classifier-based and duplicate-detection methods assess distributional coverage, diversity, or memorization, but can require manual inspection or depend on classifier behavior.Near-duplicate testing found, with probability ≥50%, at least one duplicate in batches of about 400 CelebA samples for both DCGAN and MIX+DCGAN.
- Relative and Distributional Comparisons: NDB compares proportions across data-derived bins, while GAM compares two GANs through discriminator swapping and reports relative rather than absolute performance.NDB uses image pixels directly and can be domain agnostic; GAM requires similarly performing discriminators and is expensive in practice.
- Tournament-Based Measures: Tournament win rate and skill ratings provide alternative progress measures, with skill-rating learning curves broadly agreeing with Fréchet-distance curves but sometimes disagreeing with classifier scores.The comparison used generator and discriminator snapshots across models with different seeds, hyperparameters, and architectures.
- Distributional and Topological Measures: Topological measures such as the geometry score can compare generated distributions without auxiliary networks, but may not capture visual quality when relevant transformations preserve topology.The paper therefore describes combining the geometry score with measures such as FID.
3. Discussion
The discussion compares GAN evaluation measures against practical and conceptual desiderata, finding that no single score captures all relevant aspects of generation quality. It emphasizes trade-offs among fidelity, diversity, controllability, robustness, human agreement, and computational practicality.
- Meta-measure findings: Only two measures explicitly address overfitting, while most do not evaluate disentangled representations and few provide both lower and upper bounds.The meta-measure analysis also reports that agreement with human perceptual judgments is less clear.
- Efficiency and robustness: Several highly regarded measures have high sample and computational efficiency, but sensitivity to image distortions remains less explored.Sample and computational efficiency are treated as practical concerns for monitoring training and diagnosing problems early.
- Choosing measures: No consensus identifies the best score because different measures assess different aspects of image generation, making a single universally adequate score unlikely.FID is described as comparatively plausible, while average likelihood and Wasserstein distance have important limitations in high-dimensional settings.
- Application-specific evaluation: Measure choice should depend on the application: likelihood suits compression, psychophysics and user ratings suit reconstruction or synthesis, and FCN score suits conditional GANs with correspondences.Other measures are intended for generic GANs that generate from noise vectors.
- Evaluation desiderata: Evaluation measures differ in whether they assess fidelity, diversity, controllable sampling, bounds, or sensitivity to image distortions.The paper identifies these as core properties for evaluating GAN measures.
- Fidelity and diversity: Qualitative measures such as SSIM and PSNR primarily assess visual fidelity, whereas measures such as the Birthday Paradox Test primarily assess diversity.The paper argues that either emphasis alone is insufficient because a useful score should account for both fidelity and diversity.
4. Summary and Future Work
The paper concludes that GAN evaluation remains an open problem requiring critical comparison rather than reliance on a universal score. It recommends coordinated infrastructure, controlled benchmarking, and multiple measures, while noting unresolved concerns about robustness and empirical evidence.
- Summary: The review covers the strengths and limitations of 24 quantitative and 5 qualitative GAN evaluation measures.The paper frames appropriate evaluation as important for fair comparison and for understanding, improving, and developing generative models.
- Future evaluation criteria: A good measure should remain constant under semantics-preserving rotations and translations of real images.The robustness analysis examines mixtures of real and transformed real samples, while noting that pixel-space and convolutional-space measures differ in susceptibility.
- Open problems: The lack of a universal powerful measure can hinder progress, and benchmark evidence has not favored claimed GAN superiority over the original GAN.The paper suggests borrowing ideas from natural scene statistics and cognitive vision to study generated-image plausibility.
- Future work: Future work should create a repository, conduct detailed comparative studies, and benchmark models under matched conditions using more than one measure.Matched conditions include architectures, optimization, hyperparameters, and computational budget.