Source-linked AI summary
An empirical study on evaluation metrics of generative adversarial networks
Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, Kilian Weinberger
TL;DR
GAN evaluation is challenging because meaningful quantitative scores are difficult to establish from finite samples, and existing metrics have uncertain practical reliability. The paper systematically evaluates representative sample-based metrics against desirable diagnostic properties, finding that MMD and 1-NN perform well in a suitable feature space. It also examines overfitting and the practical limitations of alternative evaluation approaches.
Problem
Meaningful GAN evaluation is difficult because the target distribution is known only through finite samples, while evaluation metrics themselves have uncertain behavior across scenarios.
Method
The paper experimentally compares six model-agnostic sample-based metrics and evaluates them using properties including discrimination, mode sensitivity, overfitting detection, and efficiency.
Results
MMD and 1-NN accuracy appear strongest overall in a pretrained ResNet feature space, while GAN overfitting is not observed unless training samples are very few.
Takeaways & Limitations
Kernel MMD is recommended in practice, and MMD and 1-NN support practical GAN comparison, model selection, and investigation of overfitting without relying on cherry-picked samples.
Takeaways & Limitations
AIS-based likelihood evaluation is excluded because it requires access to the generative model rather than only finite generated samples.
Abstract
from arXiv · showhide
Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. With a series of carefully designed experiments, we comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far they are from learning the target distribution.
1 Introduction
GAN evaluation remains difficult because it is often qualitative, while the meaningfulness of quantitative metric scores is itself uncertain. The paper addresses this by systematically testing sample-based metrics and identifying useful properties and practical guidance.
- GAN evaluation is predominantly qualitative, often relying on manual inspection of generated-image visual fidelity.
- Existing metrics may produce meaningful scores in some scenarios but invite misinterpretation in others.
- The paper evaluates evaluation metrics through experiments targeting discrimination of real and fake samples, mode dropping and collapse, and overfitting.
- MMD and the 1-NN two-sample test appear best suited because they satisfy useful discriminability, mode-sensitivity, and efficiency properties.
- The study aims to establish principles for choosing, interpreting, and designing GAN evaluation metrics in practical settings.
2 Background
GAN evaluation compares finite real and generated samples because the real density is unknown and GAN densities are usually intractable. The paper reviews model-agnostic sample-based metrics and their operational definitions, including Inception-based scores, MMD, Wasserstein distance, FID, and 1-NN testing.
- GAN framework: GANs learn a parameterized distribution Pg intended to approximate the real distribution Pr from i.i.d. real samples.
- GAN framework: In practice, the discriminator and generator are optimized with alternating gradient-descent steps using finite samples from Pr and Pg.
- Sample-based evaluation: Evaluation uses empirical sample dissimilarities because Pr is unknown and exact densities for Pg are usually intractable.
- Sample-based evaluation: The study focuses on model-agnostic metrics that treat the generator as a black box and require only finite generated samples.
- Representative metrics: The reviewed metrics include Inception Score, Mode Score, Kernel MMD, Wasserstein distance, FID, and 1-NN-based two-sample testing.
- Representative metrics: Higher Inception Score reflects confident ImageNet predictions and an approximately uniform marginal label distribution, indicating quality and diversity under that classifier.
- Representative metrics: Mode Score extends Inception Score by measuring dissimilarity between real and generated marginal label distributions.
- Representative metrics: Lower MMD indicates closer distributions for a fixed kernel, while FID compares Gaussian feature distributions through empirical means and covariances.
3 Experiments with GAN evaluation metrics
The experiments show that GAN evaluation metrics are meaningful only under suitable distance representations, with convolutional feature space generally outperforming pixel space. Across discriminability, robustness, sample efficiency, and overfitting tests, MMD and 1-NN perform reliably, while RIS, RMS, and Wasserstein have specific failures.
- 3.1 Feature space: Distance-based metrics are highly sensitive to the underlying distance, and a suitable feature space is essential for meaningful GAN evaluation.Pixel-space distances fail under small rotations and translations, whereas convolutional-space metrics remain nearly constant.
- 3.3 Discriminability: On LSUN, RIS and RMS decrease as generated samples enter real-image mixtures, while pixel-space Wasserstein and 1-NN real accuracy fail to increase.These failures violate the expected discriminability property that scores should rise with the fraction of generated samples.
- 3.3 Discriminability: When modes collapse or are dropped, convolutional-space metrics generally respond correctly, but RIS and RMS remain nearly insensitive to severe mode dropping.Pixel-space Wasserstein decreases under mode collapse, and pixel-space 1-NN real accuracy follows the opposite trend from the desired response.
- 3.5 Efficiency: All metrics are practical for 2000 samples, but Wasserstein distance does not scale to large sample sizes.Computing each metric takes about 8 seconds on an NVIDIA TitanX for 2000 samples, with most time spent extracting ResNet features.
- 3.5 Efficiency: All metrics except RIS and RMS detect increasing overfitting gaps between validation and training scores as generated samples overlap more with training data.The paper attributes RIS’s failure to ignoring the real distribution and suggests RMS’s failure reflects limited generalization and overly specific softmax features.
4 Discussions and Conclusion
The study finds that metric behavior depends strongly on feature space and that MMD and 1-NN are especially practical, while other metrics have important limitations. These metrics also reveal overfitting and mode-collapse behavior that human evaluation may miss.
- Practical implications: The metrics support studying overfitting, selecting models, and comparing GANs without relying on cherry-picked human evaluation samples.The study also reports that overfitting in DCGAN and WGAN occurs only with very few training samples.
- Metric strengths: MMD works well in pretrained ResNet features, identifies generated or noise images from real images, and has low sample and computational complexity despite bias.The authors recommend MMD for practical use.
- Metric limitations: Wasserstein distance has high sample complexity and exact computation scales as O(n^3), making it less appealing as sample size increases.Its practical disadvantage is computational cost relative to other methods.
- Metric strengths: FID performs well in discriminability, robustness, and efficiency despite modeling only the first two feature-space moments.The paper characterizes FID as a good GAN metric.
- Metric strengths: 1-NN provides a bounded score with an interpretable 50% optimum and indicates prevalent mode collapse in typical GAN models.Human evaluation and Inception Score do not effectively detect this problem.
- Conclusions: MMD and 1-NN accuracy appear well suited for GAN evaluation in a pretrained ImageNet ResNet feature space.Both perform well in discriminability, robustness, and efficiency.
- Metric limitations: Inception Score and Mode Score can be unsuitable for datasets unlike ImageNet, while Inception Score cannot detect overfitting using a holdout set.Inception Score may reward sharp and diverse images without measuring similarity to the real distribution.
A GAN Variants used in our experiments
The experiments use several representative GAN variants, including DCGAN, WGAN, WGAN with gradient penalty, and LSGAN. These models differ in generator architecture, critic constraints, regularization, or discriminator loss.
- DCGAN: DCGAN generates 64 × 64 images by projecting uniform noise into a convolutional representation followed by fractionally-strided convolutions.Its optimization minimizes Jensen-Shannon divergence between real and generated images.
- WGAN: WGAN replaces the discriminator with a real-valued critic whose Lipschitz constraint is enforced by clipping parameters.The critic approximates the Wasserstein distance between real and generated samples.
- WGAN with gradient penalty: WGAN with gradient penalty enforces the Lipschitz condition through a gradient penalty term.The cited description reports improved convergence speed and generated-image quality over WGAN.
- LSGAN: LSGAN uses a least-squares discriminator loss instead of sigmoid cross entropy.This corresponds to minimizing the Pearson χ2 divergence between real and generated distributions.
- Evaluation setting: The study evaluates metrics across feature spaces, and Wasserstein distance shows high variance while 1-NN accuracy has a small increase even when trends are otherwise similar.These observations connect model evaluation to the choice of representation.
B The choice of feature space
Feature-space choice is crucial for interpreting GAN metrics. Metrics are generally stable across several feature extractors, but random or lower-dimensional features can specifically destabilize Wasserstein distance and reduce 1-NN discriminability.
- Feature-space design: The study compares VGG, Inception, random-weight ResNet, and same-dataset-trained ResNet features with ImageNet ResNet features.These alternatives test how feature representation affects metric behavior.
- Trained feature extractors: Switching from ResNet-34 to VGG or Inception has little effect on metric scores.The broad trends remain similar across trained networks.
- Random features: Random-network features still work for MMD but make Wasserstein distance unstable and 1-NN accuracy less discriminative.Inception Score and Mode Score become meaningless when computed from random-network softmax values.
- Dataset-specific features: CIFAR-10-trained ResNet features produce lower metric variance than ImageNet features, possibly because their dimensionality is smaller.The feature dimensionality is 64 for CIFAR-10 versus 512 for ImageNet.
- Feature-space evaluation: On CelebA, overfitting is detected only with the tiny training set, where validation MMD and 1-NN scores are significantly worse.The figure reports no overfitting in the larger training settings.
C Are GANs overfitting to the training data?
The overfitting experiments compare DCGAN and WGAN using training and validation scores under full, small, and tiny CelebA training sets. The metrics show little overfitting except under the most data-scarce condition.
- Experimental setup: The experiments hold out 20,000 CelebA images for validation and construct training sets of approximately 180,000, 2,000, and 10 images.The reduced sets test whether limited data exposes overfitting.
- Experimental setup: Training and validation scores are computed from 2,000 real samples and 2,000 generated samples for each metric.Real samples come from either the training or validation set.
- Overfitting results: Training and validation scores almost overlap with 2,000 or 180,000 training samples, indicating no detected overfitting for DCGAN or WGAN.The reported overlap holds under the evaluated metrics.
- Model comparison: DCGAN outperforms WGAN and converges faster on the full training set, whereas WGAN is more stable and converges to better positions on the small training set.The comparison concerns both evaluated metrics.
- Memorization: Even memorizing 50 images is difficult for GAN models.This observation reinforces the limited evidence for memorization in the tested settings.
- Additional comparison: The table compares several GAN models on the LSUN dataset.
D Comparison of popular GAN models based on quantitative evaluation metrics
The comparison uses MMD and 1-NN accuracy in a ResNet feature space to evaluate four CelebA-trained GANs, with WGAN-GP performing best under most metrics while all models remain far from the true distribution.
- MMD and 1-NN accuracy in a 34-layer ImageNet-trained ResNet feature space were used to compare DCGAN, WGAN, WGAN-GP, and LSGAN.Scores used 2000 holdout samples and 2000 generated samples from CelebA-trained models.
- WGAN-GP performs the best under most of the metrics.
- DCGAN records 0.759 overall 1-NN accuracy on real samples, compared with 0.765 achieved by WGAN-GP.
- DCGAN's generated-sample 1-NN accuracy is 0.892, versus 0.860 for WGAN-GP, suggesting greater mode capture but more sample collapse.
- All evaluated GANs have 1-NN accuracy above 0.8 and MMD scores much larger than the ground-truth MMD of 0.019.These results indicate that even state-of-the-art GANs remain far from learning the true distribution.