Pros and Cons of GAN Evaluation Measures: New Developments

Ali Borji

Source-linked AI summary

Pros and Cons of GAN Evaluation Measures: New Developments

Ali Borji

arXiv:2103.09396v3Mar 17, 2021cs.LGcs.AIcs.CV

TL;DR

GAN evaluation remains an open problem because popular measures capture only selected aspects of generative quality and can be biased or diagnostically limited. This paper updates an earlier review by surveying newer quantitative and qualitative measures, emerging dimensions such as fairness, and links to deepfake detection. It concludes that objective and comprehensive evaluation is still unresolved despite the growing popularity of IS, FID, Precision-Recall, and PPL.

Problem
Evaluating how well generative models approximate data distributions remains difficult, while common metrics can be biased, limited in diagnostic scope, or unsuitable across domains.
Method
The paper reviews recent quantitative and qualitative evaluation measures, benchmarks, visualization and error-diagnosis techniques, fairness-related dimensions, and connections to deepfakes.
Results
Objective and comprehensive evaluation of generative models is still an open problem despite the relative popularity of IS, FID, Precision-Recall, and PPL.
Takeaways & Limitations
GAN evaluation should consider dimensions beyond image quality and diversity, including textual correspondence, perceptual response, bias, fairness, and deepfake-related concerns.
Takeaways & Limitations
FID can require usually above 50K samples, and manual inspection is subjective, time-constrained, and dependent on domain knowledge.

Abstract

from arXiv · show

This work is an update of a previous paper on the same topic published a few years ago. With the dramatic progress in generative modeling, a suite of new quantitative and qualitative techniques to evaluate models has emerged. Although some measures such as Inception Score, Frechet Inception Distance, Precision-Recall, and Perceptual Path Length are relatively more popular, GAN evaluation is not a settled issue and there is still room for improvement. Here, I describe new dimensions that are becoming important in assessing models (e.g. bias and fairness) and discuss the connection between GAN evaluation and deepfakes. These are important areas of concern in the machine learning community today and progress in GAN evaluation can help mitigate them.

1. Introduction

Evaluating generative models remains an open problem because existing measures capture different aspects of distribution quality and have important limitations. The paper reviews influential and newer quantitative and qualitative evaluation approaches.

Generative-model evaluation asks how well a model approximates a data distribution, commonly emphasizing fidelity and diversity.
The paper surveys recent quantitative and qualitative measures, benchmarks, visualization methods, and error-diagnosis techniques.
Log-likelihood can disagree with image quality, while kernel density estimation is difficult in high-dimensional spaces.
Inception Score is ImageNet-biased, misses intra-class diversity, is implementation-sensitive, and requires large samples for reliability.
FID is widely used and detects intra-class mode collapse, but its Gaussian assumption and finite-sample bias limit interpretation; samples usually exceed 50K.
Single-value metrics combine quality and diversity, making them less suitable for diagnosis; Precision-Recall separates these dimensions.

2. New Quantitative GAN Evaluation Measures

New quantitative measures extend evaluation beyond standard image-level FID by addressing spatial structure, class conditioning, computational cost, memorization, bias, and video or other modalities.

sFID supplements pooled Inception features with intermediate spatial features to retain information about spatial variability.
CAFD fits class-specific distributions, computes Fréchet distance within each class, and averages the K class distances.
FastFID speeds FID computation 25 to 500 times and is an optimization method rather than a metric.
MiFID modifies FID with a memorization penalty for generated images too similar to training data.
Bias-free FID∞ and IS∞ use extrapolation to estimate scores as if computed with infinitely many samples, providing drop-in replacements for finite-sample scores.
CleanFID standardizes image processing choices because inconsistent implementations can produce vastly different FID scores.
FVD evaluates video quality and temporal coherence using features from an I3D network trained on Kinetics datasets.

2.2. Methods based on Self-supervised Learned Representations

Self-supervised representations address the ImageNet bias of classification-based FID and can provide more transferable embeddings for evaluating generative models across domains.

Classification-pretrained FID embeddings can be misleading on non-ImageNet datasets because they are biased toward ImageNet.
Self-supervised representations provide better transfer to new tasks and domains, produce more reasonable model rankings, and often improve FID sample efficiency.

2.3. Methods based on Analysing Data Manifold

Recent evaluation measures analyze the geometry, topology, and latent-space structure of generated data rather than relying only on distributional moments. These methods assess manifold overlap, higher-order structure, perceptual smoothness, and latent-factor separability, while each has a stated scope or design limitation.

Local Intrinsic Dimensionality (LID): CrossLID measures how closely the manifolds of real and generated distributions coincide through local intrinsic dimensionality.It is reported to correlate with GAN training progress, detect mode collapse, and remain robust to small-scale noise, image transformations, and sample size.
Local Intrinsic Dimensionality (LID): CrossLID may be difficult to apply to complex, high-dimensional data where local dimensionality is hard to define.This is identified as an unresolved applicability question rather than a demonstrated failure.
Intrinsic Multi-scale Distance (IMD): IMD incorporates all distribution moments to distinguish data structures that FID and KID cannot separate when their first three moments match.It is described as intrinsic and multi-scale, capturing geometry at local and global scales and remaining applicable to unaligned data.
Manifold Topology: MTop-Divergence tracks multiscale topology discrepancies between the manifolds supporting two high-dimensional distributions.The method is applied to images, 3D shapes, and time series, and is described as domain agnostic.
Perceptual Path Length (PPL): PPL estimates perceptual differences between consecutive images along latent-space interpolation paths to assess latent entanglement and smoothness.It uses spherical interpolation, a perceptual distance such as LPIPS, and a step size of ϵ = 10^-4; lower perceptual variation is associated with smoother transitions.
Linear Separability in Latent Space: Latent-space linear separability evaluates whether individual factors of variation correspond to directions that can be separated by a linear hyperplane.The procedure trains auxiliary attribute classifiers, fits a linear SVM to labeled latent vectors, and computes conditional entropy.

2.4. Classification Accuracy Score (CAS)

Classification Accuracy Score evaluates whether synthetic data preserve information useful for recognizing labels in real images. Its results show substantial downstream accuracy loss for a state-of-the-art GAN and reveal that common GAN metrics do not predict this behavior.

Classification Accuracy Score (CAS): CAS trains an image classifier only on synthetic data and evaluates its labels on real test images.The approach uses class-conditional generative models, including GANs and VAEs, to generate training data for the classifier.
Classification Accuracy Score (CAS): 27.9% and 41.6% decreases in Top-1 and Top-5 accuracy, respectively, occur when BigGAN-deep replaces original data for classifier training.These decreases are reported relative to training with the original data.
Classification Accuracy Score (CAS): CAS automatically identifies particular classes for which generative models fail to capture the data distribution.The paper presents this class-specific diagnostic as one of the findings from the evaluation.
Classification Accuracy Score (CAS): IS and FID are neither predictive of CAS nor useful for evaluating non-GAN models in the reported experiments.

2.5. Non-Parametric Tests to Detect Data-Copying

Non-parametric data-copying tests target overfitting by comparing generated samples with training and held-out samples. The test exploits the prediction that memorizing models place generated samples closer to training data than held-out samples are.

Non-Parametric Tests to Detect Data-Copying: Data-copying describes a generative model memorizing training samples or small variations of them.
Non-Parametric Tests to Detect Data-Copying: The three-sample test compares the training set, held-out target-distribution samples, and generated samples.
Related Evaluation Measures: PPL is described elsewhere as capturing image consistency, with lower values considered better.
Non-Parametric Tests to Detect Data-Copying: Overfitted GANs are expected to produce generated samples closer on average to training samples than held-out test samples are.The comparison can also be performed separately within cells of the instance space.

2.6. Measures that Probe Generalization in GANs

Generalization probes reveal behaviors that aggregate quality metrics can miss, including mode errors, anomalous attribute combinations, and data-copying. New precision-recall variants separately characterize quality, coverage, density, and authenticity.

Controlled probes: Fixed-attribute training sets can yield generated samples with unseen feature values, such as 2–5 objects when every training image has exactly 3.With multimodal data, models may average nearby modes or memorize combinations when variety is limited.
Data-copying: Data-copying tests compare generated, training, and held-out samples: overfitted models place generated samples closer to training examples than held-out samples.The test can be conducted separately across cells because model behavior may differ by region of instance space.
Precision, recall, and extensions: Precision-recall curves expose the quality–coverage trade-off, distinguishing poor precision from mode collapse through poor recall.
Controlled probes: Training on exactly two rectangles can produce images containing one, two, or three rectangles, although a revised model generates a higher fraction of correct images.
Precision, recall, and extensions: Density and coverage address precision-recall weaknesses involving identical distributions, outliers, and arbitrary hyperparameters, providing more interpretable signals.Density counts how many real-sample neighborhood spheres contain each generated sample, rather than making a binary containment decision.
Precision, recall, and extensions: α-Precision, β-Recall, and Authenticity jointly characterize fidelity, diversity, and generalization using typical-sample supports and full precision-recall curves.

2.8. Duality GAP Metric

The duality-gap metric evaluates generative models through game-theoretic suboptimality and can monitor training without labels or a pretrained classifier. Related diagnostics also expose spectral distortions that current generators introduce.

Duality GAP Metric: The duality gap measures a generator–discriminator solution’s suboptimality relative to equilibrium and supports efficient, domain-agnostic evaluation during training.
Duality GAP Metric: The measure highly correlates with FID on natural-image datasets and extends to modalities such as text and sound.
Duality GAP Metric: Unlike classifier-based metrics, the duality gap requires neither labels nor a pretrained classifier.
Spectral diagnostics: Current generators often fail to approximate real spectral distributions because common up-scaling operations alter image spectra and create high-frequency distortions.Durall et al. illustrate this through azimuthally integrated power-spectrum statistics.

2.10. Caption Score (CapS)

Caption Score addresses a gap in text-to-image evaluation: image-only metrics can overlook whether generated images correspond to their textual prompts. It measures text-image quality and accuracy at finer granularity than FID or IS.

Caption Score: Existing image metrics may be fooled by text-to-image systems that ignore textual input while generating realistic-looking images.
Caption Score: Caption Score evaluates the quality and accuracy of text-image generation at finer granularity than FID and Inception Score.The score is defined over a text-token sequence t and an image x.
Caption Score: Higher CapS indicates better text-image generation according to the metric.

3. New Qualitative GAN Evaluation Measures

New qualitative measures assess realism through human perception, neural responses, generator omissions, latent-space steerability, internal units, and fake-image detection.

Human perceptual evaluation: HYPE uses human perception under time constraints or error rates to estimate when generated images appear real.A 50% HYPE score indicates results indistinguishable from real images; higher scores indicate hyper-realism.
Neural evaluation: Neuroscore uses RSVP and neural responses to measure perceptual reactions to real and generated target images.A convolutional neural network can predict Neuroscore directly from generated images without neural responses.
Mode-collapse diagnosis: Semantic segmentation comparisons reveal object classes and individual instances that a GAN omits at distribution and reconstruction levels.The approach exposes omissions such as people, cars, and fences in generated church images.
Latent-space evaluation: Latent-space steerability measures whether navigating toward a learned direction produces target visual transformations and matches real-data attribute distributions.The method compares generated edits with target edits and evaluates attributes such as luminance.
Interpretability and diagnosis: GAN dissection identifies generator units associated with semantic concepts, enabling interventions that expose artifacts and edit objects.Units can be forcefully activated or ablated to add or remove objects such as trees.
Universal fake detection: A detector trained on one CNN generator can generalize to unseen architectures, datasets, and training methods, revealing shared synthetic-image flaws.This supports fake-versus-real detection while indicating that current CNN-generated images contain systematic artifacts.

4. Discussion

The discussion highlights weaknesses in evaluation metrics, the need to account for bias and fairness, and the close relationship between GAN assessment and deepfake detection.

Metric limitations: IS and FID can favor memorization over imperfect models that cover more of the true distribution.Neural network divergence is proposed as a measure that discounts memorization and prefers generalization beyond training data.
Bias and fairness: Bias can enter generative models through training, evaluation, and deployment, making bias and fairness important dimensions for model comparison.The discussion emphasizes fairness despite the absence of one precise fairness definition.
Broader evaluation: GAN evaluation therefore combines quantitative scores, qualitative inspection, interpretability, and detectors that expose omissions, artifacts, or memorization.The reviewed techniques include latent-space analysis, GAN dissection, universal detection, and neural-network divergence.
Connection to deepfakes: Deepfake detection and GAN evaluation are connected because difficulty distinguishing synthetic content from real content reflects generator performance.Detection difficulty varies by category, while deep networks can still identify subtle artifacts in doctored images.

5. Summary and Conclusion

The paper concludes that GAN evaluation is evolving but remains an open problem, requiring broader coverage of scenes, dimensions, tasks, domains, and memorization.

Summary: Objective and comprehensive GAN evaluation remains an open problem despite the popularity of IS, FID, Precision-Recall, and PPL.The paper reviews recently proposed measures and summarizes them in Figure 24.
Future directions: Future evaluation should address complex scenes, since prior work has focused more heavily on faces and scenes with few objects.Bedrooms, street scenes, and nature scenes are identified as less explored settings.
Future directions: Generalization and fairness deserve more attention alongside image quality and diversity.Generalization can probe compositionality and physical logic, while fairness relates to deployment risks and societal impact.
Future directions: Metrics should be tailored to the target task because image quality and diversity can have different importance across applications.Quality may dominate synthesis or translation, whereas diversity may matter more for synthetic data augmentation.
Future directions: Reliable evaluation should diagnose errors and generalize across domains, especially where humans are less able to judge sample quality.Medical images are given as an example of a domain requiring reliable assessment.
Future directions: The degree to which generative models memorize training data remains unclear, and nearest-neighbor analysis has limitations.The conclusion identifies memorization assessment as an unresolved evaluation issue.