Source-linked AI summary
Reliable Fidelity and Diversity Metrics for Generative Models
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, Jaejun Yoo
TL;DR
Generative-model evaluation needs reliable measures of fidelity and diversity, but existing precision and recall metrics retain practical failure cases. The paper proposes density and coverage with manifold estimation, finding more interpretable and reliable signals while improving mode-dropping diagnosis and embedding choices for domain-shifted data.
Problem
Generative-model evaluation lacks reliable metrics that separately assess fidelity and diversity across high-dimensional data.
Method
The paper proposes density and coverage metrics based on a carefully designed manifold-estimation procedure and studies random CNN embeddings for domains unlike ImageNet.
Results
Density and coverage address failure cases of improved precision and recall, while coverage decreases gradually under simultaneous mode dropping and random embeddings give more sensible results for MNIST and sound data.
Takeaways & Limitations
Density and coverage provide more interpretable metrics with systematically selectable hyperparameters for evaluating fidelity and diversity.
Takeaways & Limitations
Improved precision and recall remain vulnerable to outliers and can require per-model nearest-neighbour computations for recall.
Abstract
from arXiv · showhide
Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: https://github.com/clovaai/generative-evaluation-prdc.
1. Introduction
Evaluating generative models is difficult because high-dimensional distributions admit multiple imperfect distance measures, while FID does not separate fidelity from diversity. The paper proposes density and coverage metrics and studies embedding choices, including random CNN features for domains unlike ImageNet.
- Evaluation challenges: Generative-model evaluation compares real and generated high-dimensional sample sets, but multiple distribution distances and human judgments each have limitations.Human MOS measures are affected by diverse opinions about the ideal generative model.
- Existing metrics: FID is popular and agrees well with human perceptual scores, but its single value does not separate fidelity from diversity.Fidelity concerns resemblance to real samples, whereas diversity concerns coverage of their variability.
- Existing metrics: Precision and recall introduce separate fidelity and diversity perspectives, yet the latest versions fail practical requirements for generative-model evaluation.The requirements include detecting identical distributions, resisting outliers, responding to mode dropping, and enabling easy hyperparameter selection.
- Proposed approach: The paper proposes density and coverage using a carefully designed manifold-estimation procedure to make fidelity-diversity metrics empirically reliable and theoretically analysable.The metrics are tested on generative adversarial networks.
- Embedding choices: Image-generation evaluations commonly rely on ImageNet embeddings, but dataset bias can limit fairness and create a false sense of improvement outside natural-image domains.The paper studies this issue and evaluates randomly initialized CNN embeddings on MNIST and sound generation.
2. Backgrounds
Image-generation metrics typically embed real and fake samples, estimate distributions in feature space, and quantify their discrepancy. Precision and recall diagnose fidelity and diversity separately, but improved versions retain practical vulnerabilities involving manifolds, outliers, and computation.
- Evaluation pipeline: High-dimensional image evaluation commonly embeds real and fake samples into Euclidean space, constructs distributions there, and quantifies their discrepancy.This pipeline uses a nonlinear mapping such as a CNN feature extractor before distribution comparison.
- Embeddings: Pixel-space ℓ2 distance can be misleading, motivating CNN feature embeddings as proxies for perceptual similarity.Perceptually identical images can have large pixel-space distances after a one-pixel translation.
- Embeddings: Randomly initialized CNN embeddings are proposed for MNIST and sound spectrograms, where large domain gaps from ImageNet make them more sensible evaluation measures.The paper contrasts these with ImageNet-pretrained features and reports more macroscopic views of distributional discrepancies.
- Fidelity and diversity: Single-value metrics such as IS and FID rank models but are less suitable for diagnosing the fidelity-diversity trade-off, motivating separate precision and recall measures.Precision targets how realistic generated samples are, while recall targets how well they cover real-sample variation.
- Improved precision and recall: Improved precision and recall use k-nearest-neighbour distances to estimate density, but their manifolds can overestimate distributions around outliers.This vulnerability can inflate precision or recall when unrealistic samples or real outliers enlarge the estimated manifolds.
- Improved precision and recall: P&R constructs separate real and fake manifolds, then computes precision from fake samples against the real manifold and recall symmetrically.The manifolds are formed from neighbourhood spheres whose radii depend on nearest-neighbour distances.
- Density and coverage: Density replaces precision’s binary neighbourhood-membership count with the number of real-sample neighbourhood spheres containing each fake sample.This forms a superposition of neighbourhood spheres and lies conceptually between precision and Parzen-window estimation.
3. Density and Coverage
Density and coverage are proposed as practical replacements for precision and recall, addressing outlier sensitivity and computational drawbacks while enabling analytic calibration under identical real and fake distributions. Density counts neighbourhood support around fake samples, whereas coverage measures the fraction of real samples reached by fake samples.
- Density and coverage remedy practical problems in precision and recall through a manifold estimation procedure that is empirically reliable and theoretically analysable.The paper tests the metrics on generative adversarial networks.
- Density: Density counts how many real-sample neighbourhood spheres contain each fake sample, reducing precision’s overestimation around real outliers.Unlike precision, density rewards fake samples in regions where real samples are densely packed; the paper reports 60% density versus 100% precision in a pathological example.
- Coverage: Coverage builds neighbourhood manifolds around real samples and measures the fraction of real samples whose neighbourhoods contain at least one fake sample.This avoids constructing a fake manifold and allows the manifold to be computed per dataset rather than per model, reducing nearest-neighbour computation.
- Analytic behaviour: Under identical real and fake distributions, density and coverage have analytic expected values that approach 100% as sample sizes and k increase.The expected values do not depend on the distribution type or data dimensionality; the analysis supports systematic selection of k, M, and N.
- Analytic behaviour: For identical distributions, D&C remain consistent across Gaussian and FFHQ embeddings, whereas precision and recall produce significantly different estimated values across data types.The practical setting M = N = 10 000 and k = 5 yields E[coverage] ≈0.969 > 0.95; the corresponding P&R choices lack a known exact principle.
4. Experiments
The experiments test precision, recall, density, and coverage on toy and real-world scenarios involving distribution shifts, outliers, mode dropping, latent truncation, and embedding choices. Across these checks, density and coverage better detect distributional matches, resist outliers, capture diversity loss, and diagnose fidelity-diversity trade-offs, while random embeddings can be more sensible beyond ImageNet-like domains.
- Experimental setup: The evaluation compares P&R with D&C using toy and real-world scenarios whose desired metric behaviours are defined in advance.Toy experiments use Gaussian and mixture-of-Gaussian distributions in R64; real-world experiments examine images, MNIST, and other generated data.
- Toy sanity checks: At the matched Gaussian distribution, P&R fall below one, whereas D&C remain close to one.At µ = 0, precision is 0.68 and recall is 0.67, while density is 1.06 and coverage is 0.97.
- Toy sanity checks: With one real or fake outlier, P&R vary pathologically, but D&C largely coincide with the no-outlier case.P&R are affected because their manifold estimation overestimates the manifold around inliers and the outlier.
- Fidelity-diversity diagnosis: Under StyleGAN truncation, density changes more than precision as ψ increases, providing finer-grained fidelity diagnosis while coverage and recall increase.As ψ increases, precision and density decrease, while recall and coverage increase.
- Embedding pipeline: Random embeddings produce more sensible scores when target data differ substantially from ImageNet statistics.For MNIST, density is 0.047 with T4096 versus 0.491 with R64; for sound, R64 reports recall 0.572 and coverage 0.653 versus 0.029 and 0.020 under T4096.
5. Conclusion and discussion
The paper identifies practical failure cases in existing fidelity and diversity metrics and proposes density and coverage to address them, alongside random embeddings for distributions unlike ImageNet.
- Existing precision and recall metrics can overestimate manifolds, underestimate scores for identical distributions, resist outliers, and miss certain mode dropping.
- Density and coverage address these failure cases while enabling systematic selection of their hyperparameters.
- The authors suggest density and coverage for more stable and reliable diagnosis of generative models.
- Random embeddings are argued to be better choices than ImageNet pre-trained embeddings when target distributions differ substantially from ImageNet statistics.