PaperlayerView exact revision on arXiv

Source-linked AI summary

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, J. Eric T. Taylor, Gabriel Loaiza-Ganem

arXiv:2306.04675v2cs.LGcs.CVstat.ML

TL;DR

Current generative-model metrics do not reliably reflect human judgments of image realism, particularly for diffusion models. The paper evaluates 41 models and 4.1M images with large-scale human experiments and alternative encoders, finding that DINOv2-ViT-L/14 largely resolves the mismatch while memorization remains poorly detected.

  • Problem

    The paper asks whether current metrics and their Inception-V3 representations adequately evaluate fidelity, diversity, rarity, and memorization across modern generative models.

  • Method

    The study compares 17 metrics and 9 encoders across diverse models and datasets, using psychophysics-based human realism judgments as a fidelity baseline.

  • Results

    No existing metric strongly correlates with human evaluations; diffusion models are more realistic to humans yet often rank worse by FID, while DINOv2-ViT-L/14 largely recovers the correlation.

  • Takeaways & Limitations

    Replacing Inception-V3 with DINOv2-ViT-L/14 enables richer generative-model evaluation, while memorized samples can remain undetected by current memorization metrics.

Abstract

from arXiv · show

We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.

1 Introduction

The paper questions whether current generative-model metrics adequately measure learned distributions, then tests human realism judgments, alternative encoders, diversity, rarity, and memorization across modern models.

  • Motivation: Current evaluation must assess fidelity, diversity, and memorization, while an ideal model achieves high fidelity and diversity without reproducing training samples.These properties concern resemblance to the training distribution, coverage of that distribution, and novelty relative to the training set.
  • Human evaluation: No current metric strongly correlates with human evaluators, while diffusion models significantly outperform other techniques on human-perceived indistinguishability from training data.The human evaluation establishes a fidelity baseline against which metrics are compared.
  • Alternative encoders: DINOv2 ViT-L/14 resolves the discrepancy with human evaluators, whereas SwAV and CLIP-B/32 replacements are sub-optimal.The semantic information represented by self-supervised networks depends strongly on training procedure and architecture, and supervised networks generalize poorly for this evaluation.
  • Diversity, rarity, and memorization: The FID–human discrepancy is not explained by diversity trade-offs or humans mistaking rare real images for fakes, while memorized samples appear particularly on CIFAR10 and evade current memorization metrics.The paper’s summary attributes the metric discrepancy to unfair punishment of diffusion models by Inception-V3.

2 Datasets, metrics, and encoders

The study assembles diverse generated datasets and compares standard and diagnostic metrics across multiple representation spaces, emphasizing alternatives to Inception-V3 for generative evaluation.

  • Generated datasets: The benchmark covers CIFAR10, ImageNet1k, FFHQ, and LSUN-Bedroom using diffusion, GAN, VAE, flow, Transformer-based, and consistency models.It includes 13, 11, 9, and 8 models for the four datasets, respectively.
  • Ranking metrics: FID computes Fréchet distance between 50k real and generated samples in Inception-V3 representation space, while alternatives include sFID, FID∞, IS, and KID.The study also computes Fréchet distance in alternative representation spaces, naming metrics by their image encoder.
  • Diagnostic metrics: Fidelity is measured with precision, density, and human error rate; diversity with recall and coverage; rarity with rarity score; and class-conditional diversity with Vendi score.These metrics diagnose distinct properties of generated samples and model distributions.
  • Representation spaces: The paper evaluates supervised and self-supervised encoders as alternatives to the standard ImageNet-trained Inception-V3 representation.The goal is a more general perceptual representation space across natural-image domains.
  • Representation spaces: ConvNeXt-large provides a modern supervised baseline, while ResNet50 and ViT-L/14 architectures support comparisons among self-supervised feature extractors.The selected ViT-L/14 models offer a reported tradeoff between representation quality and computation cost.

3 Human evaluation of generated data

A large psychophysics-based experiment measures human judgments of generated-image realism and compares model rankings against FID, revealing a systematic mismatch favoring diffusion models.

  • Experimental design: Each trial presents a generated or training-set image, asks participants to classify it as real or fake, and scores models by human error rate.The design follows psychophysics best practices and adapts the HYPE∞ evaluation procedure.
  • Results and analysis: FID does not track human fidelity judgments: diffusion models usually have the highest human-perceived fidelity, although GANs often achieve lower FID.Similar mismatches occur for alternatives to FID.
  • Results and analysis: Human error rate is plotted for models sorted by FID, with participant means and unbiased standard errors; a strong FID relationship would appear monotonically increasing.The figure compares perceptual rankings with lower-is-better FID rankings.
  • Results and analysis: Model type significantly affects human error rate on all four datasets, with all ANOVAs yielding F’s > 12.09 and p’s < 0.001.Diffusion models stand out on every dataset except FFHQ, where they perform comparably to other model types.

4 Improved representation spaces for generative evaluation

The study shows that encoder training and architecture strongly shape what generative-evaluation distances perceive. Self-supervised encoders, especially DINOv2, align better with human judgments than standard Inception-based evaluation.

  • 4.1 Qualitative examination of perceptual spaces: DINOv2 focuses on overall image structure while identifying important objects, providing the balance the authors seek in an evaluation encoder.By contrast, CLIP emphasizes a few main objects, while masked-image-modeling methods emphasize textures and shapes.
  • 4.1 Qualitative examination of perceptual spaces: Self-supervised representation spaces are more appropriate for generative evaluation than supervised approaches, with DINOv2 offering the best balance of object focus and holistic structure.The conclusion follows the qualitative and quantitative encoder analyses.
  • 4.2 The (mis)alignment of evaluation metrics and human assessment: FID, FD∞, and KD computed across encoders are highly correlated and therefore produce essentially the same model rankings, despite differing encoder perceptual spaces.The authors interpret this as evidence that these metrics provide similar ranking information across encoders.
  • 4.2 The (mis)alignment of evaluation metrics and human assessment: Inception-space metrics show no strong human correlation beyond CIFAR10, whereas CLIP ViT-L/14, DINOv2, MAE, and usually DreamSim align more closely with human evaluation.DreamSim is an exception on ImageNet, and the stronger alignment is reported for larger, more diverse datasets.
  • 4.2 The (mis)alignment of evaluation metrics and human assessment: FID often prefers GANs over diffusion models, while self-supervised-feature Fréchet distances do not, and this discrepancy also appears on ImageNet.The discrepancy is identified as a major source of misalignment between Inception-based evaluation and human assessment.

5 Alternative explanations: diversity, rarity, and memorization

The study rules out diversity and rarity as explanations for FID’s poor alignment with human judgments, while showing that memorization occurs selectively and is poorly measured by existing metrics.

  • Diversity and rarity: DiT-guided samples have lower per-class semantic diversity than DiT samples, despite higher overall Vendi scores across most encoders.This contrast motivates using per-class rather than overall Vendi scores for class-conditional models.
  • Diversity and rarity: DINOv2 per-class Vendi scores show that diversity differences do not explain discrepancies between FID and human evaluations.Per-class scores measure intra-class semantic diversity more meaningfully than overall Vendi scores for class-conditional ImageNet models.
  • Diversity and rarity: Human evaluators do not confuse image rarity with unrealism, supporting human error rate as a ground truth for image fidelity.Combined with the diversity analysis, this rules out diversity as an alternative explanation for FID’s weak alignment.
  • Memorization: Generative models show strong evidence of exact memorization on CIFAR10, while larger datasets such as ImageNet are not memorized by even the largest studied models.The results indicate that strong DINOv2-space performance is not necessarily achieved through training-data memorization.
  • Memorization: AuthPct, the CT score, and FLS-POG are not reliable memorization metrics because they respond to mode shrinkage or image fidelity more than memorization.Direct collection of memorized images using calibrated l2 distance is recommended instead, although it is labour-intensive and requires tuning.

6 Conclusions

The paper finds that common evaluation metrics misrepresent generative-model fidelity, particularly for diffusion models, and that DINOv2-based evaluation largely addresses this problem while memorization remains difficult to measure.

  • Conclusions: FID is not strongly predictive of human error rate, and diffusion models often receive worse FID rankings despite higher human error rates than GANs.The discrepancy is not explained by model diversity or humans judging rare real images as fake.
  • Conclusions: Replacing Inception-V3 with DINOv2 ViT-L/14 mostly addresses these evaluation deficiencies.The paper releases metrics and generated datasets to support broader evaluation.
  • Conclusions: Except on CIFAR10, the best DINOv2-scored models do not memorize their training data, while current FD-based metrics are not designed to detect memorization.None of the evaluated memorization metrics reliably separates memorization from other effects.

A Generated datasets

The study standardizes generated and reference datasets across CIFAR10, ImageNet, FFHQ, and related model evaluations, documenting preprocessing, checkpoints, and FID discrepancies.

  • Model coverage: The evaluation includes GANs, diffusion models, VAEs, normalizing flows, transformers, and consistency models across the studied datasets.The paper provides model links, checkpoints, reported FIDs, reproduced FIDs, and special notes.
  • Reproducibility: Reported and reproduced FIDs differ for several models, including StyleGAN2-ada on CIFAR10, LSGM-ODE, PFGM++, and FFHQ models.For FFHQ StyleGAN2-ada, the discrepancy is attributed to Lanczos versus box downsampling.
  • Reproducibility: The ImageNet preprocessing protocol is explicitly documented because prior literature used inconsistent approaches to raw ImageNet and FID calculation.The authors release parsing scripts to help standardize the process.
  • ImageNet preprocessing: ImageNet images were center-cropped to squares, downsampled to 256 × 256 with bicubic interpolation, and evaluated against balanced reference samples.The reference batch contained 100,000 training images, while FD used 50,000 real and 50,000 generated images.

B.3 Metrics for memorization

The paper compares several memorization diagnostics, including nearest-neighbor, hypothesis-test, likelihood, and calibrated-distance measures, while emphasizing that calibrated thresholds require tuning.

  • Memorization definition: Memorization is defined as reproducing individual training images at generation time.The paper distinguishes this form of overfitting from broader distributional behavior.
  • Memorization diagnostics: AuthPct reports the fraction of generated samples whose nearest-training distance indicates authenticity relative to training-set nearest-neighbor distances.Each generated sample is classified as authentic or inauthentic.
  • Memorization diagnostics: CT is a three-sample non-parametric test comparing training, test, and generated samples through nearest-cell relationships and a Mann-Whitney hypothesis test.The final score averages per-cell scores after k-means partitioning.
  • Memorization diagnostics: FLS-POG measures the percentage of generated samples with higher training-set than test-set log-likelihood.This operationalizes overfitting through train-test likelihood differences.
  • Limitations: The calibrated-l2 memorization ratio flags generated samples below a threshold, but both k and τ require hand-tuning across models and datasets.Consequently, the authors recommend it for detection but not for comparing models.
  • Encoders: The encoder set includes supervised, self-supervised, multimodal, masked-modeling, and perceptual-similarity approaches.Examples include ConvNeXt, SwAV, SimCLRv2, CLIP, DINOv2, MAE, data2vec, and DreamSim.

B.4.1 Determining ViT model size for generative evaluation

The ViT model-size study compares CLIP and DINOv2 encoders for generative evaluation and finds ViT-L/14 to be a strong quality–compute tradeoff, with DINOv2-L/14 selected for final evaluation.

  • Experimental design: The study evaluates FD and precision for 11 ImageNet generative models across ViT sizes and, for CLIP, across multiple training datasets.DINOv2 comparisons use a common training set across encoder sizes.
  • CLIP: CLIP-B/32 and CLIP-B/16 variants show smaller FD–human-error alignment than ViT-L/14 and ViT-G/14 variants.CLIP-L/14-DataComp-1B best aligns with the strongest CLIP model under linear evaluation.
  • DINOv2: DINOv2 ViT-S/14 has lower FD–human-error alignment than the three larger models, while ViT-B/14, ViT-L/14, and ViT-G/14 agree strongly.ViT-L/14 has the highest correlation with ViT-G/14, with ViT-B/14 also highly correlated.
  • Model-size choice: ViT-L/14 approaches the ImageNet perceptual qualities of the largest encoders at substantially lower computational cost.The authors use ViT-L/14 for ViT-based encoders and suggest DINOv2-B/14 for lower-cost metric tracking during training.
  • Human-evaluation scope: The human experiments evaluate image realism rather than other distributional properties such as diversity.The study focuses on realism for images intended for human use.

C.3 Results

Human evaluations show that diffusion models generally produce the most realistic images, while common metrics and alternative views do not consistently track human judgments.

  • Human evaluation: Human error rate measures the fraction of test images participants incorrectly classify, with lower error indicating easier discrimination from real images.The metric is interpreted as a fidelity measure for generated images.
  • Metric comparisons: Figures 10–12 extend the analysis to recall, precision, density, IS, sFID, and encoder-specific correlations.The appendix reports that additional metrics are not strongly correlated with human evaluation on datasets more complex than CIFAR10.
  • Representation analysis: Grad-CAM visualizations compare which image regions influence FD across CNN and ViT encoders, with additional heatmaps provided for several datasets.For ViT encoders, the analysis uses a Grad-CAM variation designed for ViTs.
  • Representation analysis: Intra-class Vendi scores assess how much semantic and pixel-wise diversity each encoder captures within ImageNet classes.Masked models show diversity patterns more correlated with image-space diversity, indicating greater weight on low-level features.

D.3 FD bias

The section examines finite-sample bias in FD, alternatives based on Wasserstein distance, and whether diversity and rarity analyses reflect human judgments. It finds ASW is more sample-stable but does not correlate well with human perception, while per-class Vendi detects intra-class diversity differences that global Vendi can miss.

  • FD bias: FD∞ reduces FD bias but continues decreasing as the number of samples increases, so finite-sample effects remain.The comparison uses DiT-XL-2 samples and training images across Inception, CLIP, DINOv2, and MAE representations.
  • FD bias: ASW has less dependence on sample size than FD, suggesting that batches of around 10k samples may be sufficient.The comparison uses Inception representations and is shown through relative error as sample size changes.
  • FD bias: ASW does not correlate well with human perception across encoders and does not appear sensitive to mode shrinkage.Despite its convergence and sample-complexity guarantees, ASW was not included in the main text because it is not widely used.
  • Diversity: Per-class Vendi consistently ranks DiT-XL-2 above DiT-XL-2-guided across encoders, indicating higher intra-class diversity for DiT-XL-2.The models are evaluated over individual ImageNet classes using 100 images per class.
  • Diversity: Overall Vendi can give counterintuitive rankings and very low effective-sample counts, so global diversity does not reliably capture local semantic diversity.The analysis notes that some encoders rank DiT-XL-2-guided above DiT-XL-2 overall despite the guided model's lower per-class diversity.
  • Rarity: Small correlations between rarity score and human error on CIFAR-10 and LSUN-Bedroom are attributed to defective or generated-looking real images, not diversity.Removing 6% of the fakest LSUN-Bedroom real images removes the correlation.

D.6 Memorization

This section evaluates memorization in generative models using nearest-neighbor-based checks and visual comparisons. It finds clear exact memorization on CIFAR10, while complex-dataset matches are often reconstructive and difficult to distinguish from ordinary similarity.

  • Memorization checks: More complex models are more likely to memorize on CIFAR10, while less performant models such as ResFLOW and NVAE are less likely.The memorization ratio is reported for 13 generative models using calibrated l2 distance, with dataset-specific nearest-neighbor settings.
  • Memorization checks: DDIM shows exact memorization on CIFAR10, raising concerns about training-example reproduction on that dataset.The exact memorization is illustrated through generated and corresponding training-image pairs.
  • Memorization checks: ImageNet and LSUN matches are semantically closer than alternative training neighbors but are not pixel-wise identical, complicating claims of memorization.Some generated samples are not substantially closer to the matched training image than that image's nearest neighbors.
  • Memorization checks: DINOv2 nearest-neighbor matches on complex datasets remain ambiguous because they often occur in classes with very low intra-class diversity.The authors report that exact memorization remains unclear in these cases.

D.6.1 Controlled experiment for memorization metrics

The controlled experiment gradually increases memorization by conditioning a hierarchical VAE on deeper portions of training-image posteriors. CT and AuthPct track this progression more clearly than FLS-POG.

  • Controlled experiment: Increasing the conditioning depth increases mutual information with the training image and produces a more faithful reconstruction.The procedure uses the approximate posterior for early latent variables and the prior for later variables; depth 0 uses no individual-image conditioning.
  • Controlled experiment: 30,000 samples are generated at depths 0, 1, 3, 5, 9, 15, 25, and 41 to test memorization metrics under controlled conditions.The experiment uses VDVAE with an FFHQ-256 pretrained model and evaluates AuthPct, CT, and FLS-POG.
  • Metric behavior: At depths below 15, CT is positive, whereas above 15 it becomes increasingly negative, indicating stronger detected memorization.CT compares distances to generated samples against distances to test-set samples.
  • Metric behavior: AuthPct falls from nearly 100% to 20% at depth 41, indicating that generated samples become closer to training samples than expected.Its decline is nearly equal in rate to the change in CT across depths.
  • Metric behavior: FLS-POG increases in the correct direction but changes only minimally and does not reflect the strong memorization visible at depth 41.At depth 41, training samples are almost exactly reproduced in the generated images.

D.6.2 Memorization metric quality

The section tests whether memorization metrics behave consistently across encoder spaces using CIFAR10 model results. Their trends vary substantially by encoder and metric.

  • Metric quality: Inception shows the expected association between higher memorization ratios and lower AuthPct and CT scores, but not for FLS-POG or modified CT.This pattern is specific to the encoder-level analysis of CIFAR10 results.
  • Metric quality: SwAV provides the correct directionality for all evaluated memorization metrics, while CLIP shows mixed or absent trends.CLIP is accurate for modified CT, slightly accurate for CT, and lacks discernible trends for AuthPct and FLS-POG.

D.6.3 Low-dimensional experiments for memorization metrics

The low-dimensional experiment tests whether memorization metrics distinguish memorization from mode shrinkage and underfitting. It shows that CT and AuthPct can misclassify these phenomena, while the modified CT produces the desired behavior in the tested mode-shrinkage case.

  • Experimental design: The experiment samples 1,000 training and 1,000 test points from a five-component Gaussian mixture, then evaluates metrics on 1,000 generated points.Generated datasets represent a perfect model, total mode shrinkage, perfect training-data memorization, and three underfit models with standard deviations of 1.5σ, 3σ, and 4.5σ.
  • Metric behavior: FLS-POG gives the memorized model its highest score at 80%, but scores both the true distribution and mode shrinkage around 60%.Its approximately 5% change from underfit to true distributions is comparable to the change in a VDVAE controlled experiment.
  • Metric behavior: CT incorrectly flags mode shrinkage as memorization, assigning it a score of −16.14, because generated points can lie closer to training points than test points do.The metric compares generated-to-training distances with test-to-training distances and reports highly negative scores when the former are often lower.
  • Metric behavior: The modified CT reverses the relevant argument order, CT,M(T1, G, T2) ≡ CT(G, T1, T2), and does not detect the mode-shrinkage dataset as memorized.This change is presented as producing the desired behavior in the synthetic experiment.
  • Interpretation: AuthPct incorrectly flags well-trained models, while CT incorrectly flags mode shrinkage; these findings help explain why both metrics can be nearly invariant to the training set.The comparison includes models not derived from the test set, except for the memorized model.
Loading 2306.04675v2…