PaperlayerView exact revision on arXiv

Source-linked AI summary

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, J. Eric T. Taylor, Gabriel Loaiza-Ganem

arXiv:2306.04675v2cs.LGcs.CVstat.ML

TL;DR

It remains unclear whether commonly used metrics adequately measure how well modern generative models learn image distributions. The paper evaluates human judgments and 17 metrics across generative models, finding that prevalent metrics such as FID poorly reflect human assessments and often rank diffusion models worse.

  • Problem

    It remains unclear whether current evaluation tools adequately measure how well modern generative models learn the ground-truth image distribution.

    Source: §1 · p. 1

  • Method

    The authors conduct large-scale psychophysics-based human evaluations and compare them with 17 metrics and alternative feature extractors across diverse generative models.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Prevalent metrics such as FID poorly predict human error rates and often rank diffusion models worse despite their higher human error rates.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Replacing Inception-V3 with DINOv2 ViT-L/14 mostly addresses these metric deficiencies and provides richer generative-model evaluation.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.

1 Introduction

The paper argues that standard generative-image metrics do not reliably reflect human judgments of realism. Its study finds that diffusion models are judged more realistic than GANs but rank worse on Inception-V3 metrics, and identifies DINOv2-ViT-L/14 as a richer evaluation alternative.

Study scope

  • The study compares 41 state-of-the-art generative models and 4.1M images, addressing fidelity, diversity, and whether samples are novel or memorized.An ideal model should produce high-fidelity, diverse samples without reproducing training examples.Source: §1 · p. 2

Human evaluation

  • Human evaluators judged diffusion models more indistinguishable from training data than GANs and other techniques, yet Inception-V3 metrics consistently ranked them worse than GANs.Diffusion models’ diversity also more closely resembled the training data.Source: §1 · pp. 2–3

Human evaluation; Self-supervised representations and evaluation metrics

  • No current metric strongly correlates with human evaluators; Inception-V3 Fréchet distance, kernel distance, precision, and density also correlate poorly.Common evaluation requires image representations followed by distances between real and generated samples, and Inception-V3’s ImageNet-classification features have been criticized as limited and texture-biased.Source: §1 · p. 2

Self-supervised representations and evaluation metrics

  • DINOv2 ViT-L/14 resolves the discrepancy with human evaluators, while SwAV and CLIP-B/32 replacements are sub-optimal; representation quality depends on training procedure and architecture.Supervised networks do not provide a perceptual space that generalizes well for image evaluation.Source: §1 · p. 3

Diversity, rarity, and memorization

  • The gap between human judgments and FID was not due to fidelity-diversity tradeoffs or humans treating rare images as fake.The authors argue human error rate is a sensible ground truth for aligning Fréchet-distance metrics.Source: §1 · p. 3
  • The study finds clear evidence of memorized samples across models, particularly on CIFAR10, while current memorization metrics and tests fail to capture them.The authors report that existing approaches do not properly detect memorization.Source: §1 · p. 3

2 Datasets, metrics, and encoders

The section establishes a broad evaluation suite spanning 41 generated datasets, 17 metrics, and multiple representation spaces. It describes the dataset controls, metric categories, and encoder choices used to compare generative models.

Generated datasets

  • The study evaluates 41 generated datasets across four image datasets and 100,000 generated images per model, using models trained without external data.The benchmark spans CIFAR10, ImageNet1k, FFHQ, and LSUN-Bedroom, with 13, 11, 9, and 8 models respectively; each model is assessed with 17 metrics.Source: §2 · p. 3

Metrics for ranking generative models

  • Ranking metrics include FID and alternatives such as sFID, FID∞, IS, KID, and FLS, with Fréchet distance also measured in alternative feature spaces.FID [48] compares 50,000 real and generated samples in Inception-V3 representations; the alternative metrics are sFID [76], FID∞[22], IS [96], KID, and FLS [53].Source: §2 · p. 3

Metrics for diagnosing fidelity, diversity, rarity, or memorization

  • Fidelity, diversity, rarity, and memorization diagnostics combine human error rate, precision, density, recall, coverage, rarity score, and Vendi score.They also directly check pixel-wise memorization and include percentage of authentic samples [1], CT score [70], and FLS’s percentage of overfit Gaussians [53]; per-class recall and coverage are difficult to estimate robustly with few samples per class.Source: §2 · p. 3

Representation spaces for generative evaluation

  • Representation spaces include Inception-V3, ConvNeXt-large, seven self-supervised methods, and DreamSim, enabling study of how training objectives and augmentations affect evaluation features.The self-supervised methods span contrastive, self-distillation, canonical correlation analysis, masked image modelling, and language-image learning; DreamSim ensembles three models fine-tuned on human image-similarity judgments.Source: §2 · pp. 3–4
  • For CNNs the study uses ResNet50, and for vision transformers it uses ViT-L/14, selecting weights with the highest ImageNet linear-classification performance.ViT-L/14 was chosen as a tradeoff between representation quality and computation cost.Source: §2 · p. 4

3 Human evaluation of generated data

The section establishes a large-scale human baseline for image realism and finds that diffusion models generally look most realistic to people, despite being downranked by FID.

Human evaluation of generated data

  • The study established a large-scale realism baseline with over 1,000 paid participants and 207k responses, statistically separating most tested models per dataset; it did not assess diversity or memorization.The experiments focused on image realism, which the authors treat as a property for which humans can provide ground truth.Source: §3 · p. 4

Experimental design

  • Across 41 models and four datasets, participants completed two-alternative forced-choice real-or-fake trials; human error rate measured the fraction of images misclassified.A participant judged either a generated image or a training-set image as real or fake.Source: §3 · p. 4

Human evaluation of generated data

  • The authors criticize a smaller concurrent benchmark [114], arguing that photo-realism and image-set judgments were more ambiguous and bias-prone than their forced-choice assessment.The concurrent study had ten times fewer participants; the authors also note observers lacked training and knowledge of the training set.Source: §3 · p. 4

Results and analysis

  • Across datasets, diffusion models were most realistic to humans except on FFHQ, yet FID downranked them; Figure 2 plots human error rates against FID-ranked models.Diffusion models performed on par with other model types on FFHQ.Source: §3 · pp. 4–5, Figure 2 · p. 5

4 Improved representation spaces for generative evaluation

Human alignment depends on the representation space: self-supervised encoders outperform Inception on complex datasets, while DINOv2 balances object focus with holistic image structure. FID can favor GANs over diffusion models even when humans and self-supervised encoders do not.

4.1 Qualitative examination of perceptual spaces

  • Self-distilled DINOv2 offers the best balance of important-object focus and holistic image structure across the examined representation spaces.Inception and ConvNeXt can focus on a single object rather than the whole image; CLIP emphasizes main objects, while masked models MAE and data2vec emphasize textures and shapes over object semantics.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 The (mis)alignment of evaluation metrics and human assessment

  • With an appropriate encoder, FD provides a strong evaluation approach, while FD, FD∞, and KD produce highly correlated model rankings.Large precision values do not consistently correspond to high human error rates, so the authors prefer FD over precision.Source: §4.2 · pp. 6–7
  • CLIP ViT-L/14, DINOv2, and MAE align far better with human judgments than Inception; DreamSim also aligns except on ImageNet.Inception metrics show no strong human-evaluation correlation outside CIFAR10. Figure 4 compares FD and precision across encoders and datasets.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
  • FID often prefers GANs over diffusion models, whereas FD from self-supervised encoders favors diffusion models, including on ImageNet.The paper identifies diffusion models as often driving Inception’s discrepancy with human evaluation.Source: Figure 4 · p. 7, §4.2 · p. 7

5 Alternative explanations: diversity, rarity, and memorization

This section tests whether diversity, image rarity, or memorization explains gaps between human judgments and generative-model metrics. It finds that diversity and rarity do not explain FID’s misalignment with human fidelity judgments, and that current memorization metrics are unreliable in practice.

5.1 Diversity and rarity

  • Diversity differences do not explain FID–human disagreement: GigaGAN’s diversity is farther from the training data than most diffusion models yet it has better FID, while FDDINOv2 aligns better with human evaluations.FDDINOv2 also reflects some diversity differences, such as DiT-guided scoring worse than DiT despite better human error rate, while still emphasizing fidelity more than diversity.Source: Figure 5 · p. 8, §5.1 · p. 8
  • Per-class Vendi scores capture intra-class semantic diversity more meaningfully than global scores, which chiefly measure inter-class diversity and can rate DiT-guided as more diverse despite its lower per-class scores.The study therefore uses per-class Vendi scores for class-conditional models such as those commonly evaluated on ImageNet.Source: §5.1 · p. 8
  • Human error on real images was not correlated with image rarity, so evaluators were not mistaking unlikeliness for unrealism and human error remains a sensible ground truth for fidelity.Together with the diversity analysis, this supports alignment between FD metrics and human error rate as a desirable property for the studied models.Source: §5.1 · pp. 8–9

5.2 Memorization

  • Less than 0.5% of DiT-XL-2 images showed close reconstructive memory; CIFAR10 models showed exact memory, while complex datasets lacked conclusive pixel-wise matches despite reconstructive examples on ImageNet and LSUN-Bedroom.The study was not designed to detect copy-paste combinations of aspects from different training images, so it cannot rule out every form of memorization.Source: §5.2 · p. 9, Figure 7 · p. 10
  • Although AuthPct, the CT score, and FLS-POG respond to memorization in controlled tests, none reliably measures it in practice; the authors recommend direct searches using calibrated l2 distance.CT and AuthPct track mode shrinkage and image fidelity more than memorization, while FLS-POG correlates poorly with estimated memorization ratios.Source: §5.2 · pp. 9–10, Figure 7 · p. 10

6 Conclusions

The assessment finds that prevalent metrics such as FID do not strongly predict human judgments and often rank diffusion models worse despite their higher human error rates; replacing Inception with DINOv2 ViT-L/14 mostly addresses these deficiencies. FD-based metrics do not detect memorization, and the authors recommend reporting calibrated-l2 memorization rates alongside DINOv2 metrics.

6 Conclusions

  • FID is not strongly predictive of human error rates, and diffusion models have higher human error rates than GANs yet often rank worse by FID; diversity and rare-real-image judgments do not explain this mismatch.The authors conclude that this discrepancy unfairly penalizes diffusion models when assessing fidelity.Source: §6 · p. 10
  • Replacing the Inception encoder with DINOv2 ViT-L/14 mostly addresses the identified metric deficiencies, and the authors provide FDDINOv2 and other metric scores for an updated model leaderboard.The table covers all models considered and appears in Appendix E.Source: §6 · p. 10
  • FD-based metrics are not designed to detect memorization; except on CIFAR10, the best-performing models by FDDINOv2 do not memorize their training data.Calibrated l2 distance [13] reliably identifies memorized samples, while other memorization metrics are not ideal.Source: §6 · p. 10
  • The authors recommend reporting memorized-sample ratios using calibrated l2 alongside DINOv2 ViT-L/14 metrics and identify automated, reliable memorization detection as future work.Source: §6 · p. 10

A Generated datasets … rarity

The appendix documents how generated datasets were assembled and standardized, then describes a 17-metric, 9-encoder evaluation framework spanning model ranking, fidelity, diversity, rarity, and memorization. It also specifies ImageNet curation and metric choices needed to reproduce or interpret the evaluations.

A Generated datasets; A.1 CIFAR10

  • Across datasets, the study generally generated 100,000 images per model, selected checkpoints and settings for the lowest reported FID, and checked reproduced FIDs against published values.For class-conditional models, sampling was balanced across classes; the CIFAR10 model list includes reported-versus-reproduced FID comparisons.Source: A Generated datasets · p. 17, §A.1 · p. 18

A.2 ImageNet

  • On ImageNet, DiT-XL-2-guided used classifier-free guidance cfg = 4.0 rather than 1.5 to study realistic images sampled with lower intra-class diversity.Its reproduced FID was 17.24; the passage reports no published FID for this configuration.Source: §A.2 · p. 18

A.2.1 A note on the curation of ImageNet

  • ImageNet curation center-crops each raw image to a square, downsamples it to 256 × 256 with bicubic interpolation, and compares 50,000 generated images against 50,000 sampled training images.The resulting reference batch has 100,000 images, with 100 sampled per class; this procedure slightly raises most reported FIDs but does not change model rankings.Source: §A.2.1 · p. 19

A.3 FFHQ

  • For FFHQ, downsampling choices can explain FID discrepancies: Lanczos produced 5.30 for StyleGAN2-ada, whereas box filtering reproduces the reported 4.30.The authors attribute the difference to Lanczos versus box filtering when resizing the original 1024 × 1024 images to 256 × 256.Source: §A.3 · pp. 19–20

A.4 LSUN-Bedroom

  • For LSUN-Bedroom, four models used publicly available 50,000-image sets, and the appendix lists reproduced FIDs alongside reported values for the evaluated models.Source: §A.4 · p. 20

B Metrics and encoders

  • The evaluation uses 17 ranking and diagnostic metrics across 9 supervised and self-supervised encoders; diagnostic measures target fidelity, diversity, rarity, or memorization.The authors caution that diagnostic metrics are useful for diagnosis but difficult to use for ranking because ranking requires combining concepts with tradeoffs.Source: B Metrics and encoders · p. 20

B.1 Metrics for ranking generative models

  • Ranking metrics include FD and FD∞, sFID, KD, IS, and FLS, addressing Gaussian moment distances, finite-sample bias, intermediate Inception features, distribution distances, classification predictions, and feature-space density estimation.Source: §B.1 · p. 21

B.2 Metrics for fidelity, diversity, and rarity; rarity

  • Precision and density proxy fidelity, while recall and coverage proxy diversity; the Vendi score measures overall sample diversity or, when class-conditioned, intra-class diversity.The authors use Vendi to examine mode concentration because class-conditional models can have high inter-class diversity while lacking intra-class diversity.Source: §B.2 · p. 22, rarity · p. 22

B.3 Metrics for memorization · B.4 Encoders

These sections define several diagnostics for training-image memorization and document the encoder implementations used to evaluate generative models.

B.3 Metrics for memorization

  • Memorization means a model emits individual images from its training data; AuthPct [1] reports the fraction of generated samples judged authentic by a nearest-neighbour test.A sample is inauthentic when its distance to the nearest training point is less than that training point’s distance to its nearest training neighbour.Source: §B.3 · pp. 22–23
  • CT [70] tests whether training samples are closer to generated samples than to test samples, averaging per-cell Mann–Whitney scores after k-means clustering.The default implementation uses k = 3 and 64-component PCA preprocessing.Source: §B.3 · p. 23
  • FLS-POG [53] reports the percentage of generated samples with higher log-likelihood under the training set than under the test set.The paper uses the default FLS implementation.Source: §B.3 · p. 23
  • The calibrated l2 memorization ratio [101] counts generated samples whose pixel-space calibrated distance falls below a threshold, but its k and τ require hand-tuning across models and datasets.The authors recommend it for detecting memorization, not comparing models, because no default configuration meaningfully flags memorization across all settings.Source: §B.3 · p. 23

B.4 Encoders

  • The encoder study seeks a more generalized perceptual representation space, including ConvNeXt-large trained on ImageNet22k, SwAV ResNet50 [14], and ImageNet1k-trained SimCLRv2 [19].SwAV clusters images while enforcing consistency across views; SimCLRv2 encourages similarity between augmented views and discourages similarity between distinct training images.Source: §B.4 · pp. 23–24
  • The study uses OpenCLIP ViT-L/14 trained on DataComp-1B [35], noting that CLIP-B/32 was a sub-optimal choice for generative evaluation.CLIP is a multimodal model trained to predict image–text pairings.Source: §B.4 · p. 24
  • The encoder set includes ImageNet1k-trained MAE ViT-L [44] and DINOv2 ViT-L/14 [80], whose training combines self-distillation with a masked latent-space objective.DINOv2 was trained on a 142M-image custom dataset; its teacher network provides targets for the masked latent representations.Source: §B.4 · p. 24
  • DreamSim [34] fine-tunes self-supervised encoders on human image-similarity judgments, whereas this work seeks representations whose distribution distances correlate with human fidelity judgments.The experiments use the full DreamSim ensemble, concatenating fine-tuned DINO, CLIP, and OpenCLIP representations.Source: §B.4 · p. 24

B.4.1 Determining ViT model size for generative evaluation … C.2 Comparison to HYPE∞

The authors recommend ViT-L/14 as a quality–compute tradeoff for generative evaluation, while describing an absolute human evaluation protocol designed to assess generated-image realism. Their experiments used Prolific participants and made several design changes relative to HYPE∞.

B.4.1 Determining ViT model size for generative evaluation

  • CLIP-L/14-DataComp-1B and DINOv2-L/14 aligned closely with their respective largest models; ViT-L/14 offered a lower-cost tradeoff and was used for ViT-based evaluation.For DINOv2, ViT-L/14 had the highest correlation with ViT-G/14, though ViT-B/14 was also highly correlated; the authors suggest DINOv2-B/14 for tracking during training and DINOv2-L/14 for final leaderboard values.Source: Figure 8 · p. 25, §B.4.1 · p. 25
  • On a Titan V, representing 50k samples took ∼4 minutes with ViT-B/14, ∼15 minutes with ViT-L/14, and ∼85 minutes with ViT-G/14.The authors note that evaluation metrics often need to be tracked across training setups, hyperparameters, and epochs, motivating lower-cost encoders.Source: §B.4.1 · p. 25

C.1 Test description

  • The human experiments targeted generated-image realism, not diversity, and compared each model’s samples directly against the training dataset as a consistent absolute baseline.The authors describe absolute comparisons as scaling linearly with models tested and allowing new models to be evaluated independently of earlier ones.Source: §C.1 · pp. 25–26
  • Each experiment included 40 practice trials and 200 test trials in which participants judged single images as real or generated, with feedback after each response.The authors used a long training phase and feedback to help participants make accurate judgments; preliminary tests found no significant difference between error rates in the first and second test halves.Source: Figure 9 · p. 26, §C.1 · p. 27
  • Participants were recruited through Prolific, required to be fluent in English and have at least a bachelor’s degree, and were paid a median £7.60 per hour.Each experiment took a median of 15 minutes; no other participant filters were used.Source: §C.1 · p. 27
  • Across 41 experiments, 1,036 participants provided over 207k test responses, making this the largest human study of generated-image quality the authors knew of.Each experiment had at least 25 unique recruits; practice responses were not included in the reported test-response total.Source: §C.1 · p. 28
  • For each participant, 100 real and 100 generated images were sampled from pools of 2,000 each, with class distributions respected where applicable.Training and generated images were sampled separately for each experiment, and the selected images were presented in randomized order.Source: §C.1 · p. 28

C.2 Comparison to HYPE∞

  • Compared with HYPE∞, the study doubled images shown per participant, used Prolific instead of MTurk, and omitted a mixed-model prescreening test to preserve absolute comparisons.The authors also note that Prolific required fair base pay based on time, so participants could not be compensated solely for correct answers.Source: §C.2 · p. 29

C.3 Results … D.2.2 Quantitative representation spaces

The supplemental results describe human error-rate evaluation across 41 models and probe how metric correlations and representation spaces shape generative-model evaluation. They find distinct perceptual and diversity properties across encoders, linked to differences in training objectives and semantic information.

C.3 Results

  • Human error rate measures the fraction of 200 test images incorrectly classified, and error on real images rises for models with higher fidelity.The evaluation covered 41 models; the authors interpret higher human error as consistent with images being harder to distinguish from real ones.Source: §C.3 · p. 29

D.1 Additional scatter plots and correlations

  • Figure 10 shows Fréchet distance and human error rate across encoders and datasets, with points colored by recall, precision, and density; Figures 11–12 show additional metric correlations.Shaded Figure 10 panels indicate correlations that are not both strong (|r| ≥ 0.5) and significant (p ≤ 0.05).Source: §D.1 · p. 31, Figure 10 · p. 32, Figure 11 · p. 33

D.2.1 Experimental details on GradCAM and additional heatmaps

  • The FD heatmaps use Grad-CAM on an image added to 49,999 generated images alongside 50,000 real-image representations; ViT encoders use a Grad-CAM variation.Heatmap signs for CLIP, DINOv2, ConvNeXt, and SimCLRv2 were flipped for color consistency; additional heatmaps cover ImageNet, FFHQ, and LSUN-Bedroom using the real dataset as the generated set.Source: §D.2.1 · p. 31, Table 8 · p. 31, Figure 13 · p. 34

D.2.2 Quantitative representation spaces

  • The masked encoders MAE and data2vec score highest on BAPPS perceptual alignment, while SimCLR ranks lowest, reflecting differences in self-supervised training objectives.The authors note that jitter-like distortions may preserve semantics and should not necessarily change representations, though they form only a minor part of the dataset.Source: §D.2.2 · p. 31, Figure 15 · p. 36
  • On datasets more complex than CIFAR10, additional Inception-V3 metrics including IS and sFID are not strongly correlated with human evaluation.The figure notes that some metrics are missing when a dataset lacks a validation or test set.Source: Figure 12 · p. 33
  • MAE and data2vec diversity aligns with image-space diversity, whereas ConvNeXt, CLIP, SwAV, and DINOv2 align with Inception but less with image space.The authors interpret the first pattern as greater emphasis on low-level image features, and the second as more object-based semantic information; CLIP has the least pixel-space and greatest Inception-space alignment among ViTs.Source: §D.2.2 · p. 36, Figure 15 · p. 36

D.3 FD bias · D.4 Diversity · D.5 Rarity

FD∞ reduces but does not eliminate finite-sample bias, while ASW’s sample stability does not translate into strong alignment with human perception. Per-class Vendi scores capture guided-model diversity loss, and rarity does not explain human error rates except where rare, defective real images create small dataset-specific correlations.

D.3 FD bias

  • FD∞ has less bias than FD but does not eliminate it; self-supervised encoders are less biased than Inception, with DINOv2 at 20k samples matching Inception’s relative error at 50k.The authors nevertheless retain 50k samples for FD across encoders for consistency with FID.Source: §D.3 · pp. 36–37, Figure 16 · p. 37
  • ASW depends less on sample size than Inception FD, suggesting batches around 10k may suffice, but it correlates poorly with human perception and is not affected by mode shrinkage.Despite its theoretical guarantees, the authors exclude ASW from the main text and recommend using FD as-is, hypothesizing that similar bias across datasets may preserve model rankings.Source: Figure 16 · p. 37, §D.3 · pp. 37–38
  • ASW approximates sliced Wasserstein with convergence and sample-complexity guarantees, but FD’s strong correlation with KD suggests its two-moment and sample-complexity limitations are not borne out in practice.The authors hypothesize that encodings are approximately Gaussian and that similar FD bias across generated datasets would not affect model rankings.Source: §D.3 · pp. 37–38

D.4 Diversity

  • Per-class Vendi scores ranked DiT-XL-2 above DiT-XL-2-guided for every encoder, indicating greater intra-class diversity in the unguided model.This result motivated the authors’ use of per-class Vendi for diversity analysis.Source: Figure 17 · p. 38, §D.4 · p. 38
  • Overall Vendi scores did not consistently rank DiT-XL-2 above DiT-XL-2-guided across encoders, supporting DINOv2 as a choice for generative evaluation.Only DINOv2, SimCLRv2, and ConvNext favored DiT-XL-2; SimCLRv2 showed minimal differences across ImageNet train and both generated sets, and SimCLRv2 and ConvNext had FD rankings misaligned with human perception.Source: §D.4 · p. 38, Table 10 · p. 39

D.5 Rarity

  • The rarity analysis validates that human error rates measure image fidelity rather than image unlikeliness, since dataset-specific correlations arose from humans correctly identifying generated-scene or low-quality real images.The small proportion of rare defects affects all model evaluations similarly and does not change their rankings.Source: §D.5 · pp. 39–40
  • Human error rate showed no correlation with rarity score on ImageNet or FFHQ, while small significant correlations on CIFAR-10 and LSUN-Bedroom stemmed from defective real images.Removing 6% of the “fakest” LSUN-Bedroom real images eliminated the correlation; these rare defects barely affected average error rates or model rankings.Source: Table 11 · p. 39, Figure 18 · p. 40, §D.5 · p. 40

D.6 Memorization

The appendix finds evidence of exact memorization on CIFAR10, while matches on more complex datasets are sparse and ambiguous. It also cautions that current metrics may misidentify reconstructive memory as memorization.

Memorization check with calibrated l2 distance

  • DDIM showed exact CIFAR10 memorization, but sparse matches on ImageNet and LSUN were semantic, potentially metric failures or generalization.For some matches, generated images were not much closer to the identified training sample than that sample’s nearest neighbors; the authors therefore withheld a total count and recommend continued monitoring as models grow more complex.Source: §D.6 · p. 41
  • DINOv2-space searches found well-matched examples on complex datasets, but exact memorization remained unclear, especially in classes with low intra-class diversity.The results qualitatively resembled pixel-wise matches, and training samples in these classes were extremely similar to one another.Source: §D.6 · p. 41
  • More complex models were more likely to memorize on CIFAR10, while less-performing ResFLOW and NVAE were less likely.Figure 19 reports memorization ratios for 13 models; the analysis used k = 50 for CIFAR10, FFHQ, and LSUN, and k = 3 with intra-class search for ImageNet.Source: Figure 19 · p. 41, §D.6 · p. 41

D.6.1 Controlled experiment for memorization metrics

In a controlled VDVAE experiment, increasing posterior depth made samples more faithful to training images. CT and AuthPct detected memorization, while FLS-POG did not flag it strongly enough.

D.6.1 Controlled experiment for memorization metrics

  • At depth 41, AuthPct fell from close to 100% to 20%, while CT became increasingly negative above depth 15; FLS-POG rose only minimally.Without mode collapse or mode shrinkage, CT and AuthPct flagged memorization in this experiment, but FLS-POG did not flag it strongly enough.Source: Figure 23 · p. 44, §D.6.1 · p. 45
  • Using a 62-layer pretrained VDVAE [21], deeper approximate-posterior conditioning increased mutual information with the training image and produced more faithful reconstructions.Depth 0 samples from the prior without conditioning; depth 62 corresponds to a close reconstruction of the original image.Source: §D.6.1 · pp. 42, 44
  • The experiment generated 30,000 samples at depths 0, 1, 3, 5, 9, 15, 25, and 41, using 63,000 training and 7,000 test images.AuthPct, CT, and FLS-POG were evaluated at each depth.Source: §D.6.1 · p. 44

D.6.2 Memorization metric quality

Memorization-metric performance varied substantially across encoders. Overall, modified CT performed best, followed by CT, AuthPct, and FLS-POG.

D.6.2 Memorization metric quality

  • Modified CT performed best, followed by CT, AuthPct, and FLS-POG, which was the weakest memorization detector.This ranking was consistent with the earlier experiment showing FLS-POG was the weakest detector.Source: §D.6.2 · p. 45
  • SwAV showed the correct directionality for all four metrics, while other encoders had mixed trends; Inception and DINOv2 showed no discernible or strong trends for some metrics.Inception’s trends were discernible for AuthPct and CT but not FLS-POG or modified CT; DINOv2 showed a correct correlation for modified CT but no strong trends for the others.Source: §D.6.2 · p. 45

D.6.3 Low-dimensional experiments for memorization metrics · D.7 Model samples · E FDDINOv2 scores leaderboard

In a 2D toy experiment, existing memorization metrics confuse memorization with mode shrinkage or well-trained models, while a modified CT score avoids flagging mode shrinkage. The block also provides model-sample galleries and a DINOv2-ViT-L/14 leaderboard.

D.6.3 Low-dimensional experiments for memorization metrics

  • FLS-POG gives the memorized model 80% and other models around 60%, while AuthPct flags the perfect, mode-shrunken, and slightly underfit models as memorized.FLS-POG’s similar scores for true and mode-shrinkage samples fail to flag mode shrinkage; AuthPct incorrectly flags models producing samples on the true manifold.Source: Table 12 · p. 45, §D.6.3 · p. 47
  • The toy test uses a mixture of five 2D Gaussians and compares true-distribution, mode-shrinkage, memorized, and three upscaled-standard-deviation underfit models.The experiment draws 1,000 training and 1,000 test points; its motivating concern is that metric scores against test data resemble those against training data.Source: §D.6.3 · pp. 45–47
  • CT assigns mode shrinkage a −16.14 score and incorrectly flags memorization; modified CT swaps the generated and training data roles and no longer detects mode shrinkage as memorized.The original CT is asymmetric because the training set is used in both distance comparisons, while the test set and generated dataset are each used once.Source: §D.6.3 · pp. 47–48

D.7 Model samples

  • Figures 26–29 show samples from 41 generative models across ImageNet, CIFAR10, LSUN Bedroom, and FFHQ, with rows sorted by human-experiment error rate.The ImageNet figure covers great white shark, English Springer, comic book, and volcano classes; CIFAR10 shows selected classes for class-conditional models.Source: §D.7 · p. 48, Figure 26 · p. 48, Figure 27 · p. 49

E FDDINOv2 scores leaderboard

Related papers

Loading 2306.04675v2…