PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study on the Evaluation of Generative Models

Eyal Betzalel, Coby Penso, Aviv Navon, Ethan Fetaya

arXiv:2206.10935v1cs.LGcs.CV

TL;DR

Implicit generative models are difficult to evaluate because they lack likelihood values and commonly used metrics such as IS and FID have unclear probabilistic meaning. The paper builds a likelihood-evaluable synthetic benchmark to compare these metrics with classical divergences and studies their feature representations. It finds that FID and IS capture important trends but can be volatile for close-model rankings, while CLIP features outperform Inception features on non-ImageNet datasets.

  • Problem

    Evaluating implicit generative models is challenging because they do not provide likelihood values, while the meaning and usefulness of common IS and FID scores remain unclear.

    Source: §1 · p. 1

  • Method

    The paper creates a high-quality synthetic Image-GPT dataset with exact likelihood evaluation, compares empirical metrics with KL and reverse KL, and studies Inception versus CLIP features.

    Source: §1 · p. 2

  • Results

    FID and IS correlate with probabilistic divergences but are volatile for fine-grained rankings, while CLIP features outperform Inception features on non-ImageNet datasets.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors recommend avoiding IS, using FID∞ and multiple metrics, replacing Inception with CLIP in FID, and using NotImageNet32 as a generative-model test-bed.

    Source: §6 · pp. 8–9

    The synthetic benchmark applies only to likelihood-based models, and KL and reverse KL cannot capture all generative-model complexities.

    Source: §3 · p. 4

Abstract

from arXiv · show

Implicit generative models, which do not return likelihood values, such as generative adversarial networks and diffusion models, have become prevalent in recent years. While it is true that these models have shown remarkable results, evaluating their performance is challenging. This issue is of vital importance to push research forward and identify meaningful gains from random noise. Currently, heuristic metrics such as the Inception score (IS) and Frechet Inception Distance (FID) are the most common evaluation metrics, but what they measure is not entirely clear. Additionally, there are questions regarding how meaningful their score actually is. In this work, we study the evaluation metrics of generative models by generating a high-quality synthetic dataset on which we can estimate classical metrics for comparison. Our study shows that while FID and IS do correlate to several f-divergences, their ranking of close models can vary considerably making them problematic when used for fain-grained comparison. We further used this experimental setting to study which evaluation metric best correlates with our probabilistic metrics. Lastly, we look into the base features used for metrics such as FID.

1 Introduction

The paper studies how well common generative-model metrics reflect probabilistic differences and whether their feature representations transfer beyond ImageNet. Using a likelihood-evaluable synthetic benchmark, it finds that FID and IS correlate with divergences but can be volatile for fine-grained comparisons.

Motivation

  • Standard metrics such as IS and FID are widely used, but their relation to classical probabilistic metrics, transfer across image domains, and implementation sensitivity remain unclear.These limitations motivate a controlled evaluation framework rather than relying only on visual inspection or heuristic scores.Source: §1 · p. 1

Alternative evaluation

  • Human evaluation directly measures judgments relevant to applications but is costly, time-consuming, and does not measure diversity.A model producing a single good output can receive a good human-evaluation score.Source: §1 · p. 2

Method

  • The study creates a high-quality synthetic dataset from Image-GPT on which exact likelihood values and classical divergences can be estimated.The dataset is trained on natural ImageNet images and is intended as a test-bed for generative-model evaluation.Source: §1 · p. 2

Results

  • FID and IS correlate with KL and reverse KL divergences but are much more volatile, making them potentially unsuitable for fine-grained model comparisons.The study compares empirical metrics with probabilistic divergences on likelihood models trained using the synthetic test-bed.Source: §1 · p. 2

Feature representations

  • On non-ImageNet datasets, CLIP features are better suited than Inception features for FID according to quantitative and qualitative comparisons.The comparison targets datasets such as human faces and bedrooms, which differ substantially from ImageNet.Source: §1 · p. 2

2 Background

The background introduces KL divergences and feature-based metrics used to evaluate generative models. It also reviews variants addressing Gaussian assumptions, sample bias, preprocessing, and feature-domain mismatch.

2.1 KL-Divergence

  • KL divergence favors inclusive models, whereas reverse KL favors exclusive models, so the two divergences capture complementary aspects of distributional behavior.KL can be estimated from samples up to a constant, while reverse KL cannot be directly optimized without access to the data distribution.Source: §2.1 · p. 2

2.2 Inception Score

  • Inception Score evaluates ImageNet-pretrained classifier outputs, rewarding diverse class marginals and confident per-image predictions.Higher IS results when the marginal class distribution is uniform and conditional predictions have low entropy.Source: §2.2 · p. 2

2.3 Fréchet Inception Distance

  • FID fits Gaussians to Inception pool3 features for real and generated images and computes their Fréchet distance.Its quality depends on both the informativeness of Inception features and the reasonableness of the Gaussian assumption.Source: §2.3 · p. 3

2.4 Kernel Inception Distance

  • KID relaxes FID’s Gaussian assumption by measuring squared MMD with a polynomial kernel on Inception representations.Because it is non-parametric, KID also requires fewer samples than FID’s covariance estimation.Source: §2.4 · p. 3

2.5 FID∞& IS∞

  • FID∞ and IS∞ address model-dependent estimation bias, while Clean FID addresses score variation caused by inconsistent image resizing and aliasing.These variants target different sources of unreliable comparisons: finite-sample bias and preprocessing differences.Source: §2.5 · p. 3, §2.6 · p. 3

2.7 Related work

  • Related work distinguishes likelihood-based models from implicit models and includes surveys, critiques of IS, alternative feature distributions, and comparisons where KID outperforms FID and IS.The reviewed literature situates the paper among broader efforts to improve generative-model evaluation.Source: §2.7 · p. 3

3 Synthetic dataset as a benchmark

The authors construct NotImageNet32 from Image-GPT samples and use likelihood-based models with exact or estimated likelihoods to compare generative-model metrics against divergences.

  • 100K Image-GPT samples trained on ImageNet32 form NotImageNet32, split into 70K training and 30K test images.Source: §3 · p. 3
  • Image-GPT clusters each pixel’s RGB values into 512 indexes, which are mapped back to RGB for visualization.Source: §3 · p. 3
  • PixelSnail and VD-VAE model sets with varied sizes and training epochs provide strong and weak likelihood-based models for comparison.Source: §3 · p. 4
  • The benchmark estimates KL and reverse KL divergences between the data and model distributions to capture complementary generative-model errors.Source: §3 · p. 4
  • The test-bed is limited to likelihood-based models, excluding implicit models such as GANs, and KL divergences do not capture every generative-model complexity.Source: §3 · p. 4

4 Comparison between evaluation metrics

Across PixelSnail and other trained models, FID and IS correlate with KL-based measures but are more volatile and less reliable for fine-grained ranking than KL and RKL.

4.1 Volatility

  • FID and especially IS are much more volatile than KL and RKL, assigning different scores to models with similar divergences.After 15–20 epochs, KL and RKL change slowly while FID and IS remain volatile as model capacity increases.Source: §4.1 · p. 4, Figure 3 · p. 5, Figure 4 · p. 5
  • The study compares KL, RKL, FID, and IS across four PixelSnail variants during training, using metric values recorded at different epochs and model capacities.The evaluation setting includes likelihood-based divergences alongside empirical metrics for models saved every five training epochs.Source: §4.1 · p. 4, §4.2 · p. 5
  • For high-quality models, FID and IS can change substantially without a comparable change in KL or RKL.This pattern makes the empirical metrics ill-suited to fine-grained comparisons between close models.Source: §4.1 · p. 5, Figure 5 · p. 6

4.2 Ranking correlation

  • KL and RKL have the strongest reported ranking agreement, with Kendall’s τ of 0.889, while FID and IS have low ranking correlations and FID outperforms IS.The ranking analysis focuses on models trained for 15–45 epochs to emphasize fine-grained comparisons.Source: §4.2 · p. 5, Table 1 · p. 6
  • FID extensions achieve better ranking correlations than the original FID and IS metrics.Among Inception-based metrics, FID∞ has the highest correlation with KL and RKL, whereas IS/IS∞ has the lowest correlation across the compared models.Source: §4.2 · pp. 5–6
  • Most Inception-based ranking correlations are relatively low, so changing the metric can produce significantly different model rankings.The stated exception is the correlation between KID and Clean FID.Source: §4.2 · p. 6, Table 1 · p. 6

5 Is Inception all we need?

The section evaluates whether Inception features are suitable for generative-model comparison across domains, contrasting them with CLIP using qualitative and normality analyses. CLIP anomalies more often appear semantically anomalous, and CLIP features achieve better normality-test scores across datasets.

5 Is Inception all we need?

  • Because generative models are evaluated on domains distinct from ImageNet, the section questions whether ImageNet-trained Inception features are appropriate for faces, pets, and bedrooms.CLIP is presented as a broader-domain alternative because it was trained on 400M images paired with captions.Source: §5 · p. 7

5.1 Qualitative analysis of the latent representation

  • The qualitative procedure ranks generated samples by likelihood under Gaussians fitted separately to Inception and CLIP feature representations.The Gaussian models are fitted from 20K randomly selected real-data images after sampling 10K images from a generative model.Source: §5.1 · p. 7
  • CLIP low-probability images look like anomalies, whereas most Inception low-probability images remain valid Wild-dataset images, indicating CLIP features are better suited for model comparison.The qualitative analysis used 10K generated samples and Gaussian models fitted to real-data features.Source: §5.1 · pp. 7–8

5.2 Normality test for latent representation

  • The normality test randomly projects feature matrices to one dimension, applies D’Agostino’s K-squared test, and averages p values over T = 1000 projections.Inception representations have dimension 2048 and CLIP representations dimension 512.Source: §5.2 · p. 8
  • Table 2 indicates that Inception features are non-Gaussian while CLIP achieves better normality-test scores across the evaluated settings.Inception performs best on the synthetic dataset rather than the original ImageNet dataset, which the authors hypothesize reflects shared characteristics between neural-network-generated images and the Inception network.Source: §5.2 · p. 8, Table 2 · p. 8

6 Conclusions

The conclusions find that empirical metrics capture important trends but can be volatile for fine-grained comparison, while CLIP features outperform Inception features especially beyond ImageNet benchmarks. The authors recommend alternative metrics, multiple-metric evaluation, CLIP-based FID, and a synthetic test-bed.

6 Conclusions

  • Empirical metrics correlate with KL and reverse-KL divergences but are more volatile, so metric gains do not always correspond to observable probabilistic improvements.The conclusions also report that IS and its IS∞ extensions perform significantly worse than the other metrics.Source: §6 · p. 8
  • The authors recommend dropping Inception Score, using FID∞ instead of FID, and combining FID∞, KID, and Clear FID to control score volatility.The recommendation targets both the poor performance of IS variants and instability in individual metric scores.Source: §6 · p. 8
  • They recommend replacing Inception with CLIP in FID and advocate NotImageNet32 as a generative-model test-bed.The authors state that code for CLIP-based FID was made available.Source: §6 · p. 9

A Qualitative analysis on Additional Datasets

Additional qualitative analyses extend the anomaly-detection assessment to Dogs and CelebA. The supplied passages identify the datasets and corresponding Inception and CLIP low-probability figures, but do not state the visual outcomes.

A Qualitative analysis on Additional Datasets

  • The additional analysis applies the section 5.1 anomaly-detection procedure to AFHQ Dogs and CelebA, reporting results as consistent with those in section 5.The passage does not provide the specific comparisons shown in the figures.Source: A Qualitative analysis on Additional… · p. 11
  • The figures separately show low-probability samples for Inception and CLIP on Dogs and CelebA.The supplied figure passages identify the plotted metric and dataset but not which representation produces more anomalous-looking samples.Source: Figure 8 · p. 11, Figure 10 · p. 11, Figure 9 · p. 11

B Volatility Analysis of High Quality Models

The volatility analysis examines metric behavior during training of high-quality models. FID can change dramatically even when KL and reverse-KL change little, indicating volatility in the method.

B Volatility Analysis of High Quality Models

  • FID changes dramatically during training although KL and reverse KL change little, indicating volatility in FID for high-quality models.Figure 12 tracks evaluation metrics across four PixelSNAIL and two VD-VAE models of varying sizes, focusing on high-quality models.Source: B Volatility Analysis of High Quality… · p. 12, Figure 12 · p. 12

C Technical details on experiment’s generative models architecture

The experiment varies model hyperparameters to compare generative-model performance and documents the chosen parameter sets for reproducibility.

C.1 PixelSnail

The supplementary architectures use PixelSnail and VD-VAE models with specified building blocks, optimization settings, and hyperparameter conventions.

C.1 PixelSnail

  • PixelSnail combines residual convolutional blocks with attention blocks using key-value lookup and softmax attention.The model uses Adam with learning rate 0.0001, a MultiplicativeLR scheduler, and mean cross-entropy over 512 discrete clusters.Source: §C.1 · p. 12

C.2 VD-VAE

  • VD-VAE uses encoder regular and down-rate blocks plus decoder regular and mixin blocks, with skip connections from parallel encoder layers.Down-rate blocks reduce dimensionality through average pooling, while mixin blocks interpolate to a higher dimension.Source: §C.2 · pp. 12–13
  • VD-VAE notation records repeated regular blocks, pooling down-rate blocks, and interpolation mixin blocks in its hyperparameter table.Additional settings include EMA rate 0.999, learning rate 0.00005, gradient clipping at 200, and Adam parameters β1 = 0.9 and β2 = 0.9.Source: §C.2 · p. 13, Table 4 · p. 13

D Supplementary models correlation measurements

The supplementary correlation measurements show that Pearson correlation is high for most evaluation methods, while Spearman ranking correlation yields similar results.

Related papers

Loading 2206.10935v1…