PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking FID: Towards a Better Evaluation Metric for Image Generation

Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, Sanjiv Kumar

arXiv:2401.09603v2cs.CV

TL;DR

The paper asks whether FID reliably evaluates modern image-generation quality despite rich content and perceptual evaluation challenges. It analyzes FID’s representation and statistical limitations and proposes CMMD using CLIP embeddings and MMD. Experiments show that FID-based evaluations may be unreliable, while CMMD provides a more robust assessment within the tested scope.

  • Problem

    Reliable automated evaluation is needed because image quality, aesthetics, and text faithfulness are difficult to quantify and human evaluation is expensive.

  • Method

    The paper proposes CMMD, which combines richer CLIP embeddings with maximum mean discrepancy using a Gaussian RBF kernel.

  • Results

    FID-based evaluations may be unreliable, while CMMD offers a more robust and reliable assessment of image quality across the reported experiments.

  • Takeaways & Limitations

    The authors encourage researchers to rethink FID as the primary image-quality metric for modern text-to-image models.

  • Takeaways & Limitations

    FID’s evaluation is constrained by Inception’s limited representation, an incorrect normality assumption, and inconsistent sample-size behavior.

Abstract

from arXiv · show

As with many machine learning problems, the progress of image generation methods hinges on good evaluation metrics. One of the most popular is the Frechet Inception Distance (FID). FID estimates the distance between a distribution of Inception-v3 features of real images, and those of images generated by the algorithm. We highlight important drawbacks of FID: Inception's poor representation of the rich and varied content generated by modern text-to-image models, incorrect normality assumptions, and poor sample complexity. We call for a reevaluation of FID's use as the primary quality metric for generated images. We empirically demonstrate that FID contradicts human raters, it does not reflect gradual improvement of iterative text-to-image models, it does not capture distortion levels, and that it produces inconsistent results when varying the sample size. We also propose an alternative new metric, CMMD, based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings and is sample efficient. Through extensive experiments and analysis, we demonstrate that FID-based evaluations of text-to-image models may be unreliable, and that CMMD offers a more robust and reliable assessment of image quality.

1. Introduction

Evaluating modern text-to-image models is difficult because quality, aesthetics, and prompt faithfulness are perceptual, yet FID has important representation and statistical shortcomings. The paper proposes CMMD, combining richer CLIP embeddings with MMD, as a more reliable alternative.

  • Human evaluation remains the gold standard because image-generation quality, aesthetics, and text faithfulness are difficult to quantify.
  • FID can contradict human raters, miss gradual improvements and complex distortions, and vary inconsistently with sample size.
  • CMMD uses CLIP embeddings trained on 400 million image–text pairs, compared with Inception features trained on about 1 million ImageNet images across 1000 classes.
  • MMD is distribution-free and unbiased, whereas Fréchet distance assumes multivariate normality and FID has model-dependent finite-sample bias.
  • The paper calls for reevaluating FID as the primary metric and presents CMMD as a more robust alternative for modern image-generation models.

2. Related Works

Prior work established FID, KID, and related metrics for evaluating generated images, while also identifying bias and preprocessing sensitivity. This paper extends those concerns to normality violations and broader evaluation failures.

  • Image-generation evaluation has used metrics including IS, KID, FID, perceptual path length, Gaussian Parzen windows, and HYPE.
  • FID and KID compare real and generated image distributions, while IS uses Inception-v3 class probabilities without requiring real images.
  • When normality assumptions are violated, Fréchet distances can remain misleadingly zero as distributions progressively depart, whereas MMD captures the departure.
  • Earlier studies found that FID is biased with model-dependent bias and that compression or resizing can substantially change its value.

3. Limitations of FID

FID estimates distances between real and generated Inception embeddings using a closed-form formula that is valid only under multivariate normality. The paper shows that this assumption and finite-sample covariance estimation can produce misleading or unstable evaluations.

  • 3.1. The Fréchet Distance: The simplified Fréchet formula is strictly valid only when both compared distributions are multivariate normal.The general Fréchet distance is also equivalent to the Wasserstein-2 distance, but the closed form requires the normality assumption.
  • 3.1. The Fréchet Distance: FID applies a closed-form Fréchet-distance formula to estimated means and covariances of 2048-dimensional Inception embeddings.The procedure assumes the embeddings are normally distributed and estimates their statistics from real and generated image samples.
  • 3.1. The Fréchet Distance: Estimating 2048 × 2048 covariance matrices from small samples can lead to large errors in FID.This is one of the two identified error sources in estimating the distance between real and generated embedding distributions.
  • 3.2. Implications of Wrong Normality Assumptions: Keeping means and covariances fixed while separating mixture components makes the distributions increasingly different, yet normality-based Fréchet distance remains misleadingly zero.FID∞ also suffers because it relies on the same normality assumption, whereas MMD can detect the distributional change.
  • 3.3. Incorrectness of the Normality Assumption: Inception embeddings of COCO 30K are not multivariate normal: multiple modes appear visually, and three statistical tests yield p-values of virtually zero.The multimodality is consistent with Inception-v3 activations being organized for classification across 1,000 ImageNet classes.
  • 3.3. Incorrectness of the Normality Assumption: FID’s normality assumption is inappropriate for nonnormal Inception features, and CLIP embeddings also fail normality tests with virtually zero p-values.The paper’s objection is to applying the Fréchet distance with this assumption to Inception features, not to expecting CLIP embeddings to be normal.

4. The CMMD Metric

CMMD evaluates real and generated images by comparing their CLIP embeddings with squared MMD using a Gaussian RBF kernel. It is designed for diverse image content and avoids the Fréchet distance’s distributional assumption and finite-sample bias.

  • 4. The CMMD Metric: CMMD is the squared MMD distance between CLIP embeddings of reference real images and generated images.The metric uses a Gaussian RBF kernel to compare the two embedding distributions.
  • 4. The CMMD Metric: CLIP embeddings are better suited than Inception embeddings to diverse generated images and the virtually infinite variety of text-to-image prompts.CLIP learns joint image-text representations from 400 million image-text pairs, while Inception-v3 is trained on roughly 1 million ImageNet images across 1,000 classes.
  • 4. The CMMD Metric: MMD measures distribution discrepancy using a positive definite kernel and is a metric for characteristic kernels.It was originally developed as part of a two-sample test for determining whether samples come from the same distribution.
  • 4. The CMMD Metric: With a characteristic kernel, MMD is distribution-free and does not assume that the compared distributions are normal.This contrasts with the Fréchet formula, which can give erroneous results when its normality assumption is violated.
  • 4. The CMMD Metric: MMD provides an unbiased estimator, whereas finite-sample FID has model-dependent bias that can change model rankings with sample size.Removing FID bias requires reliably estimating the d × d covariance matrix.
  • 4. The CMMD Metric: CMMD uses the Gaussian RBF kernel k(x, y) = exp(−∥x − y∥2/2σ2) with bandwidth σ = 10 and scales the result by 1000.The Gaussian kernel is characteristic, its bandwidth was fixed for consistency, and the unscaled MMD values are bounded above at 2.

5. Human Evaluation

A side-by-side human evaluation overwhelmingly preferred the full Muse model, while FID and FID∞ contradicted that preference and CMMD agreed with it.

  • Evaluation design: The evaluation compared paired images from the two Muse models using identical random seeds, isolating visual quality from content and prompt alignment.Three independent raters assessed each pair across 1,633 PartiPrompts.
  • Metric comparison: COCO 30K FID and FID∞ contradicted the human preference, whereas CMMD correctly aligned with raters’ judgments.The table caption summarizes the same disagreement between FID and human evaluation and agreement from CMMD.

6. Performance Comparison

Across iterative generation and distortion experiments, CMMD tracked image-quality changes more consistently than FID, while also offering better sample efficiency through MMD.

  • Progressive generation: CMMD monotonically decreased across Muse refinement iterations, correctly tracking improvement, while FID and FID∞ incorrectly suggested degradation.The image quality itself monotonically improved over the iterations.
  • Image distortions: For VQGAN-token distortions with p = 0.2, FID changed from 21.40 to 18.02 while CMMD changed from 0.721 to 1.190 as image quality degraded.Replacing random tokens produced visibly more distorted images; CMMD correctly increased with degradation, unlike FID.
  • Progressive generation: CMMD monotonically improved across the final Stable Diffusion iterations, whereas FID mistakenly suggested lower quality in the last two iterations.The comparison focuses on subtle differences among the final five iterations of a 100-iteration model.
  • Image distortions: CMMD increased monotonically with latent-space noise, correctly reflecting degradation, while FID inconsistently suggested improved quality.The distortion procedure replaces VQGAN tokens with random tokens before reconstruction.
  • Sample size: FID requires more than 20,000 images for reliable estimation, whereas CMMD provides consistent estimates even with small image sets.This sample-efficiency difference matters for fast online evaluation during model training.
  • Runtime: MMD has O(n^2d) asymptotic complexity, but is practically efficient because its computation can be efficiently implemented.The corresponding Fréchet-distance computation is dominated by an expensive, difficult-to-parallelize matrix square root.

7. Discussion

The authors argue that FID should be reconsidered as the primary image-quality metric because it can conflict with human judgments and misread iterative improvements and distortions.

  • Discussion: The authors warn that reliance on FID could produce flawed rankings and cause promising image-generation ideas to be rejected prematurely.They propose CMMD as a more robust metric for modern text-to-image-model evaluation.
  • Discussion: They encourage researchers to rethink FID as the primary image-quality metric and consider CMMD for modern text-to-image models.This recommendation follows reported disagreements with human raters and failures on iterative improvement and obvious distortions.

A. Multivariate Normality Tests

FID relies on multivariate normality, but tests on Inception features from COCO 30K reject that assumption.

  • Normality assumption: Three statistical tests show that COCO 30K Inception features do not satisfy FID’s multivariate normality assumption.The tests are Mardia’s skewness, Mardia’s kurtosis, and Henze-Zirkler.
  • Test interpretation: The tests’ null hypothesis is that the sampled features come from a multivariate normal distribution.Different tests use different statistics to assess this hypothesis.

Mardia’s Skewness Test

Mardia’s skewness statistic tests multivariate normality using a random sample’s sample mean and covariance. On COCO 30K Inception embeddings, the test rejects normality with a p-value of 0.0.

  • Mardia’s statistic A measures multivariate skewness for a random sample of d-dimensional observations.
  • The statistic uses the sample mean and biased sample covariance matrix to evaluate the observations’ multivariate skewness.
  • Under multivariate normality, A is asymptotically chi-squared with d(d+1)(d+2)/6 degrees of freedom.
  • p-value 0.0: the normality test rejects the multivariate normality assumption for COCO 30K Inception embeddings.

Mardia’s Kurtosis Test

Mardia’s kurtosis statistic and the Henze-Zirkler test both assess multivariate normality. Both tests reject normality for COCO 30K Inception embeddings with p-values of 0.0.

  • Mardia’s statistic B measures multivariate kurtosis for a random sample of d-dimensional observations.
  • Under multivariate normality, B is asymptotically standard normally distributed.
  • p-value 0.0: Mardia’s kurtosis test rejects normality for COCO 30K Inception embeddings, with the statistic 19,023 standard deviations from the normal mean.
  • The Henze-Zirkler test measures distributional distance and has stronger power performance than alternative multivariate normality tests.
  • p-value 0.0: the Henze-Zirkler test also rejects the normality assumption for COCO 30K Inception embeddings.

B. Synthetic Experiment Details

The synthetic experiment compares an isotropic Gaussian reference with a four-component Gaussian mixture whose mean and covariance remain matched to the reference as separation increases. Fréchet-based distances remain zero despite the mixture moving farther away, exposing the consequence of the normality assumption.

  • The reference distribution is an isotropic Gaussian centered at the origin with covariance matrix σ2I2.
  • The comparison distribution is an equally weighted mixture of four Gaussians centered at (λ, 0), (0, λ), (−λ, 0), and (0, −λ).
  • As λ increases, τλ is adjusted so the mixture’s overall covariance remains σ2I2, while its mean remains the origin.
  • Fréchet-based distances remain zero as λ increases, although the mixture moves farther from the reference because its normality assumption is incorrect.
  • The mixture distribution is formed by weighting component probability density functions, with weights summing to one.
  • The mixture’s covariance remains σ2I2 when τ2λ = σ2 − λ2/2.
Loading 2401.09603v2…