PaperlayerView exact revision on arXiv

Source-linked AI summary

The Role of ImageNet Classes in Fréchet Inception Distance

Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, Jaakko Lehtinen

arXiv:2203.06026v3cs.CVcs.AIcs.LGcs.NEstat.ML

TL;DR

FID can disagree with human evaluation because its ImageNet-based feature space may emphasize incidental class-related features. The paper visualizes FID’s image sensitivities and tests classification-histogram matching, finding substantial FID reductions without clear perceptual improvement. It concludes that FID can be distorted, especially when generative models use ImageNet pre-training.

  • Problem

    FID is widely used for generative-model ranking, but its agreement with human judgment can fail when its feature space lacks general perceptual relevance.

    Source: §1 · pp. 1–2

  • Method

    The paper uses Grad-CAM and feature-space experiments to identify image regions affecting FID and test whether matching ImageNet class histograms changes FID without perceptible result changes.

    Source: §1 · p. 2, §2 · p. 3, §3 · p. 5

  • Results

    FID was reduced by 60% or more, including a 66.4% improvement, while visual comparisons showed no indisputable improvement and most gains occurred in FID’s perceptual null space.

    Source: §3 · p. 7

  • Takeaways & Limitations

    FID improvements should be interpreted cautiously when comparing different architectures or training setups, particularly with ImageNet-pre-trained generative models.

    Source: §5 · p. 9

    FID remains dependable for detecting training failures and monitoring convergence, but can mis-rank sufficiently different architectures or setups that reproduce fringe features differently.

    Source: §5 · p. 9

Abstract

from arXiv · show

Fréchet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially -- without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.

1 INTRODUCTION

FID is widely used to rank generative models because it compares real and generated image distributions in an ImageNet-pre-trained feature space. This paper investigates why FID can diverge from human judgment, visualizing its sensitivities and examining distortions caused by matching ImageNet classifications.

  • FID remains the primary metric for quantifying progress in generative modeling, despite the need for reliable model ranking and improvement detection.Source: §1 · p. 1
  • FID embeds real and generated images using the penultimate 2048-dimensional layer of an ImageNet-pre-trained Inception-V3 network, then compares Gaussian approximations of their embeddings.The comparison uses the Fréchet distance between the two Gaussian distributions.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • FID improves on Inception Score by comparing generated images with real data, and it is fast, conceptually simple, and reasonably correlated with human judgments.Source: §1 · p. 2
  • FID conflates resemblance and variation, while its value is affected by sample count, feature-network choice, and low-level image processing.Source: §1 · p. 2
  • The paper questions whether FID’s 2048-dimensional feature space has sufficient perceptual relevance, given that perceptual feature spaces can be approximately 6M-dimensional and encode spatial feature positions.Source: §1 · p. 2
  • Grad-CAM is used to visualize which image regions influence FID by augmenting precomputed feature statistics with a newly generated image.The resulting low-resolution importance maps are upsampled to the input resolution.Source: §1 · pp. 2–3, Figure 2 · p. 3

2 WHAT DOES FID LOOK AT IN AN IMAGE?

The paper visualizes which image regions influence FID by adding one image to precomputed statistics and applying Grad-CAM. FID often emphasizes ImageNet-associated regions, including fringe features outside the intended subject.

Visualization technique

  • FID sensitivity is visualized by adding one image to statistics computed from 49,999 generated images, then applying Grad-CAM to the resulting metric.The real and generated Gaussian statistics are treated as constants while the added image updates the generated set.Source: §2 · p. 3
  • The method estimates feature-map importance, forms an 8 × 8 spatial map, upsamples it, and converts it into a heatmap.Unlike standard Grad-CAM visualization, it retains both positive and negative effects on FID rather than applying a ReLU.Source: §2 · p. 4

Individual images

  • In FFHQ, FID typically highlights regions outside the human face because ImageNet lacks “person” and “face” categories.Other fringe features and classes are therefore activated for generated face images.Source: §2 · p. 4
  • In LSUN CAT, ImageNet’s multiple cat categories help FID focus more closely on the subject, although background fringe features remain relevant.A very low FID would require matching detections such as “washbasin” between real and generated images.Source: §2 · p. 4

Aggregated images

  • Across FFHQ images, aggregate FID sensitivity is strongest for ImageNet objects outside the face, while class-conditioned maps highlight where each class typically appears.The reported examples include “bow tie”, “seat belt” and “mortarboard”.Source: §2 · p. 5, Figure 4 · p. 5

3 PROBING THE PERCEPTUAL NULL SPACE IN FID

The section tests whether FID can be reduced by resampling generated images to match ImageNet-derived statistics, without meaningful visual improvement. Top-1 and Top-N class matching, as well as broader feature matching, reveal a substantial FID perceptual null space.

Top-1 histogram matching

  • Top-1 histogram matching consistently improves FID by approximately 10% across multiple datasets.Resampling matches the Top-1 class histogram of generated images to that of the training data.Source: §3 · p. 6
  • Alternative feature spaces do not confirm a clear quality improvement, and the FID decrease closely tracks the degree of ImageNet pre-training.FIDResNet-50 drops almost as much as the original FID, whereas FIDSwAV and FIDCLIP decrease substantially less.Source: §3 · p. 6

Matching all fringe features

  • The general resampling method selects weighted subsets from 250k generated candidates and samples 50k images according to optimized weights.Weights are optimized to reduce FID using weighted generated-feature means and covariances, then used as sampling probabilities.Source: §3 · pp. 6–7
  • FID decreases drastically after matching fringe features, while uncurated samples show no indisputable visual improvement and alternative feature spaces improve much less.The authors conclude that most of the FID improvement occurs in a large perceptual null space and can be manipulated through ImageNet classification probabilities.Source: §3 · p. 7
  • The 66.4% FID reduction cannot be simply attributed to discarding visibly artifact-laden images, because the lowest- and highest-weight sets look broadly similar.Both sets contain realistic images and images with clear visual artifacts in roughly equal proportions.Source: §3 · p. 7, Figure 5 · p. 7

Top-N histogram matching

  • Matching Top-N class histograms rapidly reduces FID, showing that FID is substantially determined by the co-occurrence of top ImageNet classes.Classes ranked near the top have greater influence than middle-ranked classes, while FIDCLIP remains almost indifferent to the apparent improvements.Source: §3 · pp. 7–8

4 PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS

The authors examine whether ImageNet pre-training can produce misleadingly low FID, using Projected FastGAN and StyleGAN2 as a practical comparison. Despite similar FID and Recall, FastGAN receives worse human-quality assessments, while FIDCLIP favors StyleGAN2.

  • FID = 5.28, Recall = 0.45, FIDCLIP = 4.67 for one comparison, versus FID = 5.30, Recall = 0.46, FIDCLIP = 2.76 for the other.Source: §4 · p. 9
  • Projected FastGAN and StyleGAN2 achieve similar FID, although FastGAN samples contain more artifacts.Source: Figure 7 · p. 9, §4 · p. 9
  • Projected FastGAN uses an ImageNet-pre-trained EfficientNet in its discriminator and is compared with StyleGAN2 on FFHQ.Source: §4 · p. 9
  • FIDCLIP agrees with human assessment: StyleGAN2 is rated significantly better than Projected FastGAN.Source: §4 · p. 9
  • The authors suspect interference from ImageNet pre-training and warn that it can make FID unreliable in practice, while not calling ImageNet pre-training inherently bad.Source: §4 · p. 9

5 CONCLUSIONS

The conclusions distinguish FID's useful monitoring behavior from its vulnerability when sufficiently different models reproduce fringe features differently. The authors recommend checking FID improvements with feature spaces not trained on ImageNet.

  • FID reliably indicates large training failures and generally tracks subjective improvements during convergence monitoring or trivial hyperparameter changes.Source: §5 · p. 9
  • When architectures or training setups differ substantially, reproducing fringe features can lower FID without improving human-observable quality.Source: §5 · p. 9
  • ImageNet pre-training may compromise FID's validity as a quality metric, so improvements should be verified with a non-ImageNet-trained Fréchet distance.Source: §5 · p. 9
  • Suggested alternative feature spaces include CLIP, self-supervised SwAV, and an uninitialized network.Source: §5 · p. 9

A NUMERICAL SENSITIVITY OF FID

FID is sensitive to evaluation protocol, feature-network choice, and image preprocessing. The appendix also examines feature-space correlations and image regions that influence FID.

Number of samples and bias

Network architecture

Image processing flaws

  • Pre-logit and logit FIDs are highly correlated because the spaces differ by one affine transformation without nonlinearities.Source: Figure 9 · p. 15
  • Noise in regions important to FID produces a larger FID increase than noise in unimportant regions.Source: Figure 10 · p. 15

B CORRELATION BETWEEN PRE-LOGITS AND LOGITS FID

The appendix finds nearly perfect correlation between FIDs computed from pre-logit features and classification logits, explained by their affine relationship. This experiment is only a guide and may not generalize across architectures or training setups.

C WHAT DOES FID LOOK AT IN AN IMAGE?

FID sensitivity is concentrated in localized regions that align with ImageNet Top-1 classifications, often outside the semantically central content such as human faces. Comparisons with Grad-CAM and noise perturbations indicate substantial overlap between regions important for ImageNet classification and regions important for FID.

Sensitivity validation

  • Noise applied to important regions increases FID more than noise applied to equally sized unimportant regions.Adding noise everywhere provides an upper bound in this test setup.Source: C WHAT DOES FID LOOK AT IN AN IMAGE? · p. 16

Image-level heatmaps

Class-conditioned heatmaps

  • For FFHQ images grouped by ImageNet class, FID is most sensitive where the Top-1 class is intuitively located and relatively insensitive to human faces.For classes such as lipstick, the highlighted regions are typically outside the face area.Source: C WHAT DOES FID LOOK AT IN AN IMAGE? · p. 16, Figure 12 · p. 18

Comparison with Grad-CAM

D PROBING THE PERCEPTUAL NULL SPACE IN FID

The paper probes FID’s perceptual null space by optimizing image sampling weights and matching ImageNet Top-N histograms. These interventions can substantially reduce FID while leaving visual appearance largely unchanged, indicating that FID is strongly influenced by ImageNet-class statistics.

Resampling method

  • Optimizing per-image sampling weights minimizes FID between real and weighted generated features, with probabilities obtained from log-parameterized weights.The method samples generated images with replacement according to the optimized probabilities.Source: D PROBING THE PERCEPTUAL NULL SPACE IN… · p. 21

Visual consequences

  • Very large FID reductions after Top-1 histogram matching or fringe-feature matching leave the visual appearance of StyleGAN2 samples largely unchanged.FIDCLIP does not confirm the improvement indicated by FID.Source: D PROBING THE PERCEPTUAL NULL SPACE IN… · p. 21

Weight interpretation

  • Low FID after pre-logit resampling cannot be explained simply by removing images with clear visual artifacts.Both low-weight and high-weight sets contain realistic-looking images and images with visual artifacts.Source: D PROBING THE PERCEPTUAL NULL SPACE IN… · p. 21

Other feature-space metrics

  • Resampling in the pre-logit feature space also strongly decreases KID and RBF-KID, which are computed in the same feature space as FID.The paper hypothesizes that these metrics share approximately the same perceptual null space.Source: D PROBING THE PERCEPTUAL NULL SPACE IN… · p. 21

Top-N histogram matching

  • FID improves rapidly even for small Top-N values, showing that FID is significantly determined by the co-occurrence of top ImageNet classes.The resulting value converges slightly above the value obtained by optimizing weights directly in pre-logit space.Source: D PROBING THE PERCEPTUAL NULL SPACE IN… · p. 21

E PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS

The paper examines an ImageNet-pre-trained Projected FastGAN whose FID is comparable to StyleGAN2 despite producing more distorted and artifact-laden FFHQ images. Additional resampling and histogram-matching results reinforce that FID can improve without obvious visual improvement.

Practical example

  • Projected FastGAN and StyleGAN2 are compared on FFHQ using uncurated generated-image grids.The comparison is framed around ImageNet-pre-trained feature extraction in the discriminator.Source: E PRACTICAL EXAMPLE: IMAGENET… · p. 21

Resampling evidence

  • Randomly sampled images, Top-1 histogram-matched images, and fringe-feature-weighted images show that drastic FID reductions need not improve visual fidelity.The compared samples remain largely similar in visual appearance.Source: Figure 16 · p. 23

Weight distributions

  • Images assigned the smallest and largest sampling weights both include realistic-looking examples and images with visual artifacts.Thus, the optimized weights do not separate samples solely by obvious visual quality.Source: Figure 17 · p. 24

Cross-dataset results

  • Aligning Top-N ImageNet class histograms consistently produces the largest FID decrease across the additional datasets.FIDCLIP remains almost unchanged in the same comparisons.Source: Figure 18 · p. 25

Human-visible quality

  • Projected FastGAN achieves a better FID than StyleGAN2, while its samples contain more distortions and artifacts.The figure presents uncurated samples from both models.Source: Figure 19 · p. 26

Related papers

Loading 2203.06026v3…