Source-linked AI summary
The Role of ImageNet Classes in Fréchet Inception Distance
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, Jaakko Lehtinen
TL;DR
FID can disagree with human evaluation because its ImageNet-based feature space may emphasize incidental class-related features. The paper visualizes FID’s image sensitivities and tests classification-histogram matching, finding substantial FID reductions without clear perceptual improvement. It concludes that FID can be distorted, especially when generative models use ImageNet pre-training.
Problem
FID is widely used for generative-model ranking, but its agreement with human judgment can fail when its feature space lacks general perceptual relevance.
Method
The paper uses Grad-CAM and feature-space experiments to identify image regions affecting FID and test whether matching ImageNet class histograms changes FID without perceptible result changes.
Results
FID was reduced by 60% or more, including a 66.4% improvement, while visual comparisons showed no indisputable improvement and most gains occurred in FID’s perceptual null space.
Takeaways & Limitations
FID improvements should be interpreted cautiously when comparing different architectures or training setups, particularly with ImageNet-pre-trained generative models.
Takeaways & Limitations
FID remains dependable for detecting training failures and monitoring convergence, but can mis-rank sufficiently different architectures or setups that reproduce fringe features differently.
Abstract
from arXiv · showhide
Fréchet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially -- without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.
1 INTRODUCTION
FID is widely used to rank generative models because it compares real and generated image distributions in an ImageNet-pre-trained feature space. This paper investigates why FID can diverge from human judgment, visualizing its sensitivities and examining distortions caused by matching ImageNet classifications.
- FID remains the primary metric for quantifying progress in generative modeling, despite the need for reliable model ranking and improvement detection.
- FID embeds real and generated images using the penultimate 2048-dimensional layer of an ImageNet-pre-trained Inception-V3 network, then compares Gaussian approximations of their embeddings.The comparison uses the Fréchet distance between the two Gaussian distributions.
- FID improves on Inception Score by comparing generated images with real data, and it is fast, conceptually simple, and reasonably correlated with human judgments.
- FID conflates resemblance and variation, while its value is affected by sample count, feature-network choice, and low-level image processing.
- The paper questions whether FID’s 2048-dimensional feature space has sufficient perceptual relevance, given that perceptual feature spaces can be approximately 6M-dimensional and encode spatial feature positions.
- Grad-CAM is used to visualize which image regions influence FID by augmenting precomputed feature statistics with a newly generated image.The resulting low-resolution importance maps are upsampled to the input resolution.
2 WHAT DOES FID LOOK AT IN AN IMAGE?
The paper visualizes which image regions influence FID by adding one image to precomputed statistics and applying Grad-CAM. FID often emphasizes ImageNet-associated regions, including fringe features outside the intended subject.
- Visualization technique: FID sensitivity is visualized by adding one image to statistics computed from 49,999 generated images, then applying Grad-CAM to the resulting metric.The real and generated Gaussian statistics are treated as constants while the added image updates the generated set.
- Visualization technique: The method estimates feature-map importance, forms an 8 × 8 spatial map, upsamples it, and converts it into a heatmap.Unlike standard Grad-CAM visualization, it retains both positive and negative effects on FID rather than applying a ReLU.
- Individual images: In FFHQ, FID typically highlights regions outside the human face because ImageNet lacks “person” and “face” categories.Other fringe features and classes are therefore activated for generated face images.
- Individual images: In LSUN CAT, ImageNet’s multiple cat categories help FID focus more closely on the subject, although background fringe features remain relevant.A very low FID would require matching detections such as “washbasin” between real and generated images.
- Aggregated images: Across FFHQ images, aggregate FID sensitivity is strongest for ImageNet objects outside the face, while class-conditioned maps highlight where each class typically appears.The reported examples include “bow tie”, “seat belt” and “mortarboard”.
3 PROBING THE PERCEPTUAL NULL SPACE IN FID
The section tests whether FID can be reduced by resampling generated images to match ImageNet-derived statistics, without meaningful visual improvement. Top-1 and Top-N class matching, as well as broader feature matching, reveal a substantial FID perceptual null space.
- Top-1 histogram matching: Top-1 histogram matching consistently improves FID by approximately 10% across multiple datasets.Resampling matches the Top-1 class histogram of generated images to that of the training data.
- Top-1 histogram matching: Alternative feature spaces do not confirm a clear quality improvement, and the FID decrease closely tracks the degree of ImageNet pre-training.FIDResNet-50 drops almost as much as the original FID, whereas FIDSwAV and FIDCLIP decrease substantially less.
- Matching all fringe features: The general resampling method selects weighted subsets from 250k generated candidates and samples 50k images according to optimized weights.Weights are optimized to reduce FID using weighted generated-feature means and covariances, then used as sampling probabilities.
- Matching all fringe features: FID decreases drastically after matching fringe features, while uncurated samples show no indisputable visual improvement and alternative feature spaces improve much less.The authors conclude that most of the FID improvement occurs in a large perceptual null space and can be manipulated through ImageNet classification probabilities.
- Matching all fringe features: The 66.4% FID reduction cannot be simply attributed to discarding visibly artifact-laden images, because the lowest- and highest-weight sets look broadly similar.Both sets contain realistic images and images with clear visual artifacts in roughly equal proportions.
- Top-N histogram matching: Matching Top-N class histograms rapidly reduces FID, showing that FID is substantially determined by the co-occurrence of top ImageNet classes.Classes ranked near the top have greater influence than middle-ranked classes, while FIDCLIP remains almost indifferent to the apparent improvements.
4 PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS
The authors examine whether ImageNet pre-training can produce misleadingly low FID, using Projected FastGAN and StyleGAN2 as a practical comparison. Despite similar FID and Recall, FastGAN receives worse human-quality assessments, while FIDCLIP favors StyleGAN2.
- FID = 5.28, Recall = 0.45, FIDCLIP = 4.67 for one comparison, versus FID = 5.30, Recall = 0.46, FIDCLIP = 2.76 for the other.
- Projected FastGAN and StyleGAN2 achieve similar FID, although FastGAN samples contain more artifacts.
- Projected FastGAN uses an ImageNet-pre-trained EfficientNet in its discriminator and is compared with StyleGAN2 on FFHQ.
- FIDCLIP agrees with human assessment: StyleGAN2 is rated significantly better than Projected FastGAN.
- The authors suspect interference from ImageNet pre-training and warn that it can make FID unreliable in practice, while not calling ImageNet pre-training inherently bad.
5 CONCLUSIONS
The conclusions distinguish FID's useful monitoring behavior from its vulnerability when sufficiently different models reproduce fringe features differently. The authors recommend checking FID improvements with feature spaces not trained on ImageNet.
- FID reliably indicates large training failures and generally tracks subjective improvements during convergence monitoring or trivial hyperparameter changes.
- When architectures or training setups differ substantially, reproducing fringe features can lower FID without improving human-observable quality.
- ImageNet pre-training may compromise FID's validity as a quality metric, so improvements should be verified with a non-ImageNet-trained Fréchet distance.
- Suggested alternative feature spaces include CLIP, self-supervised SwAV, and an uninitialized network.
A NUMERICAL SENSITIVITY OF FID
FID is sensitive to evaluation protocol, feature-network choice, and image preprocessing. The appendix also examines feature-space correlations and image regions that influence FID.
- Number of samples and bias: FID depends strongly on sample count, making a standardized number of real and generated samples crucial for comparable results.
- Network architecture: FID is surprisingly sensitive to the exact Inception-V3 network instance, with discrepancies large enough to confuse state-of-the-art comparisons.
- Network architecture: FIDCLIP suggests that model rankings may remain stable across ImageNet classifier networks but not necessarily across feature spaces more broadly.
- Image processing flaws: Image resizing to 299 × 299 can introduce aliasing artifacts through poor pre-filtering, which noticeably affects FID.
- Pre-logit and logit FIDs are highly correlated because the spaces differ by one affine transformation without nonlinearities.
- Noise in regions important to FID produces a larger FID increase than noise in unimportant regions.
B CORRELATION BETWEEN PRE-LOGITS AND LOGITS FID
The appendix finds nearly perfect correlation between FIDs computed from pre-logit features and classification logits, explained by their affine relationship. This experiment is only a guide and may not generalize across architectures or training setups.
- Pre-logit and logit FIDs are highly correlated because the two spaces are separated by one affine transformation without nonlinearities.
- The result is a guiding experiment and is not guaranteed to hold for different GAN architectures or training setups.
C WHAT DOES FID LOOK AT IN AN IMAGE?
FID sensitivity is concentrated in localized regions that align with ImageNet Top-1 classifications, often outside the semantically central content such as human faces. Comparisons with Grad-CAM and noise perturbations indicate substantial overlap between regions important for ImageNet classification and regions important for FID.
- Sensitivity validation: Noise applied to important regions increases FID more than noise applied to equally sized unimportant regions.Adding noise everywhere provides an upper bound in this test setup.
- Image-level heatmaps: FID-sensitive regions in FFHQ and LSUN CAT images are highly localized and correlate strongly with their ImageNet Top-1 classes.The corresponding Top-1 classification is annotated for the individual images.
- Class-conditioned heatmaps: For FFHQ images grouped by ImageNet class, FID is most sensitive where the Top-1 class is intuitively located and relatively insensitive to human faces.For classes such as lipstick, the highlighted regions are typically outside the face area.
- Comparison with Grad-CAM: FID sensitivity heatmaps highlight regions that overlap heavily with Grad-CAM regions for ImageNet Top-1, Top-2, and Top-3 classifications.The comparison includes probability-weighted average Grad-CAM heatmaps for FFHQ and LSUN CAT.
D PROBING THE PERCEPTUAL NULL SPACE IN FID
The paper probes FID’s perceptual null space by optimizing image sampling weights and matching ImageNet Top-N histograms. These interventions can substantially reduce FID while leaving visual appearance largely unchanged, indicating that FID is strongly influenced by ImageNet-class statistics.
- Resampling method: Optimizing per-image sampling weights minimizes FID between real and weighted generated features, with probabilities obtained from log-parameterized weights.The method samples generated images with replacement according to the optimized probabilities.
- Visual consequences: Very large FID reductions after Top-1 histogram matching or fringe-feature matching leave the visual appearance of StyleGAN2 samples largely unchanged.FIDCLIP does not confirm the improvement indicated by FID.
- Weight interpretation: Low FID after pre-logit resampling cannot be explained simply by removing images with clear visual artifacts.Both low-weight and high-weight sets contain realistic-looking images and images with visual artifacts.
- Other feature-space metrics: Resampling in the pre-logit feature space also strongly decreases KID and RBF-KID, which are computed in the same feature space as FID.The paper hypothesizes that these metrics share approximately the same perceptual null space.
- Top-N histogram matching: FID improves rapidly even for small Top-N values, showing that FID is significantly determined by the co-occurrence of top ImageNet classes.The resulting value converges slightly above the value obtained by optimizing weights directly in pre-logit space.
E PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS
The paper examines an ImageNet-pre-trained Projected FastGAN whose FID is comparable to StyleGAN2 despite producing more distorted and artifact-laden FFHQ images. Additional resampling and histogram-matching results reinforce that FID can improve without obvious visual improvement.
- Practical example: Projected FastGAN and StyleGAN2 are compared on FFHQ using uncurated generated-image grids.The comparison is framed around ImageNet-pre-trained feature extraction in the discriminator.
- Resampling evidence: Randomly sampled images, Top-1 histogram-matched images, and fringe-feature-weighted images show that drastic FID reductions need not improve visual fidelity.The compared samples remain largely similar in visual appearance.
- Weight distributions: Images assigned the smallest and largest sampling weights both include realistic-looking examples and images with visual artifacts.Thus, the optimized weights do not separate samples solely by obvious visual quality.
- Cross-dataset results: Aligning Top-N ImageNet class histograms consistently produces the largest FID decrease across the additional datasets.FIDCLIP remains almost unchanged in the same comparisons.
- Human-visible quality: Projected FastGAN achieves a better FID than StyleGAN2, while its samples contain more distortions and artifacts.The figure presents uncurated samples from both models.