PaperlayerView exact revision on arXiv

Source-linked AI summary

Backpropagating through Fr\'echet Inception Distance

Alexander Mathiasen, Frederik Hvilshøj

arXiv:2009.14075v2cs.LGstat.ML

TL;DR

Evaluating generative models is difficult, and the paper asks whether optimizing FID can improve generated images. FastFID enables efficient FID-loss training; adding FID to GAN training improves validation FID across models and datasets, while some inspected samples improve, though when this occurs remains unclear.

  • Problem

    Evaluating generative-model performance is difficult, raising whether optimizing FID as a loss can improve generated images.

    Source: §1 · p. 1

  • Method

    FastFID efficiently computes gradients for FID, enabling GANs to jointly optimize their discriminator loss and FID with gradient descent.

    Source: §2.2 · p. 3, §3.1 · p. 6, §6 · p. 8

  • Results

    Adding FID as a GAN loss consistently improves validation FID across GANs and datasets, including SNGAN on CIFAR10 from 22 to 11, and can improve generated samples.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    FID can now be optimized directly, and training with it can improve both measured FID and some generated images.

    Source: §1 · p. 1, §6 · p. 8

    When FID-loss training improves generated images and when it does not remains unclear, and FID may be indifferent to whether heads are attached to a single dog.

    Source: §3.1 · p. 6, §4.1 · p. 7, §6 · p. 8

Abstract

from arXiv · show

The Fr\'echet Inception Distance (FID) has been used to evaluate hundreds of generative models. We introduce FastFID, which can efficiently train generative models with FID as a loss function. Using FID as an additional loss for Generative Adversarial Networks improves their FID.

1. Introduction

Generative models are difficult to evaluate, motivating the use of FID as both an evaluation metric and a training loss. FastFID enables efficient backpropagation through FID, and adding FID loss improves GAN validation FID while sometimes improving generated images.

1. Introduction

  • Generative modeling has broad applications, but evaluating generative-model performance remains a key difficulty.The introduction cites computer vision and drug discovery as application areas.Source: §1 · p. 1
  • FastFID enables efficient backpropagation through FID, addressing automatic differentiation that can add up to 10 days to training.The method supports training generative models with FID as a loss function.Source: §1 · p. 1
  • Adding FID as an additional loss improves SNGAN’s CIFAR10 FID from 22 to 11 and can outperform newer GANs.The paper reports similar FID improvements across different GANs and datasets.Source: §1 · p. 1, Figure 1 · p. 1
  • Optimizing FID can improve generated images, including adding ears, eyes, tongues, or a head to BigGAN samples.A dog image described as headless became an image with a head after FID optimization.Source: §1 · p. 1, Figure 2 · p. 1
  • The paper presents FastFID, demonstrates improved validation FID for GANs, and investigates whether FID optimization improves samples.These contributions correspond to the paper’s method, GAN experiments, and image-quality analysis.Source: §1 · p. 1

2. Fast Fréchet Inception Distance

FastFID computes the trace term in FID without explicitly forming the full matrix square root, replacing a d-dimensional computation with eigenvalues of a smaller m × m matrix. This enables differentiable FID computation with substantially lower runtime while preserving the result produced by SCIPY.LINALG.SQRTM.

2.2. Our algorithm

  • Algorithm 1 agrees with SCIPY.LINALG.SQRTM up to numerical errors while reporting numerical error at least 1000× smaller in the 32-bit experiment.The authors attribute the improvement to operating on the smaller m × m matrix rather than a full d × d Schur decomposition.Source: §2.1 · p. 2, §2.4 · p. 4
  • FastFID computes tr(√Σ1Σ2) by finding eigenvalues of a small m × m matrix M whose eigenvalues match those needed from the full d × d covariance product.Using m ≪ d, such as m = 128 and d = 2048 for FID, avoids explicitly computing the full matrix square root.Source: §2.2 · pp. 2–3

2. Fast Fréchet Inception Distance

  • FastFID modifies the standard FID pipeline by using fake and real Inception activations to form sample means and covariances, then combining covariance traces with the trace-square-root term.The same procedure extends to Fréchet ChemNet Distance by replacing Inception with ChemNet.Source: §2 · p. 2, §2.2 · p. 3

2.3. Experimental Speed-Up

  • At least 25× faster for every tested m, and sometimes hundreds or thousands of times faster, FastFID computes the same quantity as prior algorithms.The comparison uses repeated timing experiments across batch sizes.Source: §2.3 · p. 4
  • 12.15±0.80 seconds to 0.23±0.01 seconds: with batch size m = 128, FastFID reduces computation time and saves 13 days over 10^5 training steps.Over 10^6 steps, the reported reduction reaches 130 days.Source: §2.3 · p. 4

3. Training GANs with FID as a Loss

Across three GANs and datasets, adding FID to the training loss consistently improves validation FID, with FastFID enabling efficient backpropagation through FID. The experiments also show that the discriminator can suppress artifacts that arise when models optimize FID without it.

SNGAN on CIFAR10

  • 21.81±0.73 to 11.48±0.22: SNGAN's validation FID improved on CIFAR10 after 301070 training steps with the additional FID loss.This result compares SNGAN without and with FID loss; Figure 4 shows validation FID during training and generated samples.Source: §3 · p. 5, Figure 4 · p. 5

DCGAN on CelebA

  • 15.67 ± 1.38 to 11.03 ± 0.43: DCGAN's validation FID improved on CelebA at 64x64 resolution after 100264 training steps.Figure 5 contains the corresponding validation-FID curves and generated samples.Source: §3 · p. 5, Figure 5 · p. 5

SAGAN on ImageNet

  • 129.48±2.38 to 97.64±4.33: SAGAN's validation FID improved on ImageNet at 128x128 resolution after 100001 training steps.Figure 6 contains the corresponding validation-FID curves and generated samples.Source: §3 · p. 5, Figure 6 · p. 5

Experimental Conclusion

  • Validation FID improved in all three experiments when FID was added to the GAN loss: SNGAN on CIFAR10, DCGAN on CelebA, and SAGAN on ImageNet.The experiments used three GAN architectures and three datasets, with repeated runs reporting mean ± standard deviation.Source: §3 · p. 5

Training

  • FastFID makes FID practical as a training loss by enabling fast backpropagation through FID, while the joint objective minimizes both discriminator loss and FID.Training uses a mini-batch of 128 fake samples and computes FID against the entire training data, avoiding variance from sampling a training-data mini-batch.Source: §3.1 · p. 6

Other Generative Models

  • Training Glow to minimize FID produced “unrealistic” artifacts, whereas removing SNGAN's discriminator also led to such artifacts.The authors hypothesize that the discriminator learns to detect and penalize these artifacts; with a discriminator removed after 1000 steps, SNGANFID began producing them.Source: §3.1 · p. 6, Figure 7 · p. 6

4. Can FID Loss Improve Generated Images?

The study tests whether minimizing FID improves generated images rather than merely exploiting the metric. Across repeated BigGAN experiments, optimization added perceptible semantic features, while FID analysis suggests matching feature occurrence and co-occurrence drives this behavior.

Experimental setup

  • The image-improvement experiment pretrained BigGAN for 105 iterations, discarded its discriminator, and optimized the generator using FID alone.Source: §4 · p. 7

Experimental setup and observations

  • All 64 tracked samples changed perceptibly during FID minimization, ruling out explanations based on only a few altered pixels.Source: §4 · p. 7

Experimental observations

Conclusion

  • FID minimization can improve fake images in some cases rather than necessarily causing the generator to merely fool FID.Source: §4 · p. 7

Further details

  • FID appears to incentivize feature occurrence and co-occurrence patterns that match the training data, which may explain the observed improvements.Source: §4.1 · p. 7

5. Related Work

Prior work established FID as a widely used generative-model evaluation metric, while existing faster alternatives either approximate FID, simplify covariance handling, or retain higher time complexity. FastFID instead targets efficient exact backpropagation and low-rank computation.

FID evaluation

  • FID was introduced with evidence of consistency with increasing disturbances and human judgment under a normality assumption for Inception activations (Heusel et al., 2017).Source: §5.1 · p. 8

Evaluation practice

  • The paper reports validation FID because it explicitly optimizes training FID, addressing differing conventions in published evaluations.Source: §5.1 · p. 8

FastFID and prior methods

  • To the authors’ knowledge, FastFID is the first published approach to use FID as a loss function and to reduce its asymptotic time complexity.Source: §5.1 · p. 8, §5.2 · p. 8
  • Existing alternatives approximate FID, compute a different quantity, or retain O(d^3 + d^2m) complexity, whereas FastFID exploits low rank.Source: §5.2 · p. 8

6. Conclusion

The paper makes FID directly optimizable through FastFID, enabling GAN training with FID as a loss and consistently improving validation FID across models and datasets. Evidence also suggests possible image improvement, but when this occurs remains unclear.

Conclusion

  • FastFID efficiently computes FID gradients using eigenvalue considerations, avoiding automatic-differentiation costs that can add many days to training.Source: §6 · p. 8
  • Training different GANs with FID as a loss consistently improves validation FID across different datasets.Source: §6 · p. 8
  • The paper finds some evidence that FID loss can improve generator samples, but the conditions governing when this happens remain unclear.Source: §6 · p. 8
  • The authors recommend explicitly labeling models that optimize FID to support fair comparisons in future research.Source: §6 · p. 8

Related papers

Loading 2009.14075v2…