PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Generation Via Minimizing Fr\'echet Distance in Discriminator Feature Space

Khoa D. Doan, Saurav Manchanda, Fengjiao Wang, Sathiya Keerthi, Avradeep Bhowmik, Chandan K. Reddy

arXiv:2003.11774v2cs.CVcs.LGeess.IV

TL;DR

GAN training struggles with mode collapse, vanishing gradients, and inefficient or ineffective distributional-distance estimation in high-dimensional pixel space. The paper instead minimizes distances in a low-dimensional discriminator feature space, yielding OT-GAN and Fréchet-GAN. Fréchet-GAN in particular improves visual quality and FID over representative primal and dual GANs across benchmark datasets.

  • Problem

    GANs face mode collapse and vanishing-gradient challenges, while pixel-space optimal transport is computationally expensive and ineffective for high-dimensional images.

  • Method

    The paper estimates optimal transport or Fréchet distance in a low-dimensional feature space selected from the GAN discriminator, introducing OT-GAN and Fréchet-GAN.

  • Results

    Fréchet-GAN achieves significant improvements in generated-image quality and FID over existing primal and dual GANs across MNIST, CIFAR-10, CELEB-A, and LSUN-Bedroom.

  • Takeaways & Limitations

    The results support estimating distributional distances in discriminator feature space as an effective direction for image generation.

  • Takeaways & Limitations

    Direct optimal transport remains computationally expensive, with complexity O(N 2.5 log N), limiting practical use and motivating lower-cost alternatives.

Abstract

from arXiv · show

For a given image generation problem, the intrinsic image manifold is often low dimensional. We use the intuition that it is much better to train the GAN generator by minimizing the distributional distance between real and generated images in a small dimensional feature space representing such a manifold than on the original pixel-space. We use the feature space of the GAN discriminator for such a representation. For distributional distance, we employ one of two choices: the Fr\'{e}chet distance or direct optimal transport (OT); these respectively lead us to two new GAN methods: Fr\'{e}chet-GAN and OT-GAN. The idea of employing Fr\'{e}chet distance comes from the success of Fr\'{e}chet Inception Distance as a solid evaluation metric in image generation. Fr\'{e}chet-GAN is attractive in several ways. We propose an efficient, numerically stable approach to calculate the Fr\'{e}chet distance and its gradient. The Fr\'{e}chet distance estimation requires a significantly less computation time than OT; this allows Fr\'{e}chet-GAN to use much larger mini-batch size in training than OT. More importantly, we conduct experiments on a number of benchmark datasets and show that Fr\'{e}chet-GAN (in particular) and OT-GAN have significantly better image generation capabilities than the existing representative primal and dual GAN approaches based on the Wasserstein distance.

1 Introduction

The paper addresses GAN training challenges by estimating distributional distances in a discriminator feature space rather than high-dimensional pixels, introducing Fréchet-GAN and OT-GAN. Fréchet-GAN achieves improved image quality and FID across four benchmark datasets.

  • 1 Introduction: Existing GANs can generate realistic images but face mode collapse and vanishing-gradient challenges, motivating alternative distributional-distance objectives.Wasserstein objectives address some of these issues but remain difficult to estimate efficiently.
  • 1 Introduction: Primal Wasserstein approaches remain costly or statistically difficult because optimal transport has high computational cost, high variance, and exponential sample complexity.Direct pixel-space OT is also ineffective for high-dimensional images.
  • 1 Introduction: The methods use a low-dimensional discriminator feature space to estimate distributional distances instead of directly working in pixel space.This targets image manifolds that are often intrinsically low dimensional.
  • 1 Introduction: The paper introduces OT-GAN and Fréchet-GAN by applying optimal transport or Fréchet distance, respectively, in the discriminator feature space.Fréchet-GAN is motivated by the success of Fréchet Inception Distance.
  • 1 Introduction: The paper develops the algorithmic and design details needed for efficient distance and gradient computation and reports strong benchmark performance.The contribution includes both new methods and their practical implementation.
  • 1 Introduction: Fréchet-GAN achieves significant improvements in generated-image quality and FID on MNIST, CIFAR-10, CELEB-A, and LSUN-Bedroom.The paper compares it with representative primal and dual GAN approaches.

2 Background

Primal Wasserstein approaches estimate distributional distances directly from samples, but pixel-space optimization is unsuitable for high-dimensional images. The section motivates low-dimensional discriminator feature spaces and introduces Fréchet distance as a practical alternative inspired by FID.

  • 2.1 Primal Wasserstein Distances: Minimizing OT on high-dimensional pixel space produces mode collapse or white noise, whereas it works only when inputs are already low-dimensional and Frobenius distance is suitable.The cited comparison concerns CIFAR-10 versus MNIST generated images.
  • 2.1 Primal Wasserstein Distances: OT directly estimates empirical Wasserstein distance but costs O(N^2.5 log N), has high variance and exponential sample complexity, and remains impractical for high-dimensional images.
  • 2.1 Primal Wasserstein Distances: SWG uses random one-dimensional projections and a discriminator-selected low-dimensional feature space, but random projections can lose manifold information and require many directions.SWG’s feature space is defined by an intermediate discriminator output D′(x).
  • 2.1 Primal Wasserstein Distances: SWG costs O(KN log(N)) and can require K = 10,000 projections for batch size 64, reducing its practical computational advantage over OT.Max-SWG reduces computational complexity and memory footprint and can generate high visual quality.
  • 2.2 Fr´echet distance: Fréchet distance compares Gaussian distributions through their means and covariances and motivates FID, which computes it on InceptionV3 features of real and generated images.Lower FID is generally associated with more realistic visual quality.

3 Approach

The approach replaces pixel-space Wasserstein optimization with distances computed in a discriminator feature space, yielding OT-GAN and Fréchet-GAN. Fréchet-GAN further uses efficient, numerically stable matrix-square-root computation and gradient estimation.

  • 3.1 Scaling up OT: The discriminator feature space provides a low-dimensional representation for image distributions, avoiding the ineffectiveness of estimating OT directly in high-dimensional pixel space.The approach is motivated by low-dimensional intrinsic image manifolds and the failure of pixel-space OT on higher-dimensional images.
  • 3.1 Scaling up OT: OT-GAN minimizes optimal transport between real and generated discriminator-feature distributions, substantially improving generated MNIST and CIFAR-10 images over pixel-space minimization.The generator and discriminator optimize different objectives, so OT-GAN is not formulated as a minimax game.
  • 3.2 GAN based on Fréchet distance: Fréchet-GAN approximates Wasserstein-2 distance between discriminator-feature distributions as the Fréchet distance under a multivariate Gaussian feature-space assumption.Average pooling is motivated by the Central Limit Theorem, while max pooling also produces multivariate Gaussian outputs in the modified architecture and performs significantly better than average pooling.
  • 3.2 GAN based on Fréchet distance: The Fréchet-distance objective combines differences between feature means and covariance matrices, with the covariance square-root term dominating computation.The feature dimension is d, and the Bartels-Stewart algorithm solves the associated Sylvester equation in O(d^1.5) time rather than the naive O(d^3).
  • Computing the matrix derivative:: Newton-Schultz iterations compute the matrix square root using matrix multiplications, and the Sylvester equation provides its gradient without retaining the full iterative computation graph.The proposed implementation avoids the instability observed with SVD when eigenvalues are close and uses 10–15 iterations to achieve good numerical accuracy.
  • Fréchet-GAN Training: Fréchet-GAN alternates discriminator and generator updates: the discriminator learns a separating feature space, then the generator minimizes Fréchet distance between real and generated features.Training computes D′(D) and D′(F), evaluates the Fréchet loss, and updates the generator using the matrix-square-root value and derivative.

12 end

The implementation computes the Fréchet distance from discriminator features and uses an efficient iterative matrix-square-root procedure. Its complexity is lower than OT’s, enabling larger training mini-batches.

  • 12 end: Newton-Schultz reduces Fréchet-distance computation to O(min(Nd^2, Td^2)), compared with O(min(Nd^2, d^3)) for SVD and higher OT cost.Here N is the sample count, d is the feature dimension, and T is the iteration count.
  • 12 end: The lower computational cost of Fréchet-distance estimation allows substantially larger mini-batch sizes than OT during training.The advantage becomes especially important when N is large.

4 Experimental Results

Across synthetic and image benchmarks, OT-GAN and especially Fréchet-GAN show effective generation in discriminator feature space, with Fréchet-GAN achieving strong visual and FID results.

  • Experimental setup: The experiments evaluate OT-GAN and Fréchet-GAN against WGAN, WGAN-GP, SWG, and Max-SWG using a shared DCGAN architecture across four datasets.The study reports qualitative and quantitative results on MNIST, CIFAR-10, CELEB-A, and LSUN-Bedroom, with FID evaluated using 50,000 generated and 50,000 real images.
  • 4.1 Performance with and without the discriminator: In 8-Gaussian estimation, discriminator-based models recover the original distribution, while Fréchet-GAN converges faster than the other GANs.Without the discriminator, direct SW and OT minimization exhibit mode collapse; the experiment highlights the role of feature-space distances.
  • 4.2 Stability in generating images: Without batch normalization, Fréchet-GAN produces less mode collapse than SWG and sharper digits than Max-SWG and OT-GAN.The corresponding FID values for Fréchet-GAN increase only slightly without BN, indicating robustness to this architectural change.
  • 4.2 Stability in generating images: On multi-channel and higher-dimensional datasets, Fréchet-GAN produces visually comparable images with fewer defective artifacts and LSUN-bedroom quality comparable to Max-SWG.OT-GAN and Fréchet-GAN have visual quality comparable to existing GANs, while Fréchet-GAN attains the strongest reported overall FID pattern.
  • 4.2 Stability in generating images: Fréchet-GAN achieves the best FIDs on MNIST, CELEB-A, and LSUN-Bedroom, and a CIFAR-10 FID comparable to Max-SWG.The comparison supports direct distributional-distance minimization in discriminator feature space for image generation.

5 Conclusion

The paper proposes estimating distributional distances in a low-dimensional GAN-discriminator feature space using OT or Fréchet distance, yielding two new GAN methods. Fréchet-GAN addresses computational and numerical challenges and improves visual quality and FID relative to existing primal and dual GANs.

  • The methods estimate distributional distance in a low-dimensional feature space selected from the GAN discriminator.This design targets the low-dimensional structure of image manifolds rather than the original pixel space.
  • OT-GAN and Fréchet-GAN use Optimal Transport and Fréchet distance, respectively, as alternative distributional-distance choices.The Fréchet-distance approach is motivated by Fréchet Inception Distance and includes an efficient, differentiable calculation algorithm.
  • Fréchet-GAN achieves significant improvements in visual inspection and FID compared with existing primal and dual GANs.
  • Estimating Wasserstein distance in the low-dimensional feature space heuristically overcomes the complexity associated with high-dimensional space.
Loading 2003.11774v2…