Source-linked AI summary
Do GANs actually learn the distribution? An empirical study
Sanjeev Arora, Yi Zhang
TL;DR
The paper asks whether GANs learn the target distribution when discriminators have finite capacity, given theory suggesting near-optimal objectives can coexist with low support. It introduces a birthday-paradox-based support test and applies it to GANs, finding evidence of low-support generated distributions. The test has important practical scope limitations for noisy images and nonuniform distributions.
Problem
Finite-discriminator theory raises whether GANs can achieve near-optimal training objectives without learning the target distribution.
Method
The paper introduces a birthday-paradox test that detects duplicate generated images to estimate distribution support, combining automated similarity with visual inspection.
Results
With probability at least 50%, batches of about 400 samples contained duplicates for DCGAN and MIX+DCGAN, while ALI or BiGAN required 1000 samples.
Takeaways & Limitations
Experiments suggest that well-regarded GANs, particularly higher-visual-quality methods, learn distributions with fairly low support and exhibit mode collapse.
Takeaways & Limitations
The test is not definitive for highly nonuniform distributions and works poorly when generated images are noisy or blurry.
Abstract
from arXiv · showhide
Do GANS (Generative Adversarial Nets) actually learn the target distribution? The foundational paper of (Goodfellow et al 2014) suggested they do, if they were given sufficiently large deep nets, sample size, and computation time. A recent theoretical analysis in Arora et al (to appear at ICML 2017) raised doubts whether the same holds when discriminator has finite size. It showed that the training objective can approach its optimum value even if the generated distribution has very low support ---in other words, the training objective is unable to prevent mode collapse. The current note reports experiments suggesting that such problems are not merely theoretical. It presents empirical evidence that well-known GANs approaches do learn distributions of fairly low support, and thus presumably are not learning the target distribution. The main technical contribution is a new proposed test, based upon the famous birthday paradox, for estimating the support size of the generated distribution.
1 Introduction
GANs can reach near-optimal training objectives without learning the target distribution, because finite discriminators may be fooled by generators with very small support. The paper introduces a birthday-paradox-based empirical test to estimate support size and investigate whether this failure occurs in practice.
- Finite discriminator size can permit a training objective near optimum even when the generator supports only O(n log n/ϵ2) images.This contrasts with target distributions such as human-face images, which are expected to have much larger support.
- The theoretical result left open whether practical GAN training avoids low-support near-equilibrium solutions through backpropagation or hyperparameter choices.The paper therefore motivates empirical investigation of generated-distribution support.
- 1.1 Birthday paradox test for support size: The birthday paradox implies that duplicate samples can reveal a distribution’s approximate support size, although the test is affected by nonuniform probability mass.A distribution with a concentrated 10% mass on one image can produce collisions despite having a huge total support.
- 1.1 Birthday paradox test for support size: The proposed test samples generated images, flags highly similar pairs automatically, visually checks for duplicates, and repeats the procedure.If duplicates appear with good probability in samples of size s, the suspected support size is about s2.
- 1.1 Birthday paradox test for support size: The birthday-paradox intuition is illustrated by birthdays: among iid draws, collision probability exceeds 50% with only 23 people despite 366 possible birthdays.
2 Birthday paradox test for GANs: Experimental details
The paper applies a birthday-paradox-based near-duplicate test to estimate GAN distribution diversity, using dataset-specific similarity measures and experiments across several GAN variants. Results indicate duplicate generation, discriminator-size dependence, and limitations from image noise and sample quality.
- Test design: Near-duplicates are identified by selecting the 20 closest sample pairs under a heuristic metric and visually judging which pairs are human-perceived duplicates.For faces, the metric is pixel-space Euclidean distance; for CIFAR-10, it is Euclidean distance between CNN embeddings.
- Datasets and models: The experiments cover CelebA faces, CIFAR-10, preliminary LSUN bedrooms, and DCGAN, MIX+DCGAN, ALI, and Stacked GAN methods.The LSUN bedroom results are explicitly preliminary and inconclusive.
- Experimental limitations: The birthday-paradox test works best for sharp realistic images because injected noise makes similarity measurement and visual duplicate judgments unreliable.This limits interpretation for models that add noise to training or generated images.
- CelebA results: With probability ≥50%, batches of about 400 samples contain duplicates for both DCGAN and MIX+DCGAN, implying support below 400^2 ≈160000.ALI reaches the same collision probability only around batch size 1000, implying support around one million.
- Diversity vs Discriminator Size: Learnt-distribution diversity grows near-linearly with discriminator size in a crude fixed-hyperparameter DCGAN experiment, although it appears to plateau.The discriminator is a five-layer CNN whose channel scale is controlled by dim, with size proportional to dim^2.
- Memorization check: The observed duplicate images are not close to their nearest training-set images, suggesting low diversity rather than direct memorization of training data.Nearest neighbors were found using the same heuristic similarity measure and then visually inspected.
3 Birthday paradox test for VAEs
The birthday paradox test was difficult to apply to VAE samples because their images were not realistic or sharp enough for reliable human judgments. Collision candidates were often blurry, particularly around facial features.
- 3 Birthday paradox test for VAEs: VAE samples from CelebA were not realistic or sharp enough to determine reliably whether two images were nearly identical.This made the birthday paradox test difficult to run.
- 3 Birthday paradox test for VAEs: DCGAN bedroom samples included corrupted images with a fixed noise pattern, emphasized in red boxes.These examples illustrate a visible quality problem in generated samples.
- 3 Birthday paradox test for VAEs: Collision candidates in VAE samples were frequently blurry because features such as hair, eyes, and mouths were not distinctive.The figure highlights duplicated samples with indistinct crucial facial features.
4 Conclusions
Experiments using a Birthday Paradox test suggest that current GANs, especially higher-quality approaches, learn distributions with low support and exhibit mode collapse. The findings align with theory and motivate changes to the training objective, while leaving open whether GANs may be better suited to feature learning.
- 4 Conclusions: Current GAN approaches that produce higher visual quality still fall significantly short of learning the target distribution, with rather low generated support.The paper characterizes this as mode collapse.
- 4 Conclusions: The Birthday Paradox test provides an empirical way to assess image-distribution diversity and support size.The test connects observed duplicate samples to low diversity.
- 4 Conclusions: The empirical findings agree with theory that the GAN training objective cannot prevent mode collapse in the trained distribution.The theoretical analysis anticipated near-optimal training despite inadequate distribution learning.
- 4 Conclusions: The authors report that support size appears to scale near-linearly with discriminator capacity, but this conclusion requires more extensive experiments.The scaling result is described as rough and needing rechecking.
- 4 Conclusions: Changing the GAN training objective to avoid mode collapse remains an open problem, with ALI/BiGANs suggested as a possible direction.These methods exhibited somewhat better diversity in the reported experiments.
- 4 Conclusions: GANs and related techniques might ultimately be more useful for feature learning or another goal than for distribution learning.The paper calls for further theoretical and empirical exploration of this possibility.