Source-linked AI summary
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, Charles Sutton
TL;DR
GANs can be difficult to train and may collapse onto only a few modes of the data distribution. VEEGAN adds a reconstructor that approximately inverts the generator and autoencodes Gaussian noise, achieving substantially lower mode collapse and higher-quality samples across synthetic and real image datasets.
Problem
GANs are difficult to train and prone to mode collapse, in which generated samples cover only a few modes of the true data distribution.
Method
VEEGAN jointly trains a generator and reconstructor, mapping data to Gaussian noise while approximately reversing the generator's action through an implicit variational objective.
Results
VEEGAN is dramatically less susceptible to mode collapse and produces higher-quality samples than other state-of-the-art methods on synthetic and real-world image datasets.
Takeaways & Limitations
Autoencoding noise rather than data provides a natural ℓ2 loss and supports effective mode-collapse avoidance while retaining good sample quality.
Takeaways & Limitations
Training the variational bound is difficult because the relevant conditional generator distribution is sharply peaked and easy for a discriminator to distinguish.
Abstract
from arXiv · showhide
Deep generative models provide powerful tools for distributions over complicated manifolds, such as those of natural images. But many of these methods, including generative adversarial networks (GANs), can be difficult to train, in part because they are prone to mode collapse, which means that they characterize only a few modes of the true distribution. To address this, we introduce VEEGAN, which features a reconstructor network, reversing the action of the generator by mapping from data to noise. Our training objective retains the original asymptotic consistency guarantee of GANs, and can be interpreted as a novel autoencoder loss over the noise. In sharp contrast to a traditional autoencoder over data points, VEEGAN does not require specifying a loss function over the data, but rather only over the representations, which are standard normal by assumption. On an extensive set of synthetic and real world image datasets, VEEGAN indeed resists mode collapsing to a far greater extent than other recent GAN variants, and produces more realistic samples.
1 Introduction
GANs are powerful implicit generative models but can be unstable to train and prone to mode collapse. VEEGAN addresses this by adding a reconstructor and autoencoding Gaussian noise rather than data, reducing collapse and improving sample quality.
- GANs generate samples from difficult data distributions but are notoriously fickle to train and often miss many modes.Mode collapse occurs when the generator learns only a few modes despite missing modes appearing throughout the training data.
- VEEGAN adds a reconstructor that maps data to Gaussian noise while approximately reversing the generator's action.The generator and reconstructor are trained jointly through an implicit variational principle.
- Autoencoding Gaussian noise gives VEEGAN a natural ℓ2 loss, avoiding the difficulty of choosing an image-space autoencoder loss.This distinguishes VEEGAN from methods that autoencode data items.
- Experiments on synthetic and real-world image datasets find VEEGAN substantially less susceptible to mode collapse and better at producing high-quality samples than other state-of-the-art methods.
2 Background
GANs are implicit models trained by distinguishing generated samples from true data, with asymptotic recovery of the data distribution under ideal conditions. Their training can be unstable, and localized discriminator failures can produce mode collapse.
- Implicit probability distributions provide a sampling procedure for generating data but lack a tractable density function.
- A GAN maps normally distributed representation vectors to data and uses a discriminator to distinguish generated samples from true data samples.
- At the ideal optimum, the discriminator recovers the log density ratio and the generator-induced distribution matches the data distribution.This guarantee assumes infinite data and arbitrarily powerful networks.
- Mode collapse occurs when generated samples cover only a few data modes, and localized discriminator constancy can make generator learning impossible in those regions.
3 Method
VEEGAN jointly trains a generator and reconstructor so the reconstructor maps data to Gaussian noise while approximately inverting generation. Its computable upper-bound objective combines adversarial density-ratio estimation with noise reconstruction and retains a consistency guarantee under sufficient model capacity.
- Core idea: VEEGAN introduces a reconstructor Fθ that maps true data to Gaussian noise and approximately inverts generator Gγ.The reconstructor is trained to satisfy both objectives simultaneously.
- Core idea: If the generator collapses a mode, reconstruction or Gaussianity mismatches provide learning signals for the generator and reconstructor.The same reconstructor exposes either non-Gaussian outputs on forgotten modes or failure to recover latent noise.
- Objective: VEEGAN replaces an intractable entropy objective with a computable upper bound involving a variational generator, reconstructor, and discriminator-based density-ratio estimate.The discriminator distinguishes samples from the generator-data joint distribution and the reconstructor-data joint distribution.
- Objective: The method uses an autoencoder loss on noise vectors z rather than data items, with representation-space losses such as ℓ2.This avoids requiring a difficult image-space reconstruction loss.
- Optimization: The final objective combines discriminator estimation with reconstruction, and the networks are optimized jointly using stochastic gradient descent.Algorithm 1 updates ω, θ, and γ together; reconstructor pre-training helps in some cases.
- Guarantee: With sufficiently powerful networks, globally minimizing the upper bound can recover the true data distribution qγ = p.Proposition 1 states this guarantee when parameters achieve the standard-normal entropy value, while universal approximation supports the required conditions.
4 Relationships to Other Methods
VEEGAN differs from related adversarial and autoencoder methods by using a data-trained reconstructor and autoencoding latent noise rather than images. Its reconstruction term can provide generator and reconstructor learning signals independently of discriminator sensitivity, while noise losses avoid problematic image-space objectives.
- Adversarial reconstruction methods: BiGAN, ALI, and LFVI also use joint-space discriminators with generator and reconstructor networks, but VEEGAN uses a different objective.The paper reports benefits over ALI/BiGAN’s logistic regression loss and LFVI’s KL-divergence.
- Adversarial reconstruction methods: VEEGAN’s reconstruction term remains informative when the discriminator is constant, unlike objectives whose generator and reconstructor signals are mediated only by discriminator output.The paper connects this design to VEEGAN’s lower susceptibility to mode collapse than ALI.
- InfoGAN: InfoGAN reconstructs only part of the latent code and does not train its reconstructor on the true data distribution.The paper suggests this difference may relate to InfoGAN requiring stabilization tricks that VEEGAN does not require.
- Adversarial autoencoders: Other adversarial autoencoder methods reconstruct images using losses such as λd(x, Gγ(Fθ(x))), whereas VEEGAN reconstructs noise vectors.Image-space loss selection is problematic, while ℓ2 loss on standard-normal noise is described as natural.
5 Experiments
The experiments evaluate VEEGAN on synthetic mixtures, Stacked-MNIST, and CIFAR-10 using mode-coverage and sample-quality measures. Across these settings, VEEGAN generally captures more modes while maintaining higher-quality or more realistic samples than competing methods.
- Synthetic datasets: VEEGAN captures the greatest number of modes on all synthetic datasets while consistently generating higher-quality samples.The evaluation uses mode count and high-quality-sample percentage; Figure 2 shows VEEGAN distributions are sharper and closer to the true distributions.
- Synthetic datasets: Mode coverage and sample quality must be measured together because either metric alone can be misleading.A GAN can obtain 99.3% sample quality by collapsing onto one mode, while Unrolled GAN can cover modes with overly dispersed samples.
- Stacked-MNIST: Stacked-MNIST creates 1000 modes by stacking three randomly sampled MNIST digits across the color channels.The dataset is designed to increase the number of discrete modes.
- Evaluation protocol: The synthetic evaluation estimates modes with a trained classifier and reports KL divergence between generator and data distributions.Results average 26,000 samples across five runs.
- CIFAR-10: On CIFAR-10, ALI and VEEGAN achieve the best IvOM, while VEEGAN samples appear qualitatively more realistic than those of other methods.VEEGAN captures small details that other methods miss, whereas VEEGAN +DAE produces meaningless or implausible images.
6 Conclusion
The paper presents VEEGAN as a variational GAN-training principle combining joint-space KL divergence with a representation-space autoencoder. Its guarantee is attained when the reconstructor matches the relevant joint distribution and inverts the generator on the data distribution.
- Conclusion: VEEGAN combines joint representation–data KL divergence with an autoencoder over the representation space.The training principle is motivated by a variational argument.
- Theoretical guarantee: If parameters attain O(γ*, θ*) = H[p0], they minimize O and satisfy the proposition’s equality conditions.Here H denotes the Shannon entropy of the prior p0.
- Objective: The objective decomposes into a joint-space KL term, the negative log prior term, and a reconstruction-distance term.The reconstruction term is nonnegative, and the entropy term is defined by H[p0].
- Theoretical guarantee: The bound is attained when qγ(x|z)p0(z) equals pθ(z|x)p(x) and Fθ inverts Gγ on the data distribution.The inversion requirement applies on the generator’s data range, not necessarily outside it.
C Discriminator Architecture for ALI and VEEGAN
ALI and VEEGAN augment the DCGAN discriminator to process image–noise pairs, while VEEGAN can also support inference through its reconstructor.
- The ALI and VEEGAN discriminator operates on pairs of images and noise vectors.Its architecture is documented in Table 3.
- The discriminator flattens DCGAN’s final convolutional layer and concatenates it with the input noise vector.
- For inference, VEEGAN’s reconstructor extracts latent vectors from real images, which the generator then uses to produce reconstructed samples.
E Adversarial Methods for Autoencoders
The data-autoencoder variant of VEEGAN was evaluated against the noise-autoencoding approach, but parameter sweeps failed to produce meaningful CIFAR-10 images.
- A data-autoencoder version of VEEGAN used the reconstructor as an inference network to contrast data and noise autoencoding.
- Parameter sweeps over λ failed to produce meaningful images for any tested value in the data-autoencoder experiment.
- Figure 5 compares CIFAR-10 samples from GANs using data autoencoders.
- Figure 5d was generated entirely from the ℓ2 loss.
F Stacked MNIST Qualitative Results
The Stacked MNIST section presents qualitative samples from trained models and compares all four methods visually.
- Figure 6 provides the visual comparison across all four methods.
- The results comprise qualitative samples from trained models on the Stacked MNIST dataset.
G CelebA Random Sample from ALI and VEEGAN
On CelebA, the simple untuned architecture produced meaningful images with VEEGAN but not with ALI, within this experiment’s limited setup.
- VEEGAN generated high-quality CelebA face images, whereas ALI failed to produce meaningful images with the simple architecture.
- The comparison reused the CIFAR-10 architectures and hyperparameters without extensive tuning on the larger CelebA dataset.
- The experiment does not establish that ALI generally fails on CelebA, because higher-capacity networks and suitable hyperparameters can produce good images.
H CIFAR 10 Random Sample from VEEGAN
The CIFAR-10 sample comparison presents randomly generated samples from four methods: DCGAN, ALI, Unrolled GAN, and VEEGAN.
- Randomly generated CIFAR-10 samples are shown for all four methods.
- The comparison includes DCGAN on CIFAR-10.
- The comparison includes ALI on CIFAR-10.
- The comparison includes Unrolled GAN on CIFAR-10.
- The comparison includes VEEGAN on CIFAR-10.