PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Approaches for Auto-Encoding Generative Adversarial Networks

Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed

arXiv:1706.04987v2stat.MLcs.LG

TL;DR

GANs can produce compelling samples but suffer mode collapse, while VAEs support inference without collapse but often produce blurry images. The paper combines variational inference with discriminator-based synthetic likelihoods and implicit posteriors to derive α-GAN, whose evaluation shows strong but metric-dependent performance. The approach also supports reconstruction and representation-learning applications within the discussed scope.

  • Problem

    GANs face mode collapse and VAEs often produce blurry images, motivating a method that addresses these complementary limitations while learning latent representations.

  • Method

    The paper combines the variational lower bound with density-ratio estimation, replacing intractable likelihoods with synthetic likelihoods and posterior distributions with implicit distributions learned using discriminators.

  • Results

    The paper reports that α-GAN performs well across a battery of evaluation measures, while CIFAR-10 results show that conclusions depend strongly on the evaluation metric.

  • Takeaways & Limitations

    The hybrid framework provides a unified unsupervised objective combining adversarial learning with reconstruction, with potential uses in inpainting and representation learning.

  • Takeaways & Limitations

    The method relies on an explicit zero-mean Laplace likelihood with an L1 reconstruction loss, and sample-diversity scores must be interpreted relative to the input-data diversity.

Abstract

from arXiv · show

Auto-encoding generative adversarial networks (GANs) combine the standard GAN algorithm, which discriminates between real and model-generated data, with a reconstruction loss given by an auto-encoder. Such models aim to prevent mode collapse in the learned generative model by ensuring that it is grounded in all the available training data. In this paper, we develop a principle upon which auto-encoders can be combined with generative adversarial networks by exploiting the hierarchical structure of the generative model. The underlying principle shows that variational inference can be used a basic tool for learning, but with the in- tractable likelihood replaced by a synthetic likelihood, and the unknown posterior distribution replaced by an implicit distribution; both synthetic likelihoods and implicit posterior distributions can be learned using discriminators. This allows us to develop a natural fusion of variational auto-encoders and generative adversarial networks, combining the best of both these methods. We describe a unified objective for optimization, discuss the constraints needed to guide learning, connect to the wide range of existing work, and use a battery of tests to systematically and quantitatively assess the performance of our method.

1 Introduction

The paper proposes a principled hybrid of auto-encoders and GANs, combining variational inference with adversarial learning to address their complementary limitations. It introduces α-GAN as a unified, unsupervised framework and evaluates it systematically against existing GAN models.

  • GANs generate compelling samples and scale to large datasets but can suffer optimization instabilities and mode collapse.
  • Existing AE-GANs combine auto-encoder objectives with GAN training through discriminator, auxiliary-generator, or related hybrid strategies.
  • The proposed approach combines VAEs and GANs to pair VAEs’ inference and mode-collapse resistance with GANs’ flexible distributional assumptions.
  • The method is purely unsupervised, requires no pre-training or external classifiers, and is evaluated with a battery of measures against DC-GAN, Wasserstein GAN, and AGE.
  • The paper develops α-GAN, a principled objective for auto-encoding GANs, and discusses practical constraints needed for optimization.

2 Overcoming Intractability in Generative Models

The section presents GAN discriminators and variational inference as complementary tools for learning when marginal likelihoods are intractable. Synthetic likelihoods extend variational inference to likelihood-free models by replacing unavailable likelihood terms with discriminator-estimated density ratios.

  • Latent Variable Models: Latent variable models generate observed samples by transforming latent noise through a conditional distribution or generator.
  • Latent Variable Models: Implicit models make no observation-noise assumptions and represent the likelihood with a point mass at the generator output.
  • The marginal likelihood integrates out latent variables but is intractable in both implicit and prescribed models, complicating parameter learning.
  • Generative Adversarial Networks: GANs avoid computing the marginal likelihood by training a discriminator whose ability to distinguish real from generated samples supplies the learning signal.
  • Variational Inference: Variational inference approximates the marginal likelihood with a lower bound optimized over a variational distribution qη(z|x).
  • Synthetic Likelihoods: When likelihoods are unknown, synthetic likelihoods replace them with substitutes estimated through discriminator-based density-ratio comparisons.

3 A Fusion of Variational and Adversarial Learning

The paper transforms a VAE objective toward a GAN-like hybrid by using implicit variational distributions, explicit or synthetic likelihoods, and discriminator-based losses. The resulting system combines reconstruction with adversarial training and uses multiple networks to optimize the model.

  • The hybrid construction starts from the VAE and progressively incorporates GAN mechanisms.
  • Implicit Variational Distributions: Implicit variational distributions avoid restrictive Gaussian assumptions by using density-ratio estimation to replace the KL-divergence term.
  • Implicit Variational Distributions: A latent classifier distinguishes encoder-produced latent variables from samples drawn from a standard Gaussian.
  • Likelihood Choice: The explicit likelihood choice is a zero-mean Laplace distribution, yielding an L1 reconstruction loss based on the generator output.
  • Likelihood Choice: Synthetic likelihoods provide an adversarial alternative, while explicit likelihoods assign mass across the output support and can help prevent collapse; the final loss can weight both.
  • Hybrid Loss Functions: The objective combines an L1 reconstruction term with adversarial terms for generated data and latent codes.
  • Hybrid Loss Functions: α-GAN requires a generator, encoder, synthetic-likelihood discriminator, and latent-code discriminator, with alternating updates for optimization.
  • Improved Techniques: Using reverse-KL discriminator losses, and treating samples as fake alongside reconstructions, is reported to improve optimization stability, learning speed, and performance.

4 Related work

The section situates α-GAN among hybrid and pure generative latent-variable models, contrasting how related methods combine inference, reconstruction, and adversarial objectives.

  • Hybrid VAE-GAN approaches can be classified by whether the density-ratio trick targets the likelihood, prior approximation, or both.
  • DCGAN and WGAN-GP are pure GAN variants without auto-encoder loss or inference, while WGAN-GP uses a Wasserstein-distance critic.
  • ALI and BiGAN perform inference through an adversarial game between encoder and decoder using a discriminator over x, z pairs.
  • α-GAN matches distributions in both x and z spaces, but adds pointwise reconstruction only in x space rather than code reconstruction in z space.
  • MRGANs use separate discriminators for data versus reconstructions and data versus samples, whereas α-GAN passes both through the discriminators.

5 Evaluation metrics

Evaluating generative models, especially GANs, is difficult because likelihood is unavailable, motivating proxy metrics with different strengths and weaknesses. The section describes inception score, MS-SSIM-based diversity, and an independent Wasserstein critic for assessing generated samples.

  • Likelihood-free GAN evaluation motivates using multiple proxy metrics whose strengths and weaknesses are assessed experimentally.The paper identifies generative-model evaluation as challenging and specifically attributes GAN difficulty to the lack of likelihood.
  • Inception score evaluates whether generated samples are highly classifiable and diverse with respect to class labels using a pretrained classifier.It averages KL divergences between conditional label distributions for samples and the marginal label distribution.
  • MS-SSIM addresses within-class mode collapse that inception score can miss by measuring similarity between class-conditional generated samples.Because the models are not class conditional, the paper uses MS-SSIM for evaluation on CelebA.
  • The reported sample diversity score is 1-MSSSIM, interpreted relative to input-data diversity because excessive diversity may indicate failure to capture the data distribution.The paper applies this metric on CelebA, where data variability is smaller, and cautions against comparing it without the input-data reference.
  • An independent Wasserstein critic distinguishes held-out validation data from generated samples, measuring both overfitting and mode collapse.A generator that memorizes training data or suffers mode collapse makes the critic’s discrimination task easier.

6 Experiments

Experiments compare α-GAN with AGE and pure GAN variants across ColorMNIST, CelebA, and CIFAR-10 using visual and numerical evaluations. Results show competitive performance, reconstruction advantages, metric disagreement, and strong sensitivity to network architecture.

  • Experimental setup: The study compares α-GAN, AGE, DCGAN, and WGAN-GP across ColorMNIST, CelebA, and CIFAR-10 using visual samples and numerical metrics.The evaluation reports best values and quartiles across hyperparameter sweeps.
  • ColorMNIST: On ColorMNIST, most tested hyperparameters exceed the best DCGAN result on the independent Wasserstein critic, but WGAN-GP scores best.Higher values are better for this metric, and the result is supported by generated samples.
  • CelebA: On CelebA, α-GAN is competitive with WGAN-GP and AGE, while WGAN-GP produces the best results and has a narrower spread.The comparison uses the independent Wasserstein critic and sample diversity score.
  • Reconstructions: α-GAN and AGE reconstruct inputs, unlike WGAN and DCGAN, and α-GAN produces better reconstructions than AGE.The reconstruction comparison is reported in Appendix C.
  • CIFAR-10: On CIFAR-10, WGAN-GP performs best on the independent critic and CIFAR-10 Inception score, whereas α-GAN performs best on the ImageNet-based Inception score.The two Inception metrics produce different rankings, and 15% of hyperparameter-sweep jobs were ranked in the top 50% by ImageNet Inception but in the bottom 50% by the CIFAR-10-trained metric.
  • Experimental insights: Network architecture strongly affects quality across algorithms, with minor changes sometimes causing catastrophic mode collapse.The authors report that sufficient capacity can make DCGAN robust on the tested datasets.

7 Discussion

The paper combines variational lower bounds with the density-ratio trick to derive α-GAN, which pairs adversarial and reconstruction losses. It frames this hybrid as addressing blurriness and mode collapse while using complementary evaluation metrics.

  • α-GAN combines an adversarial loss with a data reconstruction loss derived from a variational lower bound and the density-ratio trick.
  • From the VAE perspective, the learned adversarial loss is intended to address blurry samples.
  • From the GAN perspective, reconstruction loss grounds the generator using perceptual similarity on the data to address mode collapse.
  • Evaluation across three datasets and multiple metrics indicates that the employed metrics are complementary and assess different GAN failure modes.
  • VAE-GAN hybrids can support reconstruction-based inpainting and inference-network-based representation learning, including disentangled representations with an appropriate latent prior.

A Model Samples

This section presents the training procedure and visual examples for model samples and reconstructions. The procedure updates the encoder and generator while discriminators distinguish data, generated or reconstructed samples, and latent codes.

  • Figure 7 enlarges the CelebA samples from Figure 4 for DCGAN, WGAN-GP, AGE, and α-GAN.
  • The overall training procedure is summarized in Algorithm 1.
  • Reconstructions from α-GAN and AGE are shown for CelebA and CIFAR-10.
  • The encoder samples z-hat from qη(z|x) and reconstructs x as x-hat = Gθ(z-hat).
  • The encoder is updated using reconstruction and generation loss from the code discriminator.
  • The generator is updated using reconstruction and generation loss.
  • The data discriminator treats real data as real and reconstructions and generated samples as fake, while the code discriminator treats prior codes as real and variational codes as fake.

D Ablation experiment: code discriminator and the empirical KL

The ablation compares density-ratio estimation of the KL term with an empirical KL approximation in α-GAN. It evaluates visual samples, prior matching, and autoencoder-code statistics, while leaving combined approaches for future work.

  • The empirical KL approximation uses per-dimension mini-batch means m_i and variances s_i for latent vectors under a normal prior.
  • Replacing the code discriminator with the KL approximation enables comparison through visual inspection and prior-matching evaluation.
  • The ablation also examines autoencoder-code means and covariance distributions for each latent.
  • The approximation is divided by latent size to allow the same hyperparameters across different latent sizes.
  • The authors leave investigating the effects and combining the two KL-estimation approaches for future work.

G Relationships between different metrics

This section relates evaluation metrics to model assessment using independent Wasserstein criticism and reconstruction examples. The supplied passages specify the reconstruction layout but do not state its outcome.

  • The study assesses correlation between sample quality and model quality according to an independent Wasserstein critic.
  • Figure 9 places CIFAR-10 data on the left and reconstructions on the right.

H Training details: hyperparameters and network architectures

The experiments used fixed optimization settings alongside model-specific loss, prior, latent-size, and architecture choices. Additional figures examined latent matching, training behavior, sample quality, and diversity for α-GAN.

  • Optimization settings: All models used a fixed learning rate, Adam with β1 = 0.5 and β2 = 0.9, batch size 64, and batch normalization.ColorMNIST models were trained for 100000 iterations.
  • Model-specific choices: AGE used l1 data reconstruction, cosine code reconstruction, and a uniform unit-ball prior, whereas α-GAN used l1 reconstruction and traditional GAN losses with a normal prior.The encoder output was projected to the unit ball for AGE.
  • Network architectures: Baselines shared discriminator and generator architectures and controlled latent counts, while α-GAN used a transposed-generator convolutional encoder and a three-layer, 750-unit-per-layer code discriminator.The encoder had no activation function after its output, and discriminator leaky-unit slope was 0.2 while generators used ReLUs.
  • Dataset-specific architectures: ColorMNIST experiments swept latent sizes 10, 50, and 75, using common discriminator and generator architectures described in Tables 3 and 4.Table 3 notes dropout of 0.8 after the final convolution only for DCGAN.
  • Evaluation setup: CelebA and CIFAR-10 used WGAN architectures, while a VGG-style CIFAR-10 classifier reported inception scores with specified normalization, cropping, and momentum-training schedules.The WGAN-GP inception-score architecture differed from the one used here, which was based on the classifier described in Table 5.
Loading 1706.04987v2…