Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder; Sebastian Nowozin; Andreas Geiger

Source-linked AI summary

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder, Sebastian Nowozin, Andreas Geiger

arXiv:1701.04722v4Jan 17, 2017cs.LG

TL;DR

Standard VAEs depend on inference models that may not capture true posteriors, limiting the quality of learned generative models. AVB addresses this with adversarial training for arbitrarily flexible inference networks and recovers the true posterior and maximum-likelihood generative parameters in the nonparametric limit.

Problem
Standard VAE inference models may be insufficiently expressive to capture true posterior distributions, affecting visual results and likelihood bounds.
Method
AVB uses adversarial training with arbitrarily flexible neural-network inference models and an auxiliary discriminator to approximate the VAE objective.
Results
In the nonparametric limit, AVB recovers the true posterior and a maximum-likelihood assignment for the generative-model parameters.
Takeaways & Limitations
AVB enables Variational Bayes with neural samplers and supports rich posterior distributions for latent-variable modeling.
Takeaways & Limitations
In practice, the discriminator may fail to approximate its optimum because AVB contrasts substantially different densities, where logistic regression is less effective for likelihood-ratio estimation.

Abstract

from arXiv · show

Variational Autoencoders (VAEs) are expressive latent variable models that can be used to learn complex probability distributions from training data. However, the quality of the resulting model crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VAEs and Generative Adversarial Networks (GANs). We show that in the nonparametric limit our method yields an exact maximum-likelihood assignment for the parameters of the generative model, as well as the exact posterior distribution over the latent variables given an observation. Contrary to competing approaches which combine VAEs with GANs, our approach has a clear theoretical justification, retains most advantages of standard Variational Autoencoders and is easy to implement.

1. Introduction

The paper motivates AVB by the limited expressiveness of standard VAE inference models and introduces adversarial training to support flexible inference while preserving a maximum-likelihood foundation.

Standard VAEs provide both generative and inference models, whereas GANs often produce sharper natural-image results and VAEs often achieve better log-likelihoods.
Limited inference-model expressiveness can prevent VAEs from capturing true posteriors, contributing to blurry images and weaker likelihood bounds.
AVB trains VAEs with arbitrarily flexible neural-network inference models through adversarial training.
In the nonparametric limit, AVB recovers the true posterior and a maximum-likelihood assignment for the generative-model parameters.
AVB also provides a Variational Bayes method using neural samplers with intractable densities to approximate complex posterior distributions.
The paper empirically reports rich posterior learning and compelling samples on complex datasets.

2. Background

The VAE background defines the generative and inference models, explains the ELBO approximation to maximum likelihood, and identifies inference-model expressiveness as a central limitation.

A VAE combines a generative model pθ(x | z), a prior p(z), and an approximate inference model qφ(z | x).
The variational lower bound, or ELBO, becomes exact when the inference model equals the true posterior.
In general, the inference model differs from the true posterior, leaving only a lower-bound relationship.
Maximum-likelihood training seeks to optimize the marginal log-likelihood, but computing it usually requires intractable marginalization over latent variables.
Variational Bayes replaces the intractable maximum-likelihood problem with optimization of the variational lower bound.
The lower bound depends on qφ(z | x)'s expressiveness; diagonal-covariance Gaussian inference can restrict dependence on z and yield blurry natural-image outputs.

3. Method

AVB replaces the restrictive explicit inference model with a flexible black-box sampler and an auxiliary discriminator, training them as a two-player game. Under stated expressiveness assumptions, equilibria recover the variational optimum and the true posterior.

3.1. Derivation: AVB uses adversarial training with a black-box inference model qφ(z | x) to obtain an approximate maximum-likelihood assignment and posterior approximation.Noise is supplied as an input to the inference network, allowing it to learn complex probability distributions.
3.1. Derivation: The auxiliary discriminator T(x, z) implicitly represents a variational term by distinguishing independently sampled pairs pD(x)p(z) from inference-model pairs pD(x)qφ(z | x).Its optimal value is the negative of the corresponding variational term.
3.2. Algorithm: The algorithm jointly applies SGD updates to generative, inference, and discriminator parameters after sampling data, prior latents, and Gaussian noise.The procedure computes θ-, φ-, and ψ-gradients before updating all three parameter sets.
3.2. Algorithm: AVB treats discriminator and variational objectives as a two-player game because maintaining the exact optimal discriminator during optimization is computationally challenging.At a Nash equilibrium, the game yields a stationary point of the variational objective.
3.3. Theoretical results: If T can represent any function, every Nash equilibrium defines a global optimum of the variational lower bound; with an expressive qφ, it also recovers the true posterior.Under these nonparametric assumptions, the optimal discriminator corresponds to pointwise mutual information between x and z.

4. Adaptive Contrast

Adaptive Contrast improves adversarial density-ratio estimation by contrasting the inference model with an adaptive approximation rather than the prior. This makes the adversary’s task easier while preserving the AVB training framework.

Motivation: In practice, AVB’s discriminator may fail to approximate its optimum because it contrasts substantially different densities.The compared densities are pD(x)qφ(z | x) and pD(x)p(z), whereas logistic regression estimates likelihood ratios best for similar densities.
Method: Adaptive Contrast introduces a known-density auxiliary distribution rα(z | x) that approximates qφ(z | x).A diagonal Gaussian matching qφ(z | x)’s mean and variance is one proposed choice.
Method: Contrasting qφ(z | x) with rα(z | x) replaces comparison against the prior and reduces the divergence the adversary must model.When rα approximates qφ well, KL(qφ(z | x), rα(z | x)) is usually much smaller than KL(qφ(z | x), p(z)).
Method: The Gaussian auxiliary distribution can be standardized so the adversary models deviations from Gaussianity rather than location and scale.The reparameterization uses normalized latent variables with mean 0 and variance 1.

5. Experiments

The experiments evaluate AVB for posterior approximation and generative modeling. AVB captures multimodal posteriors, improves synthetic-data metrics, and achieves strong binarized-MNIST results with perceptually close samples.

Experiments: AVB is evaluated both for black-box variational inference and for learning generative models.The variational-inference setting fixes the generative model and observation while learning qφ(z | x).
5.2. Generative Models: AVB produces visually sharp celebA faces, reasonably sharp reconstructions, and realistic outputs along latent-space interpolations.The celebA experiment is reported as an additional evaluation in the supplementary material.
5.1. Variational Inference: On Eight Schools, AVB clearly outperforms Gaussian-inference methods in KL divergence to HMC ground truth and captures posterior multi-modality.The qualitative comparison examines two marginals of the 10-dimensional posterior.
5.2. Generative Models: On the synthetic dataset, AVB learns a complex posterior while the standard VAE partitions latent space into a mixture of four Gaussians.The dataset contains four 2 × 2 binary images and uses a 2-dimensional latent space.
5.2. Generative Models: AVB achieves a higher likelihood score near the synthetic dataset’s optimum of −log(4) than the standard VAE.The synthetic-data comparison also reports lower reconstruction cross-entropy and a variational lower bound close to true log-likelihood.
5.2. Generative Models: Using Adaptive Contrast, AVB improves on both a standard VAE and AVB without Adaptive Contrast.The authors report state-of-the-art results on binarized MNIST and perceptually close random samples.

6. Related Work

AVB combines adversarial training with the standard VAE objective to support highly expressive inference models while preserving a maximum-likelihood perspective. The approach differs from related adversarial methods in its probabilistic-inference goal and joint optimization.

AVB approximates the VAE’s Kullback-Leibler divergence with an adversary rather than a closed-form formula.
Expressive inference models: The related-work comparison notes that normalizing flows and auxiliary-variable VAEs increase posterior flexibility, while AVB targets arbitrarily expressive inference models.
Connection to Adversarial Autoencoders: Restricting the adversary to functions independent of x approximates the AVB formulation but need not recover the true posterior for each observation.The aggregated latent marginal may be correct even when individual observations contribute inaccurately.
Connection to Adversarial Autoencoders: Unlike Adversarial Autoencoders, AVB seeks to improve the ELBO through better probabilistic inference.This supports settings focused on the inference network itself and enables Adaptive Contrast.
f-divergences: AVB can be formulated as a mini-max zero-sum game using any f-divergence for discriminator training.The authors observed unstable training with one objective and therefore used the standard GAN objective corresponding to the Jensen-Shannon divergence.
Relation to BiGANs: BiGANs apply an adversary to pairs of observations and latent codes, whereas AVB jointly optimizes the generative and inference models.

7. Conclusion

The paper presents adversarial training as a procedure for making VAE inference models substantially more flexible. It identifies neural-network architectures and contrasting distributions as directions for further progress.

Adversarial training allows the inference model to represent almost any family of conditional distributions over latent variables.
The authors propose investigating adversary, encoder, and decoder architectures together with better contrasting distributions.

Supplementary Material for Adversarial Variational Bayes: Unifying Variational Autoencoders and

The supplementary material includes the paper’s Generative Adversarial Networks section.

The supplementary material contains a section titled “Generative Adversarial Networks.”

Abstract

The supplementary material accompanies the main AVB paper with omitted proofs, experiment details, and additional results.

The supplementary material provides omitted proofs, further experiment details, and additional results for AVB.

I. Proofs

The proofs establish the optimal discriminator and show that, under a nonparametric discriminator, Nash equilibria maximize the variational lower bound.

Proposition 1: For fixed pθ(x | z) and qφ(z | x), Proposition 1 identifies the optimal discriminator through the density ratio qφ(z | x) / p(z).Its sigmoid equals qφ(z | x) / [qφ(z | x) + p(z)].
Proposition 2: Gradients with respect to a fixed optimal discriminator provide unbiased gradients of the ELBO.This proposition supplies the practical justification for optimizing the variational objective through the adversarial formulation.
Proposition 3: If the discriminator can represent any function and the players reach a Nash equilibrium, the resulting (θ∗, φ∗) is a global optimum of the variational lower bound.The proof derives the optimal discriminator, substitutes it into the game objective, and shows that any non-maximizing pair would be contradicted by a better pair.
Proposition 3: The contradiction argument assumes a competing (θ′, φ′) with a strictly larger variational lower bound than the equilibrium pair.Together with the discriminator characterization, this completes the theorem.

II. Adaptive Contrast

Adaptive Contrast compares the inference model with an adaptive distribution and uses a moment-efficient encoder architecture. Experiments report competitive likelihoods and sharper learned representations across MNIST and celebA settings.

Adaptive Contrast: Adaptive Contrast replaces comparison with the prior by comparison with an adaptive distribution, while treating its μ(k) and σ(k) as fixed during backpropagation.Algorithm 2 alternates sampling, gradient computation for θ, φ, and ψ, and SGD updates.
Encoder architecture: The Adaptive Contrast encoder represents z as a linear combination of noise-basis vectors whose coefficients depend on x.The basis vectors come from small networks applied to Gaussian noise, while coefficients come from a convolutional network applied to x.
Encoder architecture: Sampling basis-vector moments once per mini-batch enables efficient computation of qφ(z | x) moments for every data point in that mini-batch.This architecture supports the moment estimates required by Adaptive Contrast.
celebA: For celebA, AVB produces visually sharp face samples, reasonably sharp reconstructions, and realistic images along latent-space interpolations.The model uses a 64-dimensional latent space with a standard-Gaussian prior and deep convolutional networks.
MNIST: AVB and auxiliary-variable VAE achieve better approximate ELBOs than standard VAE in the first fixed-decoder MNIST experiment.The reported AVB + AC row gives ELBO ≈−85.1 ± 0.2 and AIS −83.7 ± 0.3, versus VAE ELBO −88.9 ± 0.2 and AIS −85.0 ± 0.3.
MNIST: Across the reported MNIST decoder settings, AVB + AC has competitive AIS values and often improves reconstruction error or approximate ELBO relative to comparison methods.The table reports AVB + AC rows for three settings, including AIS values −83.7 ± 0.3, −89.7 ± 0.3, and −81.7 ± 0.3.
MNIST: With a 32-dimensional latent space and the shallow convolutional decoder, AVB improves approximate ELBO and reconstruction error, while all methods achieve similar log-likelihoods.The authors attribute the lack of improvement from auxiliary variables and IAF to optimization challenges.