PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Encoding Variational Bayes

Diederik P Kingma, Max Welling

arXiv:1312.6114v11stat.MLcs.LG

TL;DR

Intractable continuous-variable posteriors and large datasets make efficient inference and learning difficult. The paper introduces reparameterized stochastic variational estimators and AEVB, whose theoretical advantages are reflected in experiments.

  • Problem

    Continuous latent variables with intractable posteriors and large datasets challenge efficient approximate inference and learning in directed probabilistic models.

  • Method

    The paper reparameterizes the variational lower bound for differentiable stochastic optimization and uses it to train a recognition model through AEVB.

  • Results

    Theoretical advantages of SGVB and AEVB are reflected in experiments on generative image models trained on MNIST and Frey Face datasets.

  • Takeaways & Limitations

    AEVB enables efficient approximate posterior inference using simple ancestral sampling while avoiding expensive iterative inference per datapoint.

  • Takeaways & Limitations

    The naive Monte Carlo gradient estimator has very high variance and is impractical for the targeted optimization problem.

Abstract

from arXiv · show

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

1 Introduction

The section introduces a stochastic variational method for efficient inference and learning with continuous latent variables and intractable posteriors. For i.i.d. data, AEVB uses an optimized recognition model to enable efficient approximate inference by ancestral sampling without expensive iterative schemes.

  • 1 Introduction: The work addresses approximate inference and learning when continuous latent variables or parameters have intractable posterior distributions.Common mean-field variational Bayes also requires generally intractable expectations under the approximate posterior.
  • 1 Introduction: The AEVB algorithm applies the SGVB estimator to optimize a recognition model for i.i.d. datasets with continuous latent variables per datapoint.This makes inference and learning especially efficient.
  • 1 Introduction: The optimized recognition model enables efficient approximate posterior inference using simple ancestral sampling.The resulting inference procedure avoids expensive iterative inference schemes while supporting efficient model-parameter learning.

2 Method

The method targets efficient parameter estimation and approximate inference in directed models with continuous latent variables, intractable posteriors, and datasets too large for batch optimization. It introduces a recognition model and reparameterized stochastic lower-bound estimator whose parameters are optimized jointly.

  • Problem setting: The setting allows intractable marginal likelihoods, posteriors, and mean-field variational integrals, including models with nonlinear neural-network likelihoods.The method assumes differentiable parametric prior and likelihood densities with respect to θ and z.
  • Problem setting: The method is designed for large datasets where batch optimization is too costly and minibatch or single-datapoint updates are preferred.Sampling-based alternatives such as Monte Carlo EM are described as generally too slow because of their per-datapoint sampling loop.
  • Recognition model: A recognition model qφ(z|x) approximates the intractable posterior pθ(z|x) and is learned jointly with the generative parameters θ.Unlike mean-field variational posteriors, the recognition model need not be factorial and its parameters are not obtained from closed-form expectations.
  • Variational objective: The variational lower bound L(θ, φ; x(i)) is optimized with respect to both variational and generative parameters, but its naïve Monte Carlo gradient estimator has very high variance.The lower bound equals the marginal log likelihood minus a nonnegative KL divergence from the approximate to the true posterior.
  • Reparameterization: Reparameterization expresses z sampled from qφ(z|x) as z = gφ(ϵ, x), where ϵ has independent marginal p(ϵ), enabling the SGVB stochastic estimator of the lower bound.The transformation gφ(ϵ, x) is required to be differentiable under mild conditions.

3 Example: Variational Auto-Encoder

This example uses neural networks for both probabilistic encoding and decoding in a variational auto-encoder, jointly optimizing encoder and generative parameters with AEVB. Gaussian prior and approximate posterior choices enable reparameterized sampling and exact KL computation, while the true posterior remains intractable.

  • Model and optimization: The probabilistic encoder q_φ(z|x) is a neural network, with encoder parameters φ and generative parameters θ optimized jointly by AEVB.The encoder approximates the generative model’s posterior.
  • Model and optimization: The model uses a centered isotropic Gaussian prior, while p_θ(x|z) is Gaussian for real-valued data or Bernoulli for binary data, with parameters computed by a one-hidden-layer MLP.The true posterior p_θ(z|x) is intractable, and the simplifying prior choice is not a limitation of the method.
  • Approximate posterior: The approximate posterior’s mean and standard deviation are nonlinear outputs of an encoding MLP applied to each datapoint x^(i) and variational parameters φ.These outputs define µ^(i) and σ^(i).
  • Approximate posterior: Posterior samples use the reparameterization z^(i,l) = µ^(i) + σ^(i) ⊙ ε^(l), with ε^(l) sampled from N(0, I), enabling gradient-based optimization.Because both p_θ(z) and q_φ(z|x) are Gaussian, the KL divergence is computed and differentiated without estimation.
  • Decoder: The decoding term log p_θ(x^(i)|z^(i,l)) is implemented by a Bernoulli or Gaussian MLP according to the modeled data type.This provides the likelihood contribution for binary or real-valued observations.

4 Related work

AEVB is situated among prior methods combining recognition models, stochastic variational inference, and autoencoding structures. The paper contrasts its approach with wake-sleep, variance-reduction methods, and related autoencoder–probabilistic-model connections.

  • Wake-sleep: Wake-sleep is the only other known online method applicable to the same general class of continuous latent-variable models, but it jointly optimizes two mismatched objectives.Like AEVB, wake-sleep uses a recognition model to approximate the true posterior.
  • Stochastic variational inference: Prior stochastic variational-inference work introduced control-variate schemes and related reparameterizations to reduce gradient-estimator variance.These methods include applications to exponential-family posterior approximations and general variance-reduction techniques.
  • Autoencoder connections: AEVB connects directed probabilistic models trained with variational objectives to autoencoders, extending established links between linear autoencoders and generative linear-Gaussian models.Earlier work showed that PCA is maximum likelihood for a special linear-Gaussian model.
  • Autoencoder connections: Unregularized autoencoder training has been related to maximizing a lower bound on mutual information between inputs and latent representations.The connection proceeds through conditional entropy and expected log-likelihood under the autoencoding model.
  • Related autoencoder methods: DARN also learns directed probabilistic models with an autoencoding structure but applies to binary latent variables, while RMW14 independently connected autoencoders, directed models, and stochastic variational inference.RMW14 used the same reparameterization trick described in this paper and offered an additional perspective on AEVB.

5 Experiments

Experiments evaluated AEVB on MNIST and Frey Face by comparing variational lower-bound optimization and estimated marginal likelihood against wake-sleep and Monte Carlo EM. AEVB used stochastic-gradient optimization and achieved faster convergence and better lower-bound solutions across latent-space dimensionalities.

  • Experimental setup: AEVB and wake-sleep were compared on generative image models trained on MNIST and Frey Face using the variational lower bound and estimated marginal likelihood.The experiments used 500 hidden units for MNIST and 200 for Frey Face.
  • Likelihood lower bound: AEVB converged considerably faster than wake-sleep and reached a better variational lower-bound solution in all experiments.The comparison covered different latent-space dimensionalities, and increasing the number of latent variables did not produce more overfitting because of the lower bound’s regularizing effect.
  • Optimization: Parameters were jointly optimized by stochastic gradient ascent on the lower-bound estimator with a small weight-decay term corresponding to p(θ) = N(0, I).This objective is equivalent to approximate MAP estimation with the likelihood gradient approximated by the lower-bound gradient.
  • Marginal likelihood: For very low-dimensional latent spaces, AEVB, wake-sleep, and Monte Carlo EM were compared using MCMC estimates of the learned models’ marginal likelihood.The marginal-likelihood estimates used MNIST models with 100 hidden units and 3 latent variables; higher-dimensional estimates were unreliable.
  • Visualisation of high-dimensional data: A low-dimensional latent space, such as 2D, allowed the learned recognition models to project high-dimensional MNIST and Frey Face data onto a low-dimensional manifold.The paper refers to appendix A for visualizations of the resulting 2D latent manifolds.

6 Conclusion

The paper introduces SGVB, a differentiable stochastic estimator of the variational lower bound for efficient approximate inference with continuous latent variables. It also presents AEVB for efficient inference and learning on i.i.d. datasets with continuous latent variables per datapoint.

  • 6 Conclusion: SGVB estimates the variational lower bound for efficient approximate inference with continuous latent variables.The estimator is introduced as a stochastic gradient variational Bayes method.
  • 6 Conclusion: SGVB can be straightforwardly differentiated and optimized using standard stochastic gradient methods.Its differentiability enables direct optimization with standard stochastic gradient procedures.
  • 6 Conclusion: AEVB provides an efficient algorithm for inference and learning with i.i.d. datasets containing continuous latent variables per datapoint.The method is introduced for the i.i.d. dataset setting with continuous latent variables associated with each datapoint.

7 Future work

The authors identify several future directions because SGVB and AEVB apply to nearly any inference and learning problem with continuous latent variables.

  • 7 Future work: Future work includes jointly training deep hierarchical generative architectures, using neural-network encoders and decoders such as convolutional networks, with AEVB.The proposed direction targets hierarchical architectures and joint encoder–decoder training.
  • 7 Future work: Other directions include applying the methods to time-series models, global parameters, and supervised models with latent variables.The passage specifically mentions dynamic Bayesian networks as time-series models and extending SGVB to global parameters.

A Visualisations

The section visualizes latent and observed spaces learned with SGVB/AEVB, including the learned data manifold and random MNIST samples across latent-space dimensionalities.

  • Visualisations: Figures 4 and 5 visualize the latent space and corresponding observed space of models learned with SGVB.The visualizations depict learned model structure in both latent and observed domains.
  • Visualisations: Figure 4 visualizes the learned data manifold for generative models with a two-dimensional latent space learned with AEVB.Linearly spaced unit-square coordinates are transformed through the inverse Gaussian CDF to obtain latent variables z.
  • Visualisations: Figure 5 shows random samples from learned MNIST generative models for different latent-space dimensionalities.The figure compares generated samples across models with varying latent-space sizes.

B Solution of −DKL(qφ(z)||pθ(z)), Gaussian case

The variational lower bound includes a KL term that can often be integrated analytically. For a standard normal prior and Gaussian posterior approximation, the solution is specified componentwise in the latent dimensionality.

  • Gaussian case: The KL term in the variational lower bound is analytically tractable in the Gaussian case.The objective is a variational lower bound to be maximized.
  • Gaussian case: The Gaussian setup uses prior pθ(z) = N(0, I) and posterior approximation qφ(z|x(i)), with latent dimensionality J.The variational mean µ and standard deviation σ are evaluated at datapoint i.
  • Gaussian case: The KL solution is expressed componentwise using µj and σj, the j-th elements of the variational mean and standard-deviation vectors.Under a recognition model, µ and σ become functions of x and variational parameters φ.

C MLP’s as probabilistic encoders and decoders

Variational auto-encoders use neural networks as probabilistic encoders and decoders, with architecture and output distributions chosen according to the data and model. In the example, simple MLPs implement Gaussian-output encoding and Gaussian- or Bernoulli-output decoding.

  • C MLP’s as probabilistic encoders and decoders: Variational auto-encoders use neural networks as probabilistic encoders and decoders.The passage notes that encoder and decoder choices depend on the data type and model.
  • C MLP’s as probabilistic encoders and decoders: The example uses relatively simple multi-layered perceptrons (MLPs) for both encoding and decoding.The encoder uses an MLP with Gaussian output, while the decoder uses MLPs with Gaussian or Bernoulli outputs depending on the data type.
  • C MLP’s as probabilistic encoders and decoders: The decoder output distribution is Gaussian or Bernoulli, selected according to the type of data.The encoder instead has a Gaussian output.

C.1 Bernoulli MLP as decoder … F.1 Example

The paper specifies neural-network decoders and encoders, then develops marginal-likelihood estimation and full variational Bayes using reparameterized stochastic gradients. A Gaussian example yields an alternative lower-variance estimator by solving four terms analytically.

  • C.1 Bernoulli MLP as decoder: A Bernoulli decoder models pθ(x|z) with probabilities computed by a fully connected single-hidden-layer neural network.The elementwise sigmoid supplies the activation, and θ comprises the network’s weights and biases.
  • C.2 Gaussian MLP as encoder or decoder: A Gaussian encoder or decoder uses a diagonal covariance structure, with encoder inputs and outputs swapped and variational parameters φ.The corresponding MLP weights and biases are part of θ for a decoder and φ for an encoder.
  • D Marginal likelihood estimator: The marginal-likelihood estimator uses posterior samples, density fitting, and additional posterior samples, performing well when sampled dimensionality is below 5 and sampling is sufficient.Posterior samples are obtained with gradient-based MCMC such as Hybrid Monte Carlo.
  • E Monte Carlo EM: Monte Carlo EM omits an encoder and samples latent-variable posteriors using gradients, applying 10 HMC leapfrog steps, a 90% acceptance rate, and 5 weight-update steps.Marginal likelihood uses 50 posterior samples per datapoint with 4 leapfrog steps for the first 1000 training and test datapoints.
  • F Full VB: Full variational Bayes can infer both parameters θ and latent variables z by introducing a hyperprior pα(θ) and optimizing a variational lower bound.The bound equals the true marginal likelihood when approximate and true posteriors match exactly.
  • F Full VB: When expectations are intractable, conditional samples can be reparameterized using priors p(ϵ), p(ζ) and functions gφ(ϵ, x), hφ(ζ).The construction assumes certain mild conditions for the chosen approximate posteriors.
  • F Full VB: The resulting estimator depends only on parameter-independent prior samples, so it is differentiable with respect to φ and supports stochastic optimization with SGD or Adagrad.The estimator is defined through fφ and can be used to compute stochastic gradients.
  • F.1 Example: In the Gaussian example, qφ(θ) and qφ(z|x) use diagonal-covariance Gaussians parameterized as means plus elementwise-scaled standard-normal noise.Because the relevant prior and approximate-posterior terms are Gaussian, four terms can be solved analytically to construct a lower-variance estimator.
Loading 1312.6114v11…