PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to Variational Autoencoders

Diederik P. Kingma, Max Welling

arXiv:1906.02691v3cs.LGstat.ML

TL;DR

Flexible latent-variable models require tractable approximate posterior inference, especially when neural networks are components. This work introduces VAEs and extensions that jointly learn deep latent-variable and inference models. The framework supports efficient amortized inference and fast synthesis while achieving close to state-of-the-art data log-likelihood performance.

  • Problem

    Existing posterior-inference methods were relatively inefficient or inapplicable to models with neural-network components, motivating tractable inference for flexible latent-variable models.

  • Method

    VAEs jointly learn deep latent-variable models and corresponding inference models using stochastic gradient descent, with computationally tractable inference distributions and extensions such as auxiliary variables.

  • Results

    The framework enables efficient amortized latent-variable inference and fast synthesis, with close to state-of-the-art performance in data log-likelihood.

  • Takeaways & Limitations

    Combining powerful posteriors, priors, and decoders can potentially rival or surpass purely autoregressive architectures while allowing much faster synthesis.

  • Takeaways & Limitations

    Practical inference distributions must be computationally efficient to evaluate, differentiate, and sample, restricting the approximate posteriors that are feasible in high-dimensional settings.

Abstract

from arXiv · show

Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to variational autoencoders and some important extensions.

Introduction

Generative modeling seeks to represent how data are produced, supporting interpretable structure, causal generalization, and downstream representation learning. VAEs combine deep latent-variable generative models with amortized recognition models, using reparameterization gradients for efficient learning.

  • Motivation: Generative models learn joint distributions and represent hypothesized data-generating processes rather than directly learning predictors.They can encode physical laws and constraints while treating unknown nuisance variables as noise.
  • Motivation: Causal generative representations can generalize across situations better than correlations alone, such as transferring earthquake knowledge between California and Chile.
  • Motivation: Unsupervised representation learning seeks disentangled, semantically meaningful, statistically independent, and causal factors that support multiple downstream prediction tasks.VAEs can act as an auxiliary generative task or as implicit regularization on the inverse mapping from inputs to representations.
  • VAE framework: A VAE couples independently parameterized encoder and decoder models: the encoder approximates the latent posterior, while the decoder scaffolds meaningful representations.The recognition model is the approximate inverse of the generative model according to Bayes rule.
  • VAE framework: Amortized inference uses one input-conditioned recognition model for all data cases, enabling a single feedforward pass but introducing sampling noise into learning gradients.This contrasts with variational inference, which uses a separate variational distribution for each data case.
  • VAE framework: The reparameterization trick reorganizes gradient computation to reduce gradient variance, allowing VAE learning rules to optimize a single approximation to maximum likelihood.The VAE framework is inspired by the Helmholtz Machine but differs from its inefficient wake-sleep algorithm, which did not optimize a single objective.

Variational Autoencoders

Variational autoencoders jointly optimize a deep latent-variable model and an inference model by maximizing the ELBO with stochastic gradient methods. The framework uses amortized inference and reparameterization to make posterior approximation and learning tractable.

  • Encoder or Approximate Posterior: VAEs introduce a parametric encoder qφ(z|x) that approximates the generative model’s intractable posterior pθ(z|x).The encoder is also called an inference or recognition model and can be parameterized by a directed graphical model or deep neural network.
  • Encoder or Approximate Posterior: Amortized inference shares one encoder’s variational parameters across datapoints, avoiding separate iterative optimization for each case.This shared parameterization makes posterior inference more efficient for large datasets.
  • Evidence Lower Bound: The ELBO lower-bounds the data log-likelihood, with the gap determined by the KL divergence between qφ(z|x) and the true posterior pθ(z|x).The gap is zero if and only if the approximate posterior equals the true posterior.
  • Evidence Lower Bound: Maximizing the ELBO jointly improves the generative model’s marginal likelihood and the encoder’s approximation to the true posterior.The parameters θ and φ can be optimized together using stochastic gradient descent.
  • Stochastic Gradient Optimization: Reparameterization expresses z as a differentiable function of φ, x, and independent noise ϵ, enabling gradients to pass through stochastic sampling.For continuous latent variables, this yields unbiased gradient estimates suitable for minibatch SGD.
  • Extensions: More flexible transformations can keep log qφ(z|x) tractable while producing highly flexible inference models.The framework also extends Monte Carlo estimation through multiple samples and can improve bound tightness with larger sample counts.

Beyond Gaussian Posteriors

Improving inference-model flexibility must be balanced against efficient density computation, sampling, and parallelization. Auxiliary variables and normalizing flows address this trade-off through richer approximate posteriors while preserving tractable optimization where possible.

  • Requirements for Computational Tractability: Efficient inference models must compute and differentiate qφ(z|x), sample efficiently, and ideally parallelize across latent dimensions.These requirements often favor simple Gaussian posteriors, even though tight ELBOs also require sufficient flexibility to match the true posterior.
  • Auxiliary Latent Variables: Auxiliary variables augment inference and generative models with a continuous variable u, implicitly defining a potentially powerful marginal qφ(z|x).The resulting ELBO includes an additional divergence term for the auxiliary variable, although the more flexible marginal inference distribution can improve the original objective.
  • Auxiliary Latent Variables: Auxiliary variables can improve the original ELBO enough to outweigh the additional auxiliary-variable divergence cost, with prior work reporting significant model improvements.The augmented objective can initially worsen in principle, but increased inference flexibility may more than compensate.
  • Normalizing Flows: Normalizing flows transform a simple tractable-density random variable through invertible steps whose Jacobian determinants remain computable.The final iterate therefore has a more flexible distribution while retaining a tractable density calculation.
  • Normalizing Flows: The single-unit bottleneck in an earlier flow limits high-dimensional scaling because capturing dependencies requires a long chain of transformations.This limitation motivates autoregressive constructions for high-dimensional latent spaces.
  • Inverse Autoregressive Flow: IAF uses inverse autoregressive transformations whose inverse computation parallelizes across dimensions and whose triangular Jacobian gives a simple log-determinant.Combining flexibility, parallelizability, and tractable Jacobians makes IAF suitable for high-dimensional latent spaces; greater autoregressive expressivity and chain depth improve posterior fitting.
  • Inverse Autoregressive Flow: Reversing variable ordering after each IAF step improved results without changing the simple log-determinant because the reversal preserves volume.The final flow becomes more flexible as the autoregressive models become more expressive and the transformation chain becomes deeper.

Deeper Generative Models

This section reviews strategies for learning deeper and more expressive generative models in the VAE framework, including multi-latent architectures, shared inference orderings, autoregression, and invertible transformations.

  • Multiple latent variables: VAEs support generative and inference models represented as arbitrary directed graphs with multiple latent variables and topological orderings.ELBO optimization samples latent variables ancestrally, then evaluates a differentiable stochastic lower-bound estimate using SGD.
  • Choice of ordering: Shared latent-variable ordering lets inference and generative models share parameters, leading to faster learning and better solutions.This ordering is one of two alternatives illustrated in Figure 4.1.
  • Increasing expressivity: Fully autoregressive models synthesize sequentially in O(D) computation and require an input ordering that can impose an awkward bias on data such as images.Their log-likelihood computation need not scale with data dimensionality, but synthesis remains relatively slow.
  • Increasing expressivity: Latent variables are attractive for high-dimensional data because factorized conditional distributions can parallelize synthesis and log-likelihood evaluation across dimensions.Kingma et al. report a 100x improvement in synthesis speed for such a design.
  • Increasing expressivity: Expressive directed models combine latent variables, full autoregression, and invertible transformations with tractable Jacobian determinants.The best reported test-data log-likelihood models use a combination of all three approaches.
  • Chemical design: Continuous VAE representations support gradient-based optimization for designing drug-like molecules and organic light-emitting diodes.The latent representation is learned from molecular data and optimized toward desired properties.

Conclusion

Directed probabilistic models gain flexibility when conditional distributions are parameterized by differentiable deep neural networks. The VAE is presented as a framework for learning deep latent-variable models and inference models.

  • Deep neural networks make directed probabilistic models highly flexible.
  • Variational inference addresses approximate posterior estimation in latent-variable models by optimizing a lower bound on model evidence.The paper identifies inefficiency and limited applicability to neural-network components as shortcomings of existing methods.
  • The VAE combines a deep latent-variable model with an associated inference model.

64 Conclusion

A VAE combines a deep latent-variable generative model with an inference model that approximates latent posteriors. The framework uses stochastic gradients and reparameterization, and its extensions include IAF for flexible high-dimensional inference.

  • A VAE combines a deep latent-variable model with continuous latent variables and an associated inference model.
  • The inference model approximates the generative model’s latent posterior, while both models may be parameterized by deep neural networks.
  • Stochastic gradients for joint optimization are obtained through the reparameterization trick.
  • Inverse autoregressive flows provide flexible inference models that scale to high-dimensional latent spaces.
  • VAE architectures can potentially combine powerful posteriors, priors, and decoders while allowing faster synthesis than purely autoregressive architectures.
  • The framework supports discrete and continuous observed variables, amortized latent-variable inference, fast synthesis, and close-to-state-of-the-art log-likelihood performance.

A.1.1 Notation

The notation section specifies conventions for random variables, matrices, graph structure, model parameters, distributions, and common vector operations.

  • Italic lowercase letters denote random scalars, while bold lowercase letters denote random vectors or collections of variables.
  • Bold capital letters denote random matrices, and Pa(z) denotes the parents of z in a directed graph.
  • diag(x) denotes a diagonal matrix formed from vector x, while x ⊙ y denotes element-wise vector multiplication.
  • θ denotes generative-model parameters, while φ denotes variational parameters.
  • p(.) and q(.) denote probabilistic models, with p(x,y) for joint distributions and p(x|z) for conditional distributions.
  • Subscript notation such as pθ(x) denotes distribution parameters, and equality or inequality signs inside distributions indicate evaluation values.

A.1. Notation and definitions

The definitions distinguish probability density, cumulative distribution, and probability mass functions. They describe how each assigns density, cumulative probability, or mass to random-variable values.

  • A probability density function assigns probability density to possible values of continuous random variables.
  • A cumulative distribution function assigns cumulative probability density to possible values of univariate continuous random variables.
  • A probability mass function assigns probability mass to a given discrete random variable.

A.1.3 Distributions

The notation for distributions is overloaded to mean both a sampling distribution and its probability density function.

  • For example, p(x) = N(x; µ, Σ) uses distribution notation together with its density-function interpretation.
  • The notation p(x) can denote a distribution from which samples are drawn.
  • The same notation can also denote that distribution’s probability density function.

Chain rule of probability

This section develops Bayesian and optimization perspectives for latent-variable models, including EM, MCMC, and stochastic-gradient optimization. It emphasizes posterior approximation, ELBO tightness, and practical optimization choices.

  • Bayesian foundations: Bayes’ rule combines a prior p(θ) with the likelihood p(D|θ) to form the posterior over parameters after observing data D.
  • Bayesian foundations: Bayesian prediction of future data x′ is optimal when the prior and model class are correct.
  • Bayesian foundations: MAP estimation maximizes the log-posterior, while large datasets often motivate maximum likelihood because the prior’s effect diminishes with N.
  • Expectation Maximization: EM performs coordinate ascent on the ELBO; traditional variational EM uses separate local parameters per datapoint, whereas VAEs use global inference parameters.
  • Expectation Maximization: The E-step minimizes KL divergence from qφ(z|x) to the true posterior, and the ELBO becomes exact when qφ(z|x) equals pθ(z|x).
  • MCMC inference: MCMC approximates the posterior with approximately i.i.d. samples, but Hamiltonian MCMC requires an independent chain for every datapoint.
  • Stochastic optimization: Differentiable scalar objectives permit values and gradients with equal time complexity through reverse-mode automatic differentiation.
  • Stochastic optimization: SGD uses an unbiased objective estimate incorporating datapoint or posterior-sampling noise, while Adam and Adamax choose the learning-rate or preconditioning sequence.
Loading 1906.02691v3…