An Introduction to Variational Autoencoders

1.1 Motivation

• Generative modeling describes joint data distributions, incorporates physical constraints and nuisance variables, supports causal generalization, and can guide discrimination when labeled data are scarce.Generative models can provide meaningful representations for downstream prediction, while their stronger assumptions may increase asymptotic bias when the model is misspecified and data are abundant.Source: §1.1 · pp. 5–6
• VAEs couple an encoder that approximates latent posteriors with a decoder that generates data, using amortized inference for fast feedforward computation and the reparameterization trick to reduce gradient variance.Unlike ordinary variational inference, one recognition model shares parameters across data cases, although sampling still introduces gradient noise that reparameterization counteracts.Source: §1.1 · p. 7

1.2 Aim

• VAEs jointly learn deep latent-variable models and corresponding inference models using stochastic gradient descent, supporting generative, semi-supervised, and representation-learning applications.The work expands earlier work to explain VAEs in greater detail and discuss selected follow-up research, rather than providing a comprehensive review.Source: §1.2 · p. 9

1.3 Probabilistic Models and Variational Inference

• Probabilistic models represent uncertain phenomena through distributions over observed and latent variables, and learning seeks parameters θ such that pθ(x) approximates the unknown data distribution p∗(x).Conditional models instead approximate p∗(y|x), including high-dimensional prediction problems such as image, video, sound, text, and time-series modeling.Source: §1.3 · pp. 9–10

1.4 Parameterizing Conditional Distributions with Neural Networks

• Neural networks provide flexible, scalable parameterizations for conditional probability distributions, while directed graphical models factorize joint distributions over variables organized in a directed acyclic graph.Embedding neural networks inside graph conditionals enables stochastic gradient-based optimization of larger probabilistic models.Source: §1.4 · p. 11, §1.5 · pp. 12–13

1.6 Learning in Fully Observed Models with Neural Nets

• For fully observed models, maximum likelihood optimizes the data log-probability, while minibatch stochastic gradients provide an unbiased and more efficient alternative to batch gradient descent.Under the i.i.d. assumption, likelihood factorizes across datapoints; maximum likelihood is equivalent to minimizing the KL divergence between data and model distributions.Source: §1.6.1 · p. 13, §1.6.2 · p. 13, §1.6 · p. 14

1.7.2 Deep Latent Variable Models

• Deep latent-variable models parameterize latent-variable distributions with neural networks, allowing simple factors such as conditional Gaussians to produce highly complex marginal distributions pθ(x).Latent variables are unobserved model variables, and this expressivity makes deep latent-variable models attractive for approximating complicated underlying distributions p∗(x).Source: §1.7.1 · p. 15, §1.7.2 · p. 15, §1.8 · p. 16

1.8. Intractabilities

• A common DLVM factorizes a prior pθ(z) over latent variables and an observation model pθ(x|z), with the prior unconditioned on observations.The joint model structure specifies pθ(z) and/or pθ(x|z).Source: §1.8 · p. 16

1.7.3 Example DLVM for multivariate Bernoulli data

• For binary data, a simple DLVM uses a spherical Gaussian latent space and a factorized Bernoulli observation model whose probabilities lie between 0 and 1.The Bernoulli probabilities can be implemented with a sigmoid as the decoder network’s final layer, and the observation dimensionality is D.Source: §1.7.3 · p. 16

1.8 Intractabilities

• Marginal likelihood computation is typically intractable because integrating over latent variables lacks an analytic solution or efficient estimator, preventing direct parameter optimization.The posterior pθ(z|x) is likewise intractable, although the joint distribution pθ(x, z) is efficiently computable.Source: §1.8 · pp. 16–17
• Approximate inference methods address intractable posteriors and marginal likelihoods, but traditional approaches may require per-datapoint optimization or produce poor posterior approximations.The section motivates avoiding such expensive procedures; parameter posteriors p(θ|D) in neural-network directed models are also generally intractable.Source: §1.8 · p. 17

2.1 Encoder or Approximate Posterior

• VAEs introduce a parametric encoder qφ(z|x) that approximates the intractable posterior and shares variational parameters across datapoints, avoiding per-datapoint optimization loops.The encoder can be represented by directed graphical models and deep neural networks; this shared-parameter strategy is amortized variational inference.Source: §2.1 · pp. 18–19

2.2 Evidence Lower Bound (ELBO)

• Maximizing the ELBO approximately maximizes marginal likelihood while minimizing the KL divergence between qφ(z|x) and the true posterior pθ(z|x).The ELBO is a lower bound on log-likelihood, and its tightness improves as the approximate posterior approaches the true posterior.Source: §2.2 · p. 21, §2.2.1 · p. 22

2.3 Stochastic Gradient-Based Optimization of the ELBO

• The ELBO supports joint stochastic-gradient optimization of generative parameters θ and variational parameters φ using minibatches and unbiased gradient estimators.Gradients with respect to φ are difficult because qφ depends on φ, motivating the reparameterization trick for continuous latent variables.Source: §2.3 · pp. 22–23

2.4 Reparameterization Trick

• Reparameterization expresses z as a differentiable transformation of noise ϵ independent of x and φ, allowing gradients to backpropagate through sampling and yielding the AEVB algorithm.For differentiable encoders and generative models, the resulting gradient estimator is unbiased for the exact single-datapoint ELBO gradient.Source: §2.4 · pp. 23–24, §2.4.1 · p. 23, §2.4.2 · p. 24

2.4.4 Computation of log qφ(z|x)

• Invertible transformations make log qφ(z|x) computable through the noise density and the transformation’s log-Jacobian determinant, enabling flexible inference models.Triangular transformations keep the log-determinant simple; full-covariance Gaussian posteriors use a triangular Cholesky parameterization, and chains of such transformations motivate IAF.Source: §2.4.4 · p. 27, §2.5.1 · p. 29, §2.5 · p. 30

2.6 Estimation of the Marginal Likelihood

• Importance sampling estimates the marginal likelihood more accurately as the sample count L increases and converges to the true marginal likelihood as L →∞.Setting L = 1 recovers the VAE ELBO estimator; IWAE uses the resulting tighter objective, whose importance-weighted estimates scale poorly in high-dimensional latent spaces.Source: §2.6 · p. 31

2.7. Marginal Likelihood and ELBO as KL Divergences

• ELBO maximization is equivalent to minimizing DKL(qD,φ(x, z)||pθ(x, z)), while maximum likelihood minimizes the corresponding data-space divergence.The ELBO can also be interpreted as a maximum-likelihood objective in an augmented space containing stochastic auxiliary features z.Source: §2.7 · pp. 32–33, Figure 2.4 · p. 34

2.8.1 Optimization issues

• The unmodified ELBO can reach a stable equilibrium where q(z|x) ≈ p(z), so latent-cost annealing from 0 to 1 is used to facilitate optimization.Early in training, the likelihood term log p(x|z) is weak, making this equilibrium difficult to escape.Source: §2.8.1 · p. 33

2.8. Challenges

• Free bits improves benchmark log-likelihood across λ values from 0.125 to 2.The method enforces at least λ nats of information per latent-variable group on average, preventing latent subsets from using less than the allotted information.Source: §2.8.1 · p. 33, §2.8 · p. 35

2.8.2 Blurriness of generative model

• The KL-divergence direction can make generative models overly broad, because probability mass outside the support of qD,φ is only weakly penalized.Sufficiently flexible inference and generative models can counter this blurriness.Source: §2.8.2 · p. 35

2.9 Related prior and concurrent work

• Wake-sleep uses a recognition model but jointly optimizes two objectives that do not form a marginal-likelihood bound, whereas AEVB has the same per-datapoint computational complexity.Wake-sleep additionally applies to models with discrete latent variables.Source: §2.9 · pp. 35–36
• VAEs connect directed probabilistic models and autoencoders through a variationally dictated regularizer, unlike reconstruction-only criteria requiring nuisance regularization hyperparameters.Linear autoencoders correspond to a restricted linear-Gaussian case, whereas the VAE approach applies to a broader class of continuous latent-variable models.Source: §2.9 · pp. 36–37

2.9.1 Score function estimator

• The score-function estimator is applicable to discrete latent variables, but reparameterization-based estimation is favored here because continuous latents provide cheap backpropagated gradient information.The reparameterization advantage is described as removing an information bottleneck from ELBO gradients with respect to φ.Source: §2.9.1 · pp. 38–39
• Two orders of magnitude more samples are required by a sophisticated score-function estimator to match the variance of a reparameterization-based estimator.For continuous latent variables, reparameterization can exploit ∇z log pθ(x, z), whereas score-function estimators use only scalar log pθ(x, z), generally producing higher variance.Source: §2.9.1 · p. 39

3.1 Requirements for Computational Tractability

• Practical inference models must support efficient density computation, differentiation, sampling, and parallelization, while remaining flexible enough to match p(z|x) and tighten the ELBO.These constraints often motivate simple Gaussian posteriors, despite their limited flexibility.Source: §3.1 · p. 40, Beyond Gaussian Posteriors · p. 40

3.2.1 Auxiliary Latent Variables

• Auxiliary-variable methods augment inference and generative models with a continuous u, producing more flexible marginal posteriors whose gains can outweigh the added KL cost and improve models.The approach was explored by Salimans et al., 2015, and Maaløe et al., 2016.Source: §3.2.1 · p. 41, §3.2 · p. 42

3.2.2 Normalizing Flows

• Normalizing flows transform a tractable initial distribution through invertible parameterized maps, retaining computable densities when each transformation’s Jacobian determinant is tractable.The section introduces this approach through (Rezende and Mohamed, 2015).Source: §3.2.2 · pp. 42–43

3.3 Inverse Autoregressive Transformations

• Inverse autoregressive transformations use autoregressive structure to enable parallel inverse computation and simple log-determinants, making flexible normalizing flows practical in high-dimensional latent spaces.Their combination of flexibility, parallelizability across dimensions, and simple log-determinants motivates their use for high-dimensional latent variables.Source: §3.3 · pp. 44–45

3.4 Inverse Autoregressive Flow (IAF)

• IAF chains inverse autoregressive transformations initialized from a factorized Gaussian, with posterior flexibility increasing with autoregressive expressivity and chain depth.Reversing variable order after each step further improved results while preserving the simple log-density form.Source: §3.4 · pp. 45, 46, 47, 49

3.5 Related work

• Related flows trade off expressivity, dimensional scalability, and computational cost: planar and radial flows target relatively low-dimensional spaces, whereas Hamiltonian flows can approach the exact posterior but are computationally demanding.The section also notes that deep latent-variable models can be combined with IAF to benefit from both techniques.Source: §3.5 · pp. 49–50

4.1 Inference and Learning with Multiple Latent Variables

• VAEs with multiple latent variables use ancestral sampling in topological order, then optimize the differentiable scalar log pθ(x, z) − log qφ(z|x) with SGD.The resulting scalar is an unbiased stochastic lower-bound estimator on log pθ(x).Source: §4.1 · pp. 51–52

4.1.1 Choice of ordering

• Sharing latent-variable ordering between generative and inference models enables parameter sharing, faster learning, and better solutions.Although reverse ordering may seem natural because data are roots in inference and leaves in generation, shared ordering has been advantageous in multiple works (Salimans, 2016).Source: §4.1.1 · p. 52, Figure 4.1 · p. 53

4.2 Alternative methods for increasing expressivity

• The strongest expressive models combine latent variables with amortized inference, full autoregression, and invertible transformations with tractable Jacobians.For high-dimensional variables, parallelized conditional factors can yield a reported 100x improvement in synthesis speed.Source: §4.2 · pp. 54–55

4.3 Autoregressive Models

• Autoregressive models factorize distributions into ordered low-dimensional conditionals, but ancestral sampling requires sequential computation of O(D) length and depends on an input ordering.Their ordering can impose an awkward inductive bias when data, such as images, lack a natural one-dimensional order.Source: §4.3 · p. 55, §4.4 · p. 56

4.4 Invertible transformations with tractable Jacobian determinant

• Normalizing flows provide tractable likelihoods and gradients through invertible mappings, yet their dimensionality-preserving intermediate states make optimization challenging and had not matched autoregressive or VAE-based log-likelihood.NICE and Real NVP offer equal computational cost and depth in both directions, making optimization and sampling relatively cheap (Dinh et al., 2014; Dinh et al., 2016).Source: §4.4 · pp. 56–57

4.5.1 Representation Learning

• VAE-based semi-supervised learning improves classification with few labels, achieving under 1% MNIST error using 10 labeled images per class and around 45% ImageNet accuracy with 1% labeled images.The approach jointly optimizes classification and a VAE, sharing parameters and using the VAE as an auxiliary objective; it can also work with a single sample per class.Source: §4.5.1 · pp. 57–58

4.5.2 Understanding of data, and artificial creativity

• Deep generative models support latent-space exploration and plausible synthesis, including gradient-based chemical design, interpolation and imputation of natural-language sentences, astronomical pseudo-data, and semantically controlled image resynthesis.Chemical design targeted drug-like molecules and organic light-emitting diodes; sentence interpolation produced grammatically correct, locally consistent outputs; image edits included movement along a “smile vector.”Source: Chemical Design · pp. 58–59, Natural Language Synthesis · p. 59, Figure 4.3 · p. 60

4.5.3 Other relevant follow-up work

• Follow-up work broadens VAEs through Bayesian parameter posteriors, normalizing flows, auxiliary variables, multiple stochastic layers, and variance-reduction methods for discrete latent variables.Examples include Bayes by Backprop, Ladder VAEs, and Gumbel-Softmax (Jang et al., 2017).Source: §4.5.3 · pp. 61–62
• VAE objectives have been extended with importance weighting, alternative divergences, normalizing flows, black-box α-divergence minimization, and bidirectional Helmholtz machines, although importance-weighted estimators can be difficult to scale to high-dimensional latent spaces.The reparameterization-gradient literature also includes acceptance-rejection sampling, modified estimators that reduce variance by “carving up the ELBO,” and second-order estimators.Source: §4.5.3 · pp. 62, 63, 65
• Adversarial hybrids use GAN objectives for variational inference, while image-generation VAEs combine convolutional, recurrent, attention, and PixelCNN components for synthesis, compression, and controlled generation.Examples include AAE, ALI, DRAW, PixelVAE, and Variational Lossy Autoencoder, alongside applications to colorization and semi-supervised learning.Source: §4.5.3 · p. 63
• VAEs have been applied to text, sequential data, speech, handwriting, and molecules, typically using recurrent encoders or decoders and, where appropriate, incorporating context-free grammars to improve sequence modeling.Grammar VAEs incorporate grammar constraints and were applied to molecules represented textually.Source: §4.5.3 · p. 64
• Continuous latent spaces and reparameterization support model-based control, novel policy gradients, intrinsically motivated reinforcement learning, improved exploration, and iterative reasoning about objects in scenes.These applications use approximately known latent marginals, observation models, variational information maximization, or VAE components for scene reasoning.Source: §4.5.3 · p. 64
• β-VAE strengthens the KL contribution to restrict latent information flow and improve disentanglement, while other VAE applications include graphs, 3D structure, one-shot learning, state-space models, voice conversion, fair representations, and transfer learning.Disentanglement was further studied in.Source: §4.5.3 · pp. 64–65
• Overall, the breadth of extensions and applications reflects an actively researched area with frequent developments.The surveyed directions span objectives, inference methods, architectures, applications, and gradient estimators.Source: §4.5.3 · p. 65

Conclusion

• Directed probabilistic models can be made highly flexible by parameterizing their conditional distributions with differentiable deep neural networks.Source: Conclusion · p. 66
• The main contribution is an efficient, scalable framework for gradient-based variational posterior inference and approximate maximum likelihood learning.It addresses posterior inference for flexible latent-variable and Bayesian models with neural-network components.Source: Conclusion · p. 66
• Variational inference casts approximate posterior estimation as optimization of variational parameters toward the ELBO, a lower bound on model evidence or marginal likelihood.Source: Conclusion · p. 66
• The work describes the variational autoencoder and some of its extensions.Source: Conclusion · p. 66

• Generative and inference models are deep-neural-network-parameterized directed graphical models jointly optimized on the ELBO using stochastic gradients from reparameterization.The inference model approximates the generative model’s latent-variable posterior.Source: §64 · p. 67
• The VAE framework is now widely used for probabilistic modeling and artificial creativity, with basic implementations available in major deep-learning software libraries.Source: §64 · p. 67
• Inverse autoregressive flows provide scalable, flexible inference models for high-dimensional latent spaces, motivating comparison with NICE and Real NVP transformations.These alternatives have computationally cheap inverses and are proposed as a direction for further exploration.Source: §64 · p. 67
• Combining powerful posteriors, priors, and decoders could yield simple VAEs that rival or surpass purely autoregressive architectures while synthesizing much faster.Source: §64 · p. 67
• The VAE framework supports discrete and continuous observed variables, efficient amortized latent-variable inference, fast synthesis, and near-state-of-the-art data log-likelihoods.Source: §64 · p. 67

A.1.1 Notation

• Bold lowercase letters denote random vectors or collections of random variables, while italic lowercase letters denote random scalars.Source: §A.1.1 · p. 70
• Bold capitalized letters denote random matrices.Source: §A.1.1 · p. 70
• Pa(z) denotes the parents of random variable z in a directed graph.Source: §A.1.1 · p. 70
• diag(x) forms a diagonal matrix from vector x, and x ⊙ y denotes element-wise vector multiplication.The resulting vector is (x1y1, ..., xKyK)T.Source: §A.1.1 · pp. 70–71
• θ denotes generative-model parameters, whereas bold φ denotes variational parameters.Source: §A.1.1 · p. 71
• Probability distributions use p(.), q(.), or r(.); joint and conditional distributions use p(., .) and p(.|.), respectively.Distribution parameters may be written p(.; θ) or equivalently pθ(.).Source: §A.1.1 · p. 71
• Equality or inequality signs inside distributions distinguish arguments from evaluation values, while different distribution letters distinguish models and repeated letters indicate related marginals or conditionals.For example, p(x = a) denotes a PDF or PMF evaluated at a, and p(x ≤ a) denotes a CDF evaluated at a.Source: §A.1.1 · p. 71

A.1. Notation and definitions

• A probability density function assigns a probability density to each possible value of given continuous random variables.Source: §A.1 · p. 72
• The PDF is defined for continuous random variables.Source: §A.1 · p. 72
• A cumulative distribution function assigns a cumulative probability density to each possible value of given univariate continuous random variables.Source: §A.1 · p. 72
• The CDF is defined for univariate continuous random variables.Source: §A.1 · p. 72
• A probability mass function assigns a probability mass to a given discrete random variable.Source: §A.1 · p. 72
• The PMF is defined for discrete random variables.Source: §A.1 · p. 72

A.1.3 Distributions

• The notation p(x) = N(x; µ, Σ) denotes both a distribution from which samples can be drawn and its probability density function (PDF).The section explicitly assigns these two meanings to the same distribution notation.Source: §A.1.3 · p. 72

A.1.4 Bayesian Inference

• Bayes’ rule expresses the posterior over parameters as proportional to the prior times the likelihood, with the evidence p(D) constant in θ.The posterior supports predictions about future data when the prior and model class are correct.Source: §A.1.4 · p. 73

A.2.1 Maximum A Posteriori

• MAP estimation maximizes the log-posterior, while large datasets often justify maximum likelihood by omitting the diminishing-effect prior.Omitting the prior is numerically equivalent to setting p(θ) = constant.Source: §A.2.1 · p. 73, §A.2 · p. 74

A.2. Alternative methods for learning in DLVMs

• Alternative deep latent-variable learning methods include MAP, variational EM, and MCMC-EM, offering Bayesian approaches beyond maximum likelihood.The reviewed methods differ in how they represent or approximate latent-variable posteriors during optimization.Source: §A.2.1 · p. 73, §A.2.2 · p. 74, §A.2.3 · p. 75

A.2.2 Variational EM with local variational parameters

• Variational EM performs coordinate ascent on the ELBO, using separate local variational parameters per datapoint rather than the global variational parameters used by VAEs.The E-step minimizes KL divergence to the true posterior, while the M-step optimizes a bound whose tightness is D_KL(q_φ(z|x)||p_θ(z|x)).Source: §A.2.2 · p. 74

A.2.3 MCMC-EM

• MCMC-EM approximates each posterior with approximately i.i.d. samples from a Markov chain, but Hamiltonian MCMC requires an independent chain per datapoint.Posterior gradients can be computed by differentiating the log-joint distribution with respect to z.Source: §A.2.3 · p. 75

A.3 Stochastic Gradient Descent

• SGD iteratively updates differentiable model parameters using an unbiased stochastic objective estimate, with reverse-mode automatic differentiation computing scalar values and gradients efficiently.The stochastic variable ξ may represent a datapoint index or posterior sampling noise in VAEs.Source: §A.3 · p. 75

A.3. Stochastic Gradient Descent

• Adam and Adamax choose the SGD learning-rate preconditioner and are invariant to constant objective or gradient rescaling, so estimates need only be unbiased up to proportionality.Optimization stops when a stopping criterion is met; early stopping halts it when holdout-set probability begins decreasing.Source: §A.3 · p. 76