PaperlayerView exact revision on arXiv

Source-linked AI summary

Tutorial on Variational Autoencoders

Carl Doersch

arXiv:1606.05908v3stat.MLcs.LG

TL;DR

Likelihood models can identify unlikely observations without helping synthesize likely ones. This tutorial explains VAEs and their mathematics, and notes zero approximation error in at least one one-dimensional scenario under stated conditions.

  • Problem

    A model that identifies unlikely observations does not necessarily help synthesize likely ones.

    Source: §1 · p. 2

  • Method

    The tutorial presents VAEs as latent-variable models that define an output distribution conditioned on z, and explains their assumptions and training.

    Source: §1 · p. 2, §1.1 · p. 3

  • Results

    VAEs have zero approximation error in at least one one-dimensional scenario when σ is small relative to the ground-truth distribution’s CDF curvature.

    Source: §2.4.1 · p. 13

  • Takeaways & Limitations

    The result suggests future theoretical work may clarify VAE approximation error in more practical setups.

    Source: §2.4.1 · p. 13

    The Gaussian assumption for Q(z|X) does not ensure its divergence from the true posterior vanishes, so approximation error may remain nonzero for fixed P.

    Source: §2.4.1 · p. 12

Abstract

from arXiv · show

In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.

1 Introduction

The introduction frames generative modeling as learning to produce new data resembling an unknown distribution, and presents VAEs as a fast, weak-assumption neural framework. The preliminaries motivate latent variables for modeling complex dependencies and explain how conditional output distributions support likelihood-based training.

1 Introduction

  • Generative modeling aims to learn a model that can sample new examples resembling an unknown data distribution; assigning probabilities alone does not produce likely examples.The introduction illustrates this goal with generating unseen images, plants, or handwriting from examples.Source: §1 · pp. 1–2
  • VAEs [1] [3] combine weak assumptions and fast backpropagation training, with approximation error argued to be small for high-capacity models.The tutorial presents VAEs as a popular framework enabled by advances in neural networks as function approximators.Source: §1 · p. 2
  • The tutorial offers an informal VAE introduction for readers without strong variational Bayesian or minimum-description-length backgrounds, with a vision-oriented perspective.It grew out of presentations for computer vision reading groups.Source: §1 · p. 2

1.1 Preliminaries: Latent Variable Models

  • A latent variable can encode a character choice before pixel assignment, helping coordinate dependent dimensions into a coherent digit.For example, choosing a digit value first helps ensure the strokes match that character.Source: §1.1 · p. 3
  • A conditional output distribution makes sample similarity probabilistic: VAEs often use a Gaussian, allowing gradient descent to move its mean toward training examples.The output distribution need not be Gaussian; it must be computable and continuous in the model parameters.Source: §1.1 · pp. 3–4

2 Variational Autoencoders

VAEs use a learned encoder–decoder objective to make likelihood learning tractable while modeling complex distributions from simple latent variables. The section explains how the objective is optimized and interpreted, and where its approximation and regularization limits arise.

2 Variational Autoencoders

  • VAEs assume z follows N(0, I) and use a learned function to map it to latent structure and then to X, rather than hand-specifying latent dimensions.The network can learn useful structure when it improves reproduction of the training data; samples can be drawn directly from P(X) without MCMC.Source: §2 · pp. 5–6
  • Sampling z from the prior estimates P(X) inefficiently in high dimensions, so VAEs use Q(z|X) to focus sampling on codes likely to have produced X.The similarity metric is unchanged; the sampling procedure is made faster.Source: §2 · p. 7, §2.1 · p. 7

2.1 Setting up the objective

  • The VAE objective maximizes a lower bound on log P(X) while penalizing mismatch between Q(z|X) and the intractable posterior P(z|X).The bound is optimized via stochastic gradient descent; a high-capacity Q can reduce the mismatch term.Source: §2.1 · pp. 8–9

2.2 Optimizing the objective

  • A Gaussian Q(z|X) with learned mean and diagonal covariance makes the KL term closed-form, while reparameterizing z = µ(X) + Σ^1/2(X) * ϵ enables backpropagation.The reparameterization requires a continuous sampling function; discrete Q distributions do not meet that condition.Source: §2.2 · pp. 9, 11

2.3 Testing the learned model

  • At test time, VAEs generate samples by feeding z ∼ N(0, I) to the decoder, and estimate example likelihood with a lower bound sampled from Q(z|X).The likelihood is generally intractable, but Q-based sampling converges faster than prior sampling for a rough assessment of a datapoint.Source: §2.3 · pp. 11–12

2.4.1 The error from D[Q(z|X)∥P(z|X)]

  • A Gaussian Q(z|X) need not match P(z|X) for arbitrary models, but in one dimension the approximation error can be zero when σ is sufficiently small relative to CDF curvature.Such a small σ may badly scale gradients; the passage leaves the multidimensional case for future work.Source: §2.4.1 · pp. 12–13

2.4.2 The information-theoretic interpretation

  • The objective has an information-theoretic interpretation: its terms encode z and reconstruct X, with a penalty for Q being a sub-optimal encoding.The KL term D[Q(z|X)||P(z)] measures extra information about X gained by drawing z from Q rather than the prior.Source: §2.4.2 · p. 13

2.4.3 VAEs and the regularization parameter

  • VAEs generally lack a tunable regularization parameter: rescaling the prior leaves the objective and sampling model unchanged, while σ acts like one for continuous Gaussian outputs but not Bernoulli outputs.For continuous data, σ controls the expected reconstruction accuracy; the Bernoulli model uses the same units for both objective terms.Source: §2.4.3 · p. 14

3 Conditional Variational Autoencoders

Conditional variational autoencoders (CVAEs) address one-to-many generation by conditioning the generative process on an input, allowing outputs to follow a complex, multimodal distribution rather than collapsing to a regression average. Their latent-variable model can generate conditional outputs by sampling from a standard normal prior.

3 Conditional Variational Autoencoders

  • CVAEs model one-to-many mappings with a multimodal output distribution, avoiding the meaningless average-image blur produced by standard regression.The model conditions the entire generative process on the input, without requiring the output structure to be specified explicitly [8].Source: §3 · p. 15
  • A standard denoising autoencoder does not resolve this problem because it still requires a simple parametric conditional distribution for reconstructing complex data such as image patches.Its noise distribution may delete pixels, but the conditional distribution of the original sample given the noisy one remains too simple for complex image data [13].Source: §3 · p. 15
  • The CVAE defines P(Y|X) to maximize ground-truth probability using a latent variable z, with the generative process conditioned on X.At test time, z is sampled independently of X from N(0, I).Source: §3 · p. 16
  • Sampling z ∼N (0, I) at test time yields samples from P(Y|X).The prior remains independent of the input X.Source: §3 · p. 16

4 Examples

The MNIST examples show that a VAE can generate mostly realistic digits, with some class-interpolating failures, while its results depend on latent dimensionality. The section also illustrates conditional digit completion and describes the training setup.

4.1 MNIST variational autoencoder

  • The MNIST model adapts Caffe’s basic AutoEncoder example rather than published VAE structures, using ReLU nonlinearities and ADAM to speed convergence.Training was completed once; five restarts were used to select a learning rate that made the loss descend fastest.Source: §4.1 · pp. 16–17
  • Sigmoid cross-entropy models an independently Bernoulli-binarized X′, while the encoder receives original X; this practice is also used in VAE literature [6].MNIST values are constrained between 0 and 1, motivating the loss choice.Source: §4.1 · p. 17
  • Most generated MNIST digits look realistic, but many fall between digit classes because a continuous distribution passes through a smooth function, making samples difficult to evaluate.The passage gives a sample between 7 and 9 as an example.Source: §4.1 · p. 17, Figure 7 · p. 17
  • Fewer than 4 latent dimensions produce noticeably worse results; 1,000 work well, while 10,000 degrade, as stochastic gradient descent struggles to keep D[Q(z|X)||P(z)] low.The tutorial notes that, in theory, a larger latent space could ignore extra dimensions, but this does not work reliably in practice.Source: §4.1 · pp. 17–18
  • The CVAE conditions digit completion on noisy binary values in a central column, with generated images aligned to ground truth for comparison.The figure caption describes three aligned sets and highlights the conditioning input in the top two images.Source: Figure 8 · p. 18

5 MNIST conditional variational autoencoder

On a deliberately ambiguous MNIST task, the CVAE produces sharper, more believable digits than a regressor that blurs possible outputs. A 1D analysis explains how, under smoothness assumptions and arbitrarily powerful learners, a VAE can approximate the target distribution as observation noise vanishes.

5 MNIST conditional variational autoencoder

  • With a resampled binary middle-pixel column as input, the CVAE generally outputs a specific unblurred digit, while the regressor blurs across plausible digits.The setup makes the task more ambiguous; the regressor can still be suspiciously confident on wrong guesses, suggesting overfitting remains an issue.Source: §5 · p. 19

A Proof in 1D that VAEs have zero approximation error given arbitrarily powerful learners.

  • Under the stated 1D smoothness assumptions, mapping a standard normal through the target CDF’s inverse yields the target distribution as σ → 0.The construction uses f(z) = F^-1(G(z)), where F and G are the target and standard-normal CDFs, respectively.Source: A Proof in 1D that VAEs have zero… · pp. 19–20
  • The analysis identifies curvature of the inverse map g, determined mostly by the ground-truth CDF’s curvature, as the bulk source of approximation error.The proof uses a Gaussian approximate posterior centered at g(X), with standard deviation g′(X)σ; bounded derivatives make the Taylor-expansion terms vanish as σ → 0.Source: A Proof in 1D that VAEs have zero… · pp. 20–21

Related papers

Loading 1606.05908v3…