Source-linked AI summary
Tutorial on Variational Autoencoders
Carl Doersch
TL;DR
Learning complicated, multimodal distributions requires more than ranking unlikely samples or averaging ambiguous outputs. This tutorial explains VAEs’ latent-variable framework and mathematics, highlighting their fast training, popularity, and approximation behavior.
Problem
High-dimensional likelihood estimation may require extremely large samples, while useful similarity metrics are difficult to engineer or train without labels.
Method
VAEs model data with latent variables, neural-network approximators, and a Gaussian approximate posterior trained efficiently through backpropagation.
Results
In one dimension, VAEs can achieve zero approximation error when σ is sufficiently small relative to the ground-truth distribution’s curvature.
Takeaways & Limitations
Weak assumptions and fast backpropagation have contributed to VAEs’ rapid rise in popularity.
Takeaways & Limitations
A Gaussian variational posterior may not match the true posterior, so the approximation error may never vanish for a fixed model.
Abstract
from arXiv · showhide
In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. VAEs are appealing because they are built on top of standard function approximators (neural networks), and can be trained with stochastic gradient descent. VAEs have already shown promise in generating many kinds of complicated data, including handwritten digits, faces, house numbers, CIFAR images, physical models of scenes, segmentation, and predicting the future from static images. This tutorial introduces the intuitions behind VAEs, explains the mathematics behind them, and describes some empirical behavior. No prior knowledge of variational Bayesian methods is assumed.
1 Introduction
Generative modeling seeks to learn high-dimensional data distributions so it can synthesize new, similar examples rather than merely assign probabilities. VAEs address longstanding training difficulties with weak assumptions, fast backpropagation, and a typically small approximation error, while this tutorial provides an informal introduction for readers without variational-Bayesian expertise.
- Motivation: Generative models learn distributions over datapoints, such as images, by capturing dependencies among potentially millions of dimensions.For images, realistic examples should receive high probability and random-noise images low probability.
- Motivation: The practical goal is to synthesize unseen examples resembling database samples, including images, 3D plants, and handwritten text.Such tools could support applications such as populating video-game forests or assisting graphic designers.
- Challenges: Classical generative-model training faced strong structural assumptions, severe approximations, or computationally expensive inference such as Markov Chain Monte Carlo.These drawbacks made learning complicated dependencies difficult.
- Variational Autoencoders: VAEs became popular because they use weak assumptions, train quickly via backpropagation, and introduce arguably small approximation error with high-capacity models.The tutorial identifies the Variational Autoencoder framework as its subject.
- Tutorial Scope: The tutorial informally explains VAE intuitions, mathematics, and empirical behavior for readers who may lack backgrounds in variational Bayesian and minimum-description-length methods.It was developed from computer-vision reading-group presentations and has a vision-oriented bias.
- Generative Framework: VAEs model data through latent variables and a conditional output distribution, often Gaussian, so maximum likelihood can be optimized with gradient-based methods.The output distribution need not be Gaussian; it must be computable and continuous in the model parameters.
2 Variational Autoencoders
VAEs learn a deterministic transformation from simple latent variables to complex data distributions and optimize a tractable lower bound on log P(X) using stochastic gradient descent. Their practical approach addresses latent-variable design and intractable integration, but relies on approximation assumptions and can face high-dimensional sampling and posterior-shape limitations.
- Sampling and likelihood: Sampling directly from P(X) avoids Markov Chain Monte Carlo, but estimating likelihoods by sampling can require extremely large sample counts in high-dimensional spaces.Pixel-space Euclidean distance may poorly reflect perceptual similarity, making likelihood evidence difficult to obtain from samples.
- Model intuition: VAEs learn latent variables z and map simple distributions through powerful functions to generate arbitrary data distributions.Any d-dimensional distribution can be generated by mapping d normally distributed variables through a sufficiently complicated function.
- Objective and inference: The VAE objective is a computable lower bound on log P(X), optimized with stochastic gradient descent through a suitable approximate posterior Q(z|X).The bound includes a divergence term and an expectation over latent samples; sampling from Q generally converges faster than sampling from the prior.
- Optimization: Training must ensure that Q produces codes that P can reliably decode, because the reconstruction expectation depends on both model and posterior parameters.This dependency is absent from the naive gradient expression, creating a central optimization problem for VAEs.
- Approximation limits: Tractability assumes Q(z|X) is Gaussian, although the true posterior need not be Gaussian for an arbitrary decoder function.The modeled distribution converges to the true distribution only when D[Q(z|X)∥P(z|X)] approaches zero.
- Regularization: For Gaussian outputs, σ acts like a regularization parameter weighting reconstruction and divergence terms, whereas changing the prior scale alone does not change the model.The analogous regularization effect depends on the chosen output distribution and does not apply identically to binary Bernoulli outputs.
3 Conditional Variational Autoencoders
Conditional variational autoencoders model multimodal conditional outputs by introducing a latent variable z alongside the input X. At test time, sampling z from N (0, I) enables sampling outputs from P(Y|X).
- Motivation: Conditional generation is needed for tasks such as extending a person’s handwritten digit string or filling holes in images with plausible pixels.These tasks require modeling an output conditioned on an existing input.
- Motivation: Multimodal plausible outputs make standard regression produce an in-between result because its objective penalizes distance from a single ground truth.The problem applies to both choosing the next digit and extrapolating missing pixels.
- Limitations: A standard denoising autoencoder still does not solve the problem because it requires a restrictive conditional distribution for the original sample given the noisy sample.Deleting pixels can define the noise distribution, but reconstruction remains conditionally constrained.
- Model: The model defines P(Y|X) with a latent variable z ∼N (0, I) and a learnable deterministic function f.The latent variable is sampled independently of X at test time, so P(z|X) remains N (0, I).
- Model: At test time, sampling z ∼N (0, I) directly samples from the conditional distribution P(Y|X).The conditional variational autoencoder is implemented as a feedforward neural network during training.
4 Examples
The examples demonstrate VAE distribution learning on MNIST using a Caffe autoencoder adapted with ReLU nonlinearities and ADAM. The MNIST setup models randomly binarized inputs with Sigmoid Cross Entropy, and performance is generally robust to latent dimensionality except at extremes.
- MNIST: The MNIST experiment adapted Caffe’s basic autoencoder rather than using published VAE architectures, while adding ReLU nonlinearities and ADAM to speed convergence.This tests whether the framework depends heavily on initialization or network structure.
- MNIST: Sigmoid Cross Entropy models P(X|z) by treating each MNIST dimension as an independently sampled Bernoulli variable.The model therefore represents randomly binarized MNIST as X′ while giving q a summary of the original X.
- MNIST: Despite being considerably deeper than prior models, the VAE was not difficult to train and reached completion after exactly one training run.The passage notes that this setup works well in practice despite not exactly following the prescribed VAE framework.
- Latent dimensionality: Less than 4 z dimensions produced noticeably worse results, while 1,000 z dimensions worked well and 10,000 z dimensions degraded performance.The observed degradation at very large latent dimensionality is attributed to stochastic gradient descent struggling to keep D[Q(z|X)||P(z)] low.
5 MNIST conditional variational autoencoder
For MNIST digit completion from half of each digit, a conditional variational autoencoder works well, but a regressor also produces relatively crisp samples. The regressor’s performance is attributed to MNIST’s small size, which allows a similarly capable network to memorize the dataset and overfit.
- 5 MNIST conditional variational autoencoder: A conditional variational autoencoder works well for completing MNIST digits from only half of each digit.This was the originally intended demonstration.
- 5 MNIST conditional variational autoencoder: A regressor also produces relatively crisp samples for this MNIST completion task.Thus, crisp predictions are not unique to the CVAE in this setting.
- 5 MNIST conditional variational autoencoder: MNIST’s small size lets a network with capacity similar to the Section 4.1 model memorize the entire dataset, causing the regressor to overfit badly.The passage identifies dataset size as the apparent reason for the regressor’s strong behavior.