Source-linked AI summary
Towards Deeper Understanding of Variational Autoencoding Models
Shengjia Zhao, Jiaming Song, Stefano Ermon
TL;DR
VAEs can produce blurry samples on complex images and ignore latent variables, limiting both generation quality and unsupervised feature learning. The paper generalizes VAE optimization, explains these failures, and proposes sequential and generalized objectives that generate sharp LSUN samples and learn meaningful latent features.
Problem
Existing VAEs struggle with blurry samples on complex images and uninformative latent variables, especially when expressive decoders can model data without using the latent code.
Method
The paper derives a broader VAE optimization framework, then uses a sequential VAE and alternative objective to address blurry samples and latent-code neglect.
Results
The sequential VAE generates sharp LSUN bedroom samples with 2-norm pixel-space reconstruction, while the generalized framework learns meaningful latent features with comparable sample quality to the original VAE.
Takeaways & Limitations
VAE failures can arise from the inference distribution or objective design, and the proposed framework offers methods that alleviate both blurry generation and latent-code neglect.
Takeaways & Limitations
VAEs using pixel-wise 2-norm reconstruction can generate blurry or fuzzy samples on complex natural images.
Abstract
from arXiv · showhide
We propose a new family of optimization criteria for variational auto-encoding models, generalizing the standard evidence lower bound. We provide conditions under which they recover the data distribution and learn latent features, and formally show that common issues such as blurry samples and uninformative latent features arise when these conditions are not met. Based on these new insights, we propose a new sequential VAE model that can generate sharp samples on the LSUN image dataset based on pixel-wise reconstruction loss, and propose an optimization criterion that encourages unsupervised learning of informative latent features.
1. Introduction
The paper develops a broader VAE optimization framework to explain failures in complex-image modeling and latent representation learning. It proposes sequential and generalized objectives that produce sharp LSUN samples and meaningful latent features.
- Existing VAEs struggle to model complex datasets such as LSUN and ImageNet without adversarial training or supervised features.
- On natural images, VAEs tend to produce blurry samples, while more expressive decoders can ignore latent variables and hinder unsupervised feature learning.
- The paper introduces a general optimization criterion that is not always a marginal-likelihood lower bound but can recover the data distribution under suitable conditions.
- The proposed sequential VAE generates sharp samples on LSUN bedrooms using 2-norm reconstruction loss in pixel space.
- The generalized framework achieves comparable sample quality to the original VAE while learning meaningful latent features with a powerful PixelCNN decoder.
2. A Novel Derivation for VAE
The paper introduces a broader VAE optimization framework that fits distinct conditional distributions for each latent code, explaining when latent features and data-distribution recovery succeed. It identifies posterior expressiveness and latent-code regularization as central design conditions, while noting limits on ancestral sampling.
- Model family: The framework defines conditional models by combining a tractable distribution family P with a parameterized mapping fθ from latent codes to P.The resulting pθ(x|z) belongs to P, while the marginal likelihood is a mixture of distributions in P.
- Optimization criterion: Unlike the standard lower bound, the proposed criterion fits a different q(x|z) for each latent code instead of one distribution to pdata(x).This avoids the trivial solution in which every latent code maps to the same optimal member of P.
- Latent-code utilization: With sufficient representation capacity, the optimal mapping uses P to approximate q(x|z) separately for each z, encouraging latent codes with discriminative information.If q(x|z1)=q(x|z2), the objective has no motivation to map those codes to different distributions.
- Data-distribution recovery: Under suitable conditions, the learned joint distribution has marginal pdata(x), provided P can represent the posterior q(x|z) and the relevant ergodicity condition holds.The result concerns q(z)pθ*(x|z), rather than necessarily the original p(z)pθ*(x|z).
- Sampling limitation: The approach generally cannot tractably sample pdata(x) ancestrally because p(z)pθ*(x|z) need not have marginal pdata(x).An additional condition is required for the original generative model to recover the data marginal.
- Learning the inference distribution: Jointly optimizing qφ and pθ can learn feature detectors whose posteriors are representable by P, while regularization may encourage qφ(z) to approach the prior p(z).Without regularization, the resulting unregularized VAE can model pdata through a Gibbs Markov chain but does not permit direct tractable ancestral sampling.
3. Simple P Requires Discriminative q
When the inference distribution maps multiple data points to the same latent code, simple conditional families cannot represent the resulting posterior well, producing high reconstruction variance and blurry samples. Sequential VAE addresses this by progressively augmenting latent information and refining reconstructions, yielding sharper images with pixel-wise loss.
- Limitations of Gaussian conditionals P: A simple family P can poorly approximate qφ(x|z) when the encoder maps different data points to the same latent code.The resulting posterior over data points associated with a feature may be too complex for fixed-variance Gaussian or other simple conditionals.
- Limitations of Gaussian conditionals P: For L2 reconstruction, the optimal decoder output is the conditional mean, and its expected error equals the sum of coordinatewise posterior variances.The reconstruction is an M-projection onto qφ(x|z) and therefore satisfies moment-matching conditions.
- Limitations of Gaussian conditionals P: High posterior variance regions correspond to fuzzy generated samples, while low-variance regions generally produce well-formed outputs.This relationship is illustrated by plotting posterior variance over latent space alongside generated digits.
- Limitations of Gaussian conditionals P: The paper attributes VAE fuzziness to the approximation of maximum likelihood through the inference distribution, rather than directly to maximum likelihood itself.This conclusion is presented as consistent with evidence that changing divergence objectives under adversarial learning produces no major difference.
- Sequential VAE: Sequential VAE generalizes infusion training by extracting features at each step and conditioning each reconstruction on prior results plus newly inferred information.Each step uses an inference distribution for a latent code and a decoder conditioned on the previous reconstruction; training maximizes the VAE criterion at each step.
- Sequential VAE: Sequential generation progressively increases latent-code information, enabling sharper LSUN images from pixel-wise 2-norm loss while supporting multiple model variants.The reported experiments show increasingly sharp images and added detail across sequence steps, and the framework is described as general beyond infusion training.
4. Complex P and the Information Preference Property
With expressive decoders such as PixelCNN, the ELBO can produce good samples while driving mutual information to zero and ignoring the latent code. Removing regularization encourages latent-code use, but tractable ancestral sampling can fail.
- Information preference property: Complex conditional models optimized with the ELBO tend to ignore the latent code.This information preference property is associated with expressive decoders.
- Information preference property: When the decoder family can already model the data, making latent variables independent of observations eliminates the remaining divergence without learning useful features.The model can set pθ(z|x)=p(z) and qφ(z|x)=p(z).
- Alternative objective: Removing regularization creates an incentive to use the latent code and favors maximizing mutual information between x and z.This differs from the ELBO objective, which minimizes mutual information in the cited analysis.
- Experiments: For PixelCNN VAEs, ELBO ancestral samples match Markov-chain quality while mutual information falls to zero; unregularized ancestral samples are nonsensical, although Markov-chain samples remain comparable.The unregularized model prefers high mutual information, whereas the ELBO model leaves the latent code unused.
- Experiments: On CIFAR, PixelCNN is too limited to model the data directly, so both models use the latent code and KL(q(z|x)||p(z)) replaces mutual-information estimates.The mutual information was considered too difficult to estimate directly in this experiment.
- Experiments: The general framework can retain comparable sample quality while learning meaningful latent features with a powerful PixelCNN decoder.The paper demonstrates this experimentally using its broader VAE family.
5. Conclusion
The paper derives a general VAE family from a posterior-approximation perspective rather than by lower-bounding marginal likelihood. This framework explains blurry samples and latent-code neglect and motivates two models that alleviate them.
- Conclusion: The proposed VAE family is derived from variational posterior approximation rather than a lower bound on intractable marginal likelihood.The paper presents this as a new perspective for deriving VAE methods.
- Conclusion: The framework explains blurry samples and the tendency to ignore the latent code, motivating two VAE models that significantly alleviate these problems.These are the principal failure modes identified in the conclusion.
A.1. Comparison to Adversarial Training
The paper contrasts VAE behavior with adversarial training when the conditional distribution is complex and the model family is simple. It expects adversarial training to have an advantage in that setting, while carefully designed complex families can also perform well.
- Comparison to Adversarial Training: In VAE-style modeling, identical conditional distributions for different latent codes provide no incentive to use distinct members of the model family.This limits how representation capacity is exploited when q(x|z1)=q(x|z2).
- Comparison to Adversarial Training: Adversarial training can map different latent codes to different models without inference because it selects support-covered models rather than approximating q(x|z).Its objective is to generate realistic samples rather than a particular conditional set.
- Comparison to Adversarial Training: The paper expects adversarial training to outperform VAE when q(x|z) is complex and P is simple, while carefully designed complex P models can also show promise.This is presented as a scope-specific expectation, not a universal ranking.
A.2. Failure Modes for Factorized Discrete Family
For a factorized discrete decoder family, the best conditional model produces pixel-wise independently sampled outputs, so mismatch with the inferred conditional distribution appears as excessive noise.
- Factorized discrete family: For factorized discrete distributions, the optimal decoder is chosen independently across dimensions for each latent code.The cited proposition characterizes the optimal solution for this family.
- Factorized discrete family: The best achievable conditional error is the pixel-wise negative entropy.This follows from the factorized discrete decoder analysis.
- Failure mode: Mismatch between the inferred conditional distribution and a factorized discrete decoder produces excessively noisy outputs because each pixel is sampled independently.The failure mode differs from blur by appearing as independent pixel noise.
A.3. Estimating Mutual Information
The paper estimates mutual information from samples drawn from q(x, z), but the resulting estimate is limited by the sample count. This limitation also applies to the empirical data distribution itself.
- Mutual information is estimated using M samples x_i, z_i ∼ q(x, z).
- The estimate is upper bounded by log M, making it unreliable when mutual information is large.
- For a dataset with M samples, the true mutual information under the empirical data distribution is also upper bounded.
B. Proofs
The proofs characterize optimal solutions for reconstruction and distribution-matching objectives. They use sufficiently expressive model families, KL-divergence non-negativity, and posterior conditions to establish global optima and correct data marginals.
- When F is sufficiently large, a function f* can achieve the best expected log-likelihood for every latent value z.
- The resulting f* is a global maximum of L.
- If every posterior q(x|z) belongs to P, the optimum in Equation (11) is attained under that posterior-family condition.
- The constructed optima have marginal pdata(x) when the stated conditions hold.
- Under squared-error reconstruction, the optimal x* is the mean of q(x|z), and the minimum expected loss equals Var[q(x|z)].
- For factorized discrete P, the unique optimum sets each independent probability to pi = q(xi|z).
C. Experimental Setup
The experimental setup uses a sequential VAE with step-specific encoders, autoregressive decoders, and shortcut connections that refine earlier reconstructions. Experiments use convolutional PixelCNN variants with 20-dimensional latent codes on MNIST and CIFAR.
- Sequential VAE: Each sequential-VAE step encodes ground-truth x and decodes from the previous reconstruction plus a latent code.
- Sequential VAE: Shortcut connections encourage identity mappings so later steps preserve and refine earlier results.
- Architecture: Figure 7 depicts convolutional blocks followed by upsampling or downsampling, with two convolution layers before each sampling operation in the experiments.
- Architecture: The architecture uses a non-homogeneous Markov chain with weights not shared across time steps.
- Datasets and decoders: MNIST uses a simplified conditional PixelCNN, while CIFAR uses PixelCNN++; both receive a 20-dimensional latent code and are trained end to end.