Source-linked AI summary
Ladder Variational Autoencoders
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther
TL;DR
Deep VAEs are difficult to train because their multiple stochastic layers challenge purely bottom-up inference. The paper introduces Ladder-VAE, which recursively combines data-dependent approximate likelihood information with the generative distribution, and reports better generative performance, tighter likelihood bounds, and deeper latent representations. It also identifies deterministic warm-up and batch normalization as important for training deep stochastic models.
Problem
Deep hierarchical VAEs are difficult to optimize, and purely bottom-up inference only limitedly utilizes multiple stochastic latent layers.
Method
Ladder-VAE recursively corrects the generative distribution with a data-dependent approximate likelihood using matching top-down dependency structures in inference and generation.
Results
LVAE improves approximated log-likelihood, provides a tighter log-likelihood bound, and learns a deeper, qualitatively different latent representation than bottom-up VAE inference.
Takeaways & Limitations
Deterministic warm-up and batch normalization are important for optimizing deep VAEs and LVAEs, while LVAE's explicit inference–generative parameter sharing can ease optimization.
Takeaways & Limitations
Future work includes extending LVAE to semi-supervised learning and studying more elaborate iterative inference schemes.
Abstract
from arXiv · showhide
Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.
1 Introduction
The paper introduces Ladder-VAE, a structured inference model that improves deep VAE training without changing the generative model. It reports better generative performance, tighter likelihood bounds, deeper latent representations, and important training benefits from warm-up and batch normalization.
- Motivation: Deep VAEs are difficult to optimize because multiple conditional stochastic layers limit the usefulness of purely bottom-up inference.The paper frames this as a limitation of deep hierarchical stochastic models and their variational approximate posteriors.
- Method: Ladder-VAE uses the same top-down dependency structure in inference and generation, allowing bottom-up and top-down signals to interact.The inference model recursively corrects the generative distribution with a data-dependent approximate likelihood while leaving the generative model unchanged.
- Results: Compared with bottom-up VAE inference, LVAE achieves better generative performance, a tighter true-log-likelihood bound, and deeper latent-variable utilization.The paper also compares LVAE favorably with several flexible variational-inference methods.
- Results: LVAE learns a deeper and more distributed latent representation than VAE.The paper presents this as a qualitative difference in the learned hierarchical representation.
- Training: Deterministic warm-up and batch normalization are important for training deep stochastic models.These techniques are identified as key contributors to successful optimization of deep VAEs and LVAEs.
2 Methods
The paper trains hierarchical VAEs by optimizing a tractable variational likelihood lower bound, then introduces LVAE inference that combines bottom-up data signals with top-down generative information. Warm-up addresses inactive latent units during training, while batch normalization and warm-up support deeper stochastic models.
- Variational autoencoder inference model: VAEs jointly train a generative model pθ(x,z) and inference model qφ(z|x) by optimizing a variational lower bound on pθ(x).The generative model factors as pθ(x|z)pθ(z), with latent variables organized into layers.
- Variational autoencoder inference model: Bottom-up VAE inference specifies each stochastic layer as a fully factorized Gaussian conditioned on the layer below.The inference and generative distributions are computed separately, without explicit information sharing.
- Ladder variational autoencoder inference model: LVAE recursively corrects the generative distribution with a data-dependent approximate likelihood using a deterministic upward pass and stochastic downward pass.The downward pass computes approximate posterior and generative distributions, combining bottom-up and top-down signals.
- Training and evaluation: Batch normalization and warm-up are important for training deep stochastic models with multiple latent layers.The paper studies MNIST log-likelihood across VAEs and LVAEs with varying latent-layer counts, batch normalization, and warm-up.
- Ladder variational autoencoder inference model: LVAE uses the same top-down dependency structure in inference and generation while keeping the generative model unchanged and using a similar parameter count.The inference model combines approximate likelihood parameters with prior information from the generative distribution.
- Warm-up from deterministic to variational autoencoder: Warm-up gradually increases β from 0 to 1 so training begins with reconstruction error before introducing the variational regularization term.This targets inactive latent units that can remain near KL divergence zero during optimization.
3 Experiments
Experiments show that LVAE improves likelihood performance as latent depth increases, while batch normalization and warm-up are important for training deep stochastic models. LVAE also learns deeper and more distributed latent representations than VAE.
- Evaluation setup: The models were evaluated on MNIST, OMNIGLOT, and NORB using hierarchies of up to five stochastic latent layers.The largest hierarchy used layer sizes 64, 32, 16, 8, and 4.
- Generative log-likelihood performance: LVAE performance improves with each additional latent layer, reaching Ltest = −85.23 with five layers versus −87.49 for the best three-layer VAE.The improvement decreases with depth but remains consistent through the topmost layers.
- Generative log-likelihood performance: Ltest^5000 = −82.12 for LVAE compared with −82.74 for the best VAE, indicating both a higher approximate log-likelihood and a tighter lower bound.These values use 5000 importance-weighted samples.
- Generative log-likelihood performance: Comparisons with other MNIST generative models suggest LVAE is competitive, but the training procedures differ and results are not directly comparable.The comparison includes normalizing-flow VAE, importance-weighted VAE, and variational Gaussian-process VAE.
- Generative log-likelihood performance: On OMNIGLOT, LVAE achieves its best performance with five latent layers, reaching −102.11 versus −103.38 for prior work using more latent variables.The prior result used 100-50 latent variables and 50 importance-weighted samples for training, compared with LVAE’s 64-32-16-8-4 hierarchy.
- Generative log-likelihood performance: On NORB, LVAE slightly outperforms VAE, but adding more than three stochastic layers does not improve performance.The Gaussian observation models were harder to optimize, potentially limiting use of the topmost latent layers.
- Latent representations: Warm-up and batch normalization activate more latent units, while LVAE produces the deepest and most distributed representation with greater use of higher layers.VAE concentrates KL divergence in lower layers, whereas LVAE distributes it more broadly across the hierarchy.
- Latent representations: LVAE uses all five latent layers and shows progressively stronger class clustering, especially in the topmost layer.Vanilla VAE representations above the second layer collapse onto a standard normal prior.
4 Conclusion and Discussion
The paper presents LVAE as an inference model that combines data-dependent approximate likelihoods with generative priors. It reports improved likelihood, tighter bounds, and deeper latent representations, while identifying future extensions and unresolved optimization questions.
- Contribution: LVAE recursively corrects the generative distribution with a bottom-up data-dependent approximate likelihood contribution.This explicit parameter sharing makes inference a correction of the generative distribution rather than a separately fitted model.
- Findings: The parameterization increases approximate log-likelihood, provides a tighter likelihood bound, and learns a deeper, qualitatively different latent representation.The conclusion attributes these outcomes to the proposed inference model relative to VAE.
- Optimization: Deterministic warm-up and batch normalization are important for optimizing deep VAEs and LVAEs.The authors note that the benefits of batch normalization for generative performance and representation depth are not fully understood.
- Future work: Future work includes semi-supervised learning and more elaborate iterative inference schemes such as k-step LVAE inference.These directions are motivated by the learned deep structured hierarchies and possible inference extensions.
A Additional Results
Additional figures examine likelihood across datasets and configurations, latent-unit activity during training, and qualitative samples. They highlight the roles of batch normalization, warm-up, and LVAE’s deeper representation.
- Likelihood results: MNIST likelihood plots compare VAE and LVAE across latent depths with batch normalization and warm-up, including estimates using 5000 importance samples.The LVAE without batch normalization performs very poorly, with some results outside the plotted range.
- Likelihood results: OMNIGLOT likelihood plots similarly compare train, test, and 5000-sample importance-weighted log-likelihood across model depths and training settings.The figure varies batch normalization and warm-up alongside the number of latent layers.
- Generated samples: OMNIGLOT and MNIST sample figures show true data, conditional reconstructions, and samples from the prior distribution.These panels provide qualitative comparisons of reconstruction and generation behavior.