PaperlayerView exact revision on arXiv

Source-linked AI summary

Avoiding Latent Variable Collapse With Generative Skip Models

Adji B. Dieng, Yoon Kim, Alexander M. Rush, David M. Blei

arXiv:1807.04863v2stat.MLcs.CLcs.LG

TL;DR

VAEs can learn good generative models while latent variable collapse prevents useful representations, especially with powerful likelihoods. This paper adds skip connections to the generative model to strengthen latent–observation dependence. Across theoretical and empirical analyses, skip-VAEs preserve generative performance while reducing collapse and improving representations.

  • Problem

    Powerful VAE likelihoods can make the approximate posterior collapse to the prior, leaving latent variables independent of data and representations uninformative.

  • Method

    The paper adds nonlinear skip connections from latent variables to multiple layers of the likelihood network, enforcing stronger latent–observation dependence.

  • Results

    Skip-VAEs maintain similar approximate likelihood while promoting greater observation–latent dependence, reducing collapse and producing better representations.

  • Takeaways & Limitations

    Generative skip models provide useful data summaries and can further improve representations when combined with semi-amortized VAEs.

  • Takeaways & Limitations

    The image visualization used 50-dimensional latents projected with t-SNE rather than directly fitting realistic two-dimensional latent models.

Abstract

from arXiv · show

Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," especially if the likelihood model is powerful. Specifically, the lower bound involves an approximate posterior of the latent variables; this posterior "collapses" when it is set equal to the prior, i.e., when the approximate posterior is independent of the data. While VAEs learn good generative models, latent variable collapse prevents them from learning useful representations. In this paper, we propose a simple new way to avoid latent variable collapse by including skip connections in our generative model; these connections enforce strong links between the latent variables and the likelihood function. We study generative skip models both theoretically and empirically. Theoretically, we prove that skip models increase the mutual information between the observations and the inferred latent variables. Empirically, we study images (MNIST and Omniglot) and text (Yahoo). Compared to existing VAE architectures, we show that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.

1 Introduction

VAEs provide powerful unsupervised representation learning but can suffer latent variable collapse, where approximate posteriors ignore observations. The paper proposes generative skip models that preserve strong latent–observation links while maintaining generative performance.

  • VAEs aim to learn low-dimensional representations of high-dimensional data using deep generative models and variational inference.
  • Latent variable collapse occurs when the variational posterior becomes independent of the data, producing poor representations despite a good generative model.
  • The paper adds skip connections from latent inputs to multiple likelihood-network layers, enforcing dependence between latent variables and observations.
  • Skip-VAEs maintain similar approximate likelihood while promoting greater dependence between observations and latent variables.
  • Generative skip models can complement semi-amortized VAEs and further improve learned representations.
  • The approach differs from strategies that weaken the generative model, modify priors, or restrict variational distributions.

2 Latent variable collapse issue in vaes

VAEs can trade away informative latent representations when powerful likelihoods make latent variables unnecessary, producing latent variable collapse. The ELBO formalizes this tension: increasing mutual information can conflict with maximizing the objective, while simply removing KL regularization harms generative modeling.

  • Collapse mechanism: Flexible likelihoods can drive the KL term toward zero while still assigning high probability to the data, yielding poor latent representations.A sufficiently powerful likelihood can model the data without using information from z.
  • Objective tension: The ELBO creates tension because high mutual information supports useful representations, while its objective contains the negative of mutual information.This follows from the ELBO decomposition involving the KL term and mutual information.
  • Objective tension: Maximizing reconstruction-like objectives without regularization can avoid zero KL but produces poor generative models because the variational distribution diverges from the prior.The prior is the distribution used to generate samples after training.

3 Generative skip models avoid latent variable collapse

Generative skip models attach latent variables to multiple layers of the likelihood network, strengthening the link between z and x while retaining the original model’s functions. Theory and experiments show that skip-VAEs promote higher mutual information and more useful latent representations than ordinary VAEs.

  • Model design: Skip-VAEs extend VAEs by adding residual paths from the latent variable to multiple layers of the generative model.At each layer, skip functions combine z with the preceding hidden state.
  • Empirical representations: On MNIST, skip-VAEs produce clearer digit clusters and discover 7 digit classes, whereas VAE latents remain spread out.The comparison uses a 9-layer PixelCNN and a 3-layer ResNet variational network.
  • Model design: The skip construction can be applied to any VAE and uses nonlinear skip functions alongside the original generative-network functions.The paper studies a simple subclass using nonlinearities such as sigmoid or ReLU.
  • Empirical results: Skip-VAEs promote higher mutual information when trained with the ELBO objective, with advantages increasing as the generative model becomes deeper.The models are designed to maintain useful representations while preserving generative modeling.
  • Theory: Theorem 1 establishes that for any VAE, a corresponding skip-VAE can achieve higher mutual information between observations and latent variables.The proof uses the data processing inequality across the generative hierarchy.
  • Theory: Skip-VAEs increase mutual information in the generative and variational joints, thereby mitigating latent variable collapse.The paper reports this increase empirically relative to the ordinary VAE.

4 Empirical study

The empirical study evaluates skip-VAEs on image and text benchmarks using predictive-performance and latent-collapse metrics. Across settings, skip connections reduce collapse and improve latent representations while generally preserving likelihood performance.

  • Images: Skip-VAEs outperform VAEs on MNIST collapse metrics while achieving similar ELBO values as latent dimensionality increases.The evaluated metrics include KL divergence, mutual information, and active units.
  • Images: As generative-model depth increases on MNIST and Omniglot, skip-VAEs maintain better collapse metrics than VAEs with similar ELBO values.The performance difference widens as the number of layers increases.
  • Images: Even with simplified multilayer-perceptron networks, skip-VAEs produce less latent-variable collapse than VAEs.This setting uses 50 latent dimensions and varies decoder depth.
  • Latent Representations: Skip-VAEs learn clearer MNIST digit separation in latent space than VAEs, indicating more informative latent representations.The visualization uses 50-dimensional latents projected to two dimensions with t-SNE.
  • Text: On Yahoo text, skip-VAEs slightly reduce collapse relative to VAEs at similar log likelihoods, although KL remains marginally above zero and only one unit is active.Text evaluation reports ELBO and perplexity alongside collapse metrics.
  • Text: Combining skip connections with semi-amortized training further increases mutual information and can activate all latent dimensions.Skip connections improve collapse metrics as latent dimensionality increases, unlike vanilla semi-amortized VAEs in the reported comparison.

5 Conclusion

The paper proposes skip connections to reduce latent variable collapse in VAEs, strengthening dependence between observations and their latent variables. Skip-VAEs yield higher mutual information and learn useful data summaries, with significant improvement when combined with SA-VAE.

  • Skip connections promote stronger dependence between observations and their associated latent variables, reducing latent variable collapse in VAEs.
  • Skip-VAEs theoretically yield higher mutual information than corresponding models without skip connections.
  • Skip-VAEs learn useful summaries of data and significantly improve latent variable collapse when used with more sophisticated VAEs such as SA-VAE.
Loading 1807.04863v2…