PaperlayerView exact revision on arXiv

Source-linked AI summary

A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

Tim Salimans

arXiv:1602.08734v1stat.MLcs.LGstat.CO

TL;DR

The paper models deep hierarchical latent features with rectified-Gaussian units and a structured encoder that mirrors the prior dependencies. It reports a −92.5-nat variational lower bound on binarized MNIST and describes joint training without layerwise procedures, while noting that deep models have not dramatically improved over simpler single-layer models.

  • Problem

    A mean-field posterior does not preserve the generative model's layerwise dependencies, motivating a structured approximation for hierarchical latent features.

    Source: §1 · p. 2

  • Method

    The model uses rectified-Gaussian latent layers and a data-dependent structured posterior whose conditional dependencies mirror the generative prior.

    Source: §1 · pp. 1–2

  • Results

    −92.5 nats was the variational lower bound reached by the four-layer model on binarized MNIST.

    Source: §2 · p. 3

  • Takeaways & Limitations

    The structured approximation allows joint training of deep latent-variable models without stacking or other layerwise training procedures.

    Source: §3 · p. 3

    Neither this method nor the probabilistic ladder network had dramatically improved over simpler single-layer latent-variable models.

    Source: §3 · p. 3

Abstract

from arXiv · show

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability necessary for efficient stochastic gradient variational inference. To learn the parameters of the new model, we approximate the posterior of the latent variables with a variational auto-encoder. Rather than making the usual mean-field assumption however, the encoder parameterizes a new type of structured variational approximation that retains the prior dependencies of the generative model. Using this structured posterior approximation, we are able to perform joint training of deep models with many layers of latent random variables, without having to resort to stacking or other layerwise training procedures.

1 A structured variational auto-encoder model

The model builds a hierarchy of conditionally dependent latent-feature layers with rectified-Gaussian units, then uses a structured encoder whose dependencies mirror the generative prior. Its variational objective is evaluated through upward inference and downward KL calculations, enabling joint optimization of the model and encoder.

  • The generative model organizes latent features into layers that are conditionally independent given the layer above.Source: §1 · p. 1
  • Rectified-Gaussian latent units combine a point mass at zero with a positive truncated Gaussian, while remaining differentiable for fixed noise.The same parameters determine both mixture components.Source: §1 · p. 1
  • The final latent layer generates observations through a conditional distribution, such as independent Bernoulli outputs for binary data.The model defines the observed-data distribution by marginalizing over the latent features.Source: §1 · pp. 1–2
  • The variational bound is evaluated with an upward encoder pass and downward per-layer KL calculations, then optimized jointly over generative and variational parameters.The conditional KLs are analytic because the prior and posterior marginals are factorized; posterior samples continue the pass through the hierarchy.Source: §1 · pp. 2–3
  • The encoder's structured posterior mirrors the prior's layerwise dependencies, with data-dependent conditional parameters computed recursively from the data upward.The approximate-likelihood parameters are computed bottom-up, unlike the top-down prior.Source: §1 · p. 2

2 Experiments

On binarized MNIST, a four-layer model with sparse latent features reached a test variational lower bound comparable to single-layer-variable VAEs with multiple deterministic layers.

2 Experiments

  • After 500 epochs, the four-layer model reached a test variational lower bound of −92.5 nats, comparable to VAEs with single-layer random variables and multiple deterministic layers.The binarized MNIST model used 50, 100, 200, and 300 units from top to bottom.Source: §2 · p. 3
  • The model used minibatches of 150 examples to maximize Batch Normalization’s regularizing effect, while limiting units because larger models easily overfit training data.Source: §2 · p. 3

3 Conclusion

The paper combines a hierarchical generative model with a structured top-down variational approximation to jointly train deep models with sparse stochastic latent variables. However, the method has not dramatically improved over simpler single-layer latent-variable models.

3 Conclusion

  • The structured top-down variational approximation combines the hierarchical model with conjugate approximate likelihood terms, enabling joint training without stacking or other layerwise procedures.Source: §3 · p. 3
  • The proposed variational auto-encoder contains many layers of stochastic latent variables with sparse activations.Source: §3 · p. 3
  • Neither this method nor the probabilistic ladder network [5] has so far improved dramatically over simpler models with a single layer of latent variables.Developing deep generative models that more fully use the expressivity of multiple stochastic latent layers remains a future goal.Source: §3 · p. 3

Related papers

Loading 1602.08734v1…