PaperlayerView exact revision on arXiv

Source-linked AI summary

VAE with a VampPrior

Jakub M. Tomczak, Max Welling

arXiv:1705.07120v5cs.LGcs.AIstat.ML

TL;DR

VAEs can suffer from overly simple priors that over-regularize latent representations and leave dimensions inactive. The paper introduces the VampPrior and a two-level hierarchical VAE using learnable pseudo-inputs, and reports state-of-the-art or comparable results across six datasets while overcoming inactive stochastic latent variables.

  • Problem

    A standard normal prior can over-regularize VAEs, producing poor representations and few active latent dimensions.

  • Method

    The paper replaces the fixed prior with a mixture of variational posteriors conditioned on learnable pseudo-inputs and extends it to a two-level VAE.

  • Results

    The hierarchical VampPrior VAE achieves state-of-the-art or comparable results on six datasets and successfully overcomes inactive stochastic latent variables.

  • Takeaways & Limitations

    The VampPrior provides a flexible, capacity-controlled prior that improves generative models and latent-variable usage within the evaluated VAE settings.

  • Takeaways & Limitations

    Using the aggregated posterior directly may overfit and requires expensive optimization over all training points.

Abstract

from arXiv · show

Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks.

1 Introduction

The paper argues that the VAE prior critically shapes latent representations and proposes the VampPrior, a flexible prior based on variational posteriors and learnable pseudo-data. A hierarchical extension is evaluated across datasets and achieves state-of-the-art or comparable results.

  • A standard normal prior can over-regularize VAEs and produce poor hidden representations.
  • The VampPrior is a mixture of variational posteriors conditioned on learnable pseudo-data, making the prior more flexible.
  • The paper introduces a two-layer VAE with two stochastic latent-variable layers based on the VampPrior.
  • The VampPrior outperforms the standard normal prior across different VAE architectures.
  • The hierarchical VampPrior VAE achieves state-of-the-art or comparable results on six datasets.

2 Variational Auto-Encoder

A VAE uses variational inference and reparameterization to optimize a tractable lower bound on data likelihood. Its reconstruction and regularization terms jointly shape the encoder, while the prior anchors posterior representations.

  • The VAE combines an encoder posterior, a decoder likelihood, and a prior within a variational lower-bound objective.
  • VAEs use reparameterization to make variational optimization efficient and scalable for continuous latent variables.
  • The reconstruction term encourages each hidden representation to concentrate near its data-specific maximum-a-posteriori value.
  • The entropy and cross-entropy terms regularize the encoder by encouraging agreement between the aggregated posterior and the prior.
  • The prior acts as an anchor that keeps the variational posterior close to it.

3 The Variational Mixture of Posteriors Prior

The optimal VAE prior is the aggregated posterior, but using it directly is computationally expensive and may overfit. The VampPrior approximates it with a learnable mixture of posterior components, while coupling prior and posterior parameters.

  • Rewriting the VAE objective separates regularization into posterior entropy and cross-entropy between the aggregated posterior and prior.
  • The ELBO-optimal prior is the aggregated posterior over training points, but directly using it is computationally expensive and may overfit.
  • A standard normal prior can over-regularize models and leave only a few latent dimensions active.
  • The VampPrior approximates the aggregated posterior with a mixture of variational posteriors conditioned on learnable pseudo-inputs.
  • Choosing K ≪ N pseudo-inputs limits prior capacity, reduces training expense, and helps prevent overfitting.
  • Coupling the prior and posterior uses fewer parameters and lets them cooperate during training, distinguishing VampPrior from a Gaussian mixture prior.

4 Hierarchical VampPrior Variational Auto-Encoder

The hierarchical VampPrior VAE addresses inactive stochastic units by using a rich, multimodal prior in a two-layer latent model. The section also motivates comparisons with simpler priors to assess whether prior complexity and trainable pseudo-inputs are necessary.

  • Hierarchical VAE and the inactive stochastic latent variable problem: Inactive stochastic units are a common VAE training problem that becomes worse in deeper models.Top-down generative and bottom-up variational dependencies provide less data information to deeper stochastic layers, making them more prone to prior regularization.
  • Hierarchical VAE and the inactive stochastic latent variable problem: The VampPrior is proposed as a remedy because its rich, multimodal prior prevents the KL term from pulling individual posteriors toward a simple prior.
  • Hierarchical VampPrior model: Figure 1 contrasts stochastic dependencies in one-layered and two-layered models, distinguishing generative and variational parts.Generative dependencies are shown with solid lines, whereas variational dependencies are shown with dashed lines.
  • Hierarchical VampPrior model: The proposed model uses two layers of stochastic latent variables, with p(z2) specified by a VampPrior.The model uses diagonal-covariance normal distributions for z1 ∈ R^M1 and z2 ∈ R^M2, parameterized by neural networks.
  • Alternative priors: Alternative-prior experiments compare the standard Gaussian, mixture of Gaussians, and real-image-subset priors.These comparisons test the need for complex priors, the benefit of coupling the prior with the variational posterior, and the usefulness of trainable pseudo-inputs.

5 Experiments

Experiments across six image datasets evaluate VampPrior variants, hierarchical architectures, and decoder choices. VampPrior generally improves test log-likelihood, activates more latent units, and achieves state-of-the-art or comparable results in several settings while producing sharper generations and reconstructions.

  • Experimental setup: Experiments use six image datasets and compare standard-prior and VampPrior models with MLP, convolutional, and PixelCNN-based architectures.The evaluation includes static and dynamic MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces, and Histopathology patches.
  • Quantitative results: Test log-likelihood improves substantially with VampPrior over the standard normal prior in all but one comparison.The authors attribute the advantage to combining multimodality with coupling between prior and posterior.
  • Prior comparisons: The VampPrior outperforms single-Gaussian and mixture-of-Gaussians priors, while increasing pseudo-inputs or mixture components does not necessarily improve performance.Figure 2 compares HVAE models on static MNIST across varying pseudo-input or component counts.
  • Latent-unit usage: VampPrior increases active stochastic units to four times the second-level count and about 1.5 times the first-level count relative to VAE and IWAE baselines.The number of mixture components strongly affects active units at the second level.
  • Quantitative results: Two-layer VampPrior models achieve state-of-the-art results across MLP-based datasets and approach convolutional-model performance, with PixelHVAE reaching state-of-the-art on dynamic MNIST and OMNIGLOT.PixelHVAE obtains −79.78 on static MNIST, −78.45 on dynamic MNIST, −89.76 on OMNIGLOT, and −86.22 on Caltech 101 Silhouettes.
  • Qualitative results: VampPrior models produce sharper generations and reconstructions than standard VAEs, which often generate blurry or noisy images.Trained pseudo-inputs become prototypical objects, while generations associated with one pseudo-input vary in shape, thickness, and curvature.

6 Related work

The paper situates VampPrior among richer VAE priors and emphasizes its explicit mixture structure, posterior coupling, and learnable pseudo-inputs. It also contrasts VampPrior with autoregressive, memory-based, and weighted-prior approaches.

  • Alternative priors: VAEs commonly use simple standard-normal priors, while alternatives include nonparametric, autoregressive, and VampPrior-based constructions.The paper notes that richer priors can improve generative capabilities but may require sophisticated training or make contributions harder to isolate.
  • VampPrior: VampPrior is an explicit mixture of variational posteriors conditioned on learnable pseudo-inputs, rather than a prior that samples components.The paper argues that using the variational posterior in the prior is central to VampPrior’s success.
  • Comparison: Unlike a standard normal prior, VampPrior’s posterior coupling is reported to improve generative performance, whereas a concurrent memory-based VAE was comparable to the standard prior.The comparison identifies posterior use in the prior as the key difference claimed by the authors.
  • Related extensions: Weighted VampPrior components use trainable nonnegative weights summing to one, allowing the model to identify meaningful pseudo-inputs and potentially reduce overfitting.This weighted extension is presented as recent related work rather than as the core prior introduced here.

7 Conclusion

The paper concludes that VampPrior improves hierarchical VAEs by addressing inactive stochastic latent variables and achieving strong results across six image datasets. It also identifies broader data modalities and model combinations as future work.

  • Conclusion: The proposed VampPrior is a mixture of variational posteriors using freely chosen learnable pseudo-inputs, extended into a two-level generative model.The conclusion frames the prior as a critical element for improving VAEs.
  • Conclusion: VampPrior empirically improves the proposed generative model and overcomes inactive stochastic latent variables, particularly in models with multiple stochastic layers.This is the paper’s stated conclusion about hierarchical latent-variable models.
  • Conclusion: State-of-the-art or comparable-to-SOTA results were achieved on six datasets, with hierarchical VampPrior generations and reconstructions judged better than those from the standard VAE.The reported qualitative improvement concerns both generated images and test-image reconstructions.
  • Future work: The study is limited to image data, and more thorough evaluation is left for text, sound, and combinations with additional density-estimation or inference methods.The authors explicitly leave these extensions for future work.

8 SUPPLEMENTARY MATERIAL

The supplementary material documents experiments, computational costs, visual evaluations, and test log-likelihood distributions for VampPrior models across six image datasets. It reports sharper and more detailed generations and reconstructions, alongside fewer hard-to-represent examples.

  • Experimental setup: Experiments used six image datasets, including static and dynamic MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces, and Histopathology patches.Frey Faces contains 28 × 20 images; the other datasets contain 28 × 28 images.
  • Computational cost: HVAE+VampPrior was about 1.4 times slower than the standard-normal-prior model for K = 500 on MNIST and K = 1000 on OMNIGLOT.The wall-clock measurements were obtained using the authors’ implementation.
  • Computational cost: ConvHVAE and PixelHVAE training times increased by ×1.9/×2.1 and ×1.4/×1.7 for K = 500/K = 1000, respectively.These factors are reported relative to the standard prior.
  • Generations: VampPrior generations were sharper and more realistic than vanilla-VAE images, while convolutional variants captured more detail and data variation.The comparison covers HVAE, convHVAE, and PixelHVAE models with L = 2.
  • Reconstructions: VampPrior reconstructions were sharper and more detailed, with OMNIGLOT examples recovering small circles that vanilla VAE often failed to reconstruct.Convolutional networks further improved tiny details, while PixelHVAE sometimes produced decoder-driven “fantasizing.”
  • Log-likelihoods: HVAE with VampPrior and its convolutional version showed fewer hard examples than standard VAE in test log-likelihood histograms, especially convHVAE.The histograms exhibit heavy tails indicating examples that are hard to represent.
Loading 1705.07120v5…