PaperlayerView exact revision on arXiv

Source-linked AI summary

Importance Weighted Autoencoders

Yuri Burda, Roger Grosse, Ruslan Salakhutdinov

arXiv:1509.00519v4cs.LGstat.ML

TL;DR

VAEs rely on restrictive posterior assumptions that can produce overly simplified representations and limit use of modeling capacity. The IWAE retains the VAE architecture but replaces its objective with a tighter importance-weighted bound using multiple posterior samples. Empirically, IWAEs learn richer latent representations and achieve better generative performance than equivalent VAEs.

  • Problem

    VAEs assume approximately factorial posteriors whose parameters are predictable from observations, potentially constraining posterior flexibility and representation richness.

  • Method

    The IWAE uses the VAE architecture with a tighter importance-weighted log-likelihood lower bound and multiple approximate posterior samples.

  • Results

    IWAEs learn richer latent representations and achieve better generative performance than VAEs with equivalent architectures and training time.

  • Takeaways & Limitations

    Importance weighting may improve the flexibility of other generative models currently trained with the VAE objective.

  • Takeaways & Limitations

    For k > 1, no analogous lower-variance update trick applies, so IWAE updates may have higher variance in principle.

Abstract

from arXiv · show

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

1 INTRODUCITON

Deep generative models use recognition networks to approximate posterior inference, but VAEs impose strong posterior assumptions that can limit expressive power. The IWAE keeps the VAE architecture while using importance weighting and multiple samples to obtain a tighter bound and richer representations.

  • Recognition networks approximate posterior distributions over latent variables given observations more quickly than generic inference methods such as MCMC.
  • VAEs pair top-down generative and bottom-up recognition networks, jointly maximizing a variational lower bound on data log-likelihood.
  • VAEs typically assume approximately factorial posteriors whose parameters are predicted from observations through nonlinear regression, constraining model expressive power.
  • The IWAE shares the VAE architecture but uses a tighter log-likelihood lower bound derived from importance weighting.
  • Multiple approximate posterior samples give the IWAE flexibility to model posteriors that do not fit VAE assumptions, producing richer representations and higher density-estimation log-likelihoods.

2 BACKGROUND

The VAE models latent variables through tractable hierarchical generative and recognition distributions and trains them with a variational lower bound. Reparameterization expresses recognition samples as deterministic functions of fixed-noise variables, enabling backpropagation and Monte Carlo gradient estimates.

  • The VAE defines a generative process by ancestral sampling through a cascade of stochastic hidden layers, with tractable sampling and probability evaluation at each conditional.
  • The model uses stochastic hidden units h = {h1, . . . , hL}, while deterministic layers are omitted from the layer-count notation.
  • The prior and conditional latent distributions are Gaussian with diagonal covariance, while the observation model is Gaussian for real-valued data and Bernoulli for binary data.
  • The VAE maximizes a variational lower bound that trades off data log-likelihood against KL divergence from the true posterior.
  • Reparameterization makes recognition samples deterministic functions of inputs and fixed-distribution auxiliary variables, allowing gradients to use standard backpropagation.
  • A Monte Carlo estimator approximates the expectation using k samples, with importance weights w(x, h, θ) = p(x, h|θ)/q(h|x, θ).

3 IMPORTANCE WEIGHTED AUTOENCODER

IWAE retains the VAE architecture but replaces its objective with a tighter importance-weighted lower bound. Multiple recognition-network samples improve posterior flexibility, while the bound becomes tighter with more samples and approaches the true log-likelihood under a boundedness condition.

  • Motivation: The VAE objective can be too strict because it penalizes samples that fail to explain observations, requiring approximately factorial and predictable posteriors.This motivates allowing only a fraction of recognition samples to occupy high-posterior-probability regions.
  • Objective: IWAE uses the VAE architecture but trains both networks with a different lower bound based on k-sample importance weighting.The importance weights are formed as wi = p(x, hi)/q(hi|x) from independently sampled latent states.
  • Objective: The bound is a lower bound on marginal log-likelihood because the average importance weights unbiasedly estimate p(x), and Jensen’s inequality applies.The expectations are taken with respect to q(h|x).
  • Bound properties: k = 1 recovers the standard VAE objective, while increasing k can only improve bound tightness.If p(h, x)/q(h|x) is bounded, Lk approaches log p(x) as k increases.
  • Training: Multiple approximate posterior samples give IWAE more flexibility to model posteriors that violate VAE assumptions.The resulting update averages gradient contributions using normalized importance weights; the reparameterization trick supplies an unbiased gradient estimate.
  • Training: Basic IWAE computation scales linearly with k because forward and backward passes are performed independently for each sample.A stochastic alternative uses k forward passes and one backward pass per example.

4 RELATED WORK

Related work addresses deep generative modeling through Boltzmann and belief-network models, recognition networks, alternative objectives, and importance sampling. IWAE is positioned alongside reweighted wake-sleep and other methods for improving posterior inference flexibility.

  • Deep generative models: Boltzmann models offer tractable conditionals, but sampling and partition-function computation remain major obstacles.
  • Recognition networks: Wake-sleep trains generative and recognition networks with different objectives, whereas VAEs jointly use a variational objective and reparameterization to reduce update variance.
  • Alternative objectives: Generative adversarial networks use generative and recognition networks that act in opposition rather than optimizing the variational objective.
  • Importance weighting: Importance-sampling approaches include inference from the prior, graphical-model algorithms, reweighted wake-sleep, and IWAE.IWAE differs from reweighted wake-sleep by interpreting its gradient as an unbiased estimate of ∇θLk and training both networks with a single objective.
  • Posterior inference: Normalizing flows and Hamiltonian variational approximations improve posterior flexibility through more sophisticated inference algorithms than importance sampling.
  • Prior work: The authors report that an importance-weighted lower bound for training VAEs was independently explored by Laurent Dinh and Vincent Dumoulin.

5 EXPERIMENTAL RESULTS

Experiments on MNIST and Omniglot compare VAEs and IWAEs using held-out log-likelihood and active latent dimensions. Increasing the number of importance-weighted samples substantially benefits IWAE performance and yields richer latent representations, while the VAE benefits only slightly.

  • Evaluation setup: The experiments evaluate VAE and IWAE generative performance on binarized 28 × 28 MNIST and Omniglot images.The datasets use standard training and test splits.
  • Density estimation: IWAE log-likelihood improved considerably with multiple samples on both datasets, whereas increasing k helped VAE performance only slightly.The two algorithms are identical at k = 1, so their results should match up to random variability.
  • Density estimation: -82.90 nats is the MNIST log-likelihood achieved by the two-layer IWAE with k = 50.This compares with approximately -84.55 nats for deep belief networks and -84.13 nats for deep autoregressive networks, while spatially structured models reached -80.97.
  • Density estimation: -103.38 nats is the best Omniglot IWAE log-likelihood, slightly worse than the -100.46 nats reported for a 500-hidden-unit RBM.Reported values are stochastic lower bounds estimated using L5000 on the test set.
  • Latent space representation: Inactive latent dimensions were negligible to predictions: removing them changed test log-likelihood by less than 0.06 nats in all cases.Activity was defined using a covariance-based statistic with threshold 10^-2.
  • Latent space representation: For k > 1, IWAEs learned more active latent dimensions than VAEs, while the two-layer models used fewer than 10 active dimensions in their second layer.The authors speculate that more active dimensions reflect richer latent representations because they coincided with higher log-likelihood values.
  • Latent space representation: Training with the IWAE objective increased active dimensions and test log-likelihood, whereas training with the VAE objective had the opposite effect.This strongly suggests that the objective functions, rather than only optimization issues, drive latent-dimension inactivation.

6 CONCLUSION

The paper presents IWAE as a VAE variant trained with a tighter importance-weighted log-likelihood lower bound. Empirically, IWAEs learn richer latent representations and achieve better generative performance than equivalent VAEs trained for the same time.

  • IWAE is a VAE variant trained by maximizing a tighter log-likelihood lower bound derived from importance weighting.
  • IWAEs learn richer latent representations and achieve better generative performance than VAEs with equivalent architectures and training time.The paper suggests the method may improve other generative models trained with the VAE objective.

APPENDIX A

The appendix establishes properties of the importance-weighted lower bound Lk. It shows that the bound is valid and converges to the true log-likelihood as the number of samples increases under a bounded importance-ratio condition.

  • Jensen’s inequality is used to establish the log-likelihood lower-bound properties of Lk.
  • For a uniformly sampled subset of distinct indices, an averaging observation and Jensen’s inequality support the bound’s intermediate properties.
  • If p(h, x)/q(h|x) is bounded, the Monte Carlo estimator Mk converges almost surely to p(x) under the strong law of large numbers.
  • As k →∞, Lk = E log[Mk] converges to log p(x).

APPENDIX B

The appendix analyzes the log of a positive unbiased estimator and bounds its mean absolute deviation. In IWAE, the relevant bias is the gap between Lk and log p(x).

  • Taking the logarithm of a positive unbiased estimator produces a downward-biased estimator of the logarithm of the target quantity.The bias is defined as δ = log Z − E[log Ẑ].
  • Markov’s inequality bounds how much log Ẑ can overestimate log Z, supporting the subsequent mean-absolute-deviation analysis.
  • The mean absolute deviation is bounded by 2 + 2δ, where δ is the gap between the IWAE bound Lk and log p(x).

NETWORK ARCHITECTURES

The experiments use specified network architectures, including a VAE with one stochastic layer and comparisons of VAE and IWAE posterior behavior.

  • The section summarizes the network architectures used in the experiments.
  • The displayed architecture includes two 200d layers with nonlinear transformations leading to a Bernoulli visible distribution.
  • Latent activity was determined using a threshold of 10^-2.
  • For two latent dimensions, heat maps compare true and approximate posteriors for VAE and IWAE models with k = 5 and k = 50.The IWAE models prefer less regular posteriors and more spread-out posterior predictions.

RESULTS FOR A FIXED MNIST BINARIZATION

On fixed-binarization MNIST, the relative merits of VAEs and IWAEs remain unchanged, while IWAE performance improves more with increasing sample count and deeper models perform better.

  • The relative merits of VAEs and IWAEs remain unchanged on the fixed-binarization MNIST setup.The experiments use 50000 training examples and 10000 test examples.
  • The fixed-binarization setup exhibited significantly larger amounts of overfitting.
  • IWAE generative performance improved with increasing k, whereas VAE performance benefitted only slightly.
  • Two-layer models achieved better generative performance than one-layer models.

SAMPLES

The samples table compares random samples from VAE and IWAE models across one- and two-stochastic-layer architectures.

  • The table places VAE samples in the left column and IWAE samples with k = 50 in the right column.
  • The first row shows models with one stochastic layer, while the second row shows models with two stochastic layers.
  • Samples are represented as the means of the corresponding Bernoulli distributions.
Loading 1509.00519v4…