PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference Suboptimality in Variational Autoencoders

Chris Cremer, Xuechen Li, David Duvenaud

arXiv:1801.03558v3cs.LGstat.ML

TL;DR

The paper asks whether VAE inference gaps primarily reflect limited posterior-family complexity or imperfect recognition networks. It decomposes these gaps and studies encoders, approximations, decoders, and optimization across datasets, finding that amortization is often the more prominent source and that expressive approximation parameters also aid inference generalization.

  • Problem

    The paper examines whether divergence from the true posterior is driven by limited variational-distribution capacity or recognition networks' inability to produce good parameters for each datapoint.

  • Method

    The study decomposes VAE inference suboptimality into approximation and amortization gaps and evaluates encoder, approximation, decoder, and optimization choices across three datasets.

  • Results

    Amortization is often the more prominent source of inference suboptimality, while the generator accommodates approximation choices and expressive parameters reduce amortization error.

  • Takeaways & Limitations

    Expressive approximations can support inference generalization, particularly when efficient test-time inference is required, rather than merely increasing posterior complexity.

  • Takeaways & Limitations

    The analysis depends on marginal log-likelihood estimators whose importance-weight variance can be unreliable when proposals miss important target modes.

Abstract

from arXiv · show

Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

1. Introduction

The paper analyzes VAE inference suboptimality by separating approximation limitations from amortized recognition-network limitations, then studies how approximation expressiveness affects learned models and inference generalization.

  • Inference suboptimality: The inference gap is decomposed into an approximation gap from variational-family mismatch and an amortization gap from shared, rather than datapoint-specific, optimization.The inference gap is the difference between marginal log-likelihood and the ELBO.
  • Motivation: Prior work improved generative modeling with more expressive approximate posteriors, motivating analysis of how approximation expressiveness relates to learned distributions.
  • Experimental scope: The study varies encoder, posterior approximation, decoder, and optimization choices across MNIST, Fashion-MNIST, and CIFAR-10.
  • Contributions: The contributions include analyzing both gap components, showing that the generator accommodates approximation choices, and linking expressive parameters to reduced amortization error.

2. Background

VAEs replace intractable marginal-likelihood integration with an inference network and optimize the ELBO. More expressive posterior families, including normalizing flows and auxiliary-variable models, improve flexibility while retaining tractable objectives under stated constraints.

  • Variational inference: VAEs introduce qφ(z|x) to approximate p(z|x) because marginal-likelihood computation is intractable, then optimize the ELBO over model and inference parameters.The ELBO is tight when q(z|x) equals p(z|x).
  • Approximate posteriors: The standard factorized Gaussian posterior is simple and efficient, while expressive approximations extend beyond that family.
  • Normalizing flows: Normalizing flows construct complex distributions by applying successive invertible transformations to a base random variable.Expectations under the transformed density remain tractable through LOTUS.
  • Normalizing flows: Flow stacking remains tractable because determinants decompose, while the Jacobian determinant must be easily computable.
  • Auxiliary variables: Auxiliary variables increase variational flexibility without changing the generative model, although the resulting bound is looser than the regular ELBO.The extra flexibility can nevertheless produce a higher lower bound.

3. Methods

The methods decompose VAE inference suboptimality into approximation and amortization gaps, then estimate these bounds using flexible posterior families and tight marginal-likelihood estimators. The analysis also validates AIS estimates and compares amortized with datapoint-optimized variational parameters.

  • Inference-gap decomposition: The inference gap is decomposed into approximation error from the variational family and amortization error from using recognition-network parameters.The decomposition compares the ELBO at the optimal variational distribution q* with the ELBO at the amortized distribution q.
  • Flexible approximate posteriors: Flow and auxiliary-flow transformations provide expressive approximate posterior families whose Jacobian determinant can be evaluated tractably.The auxiliary-flow family introduces an auxiliary variable and is trained with a lower bound that is explicitly noted to be looser.
  • Bound estimation: Marginal log-likelihood is estimated with IWAE and AIS, retaining the larger independently computed lower bound: log p̂(x) = max(L_AIS, L_IWAE).IWAE becomes tighter with multiple importance-weighted samples, while AIS anneals from a proposal distribution toward the true posterior.
  • Bound estimation: The standard ELBOs are computed with either amortized q or datapoint-specific optimal q*, using 5000 samples for VAE and IWAE estimates.For q*, variational parameters are optimized separately for every datapoint.
  • Estimator validation: The reliability of the marginal log-likelihood estimator is a soundness constraint because importance-weight variance can be misleading when proposals miss important modes.BDMC is used as an empirical validation procedure, but it assumes simulated data from the model roughly matches real data.
  • Estimator validation: BDMC validation finds AIS gaps within 0.1 nat for MNIST and Fashion-MNIST, and within 1 nat for 3-BIT CIFAR under the reported settings.These comparisons provide empirical support for how well the AIS lower bounds reflect marginal log-likelihood on real data.

4. Related Work

Earlier work improved VAE inference through local variational optimization, expressive posteriors, or inference networks that amortize computation across datasets. This paper focuses on analyzing the errors introduced by inference networks and relating those errors to approximation and amortization gaps.

  • Inference networks: Inference networks amortize the cost of inference over large datasets, replacing local variational-parameter optimization for each datapoint.The paper analyzes the error introduced by this scaling strategy.
  • Hybrid inference: Hybrid methods locally optimize approximate inference from inference-network initialization, targeting amortization-gap effects during early training.Related work reports improved training on high-dimensional, sparse data and interprets local optimization as reducing negative effects of random initialization.

5. Experimental Results

Experiments separate inference suboptimality into approximation and amortization gaps, finding amortization is often the larger contributor and that model capacity affects both inference quality and generalization.

  • 5.1. Intuition through Visualization: Visualization distinguishes factorization limits, recognition-network limitations, and flexible approximations across several posterior shapes.Factorized Gaussians struggle with dependent or multimodal posteriors, while limited-capacity recognition networks may poorly support the true posterior.
  • 5.2. Amortization vs Approximation Gap: Across MNIST, Fashion-MNIST, and CIFAR, amortization is often the more prominent cause of inference suboptimality, especially on harder datasets.On MNIST the two gaps are nearly equal; on Fashion-MNIST amortization is larger, and on CIFAR it is much more significant.
  • 5.2. Amortization vs Approximation Gap: Larger encoders reduce the inference gap mainly by reducing the amortization gap for MNIST and Fashion-MNIST.This experiment directly tests whether increasing encoder capacity improves amortized inference.
  • 5.3. Influence of Flows on the Amortization Gap: More expressive approximate distributions reduce approximation error and also substantially reduce amortization error.The parameters added for flexible approximations can increase encoder capacity, so improvements attributed to expressiveness may partly reflect reduced amortization error.
  • 5.4. Influence of Approximate Posterior on True Posterior: The generator accommodates the chosen approximation, making the true posterior better fit that approximation and reducing the approximation gap.Increasing decoder capacity likewise decreases the approximation gap, so a more flexible generator can make accurate inference possible with a less flexible approximation.
  • 5.5.1. ENCODER CAPACITY AND APPROXIMATION EXPRESSIVENESS: On held-out data, expressive approximations can generalize inference better than larger encoders, while larger decoders trade easier inference for greater overfitting.The Flow and Larger Encoder models achieve similar validation log ˆp(x), but the Larger Encoder has a lower L[q] bound because its encoder overfits.

F F G] and LVAE[q∗

Without entropy annealing, the true posterior is more Gaussian, suggesting it better matches the expressive approximation.

  • The true posterior is more Gaussian when entropy annealing is not used.
  • Entropy annealing helps the true posterior better utilize the flexibility of the expressive approximation.

6. Conclusion

The paper finds that amortization can dominate inference suboptimality, while the generator adapts the true posterior to the approximation choice. Expressive approximation parameters also help generalize inference, and optimization techniques improve their use.

  • 6. Conclusion: Amortization can be a leading source of inference suboptimality.
  • 6. Conclusion: The generator can reduce the approximation gap by learning a true posterior that fits the chosen approximation.
  • 6. Conclusion: Parameters that increase approximation expressiveness can generalize inference rather than merely improve approximation complexity.
  • 6. Conclusion: Increasing encoder capacity reduces amortization error.
  • 6. Conclusion: Entropy annealing helps generative models better utilize flexible variational distributions.

Supplementary

The supplementary material provides experiment code, dataset preprocessing details, model architectures, optimization settings, and expressive-approximation components.

  • Datasets: The experiments use MNIST, Fashion-MNIST, and binarized CIFAR-10 datasets.
  • Datasets: MNIST and Fashion-MNIST images are 28x28 grayscale inputs rescaled to [0, 1] and modeled with Bernoulli likelihoods.
  • Architectures: MNIST and Fashion-MNIST models use 50 latent dimensions, ELU activations, two 200-unit encoder layers, and a reversed generator architecture.
  • Architectures: The large encoder setting increases hidden units from 200 to 500, while warm-up uses a linear schedule over 400 epochs.
  • Expressive approximations: Auxiliary-variable distributions and flow functions use multilayer perceptrons with ELU activations and 100-unit hidden layers.
  • Datasets: 3-BIT CIFAR binarizes 32 × 32 RGB CIFAR-10 pixels and requires 3 bits per pixel.
  • Optimization: Training uses Adam, learning rate 10^-3, batch size 100, and IWAE-based early stopping on held-out examples.

6.2. Inference Generalization

The study examines expressive flow and auxiliary-variable approximations, including their role in inference generalization. The auxiliary-variable model has a larger approximation gap than the flow model, likely because of its reverse-model KL term.

  • 6.2. Inference Generalization: The auxiliary-variable approximation uses flow transformations plus an auxiliary variable and reverse model r(v|z).
  • 6.2. Inference Generalization: The auxiliary-variable model has a larger approximation gap than the Flow model.The increase is likely due to the KL(q(v|z, x)∥r(v|x, z)) term in the auxiliary-variable lower bound.
  • 6.2. Inference Generalization: The auxiliary-variable model's gap is visualized across epochs in Figure 4.
  • 6.2. Inference Generalization: The fixed-decoder experiment compares factorized Gaussian and flow encoders while holding the true posterior constant.
  • 6.2. Inference Generalization: A small encoder with no hidden layers is used to create a large amortization gap.

6.4. Computation of the Determinant for Flow

This section describes determinant computation for flow transformations and annealed importance sampling, including its transition sequence, intermediate distributions, and likelihood-estimation role.

  • The overall flow mapping is composed of two shear mappings whose diagonal Jacobians make the composed determinant easy to compute.
  • AIS estimates a lower bound to marginal log-likelihood by transforming proposal samples through reversible transitions toward the desired distribution.
  • AIS initializes z1 from f1(z) with weight w1 = 1, then samples successive states and updates the weight during annealing.
  • The final weight satisfies E[wT] = ZT/Z1, relating the normalizing constants of the target and proposal distributions for marginal-likelihood estimation.
  • Intermediate AIS distributions are typically geometric averages controlled by a monotonically increasing schedule with β1 = 0 and βT = 1.
  • AIS has been used for model evaluation in deep belief networks and decoder-based models.

6.6. Extra MNIST Inference Gaps

On a 1000-datapoint MNIST training set, a factorized Gaussian achieves a very small inference gap, with AIS and IWAE bounds overlapping; Figure 5 compares FFG and Flow training curves.

  • 1000 randomly chosen MNIST training datapoints were used to test whether a small inference gap is achievable with a limited approximation.
  • The factorized Gaussian produced a very small inference gap, with the AIS and IWAE bounds overlapping.
  • On the 1000-datapoint dataset, FFG and Flow models showed little difference because inference was nearly equivalent.
  • AIS provided the tightest lower bound and was independent of encoder overfitting in the reported MNIST training curves.
Loading 1801.03558v3…