PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Implicit Variational Inference

Mingzhang Yin, Mingyuan Zhou

arXiv:1805.11183v1stat.MLcs.LGstat.COstat.ME

TL;DR

Variational inference can underrepresent complex posteriors, especially when its variational family is restricted and mean-field assumptions omit dependencies. SIVI expands the family with a flexible reparameterizable mixing distribution and optimizes an asymptotically exact surrogate ELBO, closely approaching MCMC accuracy across varied Bayesian tasks.

  • Problem

    Variational inference can underestimate posterior variance because restricted variational families fail to capture posterior complexity and dependencies.

  • Method

    SIVI mixes an analytic conditional variational distribution with a flexible, reparameterizable distribution over its parameters and optimizes an asymptotically exact surrogate ELBO.

  • Results

    SIVI closely matches MCMC in posterior uncertainty estimation across a variety of Bayesian inference tasks, including negative-binomial posterior samples with KS distances 0.0185 for r and 0.0200 for p.

  • Takeaways & Limitations

    SIVI provides a flexible variational family that can capture skewness, kurtosis, multimodality, and dependencies while remaining amenable to stochastic-gradient optimization.

  • Takeaways & Limitations

    Direct optimization of the lower-bound objective can cause the mixing distribution to collapse to a point mass, reducing SIVI to vanilla VI without early stopping.

Abstract

from arXiv · show

Semi-implicit variational inference (SIVI) is introduced to expand the commonly used analytic variational distribution family, by mixing the variational parameter with a flexible distribution. This mixing distribution can assume any density function, explicit or not, as long as independent random samples can be generated via reparameterization. Not only does SIVI expand the variational family to incorporate highly flexible variational distributions, including implicit ones that have no analytic density functions, but also sandwiches the evidence lower bound (ELBO) between a lower bound and an upper bound, and further derives an asymptotically exact surrogate ELBO that is amenable to optimization via stochastic gradient ascent. With a substantially expanded variational family and a novel optimization algorithm, SIVI is shown to closely match the accuracy of MCMC in inferring the posterior in a variety of Bayesian inference tasks.

1. Introduction

SIVI addresses variational inference’s limited representation power by mixing an analytic conditional distribution with a flexible, potentially implicit distribution. It also provides bounds and an optimizable surrogate ELBO for inference with this expanded family.

  • 1. Introduction: VI commonly restricts Q through mean-field factorization and conjugate exponential-family choices, simplifying optimization but limiting posterior approximation.These choices enable closed-form coordinate-ascent updates, but constrain the variational family.
  • 1. Introduction: VI can underestimate posterior variance because of variational-family mismatch, asymmetric KL divergence, and mean-field’s omission of dependencies.The factorized assumption can further amplify variance underestimation by ignoring dependencies among components.
  • 1. Introduction: Implicit variational models increase flexibility but make ELBO evaluation difficult because their intractable densities prevent direct log-density-ratio computation.Density-ratio estimation is therefore often used, adding optimization complexity.
  • 1. Introduction: SIVI expands Q by placing a mixing distribution over the parameters of an analytic conditional distribution in a semi-implicit hierarchical construction.The conditional distribution must have an analytic PDF, while the mixing distribution need not.
  • 1. Introduction: SIVI uses a reparameterizable implicit mixing distribution and derives ELBO bounds plus an asymptotically exact surrogate that supports stochastic gradient ascent.Its contributions target flexible variational distributions while avoiding direct density-ratio estimation.

2. Semi-Implicit Variational Inference

SIVI treats the conditional variational parameter as random, producing a hierarchical family whose mixing distribution may be implicit while the conditional remains explicit. This expands the family, restores dependencies under MFVI, and enables optimization through an asymptotically exact surrogate ELBO.

  • 2. Semi-Implicit Variational Inference: SIVI samples the variational parameter ψ from q(ψ), rather than treating ψ as a fixed parameter as in vanilla VI.When q(ψ) degenerates to a point mass, SIVI reduces to vanilla VI.
  • 2. Semi-Implicit Variational Inference: The semi-implicit hierarchy uses an explicit conditional q(z | ψ) and a potentially implicit mixing distribution qφ(ψ), whose marginal hφ(z) is often implicit.This conditional–mixing distinction gives SIVI its name.
  • 2. Semi-Implicit Variational Inference: The conditional q(z | ψ) must be explicit and either reparameterizable or analytically usable in the ELBO, whereas qφ(ψ) must only be reparameterizable.The mixing distribution can be generated by transforming random noise through a deep neural network, even when its PDF is not pointwise evaluable.
  • 2. Semi-Implicit Variational Inference: SIVI strictly expands the original variational family when its mixing distribution can degenerate to a point mass at arbitrary ψ0.The original family is then contained in the semi-implicit family.
  • 2. Semi-Implicit Variational Inference: For MFVI, dependent mixing parameters make the resulting latent variables conditionally independent but marginally dependent, restoring dependencies between factorization components.This expands MFVI’s representation beyond independently constrained factors.
  • 2. Semi-Implicit Variational Inference: Implicit distributions improve flexibility through stochastic sampling but complicate VI optimization when their PDFs are intractable and density ratios must be estimated.Deep neural-network transforms can capture dependencies among mixing-distribution parameters.

3. Optimization for SIVI

SIVI optimizes an implicit hierarchical variational family using surrogate ELBOs that avoid density-ratio estimation while controlling degeneracy and bounding the ELBO. The regularized lower and corrected upper bounds become asymptotically exact, supporting stochastic gradient optimization.

  • 3.2. Degeneracy and Regularization: Directly optimizing the lower bound can collapse the mixing distribution to a point mass, reducing SIVI to vanilla VI.
  • 3.2. Degeneracy and Regularization: The regularized lower bound avoids degeneracy and converges to the ELBO as K increases.It equals the original lower bound at K=0 and is asymptotically exact as K approaches infinity.
  • 3.3. Upper Bound of ELBO and Correction: SIVI’s ELBO can be bracketed by a lower bound and an upper bound, whose gap reflects divergences between the conditional and marginal variational distributions.
  • 3.3. Upper Bound of ELBO and Correction: The corrected upper bound decreases monotonically from above and converges to the ELBO as K approaches infinity.Its first value is the uncorrected upper bound, and each subsequent bound is no larger than the previous one.
  • 3.4. Optimization Algorithm: The asymptotically exact surrogate ELBO is optimized with stochastic gradient ascent using Monte Carlo samples from the hierarchy.The algorithm permits increasing K across iterations and uses samples for both mixing parameters and latent variables.
  • 3.4. Score Function Gradient in Conjugate Model: For conditionally conjugate models, SIVI can use an analytic variational distribution without requiring reparameterization, whereas non-conjugate non-reparameterizable cases remain future work.

4. Related Work

Related approaches improve variational expressiveness through dependencies, mixtures, copulas, hierarchical constructions, normalizing flows, or implicit distributions. SIVI differs by retaining an analytic conditional while allowing an implicit mixing distribution, avoiding direct density-ratio estimation.

  • Related variational methods: Prior work expands mean-field VI with dependencies, mixtures, copulas, nonconjugate methods, and hierarchical variational distributions.
  • Normalizing flows: Normalizing flows transform simple densities through invertible differentiable mappings with tractable Jacobians.
  • Implicit variational distributions: Implicit variational distributions are flexible but require density-ratio estimation, often implemented by training a classifier each iteration.
  • Hierarchical variational models: Unlike hierarchical variational models, SIVI requires an explicit conditional PDF but permits an implicit mixing distribution.

5. Experiments

Across synthetic and Bayesian inference tasks, SIVI uses a flexible implicit mixing layer to represent posterior features that simpler variational families miss. Its posterior estimates closely match MCMC in negative-binomial and logistic-regression experiments, while supporting non-reparameterizable conditional distributions.

  • 5.1. Expressiveness of SIVI: SIVI’s flexible implicit layer captures skewness, kurtosis, and multimodality even when its explicit conditional layer is deliberately misspecified.In synthetic targets, the inferred marginal variational distribution accurately reproduces these characteristics.
  • 5.2. Negative Binomial Model: With K = 1000, SIVI’s KS distances from MCMC are 0.0185 for r and 0.0200 for p, compared with 0.2695 and 0.2965 for MFVI.SIVI also captures the negative correlation between r and p that MFVI ignores; increasing K rapidly approaches MCMC accuracy.
  • 5.3. Non-reparameterizable Variational Distribution: SIVI accurately estimates posterior uncertainty in a conditionally conjugate model without requiring q(z | ψ) to be reparameterizable.Using conjugacy controls the variance of score-function gradient estimation, whereas MFVI ignores covariance structure and underestimates uncertainty.
  • 5.4. Bayesian Logistic Regression: SIVI generates 50,000 independent 22-dimensional β samples in 0.92 seconds after fitting the semi-implicit hierarchy.The samples are produced by feed-forward propagation through the inferred hierarchy.
  • 5.4. Bayesian Logistic Regression: On waveform Bayesian logistic regression, SIVI closely matches MCMC in predictive uncertainty, while MFVI underestimates it even with a full covariance matrix.The comparison uses predictive means and standard deviations computed from posterior samples.
  • 5.4. Bayesian Logistic Regression: SIVI better characterizes correlated, non-Gaussian posteriors than MFVI and SVGD, with only slight degradation when its explicit layer uses diagonal covariance.The reported comparison includes posterior marginals, pairwise structure, and correlation coefficients.

6. Conclusions

SIVI combines flexible implicit mixing with tractable optimization, and approaches MCMC accuracy across diverse inference tasks while supporting fast iid posterior sampling.

  • SIVI approaches MCMC accuracy in quantifying posterior uncertainty across a wide variety of inference tasks.It is not constrained by conjugacy and often runs faster than MCMC.
  • SIVI uses an asymptotically exact surrogate ELBO that supports optimization by gradient ascent without sacrificing the expressiveness of its semi-implicit distribution.
  • SIVI can generate iid posterior samples on the fly through the inferred stochastic variational inference network.

Supplementary Material

The supplementary material presents SIVI’s algorithmic inputs, sampling procedure, and stochastic optimization updates for learning the conditional variational parameter.

  • Algorithm 1 takes data, a joint likelihood, an explicit reparameterized conditional distribution, an implicit neural-network layer, and a randomness source as inputs.
  • At each iteration, SIVI samples mixing parameters through the implicit layer and draws conditional latent samples using reparameterized noise.
  • The algorithm accumulates subsampled stochastic estimates involving the model likelihood and prior for optimization.

A. Proofs

The proofs establish properties of SIVI’s bounds, Monte Carlo approximations, and optimization behavior, including the possibility that direct lower-bound maximization collapses the mixing distribution.

  • Directly maximizing SIVI’s lower ELBO bound may drive the mixing distribution toward degeneracy.
  • Equality holds only when the mixing distribution becomes a point mass, reducing SIVI to vanilla VI.
  • The proofs use laws of large numbers and limit arguments to establish convergence of Monte Carlo log-density estimates to the marginal variational density.
  • The appendix also develops the monotonic behavior of the finite-sample lower bound as the number of mixture samples increases.
  • A separate derivation expresses the gradient of the finite-sample correction term with respect to the implicit-layer parameters.

B. Bayesian Logistic Regression

The Bayesian logistic-regression experiments use datasets with differing training-size-to-feature ratios and derive likelihood expressions using Pólya–Gamma augmentation.

  • The experiments use waveform, spam, and nodal datasets with distinct sample sizes, feature dimensions, and training/testing splits.
  • The study expects posterior uncertainty to be larger when the training-set-size-to-feature-dimension ratio is smaller.
  • Each observation’s likelihood contribution is represented using an expectation over a Pólya–Gamma distribution, yielding an augmented likelihood.

B.1. Gibbs Sampling via Data Augmentation

The section specifies Gibbs-sampling and mean-field variational-inference updates for Bayesian logistic regression using Pólya–Gamma augmentation and conjugacy.

  • B.1. Gibbs Sampling via Data Augmentation: The augmented model uses Σ = (A + X′ΩX)−1 and µ = ΣX′(y −1/2) for its Gaussian conditional parameters.
  • B.1. Gibbs Sampling via Data Augmentation: Pólya–Gamma sampling is approximated by truncating an infinite gamma-variable sum to M = 5 terms with adjusted final-term moments.The adjustment matches the finite sum’s mean and variance to those of the infinite sum.
  • B.1. Gibbs Sampling via Data Augmentation: The fully factorized variational distribution exploits conjugacy to obtain closed-form coordinate-ascent update equations.
  • B.1. Gibbs Sampling via Data Augmentation: The variational expectations use E[ββ′] = µµ′ + Σ and E[ωi] = tanh(λi/2)/(2λi), matching the updates of Jaakkola and Jordan.

B.4. SIVI Configuration

The SIVI configuration combines neural implicit distributions with explicit Gaussian layers and evaluates performance against several inference methods using negative log evidence.

  • B.4. SIVI Configuration: SIVI uses an MLP with layer sizes [100, 200, 100], 50-dimensional Gaussian noise, K = 100, and J = 50 for the implicit component.The explicit layer is an MVN with mean variable ψ and covariance ξ, which may be diagonal or full.
  • B.4. SIVI Configuration: A separate experiment uses M = 3 stochastic hidden layers of sizes [150, 150, 150], injected-noise dimensions [150, 100, 50], and deterministic layers of size 500.It trains for 2000 epochs with mini-batches of 200 and a step-size schedule based on 0.001.
  • B.4. SIVI Configuration: The experiment uses binary pepper-and-salt noise for q_t(ϵ), while the auxiliary estimate h(z_s | x) uses M = 100.
  • B.4. SIVI Configuration: Performance is compared with other methods through negative log evidence, including MCMC, SIVI variants, mean-field variants, and SVGD.

D. Additional Figures

The additional figures visualize SIVI’s implicit distributions, convergence with increasing K, and posterior comparisons against MCMC and mean-field variational inference across datasets.

  • D. Additional Figures: Figure 8 visualizes MLP-based implicit distributions mixed with isotropic Gaussian or LogNormal distributions to approximate target distributions.
  • D. Additional Figures: SIVI’s inferred negative-binomial marginal posteriors become more accurate as K increases for both the dispersion parameter r and probability parameter p.
  • D. Additional Figures: Posterior comparisons include Bayesian logistic regression on waveform and marginal and pairwise joint posteriors for spam and nodal datasets.
  • D. Additional Figures: Figures compare MCMC, SIVI, and MFVI using boxplots, predictive sample means and standard deviations, and posterior probabilities for spam and nodal datasets.
  • D. Additional Figures: For spam and nodal, blue, green, and red denote MCMC, full-covariance SIVI, and full-covariance MFVI, respectively.
Loading 1805.11183v1…