Variational Inference: A Review for Statisticians

David M. Blei; Alp Kucukelbir; Jon D. McAuliffe

Source-linked AI summary

Variational Inference: A Review for Statisticians

David M. Blei, Alp Kucukelbir, Jon D. McAuliffe

arXiv:1601.00670v9Jan 4, 2016stat.COcs.LGstat.ML

TL;DR

Bayesian statistics needs tractable approximations to difficult posterior densities. This paper reviews variational inference, which uses KL-optimized approximate densities, and concludes that it is fast and scalable but less well understood than MCMC.

Problem
Bayesian inference requires approximating posterior densities that are difficult to compute, motivating methods for tractable approximate inference.
Method
The paper reviews variational inference, which optimizes over approximate-density families to minimize KL divergence, including mean-field, exponential-family, mixture-model, and stochastic variants.
Results
Variational inference tends to be faster and easier to scale than MCMC, with CAVI reported as orders of magnitude faster in a Gaussian-mixture image study.
Takeaways & Limitations
Variational inference is suited to large datasets and rapid exploration of many models, including massive-data settings addressed through stochastic optimization.
Takeaways & Limitations
Beyond conditional conjugacy, the reviewed variational inference methods may not compute the ELBO expectations or coordinate updates required by their derivations.

Abstract

from arXiv · show

One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this paper, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find the member of that family which is close to the target. Closeness is measured by Kullback-Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this paper is to catalyze statistical research on this class of algorithms.

1 Introduction

Variational inference approximates difficult posterior and probability densities by optimizing within a chosen family of densities, offering a faster, more scalable alternative to MCMC. Its computational advantages come with less-understood accuracy and statistical properties, motivating further research.

Motivation: Modern Bayesian models often require approximate inference because their posterior densities are difficult to compute.Bayesian inference conditions on observed data to compute p(z|x), but complex models make this computation intractable or impractical.
Comparison with MCMC: Compared with MCMC, variational inference tends to be faster and easier to scale to large data, but it lacks guarantees of asymptotically exact samples.MCMC samples from a Markov chain targeting the posterior, whereas variational inference returns the result of optimization.
Comparison with MCMC: Variational inference is particularly useful when data sets are large, models are complex, or many models must be explored quickly.MCMC remains preferable for smaller, expensive data sets when more precise samples justify higher computational cost.
Method: Variational inference posits a family of densities and selects the member minimizing KL divergence to the exact posterior.The optimized density q*(·) becomes the posterior approximation, turning inference into an optimization problem.
Limitations and open problems: Variational inference generally underestimates posterior variance, while its relative accuracy compared with MCMC remains unknown.Empirical work suggests that this limitation does not necessarily reduce accuracy for every task, including posterior predictive densities.
Scope and research agenda: The review focuses on KL(q||p)-based variational inference and presents it as a general tool for approximating intractable densities, not only Bayesian posteriors.The authors aim to catalyze research on the statistical properties, accuracy, and model suitability of variational methods.

2 Variational inference

Variational inference approximates the posterior over latent variables by optimizing within a chosen family of densities, typically by maximizing the ELBO instead of directly computing an intractable KL divergence. Mean-field coordinate optimization is efficient but can underestimate uncertainty and converge to initialization-sensitive local optima.

Inference difficulty: Exact inference can be intractable because the evidence integral may require exponential time, including for Bayesian Gaussian mixtures.For the mixture model, numerical evaluation of the K-dimensional integral has time complexity O(Kn), while summing over cluster-assignment configurations remains exponential in K.
Variational inference: Variational inference chooses the member of a density family Q that is closest in KL divergence to the exact conditional p(z|x).The resulting q*(·) serves as an approximation whose complexity depends on the chosen family.
Variational inference: Maximizing the evidence lower bound (ELBO) is equivalent to minimizing KL divergence because the evidence is constant with respect to q(z).The ELBO balances expected log likelihood against divergence from the prior.
Mean-field limitations: Mean-field approximations decouple covariance structure and commonly under-represent the target’s marginal variances.Because the ELBO’s KL direction penalizes mass placed where the target has little mass more strongly than the reverse, matching target variances can be difficult.
Mean-field inference: Under the mean-field assumption, all latent variables are independent, enabling valid coordinate updates that remove each factor from its own expectation.CAVI repeatedly updates factors, goes uphill on the ELBO, and eventually finds a local optimum.
Optimization limitations: CAVI optimizes a generally non-convex ELBO and guarantees only a local optimum, making results potentially sensitive to initialization.Multiple symmetric posterior modes from label switching can make representing one mode sufficient for latent clustering or prediction.

3 A complete example: Bayesian mixture of Gaussians

The section works through coordinate-ascent variational inference for a Bayesian Gaussian mixture, deriving assignment and component updates from the ELBO. It then demonstrates the algorithm in simulation and image-clustering analyses.

Model: The model uses K Gaussian components with latent cluster assignments, component means, fixed prior variance σ2, unit observation variance, and a uniform component prior.Each assignment is represented by a K-dimensional indicator vector.
Variational approximation: The mean-field variational family contains categorical assignment parameters ϕi and Gaussian component parameters mk and s2k.The ELBO is formed from the joint density and this variational family, with expectations computable in closed form.
Coordinate-ascent updates: CAVI alternates updates for cluster assignments and mixture components, using the ELBO to track progress and assess convergence.Assignment updates depend on the variational component parameters, while component updates use assignment probabilities as weights.
Coordinate-ascent updates: The coordinate-optimal density for each mixture mean is Gaussian, with updates closely related to a weighted complete conditional.Each observation contributes according to its variational probability of belonging to component k.
Empirical demonstrations: The simulation’s ELBO develops elbows because it is nonconvex in the variational parameters, so CAVI iteratively reaches better plateaus.The approximate predictive-density plot is smoother than the ELBO plot.

4 Variational inference with exponential families

When complete conditionals belong to the exponential family, mean-field variational inference yields tractable CAVI updates whose factors share the same family structure. Conditionally conjugate models expose local and global updates, while stochastic variational inference combines natural gradients with stochastic optimization to scale beyond massive-data costs.

General exponential-family models: CAVI alternates coordinate updates that optimize the ELBO, with convergence assessed by computing the ELBO across iterations.The ELBO may be evaluated at each iteration or at a lag, up to constants independent of variational parameters.
General exponential-family models: Exponential-family complete conditionals make CAVI updates easier to derive, and each optimal variational factor shares its corresponding conditional’s family, dimension, base measure, and log normalizer.Each factor’s parameter is set to the expected parameter of its complete conditional.
Conditionally conjugate models: Conditionally conjugate models separate global variables governing the data from local variables tied to individual data contexts, enabling corresponding global and local variational parameters.The global variational parameter indexes the prior’s exponential-family density, while each local parameter indexes its local complete conditional’s density.
Stochastic variational inference: Natural-gradient optimization enables stochastic variational inference, but full-data gradients remain prohibitively expensive because they require summing over all data and optimizing local parameters for every point.In exponential families, the natural gradient uses the inverse covariance of the sufficient statistic, equivalently the inverse Fisher information matrix.
Stochastic variational inference: SVI uses an unbiased noisy natural gradient computed from one sampled data point and one set of optimized local parameters, and existing CAVI implementations can be immediately scaled into stochastic algorithms.The same idea extends to minibatches by sampling multiple data points and rescaling.

5 Discussion

The discussion presents variational inference as an optimization-based approximation whose complexity depends on the chosen density family, and highlights its broad applications and emerging theoretical guarantees. It also identifies limitations of mean-field and exponential-family methods and open directions involving alternative divergences, richer approximations, MCMC connections, and statistical understanding.

Core framework: Variational inference approximates p(z|x) by selecting the closest member of a posited density family under KL divergence.The optimization problem’s complexity is governed by the complexity of the approximating family.
Core framework: Mean-field VI uses fully factorized densities and is particularly amenable to coordinate-ascent optimization, including Bayesian Gaussian-mixture inference.Coordinate ascent iteratively optimizes each factor and closely connects to the classical Gibbs sampler.
Scalability: Stochastic variational inference scales mean-field variational inference to massive data.The stochastic method updates global variational parameters using a step-size sequence after computing scaled coordinate updates.
Applications: VI has been applied across computational biology, computer vision and robotics, and computational neuroscience to analyze genetic, image, and high-dimensional time-series data.Examples include genome-wide association studies, regulatory networks, image manifolds, video layers, and autoregressive processes.
Statistical theory: Asymptotic studies establish consistency or normality in several models, including parametric-rate consistency, robustness to misspecification, and an O(1/n) gap between VI and maximum-likelihood estimates.These results cover Bayesian linear models, Poisson mixed-effects models, broader Gaussian approximations, and Gaussian mixtures.
Limitations and future work: Open problems include black-box inference, alternative divergences, richer variational families, VI–MCMC interfaces, and a fuller understanding of VI’s statistical properties.Mean-field independence can limit expressiveness, worsen local-optimum issues, and underestimate posterior variances; non-exponential-family models may also prevent standard expectation and coordinate-update calculations.

A Bayesian Linear Regression with Automatic Relevance Determination

This section develops conditionally conjugate Bayesian linear regression with automatic relevance determination (ARD), which assigns coefficient-specific priors to shrink irrelevant predictors. Because the full posterior is unavailable in closed form, the section derives a coordinate-ascent variational inference algorithm with normal-gamma and Gamma factors.

Automatic relevance determination: ARD assigns each regression coefficient its own precision and latent relevance variable, encouraging small values that select relevant coefficients.The hyper-prior on α_d encourages small relevance-variable values, automatically shrinking irrelevant coefficients.
Model: The model uses a conditionally conjugate Bayesian linear-regression formulation with an ARD prior and precision-parameterized Gaussian distributions.The regression coefficients are represented by β ∈ R^D, with fixed hyper-parameters a_0, b_0, c_0, and d_0.
Variational inference: The posterior p(β,τ,α | y ; x) is not available in closed form, unlike the corresponding model without α.Without α, conjugacy between the normal-gamma distribution and a normal likelihood with unknown mean and precision yields a closed-form posterior.
CAVI updates: The derived CAVI algorithm factorizes the variational approximation and obtains q(β,τ) as a normal-gamma distribution.Its optimal form follows by combining the expected log prior over α with the log likelihood and the remaining model terms.
CAVI updates: The optimal variational distribution q(α) is Gamma, and the resulting variational quantities define the CAVI procedure and the ELBO.The paper notes that additional algebra computes the ELBO.

B Gaussian Mixture Model of Image Histograms

This section specifies a D-dimensional diagonal-covariance Gaussian mixture model for n observations with K components, latent assignments, mixing proportions, component means, and precisions. It defines the model’s distributions and priors and notes a Stan implementation that marginalizes discrete assignments.

Model specification: The model is a multivariate, D-dimensional Gaussian mixture with diagonal covariance, n observations, and K mixture components.Each observation x_i is D-dimensional.
Model specification: Cluster assignments are K-indicator latent variables governed by a mixing vector π on the K-simplex.The assignment variables are z_1:n, while π determines their distribution.
Model specification: Each mixture component has D-dimensional latent means µ_k and precisions τ_k.The model collects means and precisions across all K components.
Distributions and priors: The model combines a Gaussian likelihood, categorical assignment distribution, Dirichlet mixing prior, and normal-gamma prior for means and precisions.The mixing-prior hyperparameters are fixed, while the normal-gamma prior uses m0, b0, α0, and β0.
Implementation: Stan implements the image-histogram GMM by marginalizing assignment variables because Stan does not support discrete latent variables.Figure 8 presents the corresponding Stan code.

C Latent Dirichlet Allocation

Latent Dirichlet allocation is a conditionally conjugate mixed-membership topic model in which documents contain multiple topics, each represented as a distribution over vocabulary words. Because its posterior is analytically intractable, coordinate-ascent variational inference approximates the topic assignments, document proportions, and topics with conjugate distributions.

Model: LDA models documents as containing multiple topics, with each topic represented by a distribution over words in the vocabulary.It is a conditionally conjugate topic model.
Model: LDA specifies K topics over a vocabulary of size V and uses symmetric Dirichlet priors for topics and document-level topic proportions.The fixed prior parameters are η for topics and α for topic proportions.
Variational inference: The posterior is analytically intractable because introducing the topics β breaks the otherwise conjugate relationship between topic assignments and proportions.The topic assignments z and proportions θ remain conditionally conjugate to each other.
Coordinate-ascent variational inference: CAVI uses a mean-field variational family whose factors mirror LDA’s complete-conditionals: multinomial assignments, K-dimensional Dirichlet proportions, and V-dimensional Dirichlet topics.The variational parameters are φdn for assignments, γd for proportions, and λk for topics.
Coordinate-ascent variational inference: The LDA CAVI algorithm iterates updates for each word within each document until the ELBO converges.Updates for φ and γ depend on topic parameters λ, while the topic update depends on assignment parameters φdn.