Source-linked AI summary
Operator Variational Inference
Rajesh Ranganath, Jaan Altosaar, Dustin Tran, David M. Blei
TL;DR
Classical KL-based variational inference can produce posterior approximations with undesirable statistical properties. This paper develops operator objectives and a black-box optimizer, OPVI, including a Langevin-Stein objective. That objective supports data subsampling and variational programs, and the approach is demonstrated on mixture and image models.
Problem
KL-based variational inference can yield posterior approximations with undesirable statistical properties, including underestimated variance and degenerate latent-variable configurations.
Source: §1 · p. 1Method
The paper designs operator variational objectives and develops OPVI, a black-box algorithm for optimizing them, with the Langevin-Stein operator as an example.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Operator objectives make computational and statistical tradeoffs explicit and characterize which objectives support scalable inference or flexible posterior families.
Source: §1 · pp. 1–2The posterior is the unique zero-objective solution only when it is the sole distribution in the variational family with zero expectation; otherwise optimization yields a member of the zero-expectation equivalence class.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling---allowing inference to scale to massive data---as well as objectives that admit variational programs---a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.
1 Introduction
Variational inference casts Bayesian inference as optimization [12], but the classical KL objective can sacrifice statistical quality for computational convenience. The paper develops operator-based objectives and OPVI to make computational and statistical tradeoffs explicit.
1 Introduction
- KL-based inference can underestimate posterior variance or eliminate latent-variable configurations, pathologies traced to the objective rather than only the approximating family.The KL penalty becomes infinite when the approximating distribution has support beyond the posterior.Source: §1 · p. 1
- Operators map functions to functions and provide a framework for designing variational objectives that explicitly trade computational properties against statistical properties.The paper gives the Langevin-Stein operator as one example.Source: §1 · p. 1
- OPVI is a black-box algorithm for optimizing operator objectives; its Langevin-Stein objective admits data subsampling and variational programs without tractable densities.The paper studies this objective on a mixture model and a generative model of images.Source: §1 · p. 2
- The paper makes explicit requirements for constructing variational objectives, including properties that support scalable and flexible inference.It also presents new objectives, including some not previously studied.Source: §1 · p. 2
2 Operator Variational Objectives
The section defines operator variational objectives through their components and the conditions that make posterior inference meaningful and tractable. It relates these objectives to familiar divergences, develops a Langevin-Stein example, and places KL within the same framework.
2.1 Variational Objectives
- Useful variational objectives must yield a good posterior approximation while remaining tractable when the posterior is known only up to a constant.The evidence lower bound is a classic construction; maximizing it is equivalent to minimizing KL to the posterior.Source: §2.1 · p. 2
2.2 Operator Variational Objectives
- An operator variational objective combines an operator, a family of test functions, and a distance-to-zero function in an expectation designed to signal posterior closeness.The objective is non-negative, and inference minimizes it over the variational family.Source: §2.2 · pp. 2–3
- The framework requires closeness, so the posterior has zero objective and is the unique minimizer when no other distribution in the variational family has zero expectation; it also requires tractability without the posterior normalizer.Tractability requires expressing the operator through the joint distribution and making the test-function supremum feasible.Source: §2.2 · p. 3
2.3 Understanding Operator Variational Objectives
- Operators with positive expectations include tractable forms of KL, Rényi’s α-divergence, and χ-divergence, while signed expectations connect operator objectives to generalized Stein divergences.The section frames the objective as matching expectations under the approximation and posterior.Source: §2.3 · p. 3, §2.2 · p. 3
2.4 Langevin-Stein Operator Variational Objective
- The Langevin-Stein operator is tractable from the joint distribution, has the posterior as its unique minimizer, and its objective does not share KL’s variance-underestimation pathology.The paper also notes that operator objectives are not restricted to continuous variables, describing a discrete-variable operator in Appendix C.Source: §2.4 · p. 4
2.5 The KL Divergence as an Operator Variational Objective
- Classical KL variational inference fits the operator-objective family, using an operator that trivially maps all test functions to the same function.Its tractability follows from expressing the log posterior as the joint log density minus the log evidence.Source: §2.5 · p. 4
3 Operator Variational Inference
Operator variational inference (OPVI) optimizes objectives built from operators, using a black-box algorithm to make statistical and computational tradeoffs explicit. The Langevin–Stein operator supports data subsampling and variational programs, which can approximate continuous distributions without tractable densities under stated conditions.
3 Operator Variational Inference
- OPVI uses Monte Carlo estimates and stochastic optimization to minimize over the approximating family while maximizing over the test-function class.The algorithm is black box when the operator does not require model-specific computation.Source: §3 · pp. 4–5
3.1 Data Subsampling and opvi
- Both Langevin–Stein and KL operators support data subsampling because they are linear in the log density, allowing operator estimates to decompose across data.This characterization relies on evaluating the operator and its gradient at latent-variable realizations.Source: §3.1 · p. 6
3.2 Variational Programs
- Variational programs are sampleable approximating families that may lack analytically tractable densities, usable with operators such as Langevin–Stein that do not require q’s density.Operators that require q’s density, such as KL, can limit construction of these richer approximations.Source: §3.2 · p. 6
- Under the theorem’s conditions, variational programs with an appropriate q-independent operator can converge in distribution to a posterior with a continuous quantile function.The result assumes finitely many latent variables, a unique objective root at the posterior, and a function class able to approximate continuous functions.Source: §3.2 · p. 6
4 Empirical Study
In binarized-MNIST logistic factor analysis, Langevin–Stein with a variational program performs best on completed-image log-likelihood, while the same approximation with Langevin–Stein performs worse than KL. In a two-mode mixture, the variational program captures the posterior where Gaussian approximations choose one mode.
4.1 Mixture of Gaussians
- In the two-Gaussian mixture, KL and Langevin–Stein with Gaussian approximations choose a single mode, while Langevin–Stein with a variational program captures the posterior.The program captures different distributions on the positive and negative halves of the real line.Source: §4.1 · p. 7, Figure 1 · p. 7
- The mixture example highlights a tradeoff: the Langevin–Stein objective admits a variational program but is more difficult to optimize than KL.Source: §4.1 · p. 7
4.2 Logistic Factor Analysis
- The image experiment fixes model parameters learned with KL and compares KL and Langevin–Stein using a fully factorized Gaussian or a neural-network variational program.The logistic factor analysis model is applied to binarized MNIST with latent dimensionality 10.Source: §4.2 · p. 7
- Langevin–Stein with a variational program performs best on completed-image log-likelihood among the compared methods on 100 binarized-MNIST test images.The test images had half their pixels removed at random; the variational program performs better without directly optimizing for likelihoods.Source: §4.2 · p. 8, Table 1 · p. 8
5 Summary
The paper presents operator variational objectives and a black-box algorithm for optimizing them, including objectives that support data subsampling and flexible approximating families. It demonstrates the approach on mixture and image factor models.
5 Summary
- Operator objectives combine an operator, a test-function family, and a distance function to provide a broad, tractable class of posterior-approximation objectives.The paper relates them to divergences including KL and develops a Langevin–Stein objective.Source: §5 · p. 8
- The paper identifies future directions including new variational objectives and learning model parameters with operators or operator variational objectives.Source: §5 · p. 8
A Technical Conditions for Langevin-Stein Operators
The technical conditions establish when the Langevin–Stein operator has expectation zero, using integration by parts for bounded supports and limits for unbounded supports.
A Technical Conditions for Langevin-Stein Operators
- For bounded open supports, the Langevin–Stein expectation is zero under a sufficient boundary condition: either p vanishes at the boundary or the vector field is zero there.The derivation uses integration by parts.Source: A Technical Conditions for… · p. 10
- For unbounded supports, the result extends through increasing bounded sets when absolute integrability holds, with bounded log-density gradients, vector fields, and their derivatives as sufficient conditions.The extension uses dominated convergence.Source: A Technical Conditions for… · p. 10
B Characterizing the zeros of the Langevin-Stein Operators
The section characterizes when the Langevin–Stein objective is zero: under the stated construction, zero implies that q and the posterior agree in distribution. The proof uses functions that test equality on Borel sets.
B Characterizing the zeros of the Langevin-Stein Operators
- If the Langevin–Stein objective is zero, q is equivalent to the posterior in distribution.The proof shows that the functions used to test equality on Borel sets lie in the span of the Langevin–Stein operator.Source: B Characterizing the zeros of the… · p. 10
- The proof obtains the required test functions by solving a differential equation and substituting the solution into the operator objective.It reduces the construction to one coordinate by setting the other functions to zero.Source: B Characterizing the zeros of the… · pp. 10–11
- For every Borel set A, the constructed functions test equality of q and posterior probabilities on A.The resulting induced distance captures total variation.Source: B Characterizing the zeros of the… · pp. 10–11
C Operators for Discrete Variables
The section introduces a discrete analogue of the Langevin–Stein operator for posterior distributions over discrete variables. Its expectation is zero under the posterior, and the construction extends to multiple dimensions.
C Operators for Discrete Variables
- The construction addresses the limitation that some Stein operators apply only to continuous latent variables.The section presents an operator based on a discrete analogue to the Langevin–Stein operator.Source: C Operators for Discrete Variables · p. 11
- The discrete operator replaces the Langevin–Stein differential with a discrete difference and extends to multiple dimensions by ordered indexing.Binary numbers of length n can index n binary latent variables.Source: C Operators for Discrete Variables · p. 11
D Proof of Universal Representations
The section describes representing posterior draws as a transformation of standard normal draws. With a sufficiently expressive function class, the operator objective can approach zero and the resulting variational program converges in distribution to the posterior under stated conditions.
D Proof of Universal Representations
- Transforming standard normal draws through Φ and the posterior inverse CDF yields samples distributed as the posterior.Φ maps the standard normal draw to a uniform random variable before the posterior inverse CDF is applied.Source: D Proof of Universal Representations · p. 11
- A sufficiently rich parameterization of R can approximate the transformation that produces exact posterior draws.The section notes that universal approximators such as neural networks can serve as the function class.Source: D Proof of Universal Representations · p. 11
- If p is the unique root and the equality-in-distribution conditions hold, the variational program converges in distribution to the posterior as the objective goes to zero.A sequence of parameters can drive the operator variational objective to zero without restricting the function class to local interpolation.Source: D Proof of Universal Representations · p. 11