Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

Tim Salimans; David A. Knowles

Source-linked AI summary

Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

Tim Salimans, David A. Knowles

arXiv:1206.6679v6Jun 28, 2012stat.COcs.CVstat.ML

TL;DR

Bayesian posterior distributions are often analytically intractable, while existing fixed-form approximations can impose restrictive requirements. The paper recasts variational optimization as stochastic linear regression, enabling broader posterior and approximation families; examples show competitive efficiency, while convergence depends on settings such as initialization, iteration count, and transformation choice.

Problem
Analytically intractable posteriors and restrictive analytic requirements limit the applicability of conventional fixed-form Variational Bayes.
Method
The method minimizes KL divergence through stochastic linear regression using sufficient statistics and the unnormalized log posterior, requiring a closed-form target density and samples from the approximation.
Results
The approach is highly competitive for problems with low-dimensional posterior factors or available log-posterior gradient and Hessian information.
Takeaways & Limitations
Exponential-family mixtures let computation–accuracy trade-offs be adjusted through the number of mixture components and can make approximations arbitrarily precise in principle.
Takeaways & Limitations
Finite-sample regression solutions depend on the transformation K(η), whose best general choice is not immediately obvious; poor initialization or few steps can also produce improper distributions.

Abstract

from arXiv · show

We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method can be used to approximate any posterior distribution, provided that it is given in closed form up to the proportionality constant. The approximation can be any distribution in the exponential family or any mixture of such distributions, which means that it can be made arbitrarily precise. Several examples illustrate the speed and accuracy of our approximation method in practice.

1 Introduction

Fixed-form Variational Bayes addresses intractable Bayesian posteriors by optimizing a convenient approximation, but conventional methods impose restrictive analytic and distributional requirements. The paper reframes this optimization as stochastic linear regression to broaden applicability while retaining computational efficiency.

Bayesian posteriors are often analytically intractable, requiring Monte Carlo methods or approximation to obtain moments and marginal distributions.
Fixed-form Variational Bayes minimizes the KL divergence between an exponential-family approximation and the intractable target posterior.
The paper shows that fixed-form Variational Bayes optimization is equivalent to linear regression using sufficient statistics as predictors and the unnormalized log posterior as the response.
The stochastic approximation algorithm requires no analytic integral calculations, needing only a closed-form target density up to proportionality and the ability to sample from the approximation.
Approximations may use exponential-family distributions or mixtures, which can in principle make them arbitrarily precise.
Examples show the algorithm is highly competitive with more specialized approaches despite its generality.

2 Fixed-form Variational Bayes

Fixed-form Variational Bayes restricts the ideal but usually intractable posterior optimization to a parametric family, commonly an exponential family. This improves tractability over unrestricted optimization but retains restrictive requirements for analytic expectations and entropy calculations.

The exact posterior minimizes the variational KL objective, but is usually unusable because its moments, marginals, and normalizing constant lack analytic expressions.
Factorized Variational Bayes can be efficient for conjugate exponential models but assumes posterior independence between parameter blocks.
Fixed-form Variational Bayes restricts the optimization to parametric approximations qη(x), often chosen from the exponential family.
Structured approximations can be more accurate than factorized ones, but typically require analytic evaluation of entropy, expected joint log density, and derivatives.
The paper uses the variational-to-linear-regression connection to develop an optimization algorithm intended to push back these limitations.

3 Variational Bayes as linear regression

The paper identifies the fixed-form variational optimum with a linear-regression solution. Sufficient statistics serve as predictors, the unnormalized log posterior as the response, and Monte Carlo estimates enable a stochastic algorithm without analytically computable expectations.

The adjusted approximation augments the sufficient-statistics vector with a constant and uses an intercept-like parameter to represent normalization or rescaling.
The fixed-point condition has the same form as the linear-regression maximum-likelihood estimator, with expectations over the approximating distribution.
The method assumes a nonsingular Fisher information matrix, as expected for an identifiable approximating exponential family.
The regression analogy maps design matrix X to sufficient statistics, dependent variable Y to the unnormalized log posterior, and coefficients β to natural parameters η̃.
A nonconstant base measure is handled by regressing the residual after subtracting the base-model log density.
Monte Carlo estimates replace analytically evaluated expectations, motivating the stochastic approximation algorithm.

4 A stochastic approximation algorithm

The paper reframes fixed-form variational optimization as stochastic linear regression and develops an iterative algorithm using Monte Carlo estimates of regression quantities. The method supports exact recovery in matching-form cases and practical convergence safeguards for difficult approximations.

Algorithm: Fixed-form variational optimization is solved by viewing the update as a fixed-point procedure based on stochastic regression estimates.The regression uses sufficient-statistic quantities to estimate C and g, then updates the variational parameters through C^-1g.
Algorithm: The stochastic update is, to first order, preconditioned stochastic gradient descent with an unbiased gradient estimate and a valid adaptive preconditioner.The preconditioner is independent of the iteration’s stochastic gradient approximation, so standard convergence results apply.
Exact convergence: When posterior and approximation share the same functional form, Algorithm 1 recovers the true posterior exactly in 2(k + 1) iterations under unique-support-point assumptions.After recovery, further iterations leave the approximation unchanged because the regression itself is noise free and only its support points are stochastic.
Efficiency: A fixed step size w = 1/N with averaging from halfway through optimization is used because it performs well for finite iteration budgets.The authors average statistics rather than parameters to address bias from estimating g and C with the same random numbers.
Limitations: Poor initialization or too few iterations can produce invalid distributions, while multimodal targets with unimodal approximations may yield only a local KL-divergence minimum.Increasing iterations, choosing a closer approximation family, and improving the initial guess can support convergence, but global optimality is not guaranteed for multimodal posteriors.

5 Marginal likelihood and approximation quality

The paper derives diagnostics for posterior-approximation quality and a correction to the variational lower bound for estimating marginal likelihood. These diagnostics connect approximation error to regression residuals, while the marginal-likelihood estimate relies on a residual normality assumption.

Purpose: The stochastic algorithm minimizes the KL divergence between qη(x) and the posterior, but marginal likelihood is additionally needed to assess approximation quality and perform model comparison.The paper therefore develops methods for approximating both the final KL divergence and the marginal likelihood.
Residual decomposition: At convergence, log p(x, y) decomposes into the variational approximation, an intercept and normalizer, and a regression residual r(x).The residual is defined as the error from regressing the unnormalized log posterior on the sufficient statistics of qη(x).
Marginal likelihood: The usual variational lower bound underestimates the true marginal likelihood unless the approximation is perfect, when the residual vanishes and the bound is tight.At convergence, E_qη[r(x)] = 0; Jensen’s inequality establishes the lower-bound relationship.
Limitations: The marginal-likelihood approximation cannot generally guarantee finite variance for its importance-sampling interpretation because the required likelihood-ratio boundedness may fail.This limitation motivates the residual-based approximation and its normality assumption.
Marginal likelihood: The estimated marginal likelihood adds a correction term to the variational lower bound based on the residual error in the posterior approximation.The correction uses an estimated residual variance and assumes r(x) is approximately normally distributed.
Approximation quality: The KL divergence is approximately half the mean squared regression error, linking variational approximation quality directly to the regression objective.After curvature scaling, the paper proposes R-squared as a more interpretable and cross-problem-comparable quality measure.
Empirical assessment: In an application, the KL-divergence approximation is accurate for reasonably good qη(x), and the proposed marginal-likelihood approximation is more accurate than the usual lower bound.The paper also reports R-squared values for several posterior approximations.

6 Extensions I: Improving algorithmic efficiency

The paper improves stochastic variational posterior approximation through factorized regressions, gradient and Hessian information, reparameterization, and minibatches. In the probit example, these extensions reduce computation and improve accuracy relative to basic stochastic approximation and VBEM.

6.0.2 Example: Binary probit regression: The probit experiment compares the proposed implementations with VBEM, VMP, and EP using simulated data with N = 100 and M = 5.Results are averaged over 500 datasets and multiple algorithm settings.
6.0.2 Example: Binary probit regression: The proposed approximation achieves lower RMSE than VBEM, while its log score is 0.193 versus −4.46, indicating substantially better variance estimates.The average R-squared is also reported as a measure of approximation quality.
6.1 Making use of factor structure: Factorized regressions reduce stochastic variance and the storage and inversion overhead of C, while parallel execution offers additional computational gains.The implementation replaces one high-dimensional regression with N regressions of dimension 3.
6.2 Using the gradient of the log posterior: Using gradients can make stochastic optimization more efficient, with one iteration of the gradient-based update providing about the same information as 2 × dim(x) basic iterations.This benefit depends on being able to obtain gradients analytically.
6.3–6.4 Hessian information and linear transformations: Hessian-based updates with linear transformations improve efficiency by replacing higher-dimensional natural-parameter statistics with mean-and-variance parameterization.The resulting algorithm is more efficient in both computation and storage.
6.3–6.5 Further efficiency improvements: In the probit example, the Hessian and linear-transformation implementation outperforms the gradient-and-factor-structure version and runs significantly faster than VBEM.Figure 2 evaluates RMSE against the number of likelihood evaluations for the different implementations.

7 Extensions II: Using mixtures of exponential family distributions

The paper extends fixed-form variational approximation beyond single exponential-family distributions by using conditionally tractable constructions whose marginals become mixtures. These extensions improve flexibility while retaining efficient stochastic fitting and yield highly accurate approximations in examples.

Conditional approximations: Conditionally exponential-family blocks allow joint approximations that capture dependencies while remaining straightforward to sample from.Each conditional uses preceding variables in its sufficient statistics.
Mixture flexibility: The resulting marginals are generally mixtures of exponential-family distributions, providing greater approximation flexibility than a single exponential-family form.The paper identifies this added flexibility as the main benefit of the construction.
Stochastic-volatility example: In the stochastic-volatility example, approximately 2,000 free parameters converged in 250 iterations using one (φ, σ2) sample per iteration.The single-threaded MATLAB implementation took half a second on a 3GHz processor.
Stochastic-volatility example: The stochastic-volatility posterior approximations for model parameters were nearly exact, and latent-volatility approximations were indistinguishable from the exact posterior.The comparison used a very long MCMC run as reference.
Mixture example: A single Gaussian achieved R-squared 0.82, whereas an eight-component Gaussian mixture achieved R-squared 0.997 for a skewed, heavy-tailed posterior.The mixture approximation improved as the number of Gaussian components increased.
Mixture example: For unimodal posteriors, interchangeable Gaussian-mixture components create multiple solutions, but the local optima found were equally good and sufficiently many components can approximate the posterior arbitrarily well.The claim concerns the unimodal case described in the example.

8 Conclusion and future work

The paper introduces a stochastic optimization scheme for fixed-form variational inference, extending applicability beyond conjugate models and supporting structured or mixture approximations. Its performance is competitive in settings with low-dimensional factorization or available derivatives, but it is not recommended for certain high-dimensional discrete distributions.

The scheme reinterprets fixed-form variational approximation as linear regression of target log density against sufficient statistics.The basic method requires only the target's unnormalized density, while gradient or Hessian information can also be incorporated.
Performance is highly competitive when the posterior decomposes into low-dimensional factors or gradient and Hessian information is available.These conditions correspond to the paper's demonstrated competitive cases.
The algorithm is not presently recommended for high-dimensional discrete distributions without factor structure.Extending the approach to this class is identified as future work.
The methodology supports hierarchical structured approximations and mixtures of exponential-family distributions.Varying the number of mixture components provides a computation–accuracy trade-off.
The approach removes the usual conditional-conjugacy restriction and can be combined with VBEM for models containing both non-conjugate and conjugate components.The paper notes that collapsed and uncollapsed representations can therefore be explored with the same method.

A Unnormalized to normalized optimality condition

The appendix converts the unnormalized variational condition into normalized form and expresses the optimal natural parameters through expectations and covariance statistics. At a solution, sufficient statistics are uncorrelated with the residual log-density term.

Normalization can replace the unnormalized approximation because the normalizing terms cancel.This yields the normalized q(x) representation without changing the optimality condition.
η0 equals E[Y] − E[T]η, equivalently E[log p(x,y) − log q(x)] − U(η).
η is obtained from the inverse covariance of sufficient statistics multiplied by their covariance with the dependent variable Y.The displayed expression uses E[T′T] − E[T′]E[T] and E[T′Y] − E[T′]E[Y].
At the KL-divergence solution, sufficient statistics are uncorrelated with log p(x,y) − log qη(x).This is the same fixed-point condition used in other stochastic-approximation variational Bayes applications.

B Derivation of Gaussian variational approximation

For Gaussian variational approximation, the paper specializes the regression formulation to a univariate variable with mean and variance parameters. The resulting statistics can be estimated using posterior gradients and Hessians to recover the Gaussian parameters.

The Gaussian approximation uses q(x)=N(m,V) with sufficient statistics T(x)=(x, −0.5x^2).The derivation assumes a univariate x; extension to multivariate x is conceptually straightforward but notationally more tedious.
The mean-variance parameters are φ=(m,V), while the natural parameters are η=(V^-1m,V^-1).
The derivation uses identities for φ1=m and φ2=V involving differentiable functions and the variance parameter.These identities supply the regression statistics for the Gaussian optimization problem.
The regression identities express the first natural parameter using the expected gradient of the log posterior and the Gaussian precision.Specifically, η1=Pm=g1+P^-1Eq[∇x log p(x,y)]+... as given by the paper's displayed identity.
The resulting Gaussian parameters satisfy P=−H and m=P^-1a+z.

C Connection to Efficient Importance Sampling

The paper connects fixed-form variational Bayes with Efficient Importance Sampling because both approximate an intractable posterior using a convenient distribution. It highlights a shared fixed-point condition, a tail-related warning, and the proposed algorithm's provable convergence.

Efficient Importance Sampling and variational Bayes share the goal of approximating an intractable posterior with a more convenient distribution.The methods differ in perspective, with EIS framed as importance sampling.
With all EIS weights fixed to one, Richard and Zhang's fixed-point condition becomes identical to the variational condition.The resulting algorithm is fitting a variational posterior approximation.
Variational posterior approximations generally have thinner tails than the actual posterior unless the approximation is extremely close.This makes fixed-unit-weight EIS unsuitable for general applications according to the cited discussion.
The paper's variational-fitting algorithm is provably convergent and includes variance-reduction and computational-speedup strategies.The authors suggest these strategies may also be useful for importance sampling, while leaving that application for future work.

D Choosing an estimator

The section compares three Monte Carlo estimators for the variational parameter, analyzing their bias and variance to identify when each is preferable. The shared-sample estimator ˆη2 is favored because covariance can reduce variance and it can recover exact solutions in matched exponential-family cases.

Estimator construction: The first estimator separately approximates the numerator and denominator before calculating their ratio, while the second uses the same samples for both integrals.The third, analytic estimator is available only when the first expectation can be computed analytically.
Bias and variance: The bias of the first estimator depends on Var(a)/E[a]^2, the spread of a relative to its magnitude.This dependence follows from the Monte Carlo variance relation Var(y) = Var(a)/S.
Bias and variance: The analytic estimator is unbiased, whereas the first estimator has an additional positive variance contribution and is therefore dominated by the analytic estimator.The variance comparison follows from independent sampling of the numerator and denominator for the first estimator.
Bias and variance: The shared-sample estimator typically has significantly smaller variance because its covariance term can be beneficial, while all three estimators share the analytic estimator's final variance term.Because variance and bias are O(1/S), their contributions to mean squared error are O(1/S) and O(1/S^2), respectively, making variance the greater problem.
Matched exponential families: When p and q belong to the same exponential family, positive and negative bias and variance contributions for ˆη2 cancel, yielding the exact solution in k + 1 samples.The exact recovery depends on the matched-family case and the number k of sufficient statistics.
Empirical comparison: In the exponential example with rate λ = 1.5, ˆη2 finds the exact rate with S = 2 samples, while ˆη1 often produces an invalid negative rate below 10 samples.The analytic estimator is unbiased in the same experiment, and the comparison used 50 repeats to estimate means and standard deviations.