Source-linked AI summary
Variational Boosting: Iteratively Refining Posterior Approximations
Andrew C. Miller, Nicholas Foti, Ryan P. Adams
TL;DR
Fixed variational families can fail to capture complicated posteriors, limiting approximation quality. Variational boosting iteratively enriches the family with covariance structure and mixture components, and its posterior approximations compare favorably with existing methods across accuracy and efficiency.
Problem
Fixed variational families may exclude the target distribution’s neighborhood, preventing arbitrarily close approximation and often underestimating posterior variances.
Source: §1 · p. 1Method
Variational boosting iteratively optimizes an enriched approximation, adding mixture components and covariance structure while using re-parameterization for mixture objectives.
Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 5Results
Variational Boosting closely matches NUTS posterior marginals and improves on mean field VI, while adding mixture components improves test log-probability across datasets.
Source: §4.1 · p. 16, §4.3 · p. 21Takeaways & Limitations
Adding and adapting mixture components can yield more expressive posterior approximations and better predictive results.
Source: §4.3 · p. 21, §5 · p. 22Diagonal covariance components can require many mixture components for strong correlations, and the independence restriction can introduce local optima.
Source: §3.3 · p. 8
Abstract
from arXiv · showhide
We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trade computation time for accuracy. We show how to expand the variational approximating class by incorporating additional covariance structure and by introducing new components to form a mixture. We apply variational boosting to synthetic and real statistical models, and show that resulting posterior inferences compare favorably to existing posterior approximation algorithms in both accuracy and efficiency.
1 Introduction
Fixed variational families can exclude the target’s neighborhood, leaving a mismatch that may underestimate posterior variances. The proposed method iteratively enriches the approximation, trading computation time for accuracy.
1 Introduction
- A fixed variational family may exclude the target’s neighborhood, preventing arbitrarily close approximation and often underestimating posterior variances.Source: §1 · p. 1
- Variational boosting iteratively enriches the approximating class through richer covariance structure and sequentially added mixture components.The method builds on black-box variational inference using the re-parameterization trick.Source: §1 · p. 2
- Like MCMC, the approach lets practitioners trade inference time for accuracy, while using optimization rather than specifying MCMC moves.The introduction notes that correctly specifying MCMC moves can be more restrictive than optimizing an objective.Source: §1 · p. 2
2 Variational Inference
Variational inference approximates an intractable target with a tractable distribution by optimizing an objective related to KL divergence. A fixed family can leave a nonzero gap, while the paper frames inference as a sequence of optimization problems that grows the approximating class.
2 Variational Inference
- Variational inference approximates a target distribution with a tractable family by minimizing KL divergence through a tractable objective such as the ELBO.The ELBO is a lower bound on the normalization constant.Source: §2 · p. 2
- When a fixed family excludes the target, the best approximation has a nonzero KL gap and can yield biased estimates of functions of the target variable.Source: §2 · p. 3
- MCMC gains fidelity as sample counts grow but can be computationally restrictive on large datasets, whereas VI can often use subsampled likelihood estimates for optimization.The passage notes that most MCMC algorithms require evaluating the log-likelihood over all data at each chain step.Source: §2 · p. 3
- The paper proposes reframing VI as a series of optimization problems that iteratively grows the approximating class to represent complex distributions and scale to large datasets.Source: §2 · p. 3
3 Method: Variational Boosting
This section describes variational boosting as an iterative approach to building richer mixture posterior approximations, including component fitting, covariance structure, and importance-weighted initialization. It also discusses gradient-based optimization and related work, including limitations of existing theoretical guarantees.
3.1 The re-parameterization trick and mixture distributions
- Mixture sampling involves nondifferentiable discrete component choices, so the method rewrites the ELBO as component-wise expectations and applies re-parameterized gradients to each component.This allows new components to be incorporated and fitted using the re-parameterization trick.Source: §3.1 · p. 5
3 Method: Variational Boosting
- Starting from one component, variational boosting fixes existing components and iteratively optimizes each newly added component and its mixture weight.Each iteration optimizes a new ELBO objective until convergence; the initial component is fitted using an existing black-box variational inference method.Source: §3.2 · pp. 6, 8
3.2 Adding Components
- When adding a component, the existing approximation is held fixed and Monte Carlo gradients optimize only the new component parameters and mixing weight.The new component is initialized before this ELBO optimization.Source: §3 · p. 4, §3.2 · pp. 7–8
3.3 Structured Multivariate Normal Components
- Low-rank plus diagonal covariance trades flexibility against fitting cost, addressing the O(D^3) determinant and inverse computations of full covariance while using smaller rank-r operations.The covariance rank controls this tradeoff: higher rank allows greater flexibility, while lower rank is more computationally efficient; determinant and inverse calculations reduce to an r × r matrix.Source: §3.3 · pp. 8–10
- The authors increase covariance rank until marginal-variance changes stabilize; for D = 40, rank r = 2 does an excellent job capturing 780 pairwise correlations and 40 marginal variances.The stopping heuristic monitors average absolute changes in marginal variance or standard deviation as rank is added.Source: §3.3 · pp. 9–10
3.4 Initializing Components
- Importance-weighted initialization targets regions underrepresented by the current approximation, but extreme weights can destabilize fitting, so resampling and re-weighting precede weighted EM.The largest importance weights indicate poorly represented regions; the resampling and re-weighting step breaks up large weights before initialization.Source: §3.4 · pp. 11–12
- Outlier-weight resampling stabilizes weighted EM by proposing components where the target probability is high but the current approximation is low.The proposal mixes the existing approximation with component means at outlier samples, using nonzero variance to explore their local probability; the M-step can clamp earlier components so only the new component changes.Source: §3.4 · pp. 13–14
3.5 Related Work
- Prior mixture variational methods include mean-field updates, equally weighted isotropic-normal mixtures [9], and jointly optimized exponential-family mixtures [32].The authors also describe mixtures as hierarchical variational models [30], but integrate out the discrete component identities because this is tractable.Source: §3.5 · pp. 14–15
- Sequential mixture estimation has O(1/K) error bounds, and adversarially trained deep-mixture construction has a similar bound [34], though applying these ideas is difficult.Here K is the number of mixture components; the cited arguments extend iterative approximation or use convex analysis, but the authors identify difficulties in applying them to variational boosting.Source: §3.5 · p. 14
- Concurrent boosting work [11] uses gradient-boosting to propose components and optimizes their mixture weights, but its error-bound argument assumes an optimal new component is found.The authors say a comprehensive theory must address multimodality and the nonconvexity of KL divergence.Source: §3.5 · p. 14
- Low-rank Gaussian approximations and LR+D covariances [27] capture posterior correlations, while the authors report superior approximations from combining spiked covariance with boosting new components.They fit the Gaussian low-rank components using black-box methods and joint optimization.Source: §3.5 · p. 14
4 Experiments and Analysis
Across the evaluated models, variational boosting improved posterior or predictive approximations as covariance capacity or mixture components increased. In the neural-network experiments, added components improved held-out log probability across datasets and surpassed PBP on five.
4.1 Hierarchical Binomial Regression
- Variational Boosting closely matched 20k-sample NUTS posterior marginals and improved on mean-field VI; covariance estimates also moved closer to NUTS with added components.Source: §4.1 · p. 16, Figure 4 · p. 16
4.2 Multi-level Poisson GLM
- Increasing covariance rank improved variance approximations, although only a few of 666 directions had nontrivial correlations.Source: §4.2 · p. 18, Figure 6 · p. 18
- Rank-3 mixtures represented non-Gaussian posterior shapes, and adding components improved marginal variance and nonzero-covariance estimates against MCMC.Source: Figure 7 · p. 19, §4.2 · p. 19
4.3 Bayesian Neural Network
- Every dataset improved test log-probability with a new component, and five surpassed PBP as components were added.Further components improved performance on most datasets; the comparison is reported in Table 2.Source: §4.3 · p. 21, Table 2 · p. 22
- Stochastic gradient noise made optimization more difficult, but using 20 samples mitigated it.Source: §4.3 · p. 21
5 Discussion and Conclusion
The paper presents a practical variational-inference method that adds components to represent complex posteriors over moderate parameter sets. It also identifies unresolved questions about convergence, covariance-rank selection, and natural-gradient updates.
5 Discussion and Conclusion
- The method adds components to a practical variational-inference approximation, learning rich representations of complex posteriors over a moderate number of parameters.It is described as applicable to a large number of Bayesian models of interest.Source: §5 · p. 22
- Whether successively adding fixed mixture components using ELBO objectives converges under KL(q||π) remains open, and its convergence rate is uncharacterized.Convergence is established for the alternative KL(π||q) direction in prior work.Source: §5 · p. 22
- The framework allows more flexible component distributions, including compositions of invertible maps [31] and auxiliary-variable variational models [23].Source: §5 · p. 22
- Covariance-rank selection is not greedy; nested boosting that uses rank-r results to inform rank r+1 is left for future work.Source: §5 · p. 23
- Natural-gradient updates remain future work because computing the Fisher information is less straightforward for mixture components than for a single Gaussian.Prior work reports that natural gradients can be more robust and lead to better optima [14] [17].Source: §5 · p. 23