Source-linked AI summary
Importance Weighting and Variational Inference
Justin Domke, Daniel Sheldon
TL;DR
Posterior inference is often intractable, and tighter importance-weighted bounds do not alone clarify how to use q for inference. The paper frames IWVI as augmented VI connected to NIS and investigates elliptical distributions. It reports reduced posterior-expectation estimation error with NIS and dimensionality-dependent benefits from elliptical VI.
Problem
Posterior queries are often intractable, and importance-weighted bounds leave unclear how q should approximate p for posterior expectations.
Source: §1 · pp. 1–2Method
The paper identifies IWVI as augmented VI tied to NIS and adapts defensive sampling through elliptical distributions and reparameterization.
Source: §1 · p. 2Results
Using NIS after optimizing q significantly reduces posterior-expectation estimation error, while elliptical VI improves ELBO and expectations in low dimensions and convergence reliability and speed in higher dimensions.
Source: §1 · p. 2Takeaways & Limitations
For inference, use the optimized q through NIS for posterior expectations; elliptical distributions offer dimensionality-dependent benefits.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent work used importance sampling ideas for better variational bounds on likelihoods. We clarify the applicability of these ideas to pure probabilistic inference, by showing the resulting Importance Weighted Variational Inference (IWVI) technique is an instance of augmented variational inference, thus identifying the looseness in previous work. Experiments confirm IWVI's practicality for probabilistic inference. As a second contribution, we investigate inference with elliptical distributions, which improves accuracy in low dimensions, and convergence in high dimensions.
1 Introduction
Approximate probabilistic inference is needed because posterior queries are often intractable, and importance-weighted bounds alone leave unclear how to use q for inference. The paper connects IWVI to augmented VI and NIS, and also investigates elliptical distributions for inference.
1 Introduction
- The ELBO uses Jensen’s inequality on the importance-sampling weight R to bound log p(x), and Monte Carlo estimates its expected log as in BBVI.Because p(x) = E R, the sample average RM also has mean p(x), motivating the importance-weighted bound.Source: §1 · p. 1
- For probabilistic inference, a tighter IW-ELBO does not by itself specify how q is close to p or how to use q for posterior expectations.Direct posterior queries are often intractable, motivating approximate inference.Source: §1 · pp. 1–2
- IWVI is augmented VI: maximizing the IW-ELBO minimizes a joint-distribution KL for qM and pM, with qM derived from NIS over M samples.After optimizing q, posterior expectations should be computed using NIS; this reduces their estimation error while tightening the log p(x) bound.Source: §1 · p. 2
- Earlier work connected the importance-weighted ELBO to the ELBO for the NIS distribution, while this paper identifies the gap as a conditional KL divergence.The earlier connection was that the importance-weighted ELBO is a lower bound of that NIS-distribution ELBO.Source: §1 · p. 2
- Elliptical VI gives small improvements over Gaussian BBVI in ELBO and posterior expectations at low dimensions, and improves convergence reliability and speed at higher dimensions.Its defensive-sampling approach is compatible with IWVI and has almost no additional overhead to BBVI.Source: §1 · p. 2
2 Variational Inference
Variational inference addresses two related goals: approximating a posterior and bounding the marginal likelihood. It selects a tractable q, with posterior expectations as one concrete inference target and lower-bound optimization as a route to likelihood learning.
2 Variational Inference
- VI approximates p(z|x) with a tractable distribution q(z), and its two goals are posterior inference and bounding log p(x).The ELBO lower bound supports the latter goal, while closeness to the posterior in KL-divergence supports the former.Source: §2 · p. 2
- For Bayesian inference, users specify p(z, x), observe x, and may seek posterior expectations of a chosen t(z).MCMC is commonly used for these problems, while its computational expense motivates VI.Source: §2 · p. 2
- For maximum-likelihood learning, optimizing a lower bound on log pθ(x) over θ and q addresses an intractable marginal likelihood integral.Variational auto-encoders have used this approach.Source: §2 · p. 2
3 Importance Weighting
Importance weighting uses averages of importance-sampling estimates to tighten the ELBO bound. The paper explains IWVI through augmented variational inference, clarifying the divergence it minimizes and how its marginal can support probabilistic inference.
3 Importance Weighting
- Jensen’s inequality gives a lower bound on log p(x) from any estimator R with mean p(x), and the bound tightens when R is more concentrated.The standard ELBO is recovered by using the usual single-sample importance ratio.Source: §3 · p. 3
- Averaging M i.i.d. importance-sampling estimates yields R_M and the corresponding importance-weighted ELBO, first introduced for variational auto-encoder likelihood learning.The paper calls maximizing this bound Importance Weighted Variational Inference (IWVI).Source: §3 · p. 3, §3.1 · p. 4
- As M increases, R_M places less mass near zero, producing a tighter IW-ELBO lower bound on log p(x).Figure 1 tracks the density of R_M as M changes.Source: §3 · p. 3, Figure 1 · p. 3
3.1 A generative process for the importance weighted ELBO
- IWVI constructs an augmented joint distribution from self-normalized importance sampling over M samples, while p_M pairs a posterior draw with M−1 dummy samples from q.The q_M process retains the unselected samples after selecting one in proportion to its importance weight.Source: §3.1 · pp. 3–4
- The marginal q_M(z1) approaches p(z1|x) as M increases, supporting posterior expectations through q_M(z1).The paper gives E_p(z|x)t(z) ≈ E_q_M(z1)t(z1) as the approximation.Source: §3.1 · p. 4
- The joint divergence minimized by IWVI upper-bounds the marginal NIS-to-posterior divergence because its extra conditional KL term is nonzero and irrelevant to the posterior approximation.This identifies IWVI as auxiliary variational inference and quantifies the looseness left unmeasured in earlier work [6, 2, 16, 12].Source: §3.1 · p. 4
- For M=1 a variational mean nearer one mode is preferable, whereas for large M a centered mean and Student-T heavy tails improve approximation and reported IW-ELBO and moment error.Figure 2 compares Gaussian and Student-T distributions with constant variance and two candidate means.Source: Figure 2 · p. 5
4 Importance Sampling Variance
Importance weighting changes which proposal tails are useful: small M favors the weak tails associated with VI’s mode-seeking behavior, while large M favors wider tails. For large M, the IW-ELBO looseness is governed by the variance of the importance weight under stated moment conditions.
4 Importance Sampling Variance
- For small M, VI’s mode-seeking behavior favors weak tails, whereas for large M, importance-weighting variance reduction favors wider tails.The paper relates VI’s weak-tail preference to avoiding cases where q(z) ≫ p(z, x).Source: §4 · p. 5
- For large M, IW-ELBO looseness is given by the variance of R, and the KL divergence between qM and pM asymptotically decreases at a 1/M rate.The variance result assumes E|R − p(x)|^(2+α) is finite for some α > 0 and lim supM→∞ E[1/RM] is finite.Source: §4 · pp. 5–6
- Large importance weights arise when q(z) ≪ p(z, x), motivating defensive sampling with a widely dispersed proposal.The paper expects the best form of q to vary with M.Source: §4 · p. 6
5 Elliptical Distributions
The section develops elliptical distributions from spherical random variables and then applies affine transformations to obtain general elliptical variables. It relates the Gaussian radial variable to a chi distribution.
5 Elliptical Distributions
- Elliptical distributions generalize Gaussians and include Student-T, Cauchy, and scale mixtures of Gaussians.The review assumes a density function exists.Source: §5 · p. 6
- A spherical random variable is constructed from a radial density and a direction uniform on the unit sphere.Its density is constant at each fixed radius, and integrating over a radius-matched sphere recovers the radial density.Source: §5 · p. 6
- An elliptical random variable is formed by an affine transformation using a positive definite matrix Σ and vector µ.The passage identifies its mean as µ and gives its covariance in terms of the spherical variable.Source: §5 · p. 6
- For a Gaussian, the squared radius is a sum of d independent squared Gaussian variables, so the radius has a χd distribution.Source: §5 · p. 6
6 Reparameterization and Elliptical Distributions
The section extends reparameterization beyond Gaussians to elliptical variational distributions, subject to conditions on their density generators. It also describes ways to compute gradients when radial distributions lack convenient inverse CDFs, including a Student-T-specific parametrization.
6 Reparameterization and Elliptical Distributions
- The reparameterization trick expresses samples as T(ϵ; w) from a parameter-independent base density, yielding unbiased gradient estimates from samples of ϵ.For elliptical distributions, the same approach applies when the density generator g is independent of w and ϵ comes from the corresponding spherical distribution.Source: §6 · p. 6, §6.1 · p. 7
6.1 Elliptical Reparameterization
- When the radial distribution ρw depends on w, the reparameterization can use a uniform v and the inverse CDF Fw^-1(v) for the radius.The base variables are a spherical direction u and uniform v; derivatives can be backpropagated through Aw and µw.Source: §6.1 · p. 7
6.2 Dealing CDFs without closed-form inverses
- When an inverse CDF is unavailable in closed form, sample r from ρw, set v = Fw(r), and use the implicit function theorem to compute gradients.This approach requires the CDF and a way to sample from ρw.Source: §6.2 · p. 7
6.3 Student T distributions
- For the Student-T radial variable, r is the ratio of independent χ variables, and the χν CDF offers a more available alternative to the F-distribution CDF.The experiment reparameterizes the radius as √νt/s, with t ∼ χd and s ∼ χν.Source: §6.3 · p. 7
7 Experiments
Across the experiments, IWVI improves inference over VI on the tested distributions, while elliptical IWVI (E-IWVI) offers low-dimensional accuracy gains and more reliable convergence on higher-dimensional logistic-regression data. In logistic regression, larger M improves both methods.
7 Experiments
- The experiments compare Student-T E-IWVI with Gaussian IWVI; regular VI is IWVI with M = 1.Two distributions have computable log p(x) and true posterior means and variances for evaluation; those experiments use fixed random inputs and batch L-BFGS.Source: §7 · p. 7
- On random Dirichlet distributions, IWVI reduces VI’s test-integral error to negligible levels, while E-IWVI adds accuracy in low dimensions but little at K = 20.The experiment averages results over 20 repetitions.Source: Figure 3 · p. 8
- On clutter distributions, IWVI reduces the error in estimated second moments by orders of magnitude, and E-IWVI’s benefit diminishes in higher dimensions.The experiment averages results over 50 repetitions.Source: Figure 4 · p. 8
- E-IWVI converges more reliably than IWVI, particularly on higher-dimensional logistic-regression data, and larger M improves both methods.E-IWVI converges at least as well across step sizes and was easier to tune; when step size was perfectly tuned and optimization ran many iterations, both methods performed similarly in IW-ELBO.Source: §7 · pp. 8–9, Figure 5 · p. 9
A.1 Additional Experimental Results
The appendix captions identify additional figures for the 1-D example, Dirichlet inference, and logistic-regression experiments on more datasets.
A.1 Additional Experimental Results
- Figure 6 contains additional figures corresponding to the 1-D example.Source: Figure 6 · p. 11
- Figure 7 presents more results on inference with Dirichlet distributions.Source: Figure 7 · p. 12
- Figure 8 extends the logistic-regression experiments to more datasets.The listed datasets are australian, sonar, ionosphere, w1a, a1a, mushrooms, and madelon.Source: Figure 8 · p. 13, §A.1 · p. 13
A.2 Proofs for Section 3
The proofs characterize IWVI using augmented distributions and identify the KL divergences that quantify the gaps in its bound.
A.2 Proofs for Section 3
- Theorem 1 defines qM from self-normalized importance sampling over M samples and pairs it with pM, formed from one p draw and M −1 dummy q samples.Source: §A.2 · p. 14
- The two inequality gaps are exactly KL[qM(z1)∥p(z1|x)] and KL[qM(z2:M|z1)∥q(z2:M)].Source: §A.2 · p. 15
- The chain-rule decomposition separates the joint KL into the posterior-marginal KL and a conditional KL for the dummy samples.The proof identifies pM(z1|x) with p(z1|x) and pM(z2:M|z1,x) with q(z2:M).Source: §A.2 · p. 15
A.3 Asymptotics
The appendix gives conditions for the large-sample behavior of the IW-ELBO, proves moment bounds used in the analysis, and contrasts two augmented-distribution decompositions for inference.
A.3 Asymptotics
- For large M, IW-ELBO looseness is governed by Var[R] under the theorem’s moment and reciprocal-weight conditions.The assumptions include a finite (2+α)-moment for R−p(x) and bounded lim sup E[1/RM].Source: §A.3 · p. 16
A.3.1 Proofs of Lemmas
- The proof bounds sample-mean moments using the Marcinkiewicz–Zygmund inequality, supporting the asymptotic argument.The cited lemma gives moment bounds for means of i.i.d. zero-mean variables.Source: §A.3 · p. 16, §A.3.1 · p. 18
A.3.2 Relationship of Decompositions
- Unlike the contrasted decomposition, this approach fixes pM’s marginal at the posterior and makes qM’s marginal approach it as M increases.The qM marginal is the self-normalized importance-sampling distribution and is available for inference tasks.Source: §A.3.2 · p. 20