PaperlayerView exact revision on arXiv

Source-linked AI summary

Convergence Rates of Variational Posterior Distributions

Fengshuo Zhang, Chao Gao

arXiv:1712.02519v4math.STstat.MLstat.TH

TL;DR

The paper asks how variational posteriors contract in nonparametric and high-dimensional inference, and formulates conditions on the prior, likelihood, and variational class to characterize their rates. Under conditions similar to prior mass and testing, the rate combines true-posterior contraction with variational approximation error; a modified prior-mass condition can control that error for a mean-field class.

  • Problem

    The paper studies conditions that characterize convergence rates of variational posteriors for nonparametric and high-dimensional inference.

    Source: §1 · p. 1

  • Method

    The authors formulate prior, likelihood, and variational-class conditions extending the prior-mass-and-testing framework to variational posteriors.

    Source: §1 · pp. 1–2

  • Results

    The variational posterior’s rate is the true posterior’s convergence rate plus a variational approximation error term.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For product-structured priors with a mean-field class, a modified prior-mass condition makes the variational approximation error dominated by the true posterior’s rate.

    Source: §1 · p. 2

    The general variational rate is only an upper bound, and additional regularity from the variational class can allow faster contraction than the true posterior.

    Source: §1 · p. 3

Abstract

from arXiv · show

We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the convergence rates. Under similar "prior mass and testing" conditions considered in the literature, the rate is found to be the sum of two terms. The first term stands for the convergence rate of the true posterior distribution, and the second term is contributed by the variational approximation error. For a class of priors that admit the structure of a mixture of product measures, we propose a novel prior mass condition, under which the variational approximation error of the mean-field class is dominated by convergence rate of the true posterior. We demonstrate the applicability of our general results for various models, prior distributions and variational classes by deriving convergence rates of the corresponding variational posteriors.

1 Introduction

The paper develops conditions under which variational posteriors contract, extending prior-mass-and-testing arguments to account for variational approximation error. For a corresponding prior and mean-field class, a modified prior-mass condition can make that error no larger than the true posterior’s rate.

1 Introduction

  • The framework applies prior-mass-and-testing arguments to variational posteriors, allowing well-studied problems to be revisited with similar proof techniques.The paper illustrates this with examples using different models, priors, and variational classes.Source: §1 · p. 2
  • The variational posterior’s convergence rate is the true posterior’s rate plus a variational approximation error term.Under a stronger prior-mass condition, that error term is dominated by the posterior rate.Source: §1 · p. 2
  • For product-structured priors and a mean-field class, a modified prior-mass condition can automatically dominate the approximation error by the true posterior’s rate.The paper presents this as a condition adapted to the structure of the variational class.Source: §1 · p. 2
  • Mean-field approximation under a sieve prior is related to empirical Bayes, and empirical Bayes is exactly variational Bayes for a specially designed variational class.The authors suggest the connection may indicate similar theoretical properties, rather than establishing them.Source: §1 · p. 3
  • The general variational rate is only an upper bound; with additional regularity imposed by the variational class, the variational posterior can contract faster than the true posterior.The paper says this can occur even when the variational posterior is not a good approximation to the true posterior.Source: §1 · p. 3
  • Related work studied variational approximations to tempered posteriors, whose results do not apply to usual posteriors with α = 1; similar α = 1 results appeared independently after this paper’s first version.An earlier approach covered posterior and variational rates but used conditions described as abstract and difficult to check in applications.Source: §1 · p. 3

2 Main Results

The paper studies variational posteriors from a frequentist perspective, giving conditions for their convergence and describing the rate as the sum of the true-posterior rate and variational approximation error. For the mean-field class, a distribution-mass condition can make that approximation error no larger than the true-posterior rate.

2.1 Definitions and Settings

  • The variational posterior is the KL projection of the data-dependent posterior onto a chosen variational family, whose tractability can ease computation.If the family contains all distributions, the variational posterior is the true posterior.Source: §2.1 · pp. 5–6

2.2 Results for General Variational Posteriors

  • Under the stated prior, likelihood, testing, and integrability conditions, the variational-posterior rate combines the true-posterior rate with a term for approximation error from the variational class.A more expressive variational set can reduce the approximation-error term.Source: §2.2 · pp. 6–7
  • The general theorem’s expected-loss convergence implies concentration of the full variational posterior at the same rate; for convex loss, the variational posterior mean also has that rate.The posterior-mean conclusion additionally requires convexity of the loss.Source: §2.2 · p. 7
  • A sufficient condition for the variational posterior to match the true-posterior rate is that the approximation-error term be dominated by that rate.The paper presents this as a condition that can be incorporated into a theorem for variational-posterior convergence.Source: §2.2 · p. 8

2.3 Results for Mean-Field Variational Posteriors

  • For the mean-field class, a distribution-mass condition characterizes approximation error; when the prior belongs to the mean-field class, the condition reduces to a prior-mass condition.The condition is described as usually not hard to check.Source: §2.3 · p. 9

3 Applications

The applications examine variational posteriors in Gaussian and piecewise-constant models, including how the choice of variational class relates to convergence rates. For the piecewise-constant model, they also describe a dynamic program for estimating knots and its computational cost.

3.1 Gaussian Sequence Model

  • In the Gaussian sequence model, the mean-field variational posterior attains the Sobolev minimax rate n^(-2α/(2α+1)) up to a logarithmic factor, and a prior rescaling removes that factor.The rate cannot be improved for the broad prior class described in the proposition.Source: §3.1 · pp. 11–12

3.2 Infinite Dimensional Exponential Families

  • Under a prior Π satisfying (20)–(23), for any θ* ∈ Θα(B) with α > 1/2, the Gaussian mean-field variational posterior achieves the minimax rate n^(-2α/(2α+1)) up to a logarithmic factor.The same rate result also holds for the mean-field variational posterior defined with SMF.Source: §3.2 · p. 13

3.3 Piecewise Constant Model

  • For piecewise-constant signals, the mean-field classes give only a trivial rate, whereas the Markov and joint Markov variational classes achieve the minimax rate.The dependent-class result holds under the stated prior conditions.Source: §3.3 · pp. 15–16
  • The mean-field failure arises because its independence structure cannot capture piecewise-constant parameter dependence, producing a large variational approximation error.The resulting variational posteriors are equivalent to those induced by an independent product prior.Source: §3.3 · pp. 15–16
  • The joint Markov variational solution can be computed exactly by dynamic programming.Both the Markov and joint Markov classes have satisfactory statistical properties, while the joint class offers this computational advantage.Source: §3.3 · p. 16
  • A dynamic program solves (28) using the data X1, ..., Xn and outputs the estimated knot set.The algorithm initializes segment scores, computes them over candidate intervals, and selects the knot count by maximizing the stated criterion.Source: §3.3 · p. 17
  • The dynamic program costs O(n^3), and (29) has a closed form when g(·) is conjugate.Source: §3.3 · p. 17

4 Variational Bayes with Model Selection

The section develops convergence guarantees for mean-field variational posteriors with model selection, then applies them to location-scale mixture density estimation. It also shows that latent-variable augmentation can aid computation without changing the contraction rate.

4.1 General Settings

  • The general setup indexes parameter spaces by a model dimension k, uses a prior with product structure conditional on k, and optimizes an ELBO for model selection.The dimension index can represent mixture component count; the mean-field class is chosen to exploit the prior’s structure.Source: §4.1 · pp. 18–19

4.2 Convergence Rates

  • Under prior-mass, testing, and variational-class conditions, the variational posterior rate combines the true posterior rate with a variational approximation-error term.The general theorem uses conditions C1, C2, and C3*; for a singleton model and product prior, C3* reduces to condition (2).Source: §4.2 · p. 19

4.3 Density Estimation via Location-Scale Mixtures

  • For location-scale mixture density estimation, Theorem 4.2 establishes a variational posterior contraction guarantee under prior conditions (40)–(46) and true-density conditions (B1)–(B3).The proof uses a surrogate density because mixture approximation is analyzed under a stronger divergence, then transfers the result to the true density by change of measure.Source: §4.3 · p. 22

4.4 Dealing with Latent Variables

  • Adding latent variables makes the variational lower bound looser, but the resulting variational posterior retains the contraction rate of Theorem 4.2.The latent-variable approach also facilitates computation and yields a coordinate-ascent algorithm with closed-form updates.Source: §4.4 · pp. 23–24

5 Discussion

The discussion examines connections between variational Bayes and empirical Bayes, how variational restrictions and misspecification affect convergence bounds, and a sharper Gaussian-sequence prior with an optimization algorithm. It also addresses the computational bottleneck of evaluating (62) across k and notes that restricting k to powers of two still achieves Theorem 4.3’s rate.

5.1 Variational Bayes and Empirical Bayes

  • The mean-field posterior has a product form, and its model selection differs from empirical Bayes in using approximate marginal likelihood and the combined scores of neighboring models.When the likelihood factorizes over coordinates, the mean-field marginal-likelihood approximation is exact.Source: §5.1 · pp. 25–26
  • For sieve priors, a special variational class induces exactly the empirical Bayes posterior, making empirical Bayes a variational Bayes procedure.The authors suggest this equivalence may help unify theoretical analyses of the two approaches.Source: §5.1 · p. 26

5.2 Variational Approximation as Regularization

  • When the Gaussian-sequence prior undersmooths the data (β < α), thresholding coordinates after k can yield a faster variational-posterior rate; when β ≥ α, it cannot improve the true-posterior rate.The stated mechanism is reduction of extra variance from undersmoothing; when β ≥ α, information lost through oversmoothing cannot be recovered by the variational class.Source: §5.2 · pp. 27–28
  • For sparse linear regression, the variational posterior is characterized by a penalized least-squares estimator whose center approaches LASSO as τ → 0, enabling sparse recovery despite a dense posterior.Under the stated assumptions, the estimator's risk rate is the same as LASSO's.Source: §5.2 · pp. 28–29

5.3 Model Misspecification

  • Under model misspecification, the variational-posterior convergence bound gains an extra term for the data-generating process's deviation from the likelihood model, and sufficient conditions can make approximation error dominated by the other terms.The section applies these results to derive oracle inequalities for variational posteriors in the piecewise-constant model.Source: §5.3 · pp. 30–31

A.1 Sharp Convergence Rates for Gaussian Sequence Model

  • A √n-scaled prior on Gaussian-sequence coefficients, under the stated replacement conditions, yields a mean-field variational-posterior rate without the logarithmic factor in Theorem 3.2.The prior rescaling supports the conditions used to remove the logarithmic factor.Source: §A.1 · p. 32

A.2 An Algorithm for (50)

  • For each fixed k, coordinate-ascent variational inference iteratively updates parameters using closed-form updates, then selects k by maximizing the objective evaluated at those updates.The objective can be calculated in closed form by conjugacy.Source: §A.2 · pp. 32–33
  • Selecting k from {1, 2, 4, ..., 2⌈log2 n⌉} instead of evaluating (62) for every k still achieves Theorem 4.3’s convergence rate.Computing (62) for each fixed k is straightforward and efficient with CAVI.Source: §A.2 · p. 33

A.3 Beyond the Kullback-Leibler Approximation

The appendix extends the convergence result to variational posteriors defined using divergences beyond Kullback-Leibler, under conditions on how the divergence compares with KL. Divergences not necessarily larger than KL require new analysis.

A.3 Beyond the Kullback-Leibler Approximation

  • The paper introduces a D*-variational posterior to formulate the generalized convergence result.The appendix says its results can be generalized to variational approximation using other criteria.Source: §A.3 · p. 34
  • Theorem A.2 extends Theorem 2.1 when D* is nonnegative and at least as large as KL for variational distributions against any comparison distribution, under conditions C1–C3.The stated examples include Rényi divergences with ρ ≥ 1 and the χ²-divergence.Source: §A.3 · p. 34
  • Divergences not necessarily larger than KL require new techniques and are left for future work.Alternative divergences may also be harder to optimize, though they may give better approximations.Source: §A.3 · p. 34

B.1 Proof of Theorem 2.1

The proof of Theorem 2.1 combines divergence inequalities, a posterior-tail bound, and a moment-generating-function bound to establish the result under its stated conditions.

B.1 Proof of Theorem 2.1

  • A preliminary inequality based on the basic property of KL divergence and Jensen’s inequality supports the proof’s bounds.The proof then takes a minimum over a > 0 and Q ∈ S.Source: §B.1 · pp. 34–35
  • Lemma B.2 bounds a probability term for the variational posterior defined in (3), using an auxiliary probability measure and an inequality from Lemma B.1.The proof introduces the auxiliary measure before applying the inequality.Source: §B.1 · p. 35
  • The proof uses a posterior-tail bound and a moment-generating-function bound before applying the lemmas to complete Theorem 2.1.The tail bound applies for ε ≥ ε_n; the moment-generating-function lemma controls E exp(aX) for 0 < a ≤ 1.Source: §B.1 · pp. 36–38

B.2 Proofs of Theorem 2.3, Theorem 2.4 and Theorem 4.1

These proofs establish the stated results by bounding variational-posterior terms, including a model-selection bound and a mean-field construction. For Theorem 4.1, the argument transfers prior conditions to the setting required by the theorem.

B.2 Proofs of Theorem 2.3, Theorem 2.4 and Theorem 4.1

  • Theorem 2.4 is proved by choosing a Q in the mean-field class and bounding the resulting expression using (8).The proof then states the desired bound.Source: §B.2 · pp. 38–39
  • Lemma B.5 provides a bound for the model-selection variational posterior defined by the solution of (34).The prior Π is induced by the sampling process for (k, θ(k)).Source: §B.2 · p. 39
  • Theorem 4.1 follows by bounding its terms and using that prior conditions C1 and C2 hold by assumption while C3* implies C3.The proof concludes with the desired result after applying these bounds.Source: §B.2 · pp. 40–41

B.3 Proofs of Theorem 3.1, Proposition 3.1, Theorem 3.2 and Theorem A.1

The proofs establish prior-mass, testing, and variational-approximation conditions used to derive the stated convergence results. In particular, the conditions hold at the displayed rates under the corresponding prior assumptions.

B.3 Proofs of Theorem 3.1, Proposition 3.1, Theorem 3.2 and Theorem A.1

  • Theorem 3.1 follows by applying Corollary 5.1 with the indicated coordinate partition.The proof sets Θ_j1 = R\{0} and Θ_j2 = {0}.Source: §B.3 · p. 41
  • The variational posterior in Proposition 3.1 is a product measure whose components after the selected coordinate cutoff are point masses at zero.The proof invokes Theorem 3.1 and Theorem B.6 for this structure.Source: §B.3 · p. 43
  • For a prior Π satisfying (15), conditions (C1) and (C2) hold for all ϵ ≥ n^-1/2; the proof establishes (C2) and constructs a test for (C1).Source: §B.3 · p. 43
  • For θ* ∈ Θα(B) and priors satisfying (16) and (17), conditions (C3) and (C4) hold at ϵ_n = n^-α/(2α+1)(log n)^α/(2α+1).The proof constructs a mean-field measure and bounds its divergence to establish the variational condition.Source: §B.3 · pp. 43–44
  • Theorem 3.2 and Theorem A.1 follow from Lemmas B.7–B.9.The lemmas establish the required conditions for the respective prior assumptions.Source: §B.3 · pp. 43, 45, 46

B.4 Proof of Theorem 3.3

The proof of Theorem 3.3 verifies testing and prior-concentration conditions, then establishes the variational conditions for the stated smoothness regime. Together, the cited lemmas imply the theorem.

B.4 Proof of Theorem 3.3

  • For θ* ∈ Θα(B) with α > 1/2, Lemmas B.10 and B.11 establish the conditions used to prove Theorem 3.3.The proof combines these lemmas through Theorem 2.2.Source: §B.4 · pp. 46, 51
  • The prior assigns at most ≲ exp(−C nϵ²) mass outside Θ_n(ϵ), establishing condition (C2).The proof states this bound for any C > 0 by choosing the constant in the radius sufficiently large.Source: §B.4 · p. 48
  • A covering-number bound yields tests satisfying (C1), with testing error made arbitrarily small by choosing C′ sufficiently large.The test separates θ* from parameters in Θ_n(ϵ) with Hellinger distance at least C′ϵ.Source: §B.4 · pp. 48–49
  • For (C4), the proof directly constructs Q in the Gaussian variational class and bounds its divergence to the prior.The construction uses k0 = ⌈n^(1/(2α+1))⌉ coordinates centered at θ* with variance n^-1 and sets later coordinates to zero.Source: §B.4 · p. 49

B.5 Proofs of Theorem 3.4, Theorem 3.5 and Theorem 3.6

These proofs characterize the support and optimization structure of the mean-field variational posterior for the piecewise-constant model. They also connect the remaining theorem proofs to results established elsewhere in the paper.

B.5 Proofs of Theorem 3.4, Theorem 3.5 and Theorem 3.6

  • For finite divergence to the prior, a mean-field distribution is supported on vectors with no adjacent equal coordinates.The proof argues that positive mass on adjacent equality would force a coordinate point mass and infinite divergence.Source: §B.5 · p. 51
  • Theorem 3.5 is a special case of Theorem 5.8.Its proof is deferred to Section B.10.Source: §B.5 · p. 53
  • The proof of Theorem 3.6 uses support inclusion under finite KL divergence to constrain the joint variational distribution.It applies the prior's zero-probability constraints on inconsistent latent-variable and parameter configurations.Source: §B.5 · p. 53
  • For a fixed change-point set S, optimizing over q(w) and q(θ) gives the stated solutions, which reduce the optimization problem to (28).The proof notes that the resulting distribution for θ remains in S_MC.Source: §B.5 · pp. 54–55

B.6 Proofs of Theorem 4.2 and 4.3

The proofs establish the conditions for Theorem 4.2 and use a change-of-measure argument for Theorem 4.3. Under the stated density and prior assumptions, the resulting rate for Theorem 4.2 has a logarithmic factor determined by r.

B.6 Proofs of Theorem 4.2 and 4.3

  • Lemma B.14 constructs a finite mixture approximation to f0, and Lemma B.17 supplies the density approximation used in that construction.The approximation is built from a convolution K_σ0 h_α under conditions (B1)–(B3).Source: §B.6 · pp. 55–57
  • Lemma B.16 establishes (C3*) relative to the approximating density ef0, with r determined by the prior-condition parameters.The proof sets σ0 = n^-1/(2α+1)(log n)^(r/(2α+1)).Source: §B.6 · pp. 56, 61
  • ϵ_n = n^-α/(2α+1)(log n)^(αr/(2α+1)) for Theorem 4.2 under the stated density and prior conditions.The proof verifies (C1)–(C2) and (C3*) using Lemmas B.15 and B.16.Source: §B.6 · p. 56
  • The proof of Lemma B.15 establishes prior concentration outside Θ_n(ϵ), yielding the exponential bound required for (C2).The bound is Π(Θ_n(ϵ)^c) ≲ exp(−C0 nϵ²).Source: §B.6 · pp. 56, 59
  • The proof of Theorem 4.3 also uses a change-of-measure argument to establish concentration around the target posterior.The argument reuses the approximating density and bounds developed for Theorem 4.2.Source: §B.6 · pp. 56, 62, 66

B.7 Proofs of Theorem 5.1, Corollary 5.1 and Theorem 5.2

These proofs characterize the variational posterior’s form, select its empirical-Bayes version, and establish bounds used in the subsequent theorem arguments.

B.7 Proofs of Theorem 5.1, Corollary 5.1 and Theorem 5.2

  • The mean-field variational posterior is a product measure whose coordinate densities take a piecewise form, with a possible mixture at the index k.The proof rules out two candidate forms under constraint (90), leaving the form in (89).Source: §B.7 · pp. 67–68
  • For the empirical-Bayes class, the variational posterior is bQ(bk), evaluated at the selected model index bk.The proof obtains bk by optimizing the model-index objective.Source: §B.7 · p. 70

B.8 Proof of Theorem 5.3

  • Theorem 5.3’s proof decomposes risk and develops upper and lower bounds in two cases indexed by k.The supplied proof passages do not state the resulting rate completely.Source: §B.8 · pp. 70–71

B.9 Proofs of Theorem 5.4 and Theorem 5.5

  • Inequality (94) immediately implies the generalized cone condition defined in; another consequence of (94) is the error bound.For λ = C√n log p, the condition λ ≥ 2||Xᵀ(y − Xβ∗)||∞ holds with probability at least 1 − p−C′ by a union bound argument in [5].Source: §B.9 · p. 73

B.10 Proofs of Theorem 5.6, Theorem 5.7 and Theorem 5.8

These proofs verify conditions for the piecewise constant model and establish the theorem results under the stated conditions; the proof of Theorem A.2 follows the argument for Theorem 2.1.

B.10 Proofs of Theorem 5.6, Theorem 5.7 and Theorem 5.8

  • The piecewise-constant prior samples change points uniformly conditional on their number, then samples values at those points and carries values forward elsewhere.This sampling representation is used to verify prior conditions for signals with k0 change points.Source: §B.10 · pp. 77, 79
  • For θ0 in Θ_k0(B), the proof verifies condition (C1) under (26) and condition (C3) under (26) and (27).The proof then combines these lemmas with Theorems 5.6 and 5.7 to prove Theorem 5.8.Source: §B.10 · pp. 79, 80, 82

B.11 Proof of Theorem A.2

  • The proof of Theorem A.2 applies the same argument used for Theorem 2.1 under Theorem A.2’s conditions.Source: §B.11 · p. 82
Loading 1712.02519v4…