PaperlayerView exact revision on arXiv

Source-linked AI summary

Covariances, Robustness, and Variational Bayes

Ryan Giordano, Tamara Broderick, Michael I. Jordan

arXiv:1709.02536v3stat.ME

TL;DR

MFVB can estimate some posterior means well but often misestimates variances and covariances, and MFVB-specific prior robustness measures were undeveloped. The paper derives sensitivity-based covariance estimates and robustness measures; its methods provide accurate posterior uncertainty estimates and sensitivity measures substantially faster than MCMC.

  • Problem

    MFVB can provide useful posterior means but often underestimates marginal variances, and methods for assessing Bayesian robustness had not been developed for MFVB.

    Source: §1 · p. 2

  • Method

    The paper derives formulas for the sensitivity of MFVB posterior means to model perturbations and uses them to estimate covariances and local hyperparameter sensitivity.

    Source: §1 · p. 2, §6 · p. 39

  • Results

    LRVB covariance estimates closely match MCMC approximations while taking over an order of magnitude less time, and the experiments report accurate means, covariances, and prior sensitivity measures much faster than MCMC.

    Source: §1 · p. 2, §6 · p. 39

  • Takeaways & Limitations

    The methods provide MFVB users with improved variance estimates and measures of prior robustness, with little additional coding when automatic differentiation is supported.

    Source: §6 · p. 39

    For ADVI, sensitivity calculations are not useful unless the stochastic optimization solution is sufficiently close to an optimum of the approximate objective.

    Source: §5.2.1 · p. 25

Abstract

from arXiv · show

Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale datasets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covariances. Furthermore, prior robustness measures have remained undeveloped for MFVB. By deriving a simple formula for the effect of infinitesimal model perturbations on MFVB posterior means, we provide both improved covariance estimates and local robustness measures for MFVB, thus greatly expanding the practical usefulness of MFVB posterior approximations. The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature relating derivatives of posterior expectations to posterior covariances and include the Laplace approximation as a special case. Our key condition is that the MFVB approximation provides good estimates of a select subset of posterior means---an assumption that has been shown to hold in many practical settings. In our experiments, we demonstrate that our methods are simple, general, and fast, providing accurate posterior uncertainty estimates and robustness measures with runtimes that can be an order of magnitude faster than MCMC.

1 Introduction

MFVB can yield useful posterior means but often underestimates uncertainty, and MFVB-specific robustness methods had not been developed. The paper uses the sensitivity of MFVB means to perturbations to estimate covariances and assess local robustness.

1 Introduction

  • MFVB often underestimates marginal variances, despite producing useful posterior means for some parameters, and methods for assessing Bayesian robustness had not been developed for MFVB.These inferential gaps motivate the paper.Source: §1 · p. 2
  • Under mild regularity conditions, derivatives of posterior expectations correspond directly to posterior covariances, connecting sensitivity analysis with uncertainty estimation.The paper builds on this relationship from Bayesian robustness work.Source: §1 · p. 2
  • LRVB uses the sensitivity of MFVB posterior expectations to estimate covariances and local hyperparameter sensitivities, matching MCMC covariance approximations closely while running over an order of magnitude faster.The accuracy claim assumes that MFVB estimates the relevant posterior means well under the perturbations of interest.Source: §1 · p. 2

2 Bayesian Covariances and Sensitivity

Local sensitivity measures how posterior expectations respond to small hyperparameter changes and can inform judgments about robustness. Under mild regularity conditions, this derivative is a posterior covariance, which can be estimated from MCMC draws.

2.1 Local Sensitivity and Robustness

  • Global robustness examines extrema across plausible hyperparameters but is often difficult to calculate, so local sensitivity measures expectation changes near a tentative value α0.Local sensitivity approximates global sensitivity to first order when posterior dependence on α is approximately linear nearby.Source: §2.1 · p. 4
  • Sensitivity is an objectively defined quantity that can inform robustness, while deciding how much change is acceptable and which prior variations are reasonable remains a modeler’s judgment.The local sensitivity is defined as a derivative with respect to the hyperparameter at α0.Source: §2.1 · p. 4

2.2 Covariances and Sensitivity

  • Under regularity conditions, the derivative of a perturbed posterior expectation equals a particular posterior covariance, allowing local sensitivity to be estimated from MCMC draws.The perturbation can encode changes to both the likelihood and prior; its log form typically avoids normalizing constants.Source: §2.2 · pp. 5–6

3 Variational Bayesian Covariances and Sensitivity

Variational Bayes approximates a posterior by optimizing KL divergence over a tractable distribution family, with MFVB imposing a factorization across parameter components. The paper derives variational sensitivity calculations and states the conditions under which they approximate exact posterior quantities.

3.1 Variational Bayes

  • Variational Bayes selects a tractable distribution by minimizing its KL divergence from the posterior; MFVB commonly uses a family that factorizes across parameter components.The exponential-family assumption may also be used, but does not follow automatically from factorization.Source: §3.1 · pp. 6–7

3.2 Variational Bayes sensitivity

  • The paper derives a closed-form sensitivity of variational expectations to local perturbations, using a linear system involving derivatives of the variational objective at its optimum.Under the stated assumptions, the result applies to arbitrary local perturbations satisfying the regularity conditions.Source: §3.2 · pp. 7–8

3.3 Approximating with Variational Bayes

  • Approximation quality depends on MFVB estimating the means and sensitivities of interest well; this condition is specific to the chosen function g and lacks a practical quantitative diagnostic beyond comparison with a slower accurate method.The condition is weaker than requiring the variational distribution to be close in KL divergence or all posterior means to be accurate.Source: §3.3 · pp. 8–9

4 Calculation and Uses of Sensitivity

The section develops uses of sensitivities of variational posterior means to estimate covariances and assess local prior sensitivity for MFVB, relying on accurate selected means under perturbations and a true local optimum. It recommends second-order Newton trust-region methods, including a conjugate-gradient variant when the Hessian is slow to compute.

4.1 Covariances for Variational Bayes

  • LRVB uses MFVB mean sensitivities to estimate posterior covariances, including nonzero cross-factor covariances that ordinary mean-field covariances set to zero.The estimate is positive semidefinite and symmetric; when Condition 1 holds exactly for a parameter’s mean, covariance terms involving that parameter are exact.Source: §4.1 · pp. 9–11

4.2 Linear Response Covariances in Previous Literature

  • The paper simplifies and generalizes classical linear-response methods to generic variational approximations and connects them to Bayesian robustness.Earlier work focused on moment-parameterized exponential families, while the current formulation builds on that line of work and Giordano et al. [2015].Source: §4.2 · p. 12

4.3 The Laplace Approximation and Linear Response Covariances

  • The standard Laplace covariance is a linear-response covariance for a point-mass variational family, whose raw covariance is zero; accuracy depends on Condition 1.The authors expect more expressive variational families to satisfy Condition 1 more readily and report testing this intuition in Section 5.Source: §4.3 · p. 13

4.4 Local Prior Sensitivity for MFVB

  • MFVB local prior sensitivity approximates exact posterior sensitivity when Condition 1 holds; for exponential-family priors, it reduces to the LRVB covariance estimate.This can capture cross-component effects missed by uncorrected mean-field covariances; the section also illustrates sensitivity to an LKJ prior’s concentration parameter under an inverse-Wishart variational approximation.Source: §4.4 · pp. 14–15

4.5 Practical Considerations when Computing the Sensitivity of Variational Approximations

  • Sensitivity computation can be costly because it requires Hessian calculations and linear solves, but sparsity, automatic differentiation, and conjugate gradients can help.Global/local models yield sparse Hessian structure; conjugate gradients can approximate Hessian-inverse products without forming the full Hessian, though a separate solve is needed for each column of g_η^T.Source: §4.5 · pp. 16–17
  • Sensitivity estimates may be unreliable without a true local optimum, so the authors recommend second-order Newton trust-region fitting over coordinate ascent or BFGS, which may fail to find one.When the Hessian is slow to compute, conjugate-gradient trust-region optimization uses automatic-differentiation Hessian-vector products without forming or inverting the full Hessian.Source: §4.5 · p. 17

5 Experiments

The experiments examine LRVB/LR-ADVI covariance and posterior uncertainty estimates, prior-sensitivity calculations, and their computational costs across several examples. They also describe the LR-ADVI implementation and assess the accuracy of linear sensitivity approximations in the examples considered.

5.1 Simple Expository Examples

  • The simple examples define known targets with mixtures of up to three normals, varying skewness, over-dispersion, and correlation, and compare approximations against the truth.The mixture specifies target distributions, not a generative model for data.Source: §5.1 · pp. 17–18

5.1.1 Multivariate Normal Targets

  • With a single multivariate normal target, variational and Laplace means equal the exact posterior mean, and LRVB recovers the exact covariance, unlike diagonal MFVB covariance.Laplace and LRVB covariance estimates coincide for this normal target.Source: §5.1.1 · p. 19

5.1.2 A Univariate Skewed Distribution

  • For skewed univariate targets, MFVB estimates the posterior mean accurately while Laplace places it too far left; the LRVB variance is accurate, unlike Laplace’s underestimate.Tilting increases skew, but the MFVB mean remains accurate; a naive Laplace approximation centered at that mean can imply negative variance.Source: Figure 2 · p. 19, §5.1.2 · p. 20

5.1.3 A Univariate Over-dispersed Distribution

  • For symmetric, over-dispersed univariate targets, MFVB and Laplace means are correct, but tilting makes the target skewed and LRVB variance outperforms Laplace variance.The Laplace mean shifts too little as tilting increases, leading to underestimated variance.Source: §5.1.3 · pp. 20–21, Figure 4 · p. 21, Figure 3 · p. 21

5.1.4 A Bivariate Over-dispersed Distribution

  • In correlated, over-dispersed bivariate targets, LRVB improves on both Laplace and badly underestimated MFVB variance, though it does not fully recover the correct variance.Mean-field underestimates the marginal variance of θ1, while Laplace cannot account for skewness introduced by tilting.Source: §5.1.4 · p. 22, Figure 6 · p. 22, Figure 5 · p. 22

5.2 Automatic Differentiation Variational Inference (ADVI)

  • ADVI uses a factorizing normal approximation and black-box optimization; in the tested examples, Stan’s fullrank version approximated MCMC posterior means worse than meanfield ADVI.ADVI also uses unconstraining transforms, the re-parameterization trick, and stochastic gradients.Source: §5.2 · p. 23

5.2.1 Linear Response for ADVI (LR-ADVI)

  • LR-ADVI directly optimizes its fixed-draw Monte Carlo KL objective because stochastic-gradient optima were insufficiently accurate for sensitivity calculations requiring an objective optimum.Direct optimization enables second-order methods and Hessian evaluation at that objective’s optimum; the examples used M = 10 draws, and the covariance procedure also accommodates nonlinear constraining transforms.Source: §5.2.1 · pp. 25–26

5.2.3 Election Model Accuracy / 5.2.4 Sesame Street Model Accuracy

  • In the Election model, Laplace and ADVI reasonably match MCMC, with LR-ADVI slightly more accurate for standard deviations; in Sesame Street, only LR-ADVI estimates uncertainty accurately.The Sesame Street MAP estimate underestimates random-effect variability, making the Laplace Hessian numerically singular; ADVI underestimates standard deviations.Source: §5.2.3 · p. 27, Figure 7 · p. 27, §5.2.4 · p. 27

5.2.5 Radon Model Accuracy / 5.2.6 Ecology Model Accuracy

  • For Radon, only LR-ADVI accurately approximates MCMC standard deviations; for Ecology, it remains most accurate despite some errors when ADVI means are inaccurate.The Laplace covariance is numerically singular for Radon and degenerate for Ecology.Source: §5.2.5 · p. 28, §5.2.6 · p. 28, Figure 9 · p. 29

5.2.7 Timing Results

  • LR-ADVI is typically about five times faster than MCMC in the Stan experiments, while remaining slower than ADVI alone and the Laplace approximation.The trust-region optimization accounted for an appreciable share of LR-ADVI runtime.Source: §5.2.7 · p. 28, Figure 11 · p. 31

5.3 Criteo Dataset / 5.3.1 Data and Model

  • The illustrative application evaluates posterior covariances and prior sensitivities in a logistic GLMM for binary conversions, using five numeric covariates and advertiser identity-linked random effects.The subsample included 61,895 rows across 5,000 advertisers, with no more than 20 rows per advertiser.Source: §5.3 · p. 29, §5.3.1 · pp. 30–31

5.3.2 Inference and Timing / 5.3.3 Posterior Approximation Results

  • The Criteo MFVB fit took 57 seconds, roughly 370 times faster than MCMC; sparse Hessian construction and solving nevertheless took 323 and 173 seconds, respectively.A conjugate-gradient alternative took 9.4 seconds; the experiment used MCMC as ground truth for posterior means and covariances.Source: §5.3.2 · pp. 32–33, §5.3.3 · p. 34

5.3.4 Posterior Means

  • MFVB closely approximates Criteo posterior means, whereas MAP performs poorly for random effects; LRVB variances match the exact posterior closely.MCMC mixing was poor for β1 and µ, making their ground-truth estimates somewhat suspect.Source: §5.3.4 · pp. 34–35, Table 2 · p. 34, Figure 13 · p. 35

5.3.5 Posterior Covariances

  • LRVB marginal variances closely match the exact posterior, and its off-diagonal covariances are quite accurate, unlike the uncorrected MFVB estimates.The uncorrected MFVB variances for β are particularly inaccurate, and its off-diagonal covariances are zero by definition.Source: §5.3.5 · p. 36, Figure 15 · p. 37, Figure 16 · p. 37

5.3.6 Parametric Sensitivity Results

  • MFVB normalized prior sensitivities correspond very closely to MCMC, though MFVB means appear slightly more sensitive to prior parameters.The comparison assumes MFVB and MCMC posterior means continue to match to first order near the original prior parameters.Source: §5.3.6 · pp. 37–38, Figure 17 · p. 38
  • −0.39 and −0.35 standard deviations are the expected changes in Epα [µ] and Epα [β1], respectively, for each unit increase in τµ.The passage notes that this could be practically significant if such a change in τµ is subjectively plausible.Source: §5.3.6 · p. 38
  • Even when changes in τµ shift Epα [µ] and Epα [β1] by far more than two standard deviations, the linear approximation holds up reasonably well in this example.A randomly selected random effect is also sensitive, but not to a practically important degree relative to its posterior standard deviation; Epα [β2] is insensitive.Source: §5.3.6 · p. 38
  • The linear approximation’s accuracy is not guaranteed to hold as well in general, and evaluating its linearity reliably without refitting remains future work.Source: §5.3.6 · pp. 38–39
  • 27.2 seconds was the average time for each new MFVB fit after initializing optimization near the perturbed optimum, whereas repeatedly re-estimating the MCMC posterior would have been extremely time-consuming.The passage also notes that importance sampling would be useless for prior changes that moved the posterior far from the original draws.Source: §5.3.6 · p. 39

6 Conclusion

The paper uses MFVB posterior-mean sensitivity to model perturbations to provide improved variance estimates and prior-robustness measures, with accurate estimates calculated much faster than MCMC in its experiments.

6 Conclusion

  • The experiments report accurate posterior means, covariances, and prior sensitivity measures, calculated orders of magnitude faster than MCMC.The methods are fast and scalable when MFVB models use software supporting automatic differentiation, with little additional coding beyond the MFVB objective.Source: §6 · p. 39
  • MFVB posterior-mean sensitivity to model perturbations provides improved variance estimates and measures of prior robustness.These are two practical tools for MFVB posterior approximations.Source: §6 · p. 39
  • The approach requires little additional coding beyond the MFVB objective when the model is implemented with automatic differentiation.Under that condition, the methods are also described as fast and scalable.Source: §6 · p. 39

A Proof of Theorem 1

The appendix justifies the derivative–covariance relationship under regularity assumptions and shows that MCMC importance-sampling sensitivity estimates match direct sample covariance estimates.

A Proof of Theorem 1

  • Under Assumption 1, the proof exchanges differentiation and integration, relying on the Lebesgue dominated convergence theorem.The proof then uses these facts to compute the result.Source: A Proof of Theorem 1 · p. 40

B Comparison With MCMC Importance Sampling

  • With IID draws from p0(θ), importance sampling estimates local sensitivity in Eq. (1) equivalently to estimating covariance in Eq. (4) directly.This equivalence is stated under Assumption 1.Source: B Comparison With MCMC Importance… · p. 40
  • At α0, the normalized importance weights recover the ordinary sample mean, and their derivatives give the sample version of the covariance in Theorem 1.The passage identifies the weight derivatives as precisely that sample covariance.Source: B Comparison With MCMC Importance… · p. 41

C Our Use of the Terms “Sensitivity” and “Robustness”

The section treats local sensitivity as a measurable quantity that informs, but does not determine, robustness judgments, whose interpretation depends on plausible perturbations and the use of the posterior mean.

C Our Use of the Terms “Sensitivity” and “Robustness”

  • Sensitivity is a measurable quantity, while robustness is a subjective judgment informed by sensitivity and other considerations.The plausible set of prior changes and the practical importance of a resulting mean change require judgment.Source: C Our Use of the Terms “Sensitivity”… · p. 41
  • A prior sensitivity swamped by posterior standard deviation may be negligible, while a larger change may still be harmless if it does not affect an actionable decision.Intermediate cases remain a matter of judgment.Source: C Our Use of the Terms “Sensitivity”… · p. 42
  • Local sensitivity approximates global robustness only when the posterior expectation is roughly linear over a sufficiently small perturbation set.This limitation is inherent to the local approach, though the measures may still provide intuition about possible model failure modes.Source: C Our Use of the Terms “Sensitivity”… · p. 42
  • A one-size-fits-all robustness summary is not prescribed because plausible parameter scaling and perturbations vary by application and require subjective judgment.The local sensitivity itself is mathematically well-defined, but its relationship to robustness remains a matter of judgment.Source: C Our Use of the Terms “Sensitivity”… · p. 43

D Proof of Theorem 2

Theorem 2’s proof differentiates the variational optimum with respect to hyperparameters by applying first-order conditions and the implicit function theorem under stated regularity and strict-optimality assumptions.

D Proof of Theorem 2

  • The proof defines KL(η, α) as the divergence from q(θ; η) to pα(θ), then uses optimality and differentiability assumptions to obtain first-order conditions.The optimum η*(α) is assumed interior and optimal over the specified hyperparameter set.Source: D Proof of Theorem 2 · p. 43
  • Strict optimality makes the relevant second-derivative matrix invertible, allowing the first-order condition to be differentiated with respect to α.Continuity assumptions and the implicit function theorem establish that η*(α) is continuously differentiable.Source: D Proof of Theorem 2 · pp. 43–44
  • The proof applies the chain rule to the posterior expectation Eqα[g(θ)] through the variational optimum η*(α).The differentiability assumptions on the expectation and objective support this step.Source: D Proof of Theorem 2 · pp. 43–44

E Exactness of Multivariate Normal Posterior Means

For a full-rank multivariate normal posterior, MFVB recovers exact posterior means, and linear response recovers the exact posterior covariance. The argument uses linear perturbations, and approximate normality may also arise under a Bayesian central limit theorem.

E Exactness of Multivariate Normal Posterior Means

  • MFVB posterior means are exact for a multivariate normal with full-rank covariance, and its linear response covariance equals the exact posterior covariance.The result is established for perturbations of the form ρ(θ, α) = α⊺θ.Source: E Exactness of Multivariate Normal… · pp. 44–46
  • A multivariate normal posterior can arise from a known-covariance normal likelihood with a conjugate normal prior, and approximate normality may hold under a Bayesian central limit theorem.The passage gives these as examples of settings relevant to the result.Source: E Exactness of Multivariate Normal… · p. 44
  • The exactness result applies to finite linear perturbations of the posterior, for which the MFVB posterior mean satisfies the stated condition with equality.The perturbation is ρ(θ, α) = α⊺θ with α0 = 0.Source: E Exactness of Multivariate Normal… · p. 46

F ADVI Model Details

This appendix section supplies Stan code for the ADVI models; its election model uses state effects and covariates in a Bernoulli-logit specification.

F ADVI Model Details

  • The appendix provides Stan code for the models used in Section 5.2 and directs readers to the Stan manual for model interpretation and unconstraining transforms.The associated data are referenced in the Stan example models wiki.Source: F ADVI Model Details · p. 46

F.1 Election Model (election88.stan)

  • The election model predicts outcomes from black and female indicators plus state effects, with a Bernoulli-logit likelihood.The code also specifies normal priors for the coefficients and state effects.Source: §F.1 · p. 47

F.2 Sesame Street Model (sesame street1)

The appendix gives model-code details for Sesame Street, radon, and ecology applications, then derives closed-form expressions for an LKJ prior with a Wishart variational approximation.

F.2 Sesame Street Model (sesame street1)

  • The model defines two linked outcomes: outcome 2’s predictor combines a site-indexed term and z, while outcome 1’s predictor combines a site-indexed term with outcome 2.The code sets Sigma_yt[1,1] to sigma_y squared and Sigma_yt[1,2] to rho_yt times sigma_y times sigma_t.Source: §F.2 · p. 48

F.3 Radon Model (radon vary intercept floor)

  • The radon model combines county-level intercepts and two predictors in a normal outcome model.The code gives the county intercepts a normal distribution and specifies a normal likelihood.Source: §F.3 · p. 49

F.4 Ecology Model (cjs cov randeff)

  • The ecology model uses capture histories and models survival and detection with Bernoulli terms.The code also includes a covariate for survival and generated yearly survival estimates.Source: §F.4 · pp. 50, 52

G LKJ Priors for Covariance Matrices in Mean Field Variational Inference

H Logistic GLMM Model Details

The appendix supplies additional model-analysis details, including a log likelihood and a convention for constants. It uses four-point Gauss–Hermite quadrature and reports no measurable change from increasing the number of points.

H Logistic GLMM Model Details

  • The appendix presents the log likelihood and uses “Constant” for terms independent of the prior parameters, parameters, or data.The appendix also states that prior parameters were selected, without specifying them in the passage.Source: H Logistic GLMM Model Details · p. 54
  • Under the variational approximation, ρit is normally distributed given xit.The supplied passage does not state further distributional details.Source: H Logistic GLMM Model Details · p. 55
  • Using four Gauss–Hermite quadrature points, increasing the point count did not measurably change the results.The integration points and weights were calculated with numpy.polynomial.hermite.Source: H Logistic GLMM Model Details · p. 55

Related papers

Loading 1709.02536v3…