Source-linked AI summary
Projected Stein Variational Gradient Descent
Peng Chen, Omar Ghattas
TL;DR
High-dimensional Bayesian inference is challenged by rapidly growing computational complexity, and kernel-based SVGD also loses accuracy in high dimensions. The paper constructs a data-informed low-dimensional projection from likelihood-gradient information and applies SVGD to its coefficients. It reports improved accuracy over SVGD and scalability across several problem sizes and computational axes.
Problem
Method
pSVGD adaptively constructs a low-dimensional data-informed subspace using log-likelihood gradient information and transports the projected parameter coefficients with SVGD.
Source: §1 · p. 2, §3.1 · p. 3Results
pSVGD is reported to be more accurate than SVGD and scalable with respect to parameters, samples, data points, and processor cores.
Source: §1 · p. 2, §4.1 · p. 7, §4.3 · p. 8Takeaways & Limitations
For the tested high-dimensional inference problems, pSVGD provides a scalable approach that uses a low-dimensional subspace to address the curse of dimensionality.
Source: §5 · p. 9The practical construction assumes that the posterior effectively differs from the prior in the dominant subspace, with negligible likelihood variation in its complement.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
The curse of dimensionality is a longstanding challenge in Bayesian inference in high dimensions. In this work, we propose a projected Stein variational gradient descent (pSVGD) method to overcome this challenge by exploiting the fundamental property of intrinsic low dimensionality of the data informed subspace stemming from ill-posedness of such problems. We adaptively construct the subspace using a gradient information matrix of the log-likelihood, and apply pSVGD to the much lower-dimensional coefficients of the parameter projection. The method is demonstrated to be more accurate and efficient than SVGD. It is also shown to be more scalable with respect to the number of parameters, samples, data points, and processor cores via experiments with parameters dimensions ranging from the hundreds to the tens of thousands.
1 Introduction
High-dimensional Bayesian inference remains difficult because computational costs can grow rapidly with parameter dimension, while existing deterministic methods can depend on smoothness or sparsity. The paper introduces pSVGD, which exploits a low-dimensional data-informed subspace and develops adaptive, parallel algorithms for it.
Motivation and related work
- Existing high-dimensional approaches face trade-offs: MCMC remains costly for expensive likelihoods, while sparse quadrature can deteriorate without smooth or sparse posteriors.Kernel-based Stein methods also lose convergence and accuracy at high dimensions; localized SVGD and parameter projection are cited as partial remedies.Source: §1 · pp. 1–2
Contributions
- pSVGD projects parameters onto a low-dimensional subspace built from log-likelihood gradients, then transports projection coefficients toward their posterior.The authors also prove equivalence between coefficient-space projected transport and transport in projected parameter space.Source: §1 · p. 2
- The method targets a posterior that effectively differs from the prior only in a low-dimensional subspace, a property associated with ill-posed or over-parameterized problems.The paper reports accuracy comparisons with SVGD and scalability tests across parameters, samples, data points, and processor cores.Source: §1 · p. 2
- Compared with pSVN, pSVGD uses likelihood-gradient information rather than Hessians, has a sharper projection-error upper bound, and examines additional benchmarks and scalability axes.The paper notes that Hessians can be challenging for complex models and practical codes.Source: §1 · p. 2
2 Preliminaries
Bayesian inference seeks posterior samples and statistical quantities from observed data, while SVGD approximates the posterior by transporting prior samples. Its kernel representation can itself become a high-dimensional bottleneck.
Bayesian inference
- Bayesian inference updates a prior over unknown parameters using observed data to obtain a posterior, from which samples and quantities such as means or variances are sought.The posterior normalization constant is typically intractable, especially at large dimension.Source: §2 · p. 2
Stein variational gradient descent
- SVGD approximates the posterior by minimizing KL divergence over a function set of transported priors, using an invertible transport map.The practical method uses a tensor-product RKHS and updates samples through a kernel-based rule.Source: §2 · pp. 2–3
- At high dimension, the SVGD kernel suffers from the curse of dimensionality, which can make samples unrepresentative of the posterior.A Gaussian kernel is one stated choice, with bandwidth based, for example, on median sample distances.Source: §2 · p. 3
3 Projected Stein Variational Gradient Descent
pSVGD addresses high-dimensional inference by focusing transport on a low-dimensional subspace where the posterior differs from the prior. It constructs that subspace from likelihood-gradient information, transports coefficients with SVGD, and adaptively approximates the required quantities for practical parallel computation.
3 Projected Stein Variational Gradient Descent
- The method exploits the premise that, in many ill-posed or over-parameterized problems, the posterior effectively differs from the prior only in a relatively low-dimensional subspace.Likelihood variation in the complement is treated as negligible.Source: §3 · p. 3
3.1 Dimension reduction by projection
- Generalized eigenvectors of the gradient information matrix rank parameter directions by data sensitivity, and small eigenvalues indicate negligible likelihood variation.The rank-r projection retains dominant directions; under stated prior and likelihood assumptions, posterior projection error is bounded by the truncated eigenvalues and is small when they decay rapidly.Source: §3.1 · pp. 3–4
3.2 Projected Stein Variational Gradient Descent
- pSVGD applies SVGD to the lower-dimensional projection coefficients, with a kernel distance weighted by the likelihood eigenvalues and the prior.Under specified conditions, coefficient-space transport is equivalent to transport of the projected parameter.Source: §3.2 · pp. 4–5
3.3 Practical algorithms
- Practical construction is challenging because the profile function and gradient information matrix involve high-dimensional integrals, including posterior expectations.The algorithm approximates these quantities adaptively using transported samples and freezes complement-space samples under the assumption that likelihood variation there is small.Source: §3.3 · p. 5
- The parallel algorithm distributes samples across processor cores, gathers information with MPI, and updates the adaptive projection basis as transport proceeds.With K cores and N samples per core, the implementation produces M = NK samples in total.Source: §3.3 · pp. 5–6
4 Numerical Experiments
The experiments compare pSVGD with SVGD on conditional diffusion, high-dimensional logistic regression, and a PDE benchmark. They report improved accuracy in the first two examples and accurate PDE approximation with scalability across several problem sizes.
4.1 Conditional diffusion process
- At iteration 100, pSVGD’s posterior mean is closer to the true parameter and its tighter 90% confidence interval covers the truth better than SVGD’s.The figure also compares their samples and solutions against the synthetic truth and noisy observations.Source: §4.1 · pp. 6–7, Figure 1 · p. 7
4.2 Bayesian logistic regression
- On the 10,000-attribute cancer-classification task, pSVGD’s prediction is more accurate than SVGD’s, with a rapidly decaying eigenvalue spectrum indicating a dominant low-dimensional subspace.The gradient-information estimate uses 32 samples; more samples produce a similar eigenvalue decay.Source: Figure 2 · p. 7, §4.2 · p. 7
4.3 Partial differential equation
- About 50 of 289–16,641 dimensions are retained with relative projection error Er < 10^-6, including over 300× reduction at d = 16,641.The PDE benchmark reports about seven orders of magnitude of eigenvalue decay.Source: Figure 3 · p. 8, §4.3 · p. 8
- pSVGD scales with parameter dimension and sample count, weakly with data-point growth, and achieves near O(K^-1) CPU-time decay across processor cores.For the core-scaling comparison, the reported work fixes KN = 1024 samples.Source: Figure 4 · p. 8, §4.3 · p. 8
5 Conclusions
The conclusion presents pSVGD as a high-dimensional Bayesian inference method whose posterior projection error is bounded by truncated eigenvalues. It reports equivalence to SVGD on projected parameters under suitable assumptions and scalability across four problem-size axes in a benchmark.
5 Conclusions
- pSVGD bounds posterior projection error using the truncated, fast-decaying eigenvalues.Under suitable assumptions, pSVGD for the coefficients is equivalent to SVGD for the projected parameter.Source: §5 · p. 9
A Proof of Theorem 1
The proof relates transport on the projected parameter to transport on its reduced coefficients. It establishes this equivalence under the stated prior condition and by matching the projected kernels and Stein operators.
A Proof of Theorem 1
- The proof sets up the projected parameter x_r and its prior and posterior before expressing its transport map in a steepest direction.The parameter is decomposed as x = x_r + x⊥, with x_r = Prx.Source: A Proof of Theorem 1 · p. 11
- The projected kernel and Stein operators are related through the projection, providing the identities used to compare the two transports.The proof uses κ(x_r, x̃_r) = k(Prx, Prx̃) and the equivalence of the Stein operators.Source: A Proof of Theorem 1 · pp. 11–12
- With the stated prior condition, the proof establishes equivalence of the coefficient and projected-parameter transports, including their steps by induction.The condition identifies the projected prior π0(w) with p0(Prx).Source: A Proof of Theorem 1 · p. 12
B A linear inference problem
This linear inference example has an analytically available Gaussian posterior, allowing sample variances from SVGD and pSVGD to be checked against ground truth. As dimension increases, SVGD deteriorates while pSVGD performs well across the tested dimensions.
B A linear inference problem
- The analytic Gaussian posterior provides the ground truth for assessing the accuracy of the proposed algorithm on a linear inference problem.The parameter-to-observable map is linear, and the posterior covariance is given in equation (60).Source: B A linear inference problem · pp. 12–13
- The parameter is a finite-element discretization of a diffusion-reaction equation, with a Gaussian prior whose covariance discretizes (−0.1∆ + I)⁻¹.The continuous parameter and solution use piecewise linear elements on a uniform mesh of size d, with boundary conditions u(0) = 0 and u(1) = 1.Source: B A linear inference problem · p. 12
- The observation map samples s equally spaced components of the solution, and the example assumes additive 1% Gaussian observation noise.For s = 15, the observations are u(1/16) through u(15/16).Source: B A linear inference problem · p. 12
- As dimension increases, SVGD deteriorates in sample-variance RMSE while pSVGD performs well across the tested dimensions.The comparison uses 256 samples, 200 iterations, and parameter dimensions d = 2n + 1 for n = 4, 6, 8, 10.Source: B A linear inference problem · p. 13, Figure 5 · p. 13
C Application in COVID-19
The COVID-19 application uses pSVGD to infer time-varying social-distancing contact reduction from New York hospitalization data. Compared with SVGD, pSVGD gives more accurate hospitalization predictions and reduction-factor estimates closer to the deterministic optimum.
C Application in COVID-19
- The model has eight compartments for COVID-19 transmission and outcomes, and the inferred time-dependent α(t) represents the effective contact reduction from social distancing.Hospitalized cases in New York, using a seven-day moving average, provide the observation data; the log-data observation noise is assumed i.i.d. N(0, 1).Source: C Application in COVID-19 · pp. 13–14, Figure 6 · p. 14
- The procedure first optimizes the model parameters against the logarithm of observed hospitalizations, then freezes all parameters except α(t).The remaining social-distancing parameter is inferred using pSVGD and SVGD.Source: C Application in COVID-19 · pp. 14–15
- pSVGD predicts hospitalized cases more accurately with tighter confidence intervals than SVGD, and its reduction-factor mean is closer to the deterministic optimum.The pSVGD 90% confidence interval covers the deterministic optimum, whereas SVGD’s does not; SVGD’s reduction factor is nearly 1 for a period, corresponding to complete shutdown without transmission.Source: C Application in COVID-19 · p. 15, Figure 7 · p. 15
- The 96-dimensional daily parameter uses a Gaussian-process prior centered on the transformed deterministic optimum, with pSVGD updating its bases every 10 iterations.The experiment uses 128 samples across 16 processor cores and runs for 200 iterations; Figure 7 reports fast eigenvalue decay and a small intrinsic dimension.Source: C Application in COVID-19 · p. 15, Figure 7 · p. 15