Source-linked AI summary
Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature
Tom Gunter, Michael A. Osborne, Roman Garnett, Philipp Hennig, Stephen J. Roberts
TL;DR
Probabilistic inference depends on numerical integration, while Bayesian quadrature’s computational cost has limited its usefulness when likelihood evaluations are cheap. This paper introduces WSABI, using a square-root likelihood model and active sampling; it reports faster wall-clock convergence than Monte Carlo benchmarks on regression and classification tasks.
Problem
Bayesian quadrature offers model-based integration, but its computational overhead can outweigh its sampling benefits when likelihood evaluations are cheap.
Source: §1 · p. 2Method
WSABI places a GP on the square root of the non-negative integrand and samples where expected posterior variance after the quadratic transform is greatest.
Source: §1 · p. 2Results
WSABI demonstrates faster wall-clock convergence for regression and classification benchmarks than the Monte Carlo state-of-the-art.
Source: §6 · p. 8Takeaways & Limitations
The authors present WSABI as a fast Bayesian quadrature scheme intended to remain useful even when likelihood evaluations are inexpensive.
Source: §1 · p. 2, §6 · p. 8The square-root GP gives each likelihood a non-central χ² marginal, leaving the integral posterior non-closed-form and requiring approximation schemes.
Source: §3 · p. 3
Abstract
from arXiv · showhide
We propose a novel sampling framework for inference in probabilistic models: an active learning approach that converges more quickly (in wall-clock time) than Markov chain Monte Carlo (MCMC) benchmarks. The central challenge in probabilistic inference is numerical integration, to average over ensembles of models or unknown (hyper-)parameters (for example to compute the marginal likelihood or a partition function). MCMC has provided approaches to numerical integration that deliver state-of-the-art inference, but can suffer from sample inefficiency and poor convergence diagnostics. Bayesian quadrature techniques offer a model-based solution to such problems, but their uptake has been hindered by prohibitive computation costs. We introduce a warped model for probabilistic integrands (likelihoods) that are known to be non-negative, permitting a cheap active learning scheme to optimally select sample locations. Our algorithm is demonstrated to offer faster convergence (in seconds) relative to simple Monte Carlo and annealed importance sampling on both synthetic and real-world examples.
1 Introduction
Probabilistic inference often requires numerical integration, but Bayesian quadrature’s computational overhead can outweigh its sampling benefits when likelihood evaluations are cheap. The paper introduces a square-root likelihood model and an active sampling strategy, and compares WSABI with standard Monte Carlo methods under fixed compute budgets.
1 Introduction
- Inference requires numerical integration over non-negative likelihoods and priors, including for model evidence, predictive test likelihoods, and marginalization over parameters or variables.Source: §1 · p. 1
- Monte Carlo estimators may use little prior information about likelihood surfaces; their convergence diagnostics for partition functions can be unreliable, and AIS can perform poorly with misspecified tuning parameters.Source: §1 · p. 1
- Bayesian quadrature places a GP prior over likelihood functions and can use active sampling to improve sample efficiency over naive randomized sampling.Source: §1 · p. 2
- Bayesian quadrature’s model and active-sampling costs can erase its effective-sample-rate advantage when likelihood evaluations are cheap.The paper aims to make BQ cheap and accurate enough to remain useful in that setting.Source: §1 · p. 2
- The square-root warp models the square root of the integrand with a GP, enforcing non-negativity and addressing the high dynamic range of likelihoods without the approximation used for log-likelihood modeling.Earlier BQ work placed a GP directly on the likelihood without enforcing non-negativity a priori.Source: §1 · p. 2
- Fast active sampling selects locations where expected posterior variance of the integrand after the quadratic transform is greatest, balancing exploration and exploitation to approximately reduce integral entropy.The cited alternative targets expected variance reduction of the integral itself.Source: §1 · p. 2
- The paper compares WSABI with non-negative standard Monte Carlo techniques on simulated and real examples, evaluating error against ground truth at a fixed compute budget.Source: §1 · p. 2
2 Bayesian Quadrature
Bayesian quadrature uses a model of the integrand to infer both its functional form and the integral from sampled function values. With a GP prior, it can yield analytic integral moments for suitable priors, while sampling strategies can exploit information from observed values.
2 Bayesian Quadrature
- Bayesian quadrature infers both the integrand’s functional form and the integral from sampled points, typically using a Gaussian prior density.An importance-reweighting identity permits integration against another prior density; the discussion focuses on one dimension, with results stated to extend to higher dimensions.Source: §2 · p. 2
- In the GP model, the output length-scale controls output standard deviation and the input length-scale controls the autocorrelation range of function evaluations.Source: §2 · p. 2
- With a GP prior directly on the integrand, Gaussian identities give analytic posterior mean and variance for the integral.Source: §2 · p. 2
- These integral moments are analytic when the prior density is Gaussian, a Gaussian mixture, or a polynomial, among other forms.Source: §2 · p. 3
- For a GP placed directly on the integrand, information-gain-optimal sample locations depend only on previous locations and can be precomputed, whereas log-likelihood GP sampling depends on observed function values [12].Source: §2 · p. 3
- With 15 grid samples, points equidistant from existing samples are equally informative under Bayes–Hermite quadrature; increasing dimensionality makes exploration harder, motivating sampling that accounts for existing probability mass.Source: §2 · p. 3, Figure 1 · p. 3
3 Square-Root Bayesian Quadrature
The section models non-negative likelihoods by placing a Gaussian process prior on their square roots, then approximates the resulting intractable integral posterior. It develops linearisation and moment-matching approximations and uses them to obtain closed-form integral estimates.
3 Square-Root Bayesian Quadrature
- The square-root transform preserves likelihood non-negativity, halves the modeled function’s dynamic range, and extends the GP input length-scale for better extrapolation.Likelihoods are non-negative; the transform helps address their large variations.Source: §3 · p. 3
- Because the square-root model yields a non-central χ2 marginal and no closed-form integral posterior, the section introduces approximation schemes.The transformed likelihood has a GP prior, but its induced likelihood process makes direct integral inference intractable.Source: §3 · p. 3
3.1 Linearisation
- Linearisation approximates the square transform around the posterior mode ˜m_D, making the likelihood a local affine transform of the transformed process.The transform’s derivative is ˜ℓ.Source: §3.1 · p. 4
3.2 Moment Matching
- Moment matching approximates the likelihood posterior as a GP with the true χ2 process posterior’s mean and covariance.On synthetic data, the two approximations largely agree with each other and the true density, while moment matching has higher mean and variance far from observations.Source: §3.2 · p. 4
3.3 Quadrature
- Under either approximation, the posterior mean and covariance are mixtures of un-normalised Gaussians, yielding closed-form integral mean and variance expressions.The quadratic transform’s gradient helps make the linearised covariance a Gaussian mixture.Source: §3.3 · p. 4
- With identical samples, WSABI fits the true function more closely and assigns minimal probability mass to negative values.Both properties arise from the square-root transform.Source: §3.3 · p. 4
4 Active Sampling
The section frames active sampling as choosing the next function evaluation to reduce uncertainty about the integrand surface or its integral. It considers minimizing expected integral entropy and uncertainty sampling over the total integrand.
4 Active Sampling
- Active sampling selects the next function evaluation to reduce uncertainty about either the integrand surface or the integral.Both utility choices are presented as applicable to either approximation of p(ℓ).Source: §4 · pp. 4–5
4.1 Minimizing expected entropy
- Minimizing expected entropy of the integral is one proposed criterion for selecting the next sample location.The section also contrasts this target with reducing entropy of the total integrand.Source: §4.1 · p. 5
4.2 Uncertainty sampling
- Uncertainty sampling targets entropy reduction for the total integrand, with computation described as cheaper and more numerically stable than minimizing integral entropy.The square-root model balances sampling unexplored regions against regions where likelihood is expected to be high; Figure 3 shows a non-zero expected mean everywhere, while Figure 4 shows zero mean away from samples.Source: §4.2 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
- The selected samples are reported to provide a good integral estimate and be more informative than a grid under the utility for minimizing integral error..Source: §4.2 · p. 5
- Under uncertainty sampling, the moment-matched approximation has a non-zero expected mean everywhere, whereas the linearised model has zero expected mean away from samples.The figures show square-root Bayesian quadrature after 15 sequentially selected samples under utility functions (14) and (15), respectively.Source: §4.2 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
5 Results
The experiments compare WSABI with Monte Carlo and other quadrature methods under matched compute budgets across synthetic likelihoods and GP regression and classification. Results favor WSABI on convergence speed in most reported cases, while WSABI-M performs slightly better on the synthetic mixture surfaces and WSABI-L is fastest across the other test cases.
5.1 Synthetic Likelihoods
- For this likelihood, whose evaluation cost was effectively zero, SMC averaged just under 86,000 samples per likelihood, while WSABI was on average faster to converge to 10^-3 error.Figure 5 reports that WSABI-M performed slightly better.Source: §5.1 · p. 6, Figure 5 · p. 7
5.2 Marginal Likelihood of GP Regression
- On yacht hydrodynamics GP regression, WSABI converged near ground truth faster than the other methods, while AIS gained little from additional compute and BMC performed worst.The eight-dimensional hyperparameter integral was evaluated under matched compute time; Figure 7 shows the convergence comparison.Source: §5.2 · p. 7, Figure 7 · p. 7
5.3 Marginal Likelihood of GP Classification
- The GP classification experiments marginalised hyperparameters for a synthetic one-dimensional dataset and a real citation-network dataset with four-dimensional inputs and 500 examples.Both used a probit likelihood and Laplace approximation for function values.Source: §5.3 · p. 7
5.4 Synthetic Binary Classification Problem
- On synthetic binary classification, WSABI-L nearly immediately reached ground truth and achieved a tighter long-run bound than SMC; both square-root variants were more accurate and lower-variance than BMC.BMC performed well in this example, while active sample selection under BBQ prevented rapid convergence.Source: §5.4 · p. 8
5.5 Real Binary Classification Problem
- The real classification experiment used a CiteSeerx citation-network subset, representing papers from 48 venues and subsampling 500 examples, half positive.The four input dimensions were the first four graph principal components.Source: §5.5 · p. 8
- Across test cases, WSABI-L was fastest except on synthetic mixture surfaces, where WSABI-M performed slightly better; WSABI-M typically fared worse than WSABI-L as dimensionality increased.The authors attribute WSABI-M's relative performance to increased exploration and suggest more aggressive exploitation for real likelihoods.Source: §5.5 · p. 8
6 Conclusions
The paper introduces WSABI, a fast Bayesian quadrature scheme that combines a warped likelihood model with active sampling. On regression and classification benchmarks, it converges faster in wall-clock time than Monte Carlo state-of-the-art methods.
6 Conclusions
- WSABI combines a novel warped likelihood model and active sampling, and converges faster in wall-clock time than Monte Carlo state-of-the-art methods on regression and classification benchmarks.The authors describe it as the first fast Bayesian quadrature scheme.Source: §6 · p. 8