Source-linked AI summary
Elliptical slice sampling
Iain Murray, Ryan Prescott Adams, David J. C. MacKay
TL;DR
Probabilistic models often use multivariate Gaussian priors or Gaussian processes to represent strong dependencies, but inference can be difficult and Gibbs sampling can perform poorly. The paper introduces elliptical slice sampling, a generic MCMC method that removes tuning parameters by adapting its step size during each iteration. It reports empirical success across GP-based models, with performance comparable to the best possible related Metropolis–Hastings scheme and broader practical applicability.
Problem
Inference is difficult for models with strongly correlated Gaussian variables, while Gibbs sampling can perform poorly and Metropolis–Hastings requires appropriately tuned step sizes.
Method
Elliptical slice sampling samples along ellipses defined by the Gaussian prior and likelihood, shrinking an angle bracket until an acceptable state is found without rejections.
Results
Elliptical slice sampling achieved more effective samples than Neal’s Metropolis–Hastings method at its best step size, while performance varied by data type and implementation.
Takeaways & Limitations
The algorithm is a simple generic sampler with no tweak parameters that can serve as a drop-in replacement across a wide variety of Gaussian-model applications.
Takeaways & Limitations
Covariances with large constant terms can produce poorly conditioned models and ineffective auxiliary draws, so large offsets should instead be modeled as separate variables.
Abstract
from arXiv · showhide
Many probabilistic models introduce strong dependencies between variables using a latent multivariate Gaussian distribution or a Gaussian process. We present a new Markov chain Monte Carlo algorithm for performing inference in models with multivariate Gaussian priors. Its key properties are: 1) it has simple, generic code applicable to many models, 2) it has no free parameters, 3) it works well for a variety of Gaussian process based models. These properties make our method ideal for use while model building, removing the need to spend time deriving and tuning updates for more complex algorithms.
1 Introduction
Multivariate Gaussian distributions and Gaussian processes express dependencies among latent variables in probabilistic models. Because exact inference is limited and MCMC is more general, the paper introduces elliptical slice sampling as a simpler, often faster approach for strongly correlated Gaussian models.
- Motivation: Multivariate Gaussian distributions encode prior dependencies among latent variables, either directly or through Gaussian-process marginals.Gaussian processes can represent spatial or temporal coherence and support Bayesian kernel methods for non-parametric regression and classification.
- Motivation: Closed-form inference is limited to the simplest Gaussian latent-variable models, motivating approximate deterministic methods and MCMC.MCMC represents joint posterior distributions with samples and applies more generally, although it can be slower.
- Contribution: Elliptical slice sampling targets strongly dependent Gaussian models with simple generic code, no preliminary tuning, and often faster performance than competing methods.The method is presented as a drop-in replacement for MCMC samplers using Gibbs or Metropolis–Hastings and is evaluated on several GP-based likelihood models.
2 Elliptical slice sampling
Elliptical slice sampling targets posteriors combining a multivariate Gaussian prior with a likelihood, using slice sampling to adapt proposals on an ellipse without free tuning parameters. The resulting Markov chain preserves the target posterior and is especially suited to strongly dependent Gaussian-prior models.
- Target distribution: The sampler targets a posterior proportional to a multivariate Gaussian prior times a likelihood over latent variables.The latent state is f, with Gaussian covariance Σ and likelihood L(f).
- Motivation: Neal’s Metropolis–Hastings approach requires an appropriately chosen step size, motivating automatic step-size adaptation.The required step size can vary as covariance and likelihood parameters are inferred.
- Algorithm: Elliptical slice sampling draws an auxiliary Gaussian vector, searches an ellipse through the current state, and shrinks an angle bracket until an acceptable proposal is found.The bracket shrinks exponentially, so the effective step size adapts on each iteration for the current auxiliary draw and covariance.
- Algorithm: The method has no rejections in its final slice-sampling update, returning a new state unless the current state is the only point on the ellipse with non-zero likelihood.The proposal angle is sampled from a bracket that is automatically reduced after rejected proposals.
- Computational considerations: For many high-dimensional models, the extra computational cost over Neal’s Metropolis–Hastings method is small because evaluating the likelihood costs O(N).In these models, drawing the auxiliary variate is the dominant update cost.
- Validity: Reversibility, unit-Jacobian transformations, and irreducibility establish the target posterior as the unique stationary distribution.Repeated applications from an arbitrary starting point asymptotically produce draws from the target posterior.
3 Related work
The paper situates elliptical slice sampling among Gaussian-prior MCMC methods, replacing poorly mixing coordinate updates and tuned proposals with generic multivariate moves. Related approaches use control variables or alternative slice trajectories, each with distinct computational or tuning trade-offs.
- Metropolis–Hastings and slice sampling: Neal’s Gaussian-prior Metropolis–Hastings update improves on Gibbs sampling but requires an appropriately chosen step size for efficient mixing.Covariance and likelihood parameters may change during inference, potentially requiring different step sizes.
- Multivariate slice updates: Elliptical slice sampling updates all latent variables together, addressing the slow convergence expected from direct univariate slice updates.The method targets correlated Gaussian-prior variables with a multivariate transition.
- Multivariate slice updates: Straight-line slice sampling can leave a thin high-dimensional Gaussian shell faster than an ellipse passing through points within that shell.This provides the paper’s geometric motivation for preferring elliptical trajectories.
- Control variables: Control-variable sampling introduces Gaussian auxiliary variables so the original latent vector retains its marginal prior while individual control variables become easier to move.Proposals resample one control variable conditionally and then resample the latent vector, using Metropolis–Hastings acceptance.
- Control variables: Control-variable updates cost O(MN^2) per iteration after O(N^3) preprocessing, while conditional Gaussian updates can also be applied to selected latent subsets.The conditional formulation permits elliptical slice sampling or another transition operator on each selected group.
4 Experiments
Experiments compare elliptical slice sampling with control-variable and other methods across Gaussian-process classification, point-process, and regression tasks. Results show different sampling behaviors and trade-offs across dimensions, data types, effective samples, and runtime.
- Tasks and setup: The experiments evaluate Gaussian-process methods on classification, log Gaussian Cox process, and Gaussian regression tasks.The study includes USPS classification, mining-disaster counts, and synthetic regression datasets spanning one to ten input dimensions.
- Trace behavior: Elliptical slice sampling and control-variable sampling produce different log-likelihood trace behavior, with rejections occurring only for the control-variable method.Trace plots alone do not reliably determine overall sampling quality.
- Comparative performance: Control-variable sampling clearly outperforms the other methods on the low-dimensional synthetic regression problems.Its performance is comparable to elliptical slice sampling on mining disasters with about 50% less runtime.
- Comparative performance: Control-variable sampling fails to sample in high dimensions and runs exceedingly slowly on USPS classification, preventing meaningful results there.The authors report that more control variables are required in higher-dimensional problems, increasing the cost of that approach.
- Comparative performance: Elliptical slice sampling yields more effective samples than Neal’s Metropolis–Hastings method with its best possible step size, but requires more runtime.On real-data problems it performs better overall, whereas Metropolis–Hastings has more effective samples per unit time on the synthetic problems in this implementation.
- Computational trade-offs: Elliptical slice sampling takes less time than straight-line slice sampling and usually performs better, while slice samplers use more likelihood evaluations than the other methods.The comparison notes that tuning could reduce elliptical slice sampling’s likelihood evaluations and may yield speedups of around ×2.
5 Discussion
The discussion presents elliptical slice sampling as a simple, generic alternative with no tuning parameters. Its broad applicability is balanced by implementation-dependent runtime comparisons and potential weaknesses when covariance or likelihood structure changes substantially.
- Caveats: Runtime comparisons are implementation-dependent, and a single tuned step size or optimized control-variable set may work poorly when Σ and L change dramatically.This scope boundary matters when model parameters vary substantially during inference.
- Conclusion: Elliptical slice sampling has no tweak parameters and can be applied across low- and high-dimensional applications.The authors characterize it as performing similarly to the best possible performance of a related Metropolis–Hastings scheme.