Source-linked AI summary
Slice sampling covariance hyperparameters of latent Gaussian models
Iain Murray, Ryan Prescott Adams
TL;DR
Non-Gaussian observations make covariance-hyperparameter sampling difficult because standard approaches can require careful tuning and mix slowly. The paper introduces robust slice-sampling representations for joint hyperparameter and latent-variable updates, reporting strong performance across the examined problems and data regimes. It also identifies simple prior whitening as a useful fallback when advanced site approximations are unavailable.
Problem
Non-Gaussian observation models make covariance-structure inference difficult, while existing hyperparameter updates can require careful tuning and mix slowly.
Method
The paper develops slice samplers using surrogate-data and post-site representations, alongside prior whitening, to update covariance hyperparameters with latent variables.
Results
The new surrogate-data and post-site representations offer state-of-the-art performance and are the first advanced methods reported as applicable to Gaussian process classification.
Takeaways & Limitations
Prior whitening should receive serious consideration for MCMC hyperparameter inference when site approximations are difficult or advanced methods are inapplicable.
Takeaways & Limitations
The experiments focus on covariances associated with input vectors, although the methods apply to arbitrary positive definite covariance functions parameterized by θ.
Abstract
from arXiv · showhide
The Gaussian process (GP) is a popular way to specify dependencies between random variables in a probabilistic model. In the Bayesian framework the covariance structure can be specified using unknown hyperparameters. Integrating over these hyperparameters considers different possible explanations for the data when making predictions. This integration is often performed using Markov chain Monte Carlo (MCMC) sampling. However, with non-Gaussian observations standard hyperparameter sampling approaches require careful tuning and may converge slowly. In this paper we present a slice sampling approach that requires little tuning while mixing well in both strong- and weak-data regimes.
1 Introduction
The paper targets covariance-hyperparameter inference in latent Gaussian models with non-Gaussian observations, presenting a sampling method designed to be broadly applicable and easy to use.
- Non-Gaussian observation models make inference for covariance parameters difficult, motivating improved Monte Carlo methods.
- The contribution is a robust slice sampler for covariance hyperparameters with few technical requirements and little tuning.
- The model uses Gaussian latent variables with covariance Σθ parameterized by unknown hyperparameters θ and likelihood L(f).
- The methods apply to arbitrary positive definite covariance functions parameterized by θ, while experiments focus on covariances associated with input vectors.
- The joint posterior integrates the likelihood, Gaussian latent prior, and hyperparameter prior, while avoiding new code or tuning for each covariance and likelihood.
2 Markov chain inference
Alternating updates of latent variables and covariance hyperparameters can mix slowly because the variables are strongly coupled or hyperparameter conditionals are narrow. The paper develops reparameterized and joint-update strategies to address this problem.
- Alternating conditional updates leave the joint posterior invariant under mild conditions, but can explore it very slowly.
- Fixed latent-variable updates can be inefficient because latent samples strongly reveal the lengthscale, making P(θ|f) sharp.
- Whitening the prior: Whitening reparameterizes the latent variables so they are independent of hyperparameters under the prior, enabling updates that change both θ and f.
- Whitening the prior: For fixed whitened variables ν, the acceptance rule depends on the likelihood L(f) rather than the Gaussian prior, and prior proposals are always accepted without data.
3 Surrogate data model
The surrogate data model introduces noisy Gaussian observations to guide joint updates of latent variables and covariance hyperparameters. Combined with slice sampling, it avoids careful proposal-scale tuning while bridging weak- and strong-data regimes.
- 3 Surrogate data model: Surrogate Gaussian observations guide joint proposals of hyperparameters and latent variables for non-Gaussian likelihoods.The auxiliary variables are noisy versions of the true latent variables.
- 3.1 Slice sampling: Slice sampling searches an adaptive bracket for scalar hyperparameters and does not require careful tuning of its initial scale.An overly large initial scale shrinks exponentially quickly to an acceptable range; stepping out can address scales that are too small.
- 3 Surrogate data model: Surrogate-data Metropolis–Hastings proposes θ′ and reconstructs f′ from whitened latent variates, then accepts using likelihood, surrogate, prior, and proposal terms.The latent variables are updated automatically to remain plausible under the proposed hyperparameters and surrogate data.
- 3 Surrogate data model: The auxiliary noise covariance Sθ controls the region of latent-variable values considered plausible during updates.The surrogate posterior is conditioned on auxiliary data and softly constrains latent-variable proposals.
- 3.2 The auxiliary noise covariance Sθ: Setting Sθ to limiting values recovers the two baseline algorithms: Sθ=0 gives fixed-latent updates, while Sθ→∞ gives prior whitening.Intermediate noise levels provide a continuum between these weak-data and strong-data limits.
- 3.2 The auxiliary noise covariance Sθ: For factorized likelihoods, site-posterior Gaussian fits set each diagonal auxiliary-noise entry to (v_i^-1−(Σθ)_ii^-1)^-1, with negative values thresholded.Moment matching or a one-dimensional Laplace approximation supplies the site-posterior variance v_i.
4 Related work
The surrogate data sampler is related to robust reparameterization and HMC approaches for jointly updating latent variables and hyperparameters. Its main distinction is centering the surrogate posterior at the current latent state, which removes the need for a point estimate and makes poor noise choices less damaging.
- 4 Related work: HMC can jointly sample real-valued variables, but it is cumbersome to implement and tune in hierarchical latent Gaussian models.The paper contrasts this with its surrogate-data approach.
- 4 Related work: Previous robust reparameterization replaces the likelihood with a Gaussian approximation centered at a point estimate such as the maximum-likelihood latent vector.The resulting approximate posterior uses a fitted covariance and surrogate center.
- 4 Related work: Under Gaussian likelihoods, the earlier reparameterized variables are independent of one another and of the hyperparameters, supporting rapid Markov-chain mixing.For non-Gaussian likelihoods, the approach hopes that Gaussian approximations produce nearly independent parameterizations.
- 4 Related work: Taylor expansions can be ill-defined for probit, logistic, or zero-count Poisson likelihoods, yielding flat or undefined Gaussian approximations.In such cases, the reparameterization may fail to reweight the prior and reduce to prior whitening.
- 4 Related work: The surrogate sampler centers its reparameterization near the current latent variables, avoiding a required maximum-likelihood estimate and remaining workable when Sθ is poorly chosen.This differs from fixed reparameterizations that can perform badly when the true posterior lies in the tails of the Gaussian approximation.
5 Experiments
The experiments compare six Gaussian-process hyperparameter samplers across regression, classification, and two Cox-process data sets. Performance is assessed by effective samples of the complete-data likelihood under three computational cost measures.
- 5 Experiments: Six methods are compared on four data sets: one regression, one classification, and two Cox-process inference problems.The experiments target mixing of covariance hyperparameters while sampling from the joint posterior.
- 5 Experiments: Each configuration uses ten independent chains, 1000 burn-in iterations, and 5000 sampling iterations.Mixing is estimated from the effective number of samples in the complete-data likelihood trace.
- 5 Experiments: The methods include fixed latent functions, prior whitening, site-matched surrogate data, Taylor-based surrogate data, and post-site/post-Taylor variants.The exact applicable set differs by experiment and likelihood model.
- 5 Experiments: For binary classification on Ionosphere, the comparison uses 200 training points, 34 dimensions, a logistic likelihood, and inferred lengthscales and signal variance.Taylor-based methods do not apply because the log-likelihood maximum is at plus or minus infinity.
- 5 Experiments: For Gaussian regression, surrogate site and Taylor methods coincide because both use auxiliary noise Sθ=0.09 I, matching the observation noise.The synthetic data contain 200 input points in 10 dimensions with observation-noise variance 0.09.
- 5 Experiments: Figure 3 reports effective samples per unit cost, with costs measured per likelihood evaluation, covariance construction, and second.Bars show means and standard errors over ten runs and are normalized to the surr-site method within each group.
6 Discussion
The proposed surrogate-data and post-site representations perform strongly across tested problems, while simple prior whitening can substantially improve hyperparameter sampling when advanced methods are unavailable.
- Both -site methods substantially outperformed two baselines on the Ionosphere classification problem, with a slight preference for surr-site.The preference reflects its lower requirement for problem-specific derivations.
- On the synthetic Gaussian regression test, post- and surr-methods performed very similarly rather than showing the expected 2–3× post-method advantage.The slight post-method time advantage was attributed to implementation details and considered unnotable.
- For the mining problem, existing post-taylor and all proposed methods performed well because the Poisson likelihoods were often close to Gaussian.For very fine discretizations, however, site reparameterizations did not always work well.
- The surrogate-data and post-site representations offered state-of-the-art performance and were the first advanced methods applicable to Gaussian process classification.The empirical study used slice sampling because it was easy to implement and use, although the representations could also accompany other MCMC methods.
- Fixing latent variables while updating covariance hyperparameters conditionally can perform exceedingly poorly, whereas prior whitening works much better when posterior smoothness matters.The authors recommend serious consideration of prior whitening even when site approximations are difficult or advanced methods are inapplicable.