PaperlayerView exact revision on arXiv

Source-linked AI summary

MCMC for Variationally Sparse Gaussian Processes

James Hensman, Alexander G. de G. Matthews, Maurizio Filippone, Zoubin Ghahramani

arXiv:1506.04000v1stat.ML

TL;DR

Gaussian-process inference must scale to large datasets while approximating non-Gaussian function posteriors and covariance-parameter uncertainty. This paper combines a variational inducing-point approximation with joint MCMC sampling, and reports reduced computational cost while approaching exact Bayesian inference.

  • Problem

    GP inference faces the combined challenges of costly covariance computations, non-Gaussian posterior approximation, and estimating covariance-function parameters.

    Source: §1 · pp. 1–2

  • Method

    A variational inducing-point approximation uses MCMC to jointly sample the function representation and covariance parameters, permitting non-Gaussian posteriors.

    Source: §1 · p. 2

  • Results

    The scheme significantly reduces computational cost while approaching exact Bayesian inference, improves accuracy over Gaussian approximations, and approximates hyperparameter distributions well across several likelihoods.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The experiments show that MCMC methods are feasible for inference in large GP problems.

    Source: §5 · p. 11

    The derivation uses a finite prediction vector large enough to include any point of interest; the infinite case is omitted but stated to yield the same solution.

    Source: §2 · p. 3

Abstract

from arXiv · show

Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper will be available shortly.

1. Introduction

The introduction frames Gaussian-process inference around three challenges: scalable computation, posterior approximation for non-Gaussian likelihoods, and covariance-parameter uncertainty. The paper combines variational inducing points with MCMC to address them together.

1. Introduction

  • GP inference must address non-Gaussian function posteriors, covariance-matrix costs that scale poorly with data, and covariance-parameter estimation.These challenges motivate methods that handle posterior approximation, computation, and parameter uncertainty together.Source: §1 · p. 1
  • Existing inducing-point methods can reduce computation, but recent schemes assume Gaussian posteriors and estimate covariance parameters by approximate maximum likelihood.Full-covariance MCMC schemes sample function values and covariance parameters but require repeated full covariance-matrix inversions.Source: §1 · p. 2
  • The proposed scheme uses variational inducing points and jointly samples the function representation and covariance parameters, allowing non-Gaussian posteriors.With sufficient inducing points, it approaches full Bayesian inference; experiments find that substantially fewer inducing points than data points suffice on several real problems.Source: §1 · p. 2

2. Stochastic process posteriors

The method represents a variational posterior through inducing-point function values and covariance parameters, then samples that distribution with MCMC. Expected log-likelihood computations and prior whitening support the resulting inference scheme.

2. Stochastic process posteriors

  • The model assumes a zero-mean GP prior with covariance function k(x, x′), and represents latent values at observed inputs separately from inducing values.The inducing inputs Z carry function responses u used in the sparse approximation.Source: §2 · pp. 2–3
  • The approximation remains a stochastic process: its prediction vector can be potentially unbounded and cover the whole domain.The derivation considers a finite but sufficiently large prediction vector, with the infinite case reported to give the same solution.Source: §2 · p. 3
  • The optimal variational distribution over inducing values and covariance parameters combines the expected log-likelihood with their GP and parameter priors.It is obtained by minimizing the KL divergence in the variational objective.Source: §2 · p. 4
  • For general likelihoods, MCMC samples the variational distribution, which is computable up to a constant in O(NM^2) computations.The finite prediction vector is assumed large enough to contain every point of interest; the infinite case is stated to yield the same solution.Source: §2 · pp. 3–4
  • Whitening writes inducing values as u = Rv with a standard-normal v, and HMC jointly samples v and covariance parameters θ.Earlier schemes alternated sampling these quantities.Source: §2 · p. 4
  • With likelihoods factorized across data-function pairs, the expected log-likelihood reduces to N one-dimensional integrals, using Gauss-Hermite quadrature when needed.The conditional function values have Gaussian means and variances computed from the inducing representation.Source: §2 · p. 4

3. Treatment of inducing point positions & inference strategy

The paper treats inducing-point positions as part of the variational approximation, formally motivating their optimization. It then initializes and tunes HMC before using samples for predictions.

3. Treatment of inducing point positions & inference strategy

  • Unlike the original inducing-point formulation, which treats Z as parameters of the approximate prior covariance, the variational formulation treats them as parameters of the approximation.The paper presents this variational treatment as protection against over-fitting.Source: §3 · p. 4
  • The variational formulation makes the optimal distribution over inducing-point positions a point mass, formally motivating optimization of Z.The derivation chooses a prior for Z that maximizes information captured by the inducing values.Source: §3 · p. 5
  • Evaluating the free-form objective gradient for inducing-point positions using samples is challenging, so the paper introduces a separate strategy.The gradient is obtained by differentiating the optimized variational objective with respect to those positions.Source: §3 · p. 5
  • A fitted Gaussian approximation initializes inducing positions, latent values, and covariance parameters before HMC sampling.The positions begin at k-means centers; after fitting with fixed positions, the method optimizes them alongside variational and covariance parameters.Source: §3 · p. 5
  • Bayesian optimization tunes HMC leapfrog count and step length, then several thousand iterations provide samples for estimating predictions.The tuning procedure runs 30 iterations of 30 samples each and retains the best parameters for the longer run.Source: §3 · pp. 5–6

4. Experiments

Experiments assess sampling efficiency and predictive or posterior behavior across binary classification, Cox processes, and multiclass tasks. Sparse variational sampling closely matches full MCMC on pine data and improves on Gaussian approximations in several reported comparisons.

4.1 Efficient sampling using Hamiltonian Monte Carlo

  • The HMC evaluation compares efficiency and convergence with Gibbs sampling, using ESS and time-normalized ESS and tuning proposals with Bayesian optimization.The comparison uses isotropic mass and proposal covariance matrices; the Gibbs baseline samples latent values with HMC and covariance parameters with Metropolis-Hastings.Source: §4.1 · p. 6

4.2 Binary Classification

  • On the image dataset, optimizing inducing locations improves over k-means, and sampling consistently improves performance over the Gaussian approximation when the number of inducing points is large enough.For small inducing-point counts, the Gaussian approximation performs well; the authors speculate that larger ARD gains reflect lengthscale uncertainty poorly represented by a Gaussian/MAP point estimate.Source: §4.2 · p. 7, Figure 1 · p. 7

4.3 Log Gaussian Cox Processes

  • On coal-mining data, the free-form variational posterior closely matches the true posterior, while the Gaussian approximation misses rate-posterior detail; covariance-parameter posteriors differ minimally.The comparison used 30 inducing points and 3,000 HMC samples.Source: §4.3 · p. 8, Figure 2 · p. 8
  • For the coal-mining held-out prediction task, free-form MCMC has average log-likelihood -1.225 versus -1.229 for the Gaussian approximation and wins on all 10 random splits.The authors attribute the improvement to marginalizing covariance-function parameters.Source: §4.3 · p. 8

Pine saplings

  • 3.4 seconds per effective sample for sparse sampling versus 554 seconds for full MCMC on the 32×32 pine grid; at 64×64, sparse sampling takes 4.7 seconds while full MCMC cannot complete one HMC step.On the 32×32 grid, the variational and full-MCMC intensities agree closely; the 64×64 full-MCMC step requires O(N^3) operations with N = 4096.Source: Pine saplings · p. 9, Figure 3 · p. 9

4.4 Multi-class Classification

  • In the toy multiclass task, free-form sampling gives more conservative contours, while posterior correlations between latent functions at decision boundaries are not captured by the Gaussian approximation.The reported inducing points also move from their k-means positions toward decision boundaries.Source: §4.4 · p. 10, Figure 4 · p. 10
  • 98.04% held-out accuracy was reported for the optimized Gaussian approximation on MNIST; sampling did not increase accuracy but improved test log-density from -0.068 to -0.064.The model used 500 inducing points initialized by k-means.Source: §4.4 · pp. 10–11

5. Discussion

The proposed scheme reduces computation while approaching exact Bayesian inference for general GP models with minimal assumptions about the posterior. Experiments demonstrate improved accuracy over Gaussian approximations and effective sampling across several likelihoods.

5. Discussion

  • The scheme significantly reduces computational cost while approaching exact Bayesian inference with minimal assumptions about posterior form.The discussion states that it makes MCMC feasible for large GP problems.Source: §5 · p. 11
  • Its accuracy improves on Gaussian approximations, and it provides a high-quality approximation to the hyper-parameter distribution.Source: §5 · p. 11
  • The MCMC scheme was effective across several likelihoods, and automatic tuning worked well over hundreds of experiments.Source: §5 · p. 11
  • The results support the feasibility of MCMC methods for inference in large GP problems.Source: §5 · p. 11

Supplementary material for:

The supplementary material reports single-lengthscale performance, sampler convergence comparisons, and visual comparisons of variational and gold-standard MCMC grids. Across the image, coal, and multiclass convergence analyses, HMC converges faster while sampling efficiency is often comparable.

5.1 Coal data

5.2 Convergence plots

5.3 Pine saplings

  • Figure 10 compares gold-standard and variational MCMC grids at 32x32, and shows a 64x64 variational grid using 225 inducing points in the non-exact case.Source: Figure 10 · p. 16
Loading 1506.04000v1…