Source-linked AI summary
Scalable Variational Gaussian Process Classification
James Hensman, Alex Matthews, Zoubin Ghahramani
TL;DR
GP classification requires approximate inference and has traditionally faced costly scaling. The paper develops variational inducing-point bounds that support stochastic optimization. Its KLSP method outperforms GFITC on benchmark datasets and enables GP classification on big data, as demonstrated experimentally.
Problem
Approximate GP classification requires scalable inference because posterior and marginal-likelihood computation generally costs O(N^3).
Method
The paper develops variational inducing-point bounds for sparse GP classification, including a KLSP bound optimized through stochastic variational inference.
Results
KLSP outperforms the state-of-the-art GFITC method on benchmark datasets and supports stochastic optimization for GP classification on big data.
Takeaways & Limitations
The variational formulation makes GP classification applicable to big data, including problems with millions of data points.
Takeaways & Limitations
The paper notes that the approach currently restricts q(u) to a Gaussian form, though this restriction can be relaxed in future work.
Abstract
from arXiv · showhide
Gaussian process classification is a popular method with a number of appealing properties. We show how to scale the model within a variational inducing point framework, outperforming the state of the art on benchmark datasets. Importantly, the variational formulation can be exploited to allow classification in problems with millions of data points, as we demonstrate in experiments.
1 Introduction
Sparse GP classification remains difficult to scale because non-conjugate inference typically retains cubic cost, while combining sparsity with classification introduces additional approximation and inducing-point challenges. The paper motivates a variational sparse scheme that supports stochastic optimization and learns inducing-input positions.
- Non-Gaussian GP classification requires approximate posterior and marginal-likelihood inference rather than closed-form computation.
- O(N^3) inference cost limits approximate GP classification because covariance-matrix factorization is generally required.
- Simply combining low-rank covariance approximations with non-conjugate inference can leave inducing-point placement and matrix-multiplication complexity unresolved.
- The proposed direction is a non-conjugate sparse GP scheme with a variational marginal-likelihood bound, stochastic optimization, and optimized inducing inputs.The variational objective is also intended to integrate with other GP models without fear of overfitting.
- The paper compares its proposals with existing methods and demonstrates empirical scalability to very large datasets.
2 Background
GP classification uses a sigmoidal likelihood whose posterior and marginal likelihood require approximation, while standard inference costs O(N^3). Inducing-point variational methods reduce this cost, but existing sparse classifiers retain important scalability and optimization limitations.
- Gaussian Process Classification: GP classification squashes a function prior through a sigmoidal inverse-link function and uses a Bernoulli likelihood for the transformed values.
- Gaussian Process Classification: The GP posterior p(f | y) and marginal likelihood p(y) must be approximated, while existing approximation schemes require O(N^3) computation.
- Sparse Gaussian Processes for Regression: Inducing inputs Z and variables u augment the latent function to reduce GP computational complexity through sparse approximations.
- Variational bounds provide objectives for optimizing covariance parameters and inducing inputs, becoming tight when inducing points equal the data points.
- The variational representation introduces q(u)=N(u | m, S), whose bound has a unique optimum and can be optimized stochastically or distributively.
- Bernoulli likelihood integrals are intractable, motivating sparse GP classifiers that build on the Gaussian-likelihood variational bounds.
- Related work: Generalized FITC improves over IVM in practice, often matching or exceeding it with fewer inducing points, but is limited to a few thousand data.
- Discussion: Optimizing inducing inputs against an approximate marginal likelihood can produce strange pathologies, motivating a direct variational approach.
3 Two stage approaches
The two-stage approaches combine sparse GP bounds with further approximations for non-Gaussian likelihoods. Mean-field inference is tractable but scales poorly in variational parameters, while the resulting scalable alternative can be ineffective.
- 3 Two stage approaches: Generalized FITC-style construction first approximates the covariance and then applies a variational approximation to handle non-conjugacy.The approach uses an augmented latent variable g for the probit likelihood and applies a sparse GP bound followed by another variational approximation.
- 3.1 Sparse mean field approach: The sparse mean-field bound has tractable expectations and gradients, with total computation bounded by O(NM^2).Its variational parameters can be optimized alongside covariance hyperparameters and inducing points; inner-loop updates cost O(NM).
- 3.1 Sparse mean field approach: Mean-field inference is unsatisfactory because its variational-parameter count scales with N, making stochastic or distributed optimization difficult.The parameters are also dependent on one another.
- 3.2 A more scalable method?: A second approach substitutes the scalable Gaussian sparse GP bound into the augmented model, yielding a tractable marginal-likelihood bound.This bound can be optimized stochastically or in distributed fashion, but is reported to be less effective than expected.
- 3.2 A more scalable method?: When inducing points equal data points, the second approach resembles a MAP-centered Laplace approximation, which is known to be relatively ineffective for classification.The authors therefore abandon this bound and construct a different scalable, empirically effective variational bound.
4 A single variational bound
The single-bound approach derives a variational marginal-likelihood bound directly, without additional latent variables or factorizing assumptions. It supports gradient optimization, stochastic or distributed computation, and O(M^2) prediction for a test latent value.
- 4 A single variational bound: The method obtains a marginal-likelihood bound directly without introducing additional latent variables or factorizing assumptions.It starts from a variational bound on the conditional distribution and requires only marginals of q(f) for the expected log-likelihood terms.
- Computations and Scalability: The expected likelihood terms can be evaluated by one-dimensional quadrature, while the objective can also be optimized using distributed computation or random mini-batches.The KL divergence contributes O(M^3) computation, with most work expected in the N data-dependent terms.
- 4 A single variational bound: The bound uses q(f, u) = p(f | u)q(u) and optimizes q(u), covariance parameters, and inducing inputs by gradient-based methods.The covariance S is represented as S = LL^T to permit unconstrained optimization while maintaining positive-definiteness.
- Predictions: O(M^2) computes the mean and variance of a test latent value, from which the test-label distribution is obtained.Prediction follows by substituting the approximate posterior into the standard probabilistic prediction rule.
- Limiting cases: When Z = X, the method recovers the full-Gaussian approximation; with Gaussian likelihood and Z = X, exact inference is recovered.With Gaussian likelihood, the bound also recovers the variational bound of Titsias [2009].
- Experiments: In Figure 1, increasing inducing points is compared across KL, mean-field, and Generalized FITC methods on the banana dataset.Rows encode methods, columns encode increasing inducing-point counts, and the rightmost column shows equivalent non-sparse results.
5 Experiments
The experiments compare the proposed MF and KL variational approximations with Generalized FITC across benchmark datasets, varying inducing-point counts and optimization time. The KL-based methods achieve strong predictive performance, favorable time-performance behavior, and stochastic optimization on large datasets.
- Benchmark comparisons: The paper evaluates MF, KL, sparse KL, and Generalized FITC using benchmark datasets, including inducing-point and numerical comparisons.The numerical comparison reports median hold-out negative log probability over ten folds with 2-σ confidence intervals.
- Increasing the number of inducing points: With 4 to 64 inducing points, KL reaches a near-optimal solution with 16 points, while MF adapts inducing inputs less effectively and Generalized FITC uses 64 points poorly.KL pulls inducing points toward the decision boundary; Generalized FITC splits some points between the boundary and the origin.
- Numerical comparison: Sparse KL and FITC show similar performance on the benchmark table, while sparse KL has overlapping or better confidence intervals and scales to very large datasets with stochastic gradient descent.The table also reports that mean-field methods often give over-confident predictions.
- Time-performance trade-off: The sparse KL method occupies the efficient frontier on the image dataset, achieving the lowest negative log probability for any given optimization time.The comparison characterizes efficiency through the trade-off between optimization time and classification performance.
- Stochastic optimization: 97.8% hold-out accuracy and 0.069 mean negative-log-probability were obtained on MNIST using stochastic optimization with 60,000 training points and 200 inducing points.The experiment used mini-batches of 10 and demonstrates fitting nonlinear, high-dimensional decision boundaries on a dataset described as beyond existing methods’ range.
- Stochastic optimization: On 5.9 million US flight records, the proposed method exceeded a linear model’s performance in a few minutes using a kernel combining Matern and linear components.The optimized kernel indicated a highly nonlinear problem, with time-of-day and time-of-year features most useful for the Matern covariance.
6 Discussion
The paper presents variational bounds for sparse GP classification and reports that KLSP outperforms GFITC while supporting stochastic optimization. It also identifies extensions to other GP models when the likelihood factorizes.
- The paper presents two novel variational bounds for sparse GP classification.
- The covariance-first variational approaches introduce inference parameters that scale linearly with N.
- KLSP outperforms the state-of-the-art GFITC method on benchmark datasets and supports stochastic optimization.
- The method can extend to other GP models when the likelihood factorizes in N, using Gauss-Hermite quadrature of the log likelihood.
Supplementary Material for :
The supplementary material reports hold-out predictive-density and error plots for image and banana datasets. These figures provide visual comparisons across methods on both datasets.
- Figure S.1 compares hold-out predictive densities of different methods on the image dataset.
- Figure S.2 compares hold-out errors of different methods on the image dataset.
- Figure S.3 compares hold-out predictive densities of different methods on the banana dataset.
- Figure S.4 compares hold-out predictive densities of different methods on the banana dataset.
- Figure S.5 presents hold-out errors of different methods on the banana dataset.
- Figure S.6 presents hold-out errors of different methods on the banana dataset.