Source-linked AI summary
Gaussian Processes for Big Data
James Hensman, Nicolo Fusi, Neil D. Lawrence
TL;DR
Gaussian processes are difficult to apply to very large data sets because inference and storage scale poorly with n. The paper combines inducing variables with stochastic variational inference to obtain a factorized GP model, enabling the stated extensions to non-Gaussian likelihoods and latent-variable models. The resulting formulation supports big-data GP inference and permits substantially larger inducing-variable sets than sparse GP representations.
Problem
Exact and existing approximate GP methods remain impractical for data sets containing millions or billions of points because their computational or storage demands scale with n.
Method
The paper combines inducing variables with stochastic variational inference so GP models factorize through globally relevant variables.
Results
The resulting model has O(m3) complexity rather than O(nm2), allowing m = 800 in the UK apartment price example and supporting richer models.
Takeaways & Limitations
The framework transfers many GP techniques to big data and is applicable to multiple-output GPs, GP classification, and GP-based latent-variable models.
Takeaways & Limitations
Implementations for GP classification and GPLVM models are left to future work.
Abstract
from arXiv · showhide
We introduce stochastic variational inference for Gaussian process models. This enables the application of Gaussian process (GP) models to data sets containing millions of data points. We show how GPs can be vari- ationally decomposed to depend on a set of globally relevant inducing variables which factorize the model in the necessary manner to perform variational inference. Our ap- proach is readily extended to models with non-Gaussian likelihoods and latent variable models based around Gaussian processes. We demonstrate the approach on a simple toy problem and two real world data sets.
1 Introduction
Gaussian processes support broad function-inference tasks but exact inference scales poorly with dataset size. The paper motivates stochastic variational inference by combining inducing variables with variational methods for data sets reaching millions or more points.
- Gaussian processes underpin regression, classification, and unsupervised-learning algorithms.
- Exact GP inference requires O(n3) complexity and O(n2) storage for n data points.For multiple outputs or tasks, the stated costs increase to O(n3p3) and O(n2p2).
- Traditionally, GP data sets exceeding a few thousand points are considered large.
- Approximate methods reduce costs to O(nm2) computation and O(nm) storage, but these demands remain prohibitive when n reaches millions or billions.Here m is a user-selected number of inducing variables.
- The paper combines inducing variables with stochastic variational inference to develop a practical algorithm for fitting GPs on very large data sets.The motivation is to address the storage problem that stochastic gradient methods already address for parametric models.
2 Sparse GPs Revisited
The sparse-GP derivation introduces inducing variables and a variational lower bound whose computation avoids the O(n3) bottleneck. The bound is tight when inducing locations represent the training data well, while exact equality requires one inducing variable per data point.
- The model observes noisy function values under independent Gaussian noise and introduces inducing variables at locations Z alongside training inputs X.The inducing vector u contains function values at the inducing points.
- The covariance construction uses K_mm between inducing points, K_nm between inducing and training points, and a residual covariance defined from these matrices.
- Jensen’s inequality produces a lower bound L1 by moving the expectation outside the log.For Gaussian noise, placing the expectation inside the log yields an expression involving K_nn with O(n3) complexity; the lower-bound route has O(m3) complexity.
- The bound’s difference from the original log likelihood is a KL divergence between the posterior mapping function distributions with and without the observations.
- When m = n and inducing locations equal training locations, the residual covariance vanishes and the bound becomes exact, but there is no computational or storage advantage.
- For m < n, optimizing Z distributes inducing points across the training data so marginal variances remain small and Jensen’s bound stays tight.
3 SVI for GPs
The paper makes Gaussian processes compatible with stochastic variational inference by retaining inducing variables as global variables and factorizing the variational bound over observations. This supports minibatch updates and extensions to latent-variable and non-Gaussian-likelihood models.
- Global Variables: Explicitly representing inducing variables u as global variables restores the factorization required for stochastic variational inference in Gaussian processes.Marginalizing u would reintroduce dependencies between observations and eliminate the global parameters.
- SVI for GPs: The construction enables online learning for Gaussian-process regression because the number of inducing variables is fixed at algorithm design time.Kernel hyperparameters and noise precision are optimized alongside variational parameters using stochastic gradient descent.
- Global Variables: The variational bound L3 decomposes into n terms, one for each input-output pair, enabling stochastic gradient methods on q(u).Data can therefore be processed individually or in mini-batches through approximate natural gradients.
- Natural Gradients: A unit natural-gradient step recovers the variational-Bayes update, while stochastic approximations permit minibatch optimization.The natural gradient for the covariance parameter also preserves positive definiteness without special parameterization.
- Latent Variables: For latent-variable GP models, the factorized bound supports alternating minibatch updates of q(X) and approximate natural-gradient updates of q(u), with parallel latent-distribution updates.The formulation addresses the poor scaling and strong variable coupling caused by marginalizing inducing variables in earlier approaches.
- Non-Gaussian Likelihoods: The factorized bound also supports non-Gaussian likelihoods, including cases where likelihood-related integrals can be evaluated exactly under existing approximations.For the probit likelihood, the resulting independent one-dimensional integrals are tractable and remain amenable to SVI.
4 Experiments
Experiments show SVIGP converging on toy problems and scaling to large apartment-price and flight-delay datasets. The approach supports many inducing variables and reveals interpretable spatial and feature-level patterns.
- Toy data: On a two-dimensional sinusoidal toy problem, SVIGP converged from random q(u) initialization after 2000 iterations.Holding covariance parameters fixed during the first epoch produced more reliable convergence.
- UK apartment prices: The UK apartment dataset contained 75,000 England-and-Wales entries, with normalized log prices regressed on latitude and longitude.The model used national and regional squared-exponential covariances, a bias term, and noise variance.
- Airline delays: The airline dataset covered almost 2 million US commercial flights, while the experiment trained on 700,000 samples and tested on 100,000.Eight flight-related variables were used, including aircraft age, distance, airtime, and timing variables.
- Airline delays: SVIGP’s prediction performance improved substantially with larger numbers of inducing inputs, enabled by complexity independent of n rather than conventional O(nm^2) scaling.Figure 8 compares root mean square errors for models using different numbers of inducing variables.
- Airline delays: Departure time was the most relevant flight-delay feature, followed closely by distance, while distance and airtime had markedly different relevances.The paper explains this difference by noting that longer flights can recover delays during departure.
5 Discussion
The method enables stochastic variational inference for Gaussian processes and supports richer models by allowing many more inducing variables. The framework also extends toward multiple-output and other GP-based models, although implementation details remain future work.
- The method enables inference in Gaussian process models using stochastic variational inference and transfers many GP techniques to big data.
- The bound becomes tight when Z = X.
- A unique q(u) parameter solution recovers the bound associated with the standard variational sparse GP.
- O(m3) complexity permits substantially increasing m beyond the sparse GP representation, yielding richer models; the UK apartment price example used m = 800.
- Increasing inducing variables and supporting unlimited data make the method suitable for multiple-output GPs.
- Details of implementations for GPLVMs and GP classification are left to future work.