Source-linked AI summary
Understanding Probabilistic Sparse Gaussian Process Approximations
Matthias Bauer, Mark van der Wilk, Carl Edward Rasmussen
TL;DR
Exact GP inference is computationally prohibitive for large datasets, motivating sparse approximations such as FITC and VFE. The paper analytically and experimentally compares their objectives and optimization behaviour, finding important differences in reliability and practical use.
Problem
Exact GP inference scales prohibitively with dataset size, creating a need to understand the differing theoretical and practical properties of FITC and VFE sparse approximations.
Method
The paper thoroughly investigates FITC and VFE through analytical characterization and illustrative practical examples of their objectives, biases, capacity allocation, and optimization behaviour.
Results
FITC can overestimate marginal likelihood, severely underestimate noise variance, waste modelling resources, and fail to recover the true posterior, whereas VFE is a true bound that behaves predictably and recovers the true posterior when possible.
Takeaways & Limitations
The authors recommend VFE while addressing optimization difficulties through careful initialization, random restarts, other optimization techniques, and comparison with FITC.
Takeaways & Limitations
VFE can find under-fitting solutions because of optimization issues, and FITC's resemblance to the full GP with many inducing inputs relies on local optima.
Abstract
from arXiv · showhide
Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.
1 Introduction
Sparse GP approximations reduce the prohibitive cost of exact inference by focusing on a small set of quantities, including inducing-point function values. This paper comparatively investigates FITC and VFE to characterize when each succeeds or fails in practice.
- Motivation: Exact GP inference costs O(N^3) time and O(N^2) memory, while sparse methods typically reduce this to O(NM^2) time and O(NM) memory for M < N.Sparse approximations retain the favourable properties of GPs while lowering computational cost.
- Scope: The paper focuses on approximations representing the posterior through function values at M inducing inputs, whose locations are jointly optimized with GP hyperparameters.Optimizing inducing inputs is presented as generally offering the best trade-off between accuracy and computational requirements.
- Scope: FITC and VFE differ in both their theoretical derivations and how inducing inputs are handled.The study treats these differences as central to understanding their distinct practical behaviour.
- Approach: The investigation analyzes hyperparameter-learning biases, modelling-capacity allocation, and optimization behaviour for the two approximations.It combines theoretical and practical analysis to characterize the methods comparatively.
- Practical aim: The paper aims to identify conditions under which each method is likely to succeed or fail and to suggest ways to diagnose or avoid practical issues.The authors position the work as a more complete comparative investigation than prior literature.
2 Sparse Gaussian Processes
Sparse GP regression uses inducing-point representations to make inference tractable, replacing full covariance operations with lower-rank approximations. FITC and VFE share similar-looking objectives but arise from different approximations and impose different corrections.
- GP regression: A GP defines a distribution over functions through its mean and covariance functions, with finite collections of function values following Gaussian distributions.The paper assumes a zero mean and uses the covariance function to determine properties such as smoothness and amplitude.
- GP regression: Regression models a latent function with a GP prior and noisy observations y at input locations X.Predictions are formed after optimizing hyperparameters through the marginal likelihood and marginalizing over the posterior.
- Computational challenge: Exact marginal-likelihood, posterior, and predictive calculations scale as O(N^3) because they require inversion of Kff + σ_n^2I.This cost is impractical for many datasets.
- Inducing-point approximation: Inducing-point methods replace the full-rank covariance Kff with the low-rank matrix Qff, reducing matrix-inversion size to M.FITC and VFE learn hyperparameters and inducing inputs through approximate marginal likelihoods.
- FITC and VFE: FITC can be interpreted through modified likelihood or prior constructions, whereas VFE bounds the marginal likelihood by introducing inducing variables u and a factorized q(u,f)=q(u)p(f|u).For VFE, variational calculus gives the optimal q(u) and the resulting lower bound.
- Common objective: The shared negative log marginal likelihood contains data-fit and complexity terms, while VFE adds a trace term that makes it a true lower bound.The trace term penalizes conditional variances at training inputs and encourages approximation of the full GP covariance structure; without it, VFE becomes DTC.
3 Comparative behaviour
FITC and VFE differ sharply in how their objectives treat noise, inducing inputs, and approximation accuracy. VFE improves toward the full GP in principle, whereas FITC can exploit heteroscedastic noise, deteriorate, or rely on local optima.
- 3.1 FITC can severely underestimate the noise variance, VFE overestimates it: FITC can explain deviations through heteroscedastic noise, encouraging severe underestimation of the homoscedastic noise variance.The estimated noise variance can become almost zero even when the mean does not pass through every observation.
- 3.1 FITC can severely underestimate the noise variance, VFE overestimates it: VFE encourages overestimation of the noise variance because of its data-fit and trace terms.Unlike FITC, its objective includes a trace term that preserves a true lower bound on the marginal likelihood.
- 3.2 VFE improves with additional inducing inputs, FITC may ignore them: VFE’s posterior and marginal-likelihood approximations become more accurate or remain unchanged whenever a new inducing input is added.Its complexity penalty increases with an additional non-zero eigenvalue, but data-fit and trace terms provide compensating gains.
- 3.2 VFE improves with additional inducing inputs, FITC may ignore them: FITC’s objective may improve or deteriorate when adding inducing inputs because reduced heteroscedastic noise lowers complexity penalties but can worsen data fit.This trade-off allows FITC to benefit from extra inputs without necessarily improving its covariance approximation.
- 3.2 VFE improves with additional inducing inputs, FITC may ignore them: Finite jitter broadens clumping pathologies and supplies gradients that let optimisation detect otherwise isolated duplicate-input configurations.Without jitter, these pathologies occur only at single points and have no gradients toward or away from them.
- 3.2 VFE improves with additional inducing inputs, FITC may ignore them: FITC can clump inducing inputs to avoid the penalty of extra inputs and retain heteroscedastic-noise benefits, effectively removing inputs at no cost.Duplicate inducing inputs reduce to fewer individually placed inputs, while VFE spreads them out and recovers the full GP more closely.
- 3.4 FITC relies on local optima: With increasing inducing inputs, FITC shows improvement, deterioration, and recovery regimes, while VFE monotonically approaches the full GP.FITC’s recovery resembles the full GP in marginal likelihood, predictive performance, and noise variance, but relies on local optima.
- 3.5 VFE is hindered by local optima: In high-dimensional pumadyn32nm experiments, FITC identified relevant lengthscales consistently, whereas VFE often failed to do so under joint optimisation.VFE recovered a good fit with correct lengthscales when initialised at the FITC solution, indicating an optimisation issue rather than an incorrect bound.
4 Conclusion
The paper finds substantial differences between FITC and VFE in objective functions and practical optimisation. VFE has stronger theoretical properties, but is harder to optimise and requires attention to dataset-dependent pathologies and local optima.
- Conclusion: FITC can overestimate the marginal likelihood, severely underestimate noise variance, waste modelling resources, and fail to recover the true posterior.These pathologies make noise variance an unreliable diagnostic for good model fitting.
- Conclusion: VFE is a true bound to the full GP marginal likelihood, improves with extra resources, and recovers the true posterior when possible.Its objective recognises good configurations, although optimisation is often more difficult.
- Conclusion: FITC pathologies do not always appear in practice because of good local optima and unintentional early stopping.Which pathologies emerge depends on the dataset.
- Conclusion: VFE is recommended based on its superior objective-function properties, with careful initialisation, random restarts, optimisation tricks, and FITC comparisons mitigating optimisation difficulties.The recommendation preserves attention to VFE’s susceptibility to local optima and harder optimisation.