Source-linked AI summary
Student-t Processes as Alternatives to Gaussian Processes
Amar Shah, Andrew Gordon Wilson, Zoubin Ghahramani
TL;DR
Gaussian processes typically rely on parametrized covariance kernels, leaving uncertainty over the kernel function insufficiently represented. The paper derives Student-t processes by placing an inverse Wishart process prior on the kernel and analytically marginalizing it, finding GP-like analytic inference with greater flexibility and observation-dependent predictive covariances. The authors report improved performance in regression and Bayesian optimization without additional computational cost.
Problem
Gaussian-process models typically use parametrized kernels, while a fully Bayesian nonparametric treatment would place a prior over the covariance kernel itself.
Method
The paper derives a Student-t process from hierarchical Gaussian processes using an inverse Wishart process prior over the kernel, with analytic marginal and predictive distributions.
Results
Student-t processes retain GP benefits while providing observation-dependent predictive covariances and improved performance over GPs in regression and Bayesian optimization without additional computational costs.
Takeaways & Limitations
The authors suggest that Student-t processes can replace Gaussian processes broadly, adding modeling flexibility orthogonal to kernel choice at no extra cost.
Takeaways & Limitations
The Bayesian optimization experiments assume functions drawn from a multivariate Student-t process with ν > 2, constant mean, and a specific ARD Matérn 5/2-plus-delta kernel.
Abstract
from arXiv · showhide
We investigate the Student-t process as an alternative to the Gaussian process as a nonparametric prior over functions. We derive closed form expressions for the marginal likelihood and predictive distribution of a Student-t process, by integrating away an inverse Wishart process prior over the covariance kernel of a Gaussian process model. We show surprising equivalences between different hierarchical Gaussian process models leading to Student-t processes, and derive a new sampling scheme for the inverse Wishart process, which helps elucidate these equivalences. Overall, we show that a Student-t process can retain the attractive properties of a Gaussian process -- a nonparametric representation, analytic marginal and predictive distributions, and easy model selection through covariance kernels -- but has enhanced flexibility, and predictive covariances that, unlike a Gaussian process, explicitly depend on the values of training observations. We verify empirically that a Student-t process is especially useful in situations where there are changes in covariance structure, or in applications like Bayesian optimization, where accurate predictive covariances are critical for good performance. These advantages come at no additional computational cost over Gaussian processes.
1 INTRODUCTION
The paper develops Student-t processes as flexible alternatives to Gaussian processes by placing nonparametric priors on covariance kernels. It derives analytic inference and identifies practical advantages in predictive uncertainty, robustness, and Bayesian optimization.
- Gaussian processes use parametrized covariance kernels, motivating a nonparametric prior over kernel functions to represent covariance uncertainty.
- The paper derives Student-t processes from hierarchical Gaussian process models and obtains analytic marginal, predictive, and marginal-likelihood derivative expressions.
- The paper introduces inverse-Wishart-process sampling to explain marginal equivalences among hierarchical Gaussian process models.
- Student-t predictive covariances explicitly depend on training-observation values, unlike Gaussian-process predictive covariances.
- Across applications, Student-t processes are reported as more robust to change-points and misspecification, with improved predictive covariances and promise for Bayesian optimization.
2 INVERSE WISHART PROCESS
The inverse Wishart process provides a consistent prior over covariance matrices of arbitrary size, avoiding a dimensionality problem that makes the Wishart process unsuitable for useful nonparametric modeling.
- The Wishart distribution is consistent under marginalization but requires ν > n −1, preventing a useful process with arbitrary-size marginals.
- The inverse Wishart distribution permits ν independent of matrix size and is consistent under marginalization.
- An inverse Wishart process assigns inverse-Wishart marginals to covariance matrices induced by finite collections of inputs.
- The inverse Wishart process is introduced as a nonparametric prior over kernels in a hierarchical Gaussian process model.
3 DERIVING THE STUDENT-t PROCESS
The paper places an inverse Wishart process prior on a Gaussian-process covariance kernel and analytically marginalizes it, yielding a consistent Student-t process with closed-form distributions.
- The generative model places an inverse Wishart process prior on a Gaussian-process kernel.
- Because the inverse Wishart distribution is conjugate to the Gaussian likelihood covariance, the covariance process can be analytically marginalized.
- The resulting multivariate Student-t distribution has mean φ and covariance K, and its marginals remain consistent.
- A Student-t process gives every finite collection of function values a joint multivariate Student-t distribution with shared degrees-of-freedom parameter ν.
4 TP PROPERTIES & RELATION TO OTHER PROCESSES
The Student-t process generalizes the Gaussian process while retaining analytic distributions and comparable computational cost. Its heavier tails, observation-dependent predictive covariance, and equivalent covariance-prior constructions provide added flexibility for nonparametric modeling.
- Relation to Gaussian process: A Gaussian process is the limiting case of a Student-t process as ν →∞, while smaller ν produces heavier tails and more extreme sample behavior.The parameter ν also controls dependence between jointly Student-t distributed variables, including tail dependence even when marginal distributions are Gaussian.
- Conditional distribution: Student-t process predictive means match Gaussian-process means for the same kernel, but predictive covariances explicitly depend on training observations.Observed squared magnitudes above their expected value scale predictive covariance up, while smaller values scale it down; the magnitude is controlled by ν.
- Another Covariance Prior: An inverse Wishart process prior and a single inverse Gamma scale prior can produce the same marginal multivariate Student-t process.This equivalence connects an infinite-dimensional covariance prior with a one-parameter construction.
- Elliptical processes: The Student-t process is the most general elliptically symmetric process with analytically representable finite-dimensional densities.Among elliptical processes, only Gaussian and Student-t processes have analytically representable densities for every finite collection.
- A New Way to Sample the IWP: The inverse Wishart density depends only on the eigenvalues of a positive definite matrix, yielding a sampling interpretation based on random orthogonal bases and exchangeable stretches.The sampling scheme draws a uniformly distributed orthogonal matrix and exchangeable eigenvalues, then forms Σ = QΛQ⊤.
- Modelling Noisy Functions: Incorporating noise into the kernel lets the Student-t process model heavy-tailed noise while retaining analytic inference and the same computational costs as a Gaussian process.The kernel construction is not equivalent to adding independent noise because ν scales both the signal and noise kernels.
5 APPLICATIONS
The paper evaluates Student-t processes against Gaussian processes for regression and Bayesian optimization, finding improved uncertainty modeling and promising optimization behavior without additional computational cost.
- 5.1 Regression: The TP provides superior predictive uncertainty when fitting data generated from a Gaussian process with Gaussian noise.This experiment trains on 80 points and tests on 20 for each of 100 sampled functions.
- 5.1 Regression: Student-t processes consistently achieve the lowest predictive MSE and highest predictive LL across the regression experiments.The table caption reports this pattern across the regression datasets.
- 5.1 Regression: For data with independent Student-t noise, the TP has a superior predictive mean and better predictive uncertainty after hyperparameter training.The TP better models the Student-t noise in this setting.
- 5.1 Regression: The TP handles the seasonality-driven changepoint behavior in Whistler snowfall data much better than the GP.The dataset contains daily snowfall amounts recorded during 2010 and 2011.
- 5.2 Bayesian Optimization: Changing only ν can produce substantially different acquisition functions and optimization proposals for TP and GP priors.Figure 5 compares a GP with TP priors at ν = 15 and ν = 5 while holding all other hyperparameters fixed.
- 5.2 Bayesian Optimization: On the sinusoidal benchmark, the TP reaches within 0.1% of the minimum in 8.1 ± 0.4 iterations, versus 10.7 ± 0.6 for the GP.The Bayesian optimization comparison also includes Branin-Hoo and Hartmann functions.
- 5.2 Bayesian Optimization: On Hartmann, the TP explores discovered modes more thoroughly before moving away, producing a step-like improvement pattern that is more pronounced in higher dimensions.The benchmark has six local minima in [0, 1]^6.
6 CONCLUSIONS
The conclusions establish the Student-t process as a tractable extension of the Gaussian process, derived through an inverse Wishart process and useful for regression and Bayesian optimization.
- 6 CONCLUSIONS: Marginalizing an inverse Wishart process placed over a GP kernel yields a Student-t process with consistent marginals and closed-form conditionals.The TP contains the GP as a special case.
- 6 CONCLUSIONS: The Student-t process is the only elliptical process other than the Gaussian process with an analytically representable density function.This gives the TP a broad analytical role among elliptical process models.
- 6 CONCLUSIONS: Regression and Bayesian optimization applications show improved performance over GPs with no additional computational costs.The conclusion presents this as an empirical result across both application types.
- 6 CONCLUSIONS: For practitioners, the TP offers increased modeling flexibility while retaining many or all of the benefits of GPs at no extra cost.Its flexibility is described as orthogonal to kernel choice.
A Proofs
The proofs establish consistency, Gaussian-process convergence, analytic-density uniqueness, and the hierarchical representation underlying Student-t processes.
- A Proofs: Multivariate Student-t distributions are consistent under marginalization, supporting process definitions with compatible finite-dimensional marginals.The proof uses principal submatrices of the inverse Wishart covariance representation.
- A Proofs: Conditioning a multivariate Student-t distribution partitions its location and scale using the observed block and a Schur-complement scale matrix.The resulting conditional form depends on the observed quadratic form through the updated degrees of freedom and scale.
- A Proofs: Analytic marginalization of the scalar mixing variable produces the multivariate Student-t distribution, with the scale parameter redundant up to normalization.The proof sets ρ = 1 without loss of generality.
- A Proofs: Among elliptical processes, analytically representable finite-dimensional densities occur only for Gaussian or Student-t processes.This is stated as a corollary for any finite collection of process values.
- A Proofs: The Gaussian and Student-t processes arise from a Gaussian scale mixture when the mixing variable is respectively constant or inverse-gamma distributed.The inverse-gamma case uses the conjugate prior r ∼ Γ^-1(ν/2, 1/2).
B Marginal Likelihood Derivatives
The paper derives marginal-likelihood derivatives for Student-t process hyperparameters, enabling gradient-based learning methods.
- B Marginal Likelihood Derivatives: Analytic derivatives of the likelihood with respect to hyperparameters support maximum likelihood and Hamiltonian Monte Carlo learning.The hyperparameters include kernel parameters and can include ν.
- B Marginal Likelihood Derivatives: The degrees-of-freedom parameter ν can be learned using gradient-based methods through a derivative involving the digamma function ψ.The displayed derivative is the marginal log-likelihood derivative with respect to ν.
C More Insight Into the Inverse Wishart Process and Inverse Gamma Priors
This section defines the Wishart distribution for positive-definite random matrices using degrees-of-freedom and scale parameters.
- A random positive-definite matrix Σ is Wishart distributed when its density is specified by parameters ν > n − 1 and K ∈ Π(n).
C.1 The Multivariate Gamma Function
This section introduces the multivariate gamma function used in the Wishart normalizing constant and relates it to the ordinary gamma function.
- The multivariate gamma function generalizes the ordinary gamma function.
- Positive definiteness is denoted by S > 0.
- A lemma establishes an explicit relationship between the multivariate gamma function and the gamma function.
- The resulting corollary is used later in the development.
C.2 Two Different Covariance Priors
The section compares two generative processes and shows that their marginal distributions are equivalent under the stated corollary.
- The two generative processes are defined for n ∈ N, ν > 2, and symmetric positive-definite n × n matrix K.
- The marginal distribution for y1 is derived for each generative process.
- Both marginal distributions are equivalent given the result in Corollary B.