Source-linked AI summary
Gaussian Process Regression with a Student-t Likelihood
Pasi Jylänki, Jarno Vanhatalo, Aki Vehtari
TL;DR
Student-t GP regression is robust to outliers but analytically intractable, and standard EP can fail to converge because the likelihood is non-log-concave. The paper develops a robust EP implementation based on parallel updates and adaptive double-loop moment matching, demonstrating its use for difficult hyperparameter MAP estimation and comparisons with alternative approximations.
Problem
Student-t GP regression requires approximate inference, while non-log-concave likelihoods can make standard EP convergence problematic, including during hyperparameter MAP estimation.
Method
The paper combines parallel EP updates with a moment-matching double-loop algorithm and adaptively selected step sizes, using fractional updates and damping when needed.
Results
The robust implementation enables type-II MAP estimation of hyperparameters from the approximate marginal likelihood and is compared with Laplace, fVB, VB, and MCMC.
Takeaways & Limitations
EP can provide good predictive performance in this challenging model, but robust convergence procedures are needed rather than relying on standard EP alone.
Takeaways & Limitations
Standard EP may fail to converge at maximum-marginal-likelihood hyperparameter values in difficult cases involving separated posterior modes caused by outliers.
Abstract
from arXiv · showhide
This paper considers the robust and efficient implementation of Gaussian process regression with a Student-t observation model. The challenge with the Student-t model is the analytically intractable inference which is why several approximative methods have been proposed. The expectation propagation (EP) has been found to be a very accurate method in many empirical studies but the convergence of the EP is known to be problematic with models containing non-log-concave site functions such as the Student-t distribution. In this paper we illustrate the situations where the standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence. We demonstrate that convergence problems may occur during the type-II maximum a posteriori (MAP) estimation of the hyperparameters and show that the standard EP may not converge in the MAP values in some difficult cases. We present a robust implementation which relies primarily on parallel EP updates and utilizes a moment-matching-based double-loop algorithm with adaptively selected step size in difficult cases. The predictive performance of the EP is compared to the Laplace, variational Bayes, and Markov chain Monte Carlo approximations.
1. Introduction
The paper addresses robust GP regression with Student-t observations, whose heavy tails reduce outlier influence but make inference analytically intractable. It focuses on EP convergence problems for the non-log-concave Student-t likelihood and develops a robust implementation for difficult hyperparameter estimation.
- Student-t observations provide a robust alternative when regression data contain outliers from measurement failures or omitted explanatory variables.
- The Student-t model’s robustness can vary continuously from heavy-tailed behavior toward the Gaussian model through its degrees-of-freedom parameter.This permits robustness to be determined through hyperparameter inference.
- Inference with the Student-t model is analytically intractable, motivating Gibbs sampling, factorizing variational approximations, and Laplace approximation.Earlier comparisons found slightly better predictive performance for Laplace than factorizing variational Bayes, with lower computational cost.
- EP is attractive for approximate inference, but the non-log-concavity of the Student-t likelihood can cause convergence problems.
- The paper develops robust EP using primarily parallel updates and a moment-matching double-loop algorithm with adaptively selected step sizes.It compares the implementation with Laplace, fVB, VB, and MCMC on simulated and real-world data.
2. Gaussian Process Regression with Student-t Observation Model
The model combines a Gaussian-process prior for a latent regression function with Student-t observation noise. This preserves robustness to outliers but removes the analytical tractability available under Gaussian noise.
- The regression model observes yi = f(xi) + εi at input locations xi, with the latent function f assigned a Gaussian-process prior.
- At observed inputs, the latent variables have a multivariate Gaussian distribution with covariance matrix entries determined by the kernel function.
- The squared-exponential kernel uses a magnitude parameter to scale variation and length-scale parameters to control correlation decay across input dimensions.
- Gaussian observation noise yields analytically tractable marginal likelihood and conditional posterior calculations, but the Gaussian model is non-robust.
- Student-t observation noise is computationally challenging because its marginal likelihood and conditional latent posterior are not analytically tractable.Approximate inference is therefore required.
3. Approximate Inference for the Student-t Model
The paper reviews MCMC, Laplace, variational, and EP approximations for Student-t GP regression, emphasizing that non-log-concavity creates multimodality and convergence risks. It presents fractional updates and double-loop procedures as tools for making EP more robust.
- MCMC: MCMC represents the posterior by sampling latent variables and hyperparameters, commonly using a scale-mixture representation of the Student-t distribution.
- MCMC: The scale-mixture Gibbs sampler can mix slowly and become trapped at small noise-scale values because the variance components and scale parameter are dependent.Reparameterization improves mixing but introduces an implicit prior for the Student-t scale parameter.
- Laplace approximation: Laplace approximation expands the conditional log posterior around its mode to construct a Gaussian approximation, then uses an approximate marginal likelihood for hyperparameter MAP estimation.
- Variational inference: Factorizing variational inference assumes posterior independence between latent values and residual variances and estimates them by maximizing a variational lower bound with EM.
- Variational inference: Factorizing variational approximations tend to underestimate posterior uncertainties, and fVB was reported to provide worse predictive variance estimates than Laplace.
- Expectation propagation: EP updates Gaussian site approximations by moment matching, but convergence is not guaranteed and non-log-concave Student-t likelihoods can produce multimodal tilted distributions.
- Double-loop algorithm: Double-loop moment matching requires positive cavity precisions so tilted distributions remain proper with finite moments, although site precisions may become negative for outliers.
- Fractional EP updates: Fractional EP uses a power η ∈ (0, 1] to flatten likelihood terms, reduce computational complexity, and improve robustness when approximation or information propagation is difficult.Standard EP is recovered when η = 1.
4. Robust implementation of the parallel EP algorithm
The robust implementation combines damped parallel EP updates with a double-loop backup, using adaptive step-size controls and numerical safeguards for difficult Student-t cases. It also handles negative site precisions, nonpositive cavity variances, and robust moment and marginal-likelihood evaluations.
- Convergence strategy: The double-loop algorithm is guaranteed to converge to a stationary point when site terms are bounded from below, but can be slower than parallel EP.The implementation therefore starts with damped parallel updates and uses the double-loop method when necessary.
- Adaptive damping: Parallel EP updates are stabilized by reducing damping when the objective fails to increase or when proposed cavity variances become nonpositive.These checks trade additional tilted-moment evaluations for stability and preserve conditions needed to verify objective improvement.
- Adaptive step size: Gradient information can determine a better damping step size when an inner-loop update does not increase the objective.The gradients with respect to site parameters yield the objective gradient along the current update direction without additional objective evaluations.
- Initialization: The implementation begins comparisons with 10 damped parallel iterations using δ = 0.8, which usually achieves convergence in most hyperparameter optimization steps with sensible initialization.If convergence is not achieved, this initialization accelerates subsequent double-loop iterations; ill-conditioned posterior covariance triggers further safeguards.
- Numerical linear algebra: Negative site precisions caused by outliers are processed in separate positive and negative blocks, with Cholesky failure prompting rejection of site parameters and step-size reduction.Regular observations reduce posterior uncertainty through U, whereas outliers increase uncertainty through V.
5. Properties of the EP with a Student-t likelihood
Student-t EP behavior depends strongly on data geometry and hyperparameters: multimodal or poorly matched settings can destabilize updates, while suitable priors and fractional methods improve robustness and predictive uncertainty.
- Outlier rejection in GP regression depends heavily on the data and hyperparameters, with unsuitable settings producing many outliers and possible multimodal posteriors.Small ν, small σ, and large lengthscale can make the model unable to explain many observations.
- The Student-t likelihood can increase posterior uncertainty for observations that are neither clearly regular nor clearly outlying.The largest uncertainty increase occurs near observations offset from the posterior mode by approximately σ√(3ν).
- In conflicting two-outlier examples, EP may assign small positive site precisions to outliers, while insufficient damping can make those precisions negative and destabilize updates.The problematic observations can increase posterior certainty despite being outliers.
- A sufficiently large lengthscale can force a unimodal posterior and allow sequential and parallel EP to converge, whereas stronger nonlinearity creates stronger multimodality and convergence difficulties.In the harder example, standard EP becomes very wide, while Laplace and fVB localize sharply near one mode.
- Damping, fractional updates, and double-loop optimization improve difficult cases, but their trade-offs include slower convergence and altered uncertainty estimates.Fractional EP with η = 0.5 can converge without damping; the double-loop method avoids oscillations but may require over 100 iterations.
- Hyperparameter optimization can fail in nonconvergent regions, with standard EP overestimating marginal likelihood and fractional EP slightly underestimating it in the harder example.The standard EP approximation at MAP values may be unevaluable when the mode lies in a nonconvergent region.
6. Experiments
Experiments compare approximate inference methods for Student-t GP regression using fixed and estimated hyperparameters across multiple datasets. EP generally provides the strongest predictive performance, while hyperparameter optimization—especially for ν—creates convergence, sensitivity, and computational challenges.
- Experimental setup: The evaluation compares LA, EP, fVB, VB, and MCMC on Friedman, Boston housing, concrete quality, and compressive strength data.Performance is assessed with predictive distributions, MAE, and MLPD using test sets or cross-validation.
- Predictive comparisons with fixed hyperparameters: With fixed hyperparameters, EP gives the most accurate predictive statistics, while fVB and VB underestimate predictive standard deviations.Differences from MCMC are clearest in predictive densities, indicating that accurate latent means and variances may not suffice for all predictive statistics.
- Predictive comparisons with estimation of hyperparameters: The Student-t model outperforms the Gaussian model in MLPD across all datasets, although approximation differences vary by dataset.The Student-t model with SM is significantly better than the Gaussian model, while EP1 performs especially well on concrete quality data.
- Predictive comparisons with estimation of hyperparameters: Hyperparameter estimation is especially challenging for ν, whose optimization is sensitive to initialization and can drift toward larger values under fVB.EP-based marginal likelihood estimates are reported as the most robust to hyperparameter initialization.
- Predictive comparisons with estimation of hyperparameters: EP is significantly better than the other approximations on all datasets except compressive strength, where differences are not significant.Either EP1 or EP2 is significantly better than LA, VB, and fVB in all datasets except for the stated compressive-strength comparison with LA1.
- Computational cost: The fastest Student-t approximations require roughly 10-fold the CPU time of the Gaussian baseline, and optimizing ν makes EP substantially slower.The increased EP cost is attributed to more double-loop iterations needed for convergence at larger numbers of optimized parameters.
7. Discussion
The discussion presents EP modifications for difficult Student-t GP regression cases, emphasizing convergence diagnostics, hyperparameter-estimation failures, and the scope of the proposed implementation.
- Non-log-concave site functions can destabilize EP, motivating damping, fractional updates, and alternative double-loop algorithms.The paper focuses on practical implementation issues that had received limited discussion.
- Conflicting outliers in uncertain regions may require very small positive site precisions, which caused problems for regular EP updates.
- Standard EP may fail to converge at maximum marginal likelihood hyperparameter estimates when separated posterior modes require a large local variance increase.These cases involved two outliers producing distant modes in the conditional posterior.
- Double-loop iterations and cavity distributions can help diagnose unsuitable models and reveal when local leave-one-out information dominates.The cavity distributions are interpreted as leave-one-out estimates of latent values.
- The study omits comparisons with Laplace and Gaussian-mixture observation models, while reporting EP as the most consistent method for estimating ν in its experiments.It also describes a grid-based approximation for improving ν estimation and presents modifications applicable to other non-log-concave likelihoods.