Source-linked AI summary
Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo
Markus Heinonen, Henrik Mannerström, Juho Rousu, Samuel Kaski, Harri Lähdesmäki
TL;DR
Standard GPs assume fixed noise variance, signal variance, and lengthscale, even though these properties can vary with input and make the analytical posterior intractable. This paper jointly models all three as input-dependent and uses gradient-based HMC and MAP inference; it improves regression when data are nonstationary and matches stationary performance otherwise.
Problem
Standard GPs keep noise variance, signal variance, and lengthscale fixed even though noise or signal smoothness can vary across inputs, making the analytical posterior intractable.
Source: §1 · p. 1Method
The framework gives all three components input-dependent GP priors and uses analytical gradients for exact-posterior HMC sampling and MAP estimation.
Source: §1 · pp. 1–2Results
The method improves regression performance on nonstationary datasets and achieves equivalent performance to a stationary model when datasets are not nonstationary.
Source: §4 · p. 8Takeaways & Limitations
On the jump dataset, signal variance did not model the dynamics, whereas nonstationary lengthscale produced a good model.
Source: §4 · p. 8The MAP solution is guaranteed only to converge to a local optimum and is adequate when the posterior is close to unimodal.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
We present a novel approach for fully non-stationary Gaussian process regression (GPR), where all three key parameters -- noise variance, signal variance and lengthscale -- can be simultaneously input-dependent. We develop gradient-based inference methods to learn the unknown function and the non-stationary model parameters, without requiring any model approximations. We propose to infer full parameter posterior with Hamiltonian Monte Carlo (HMC), which conveniently extends the analytical gradient-based GPR learning by guiding the sampling with model gradients. We also learn the MAP solution from the posterior by gradient ascent. In experiments on several synthetic datasets and in modelling of temporal gene expression, the nonstationary GPR is shown to be necessary for modeling realistic input-dependent dynamics, while it performs comparably to conventional stationary or previous non-stationary GPR models otherwise.
1 Introduction
Standard GPs keep noise variance, signal variance, and lengthscale fixed across inputs, although real-world noise and smoothness can vary, making the analytical posterior intractable. The paper introduces a fully non-stationary model and exact gradient-guided HMC and MAP inference without variational or expectation-propagation approximations.
1 Introduction
- Standard GPs hold noise variance, signal variance, and lengthscale constant, while real-world noise or signal smoothness may vary across inputs.In these cases, the analytical GP posterior becomes intractable.Source: §1 · p. 1
- Earlier GP extensions modeled input-dependent noise, lengthscale, or signal variance using ML, MAP, MCMC, or approximate inference (Kersting et al., 2007; Quadrianto et al., 2009; Goldberg, 1998; Lazaro-Gredilla and Titsias, 2011; Tolvanen et al., 2014).The paper identifies variational and expectation-propagation approximations in prior fully Bayesian methods.Source: §1 · pp. 1–2
- The framework makes noise variance, signal variance, and lengthscale simultaneously input-dependent, with GP priors.It proposes inference for the exact joint posterior of the underlying signal and all three latent functions.Source: §1 · pp. 1–2
- Analytical model gradients guide HMC-NUTS sampling, while gradient ascent provides a MAP solution without variational or expectation-propagation approximations.The experiments report that nonstationary GPR is needed for realistic input-dependent dynamics and performs comparably to stationary or prior nonstationary models otherwise.Source: §1 · p. 2
2 Heteroscedatic nonstationary GP model
The model gives the signal, lengthscale, and noise variance separate input-dependent latent functions, then uses gradient-based MAP or HMC inference and predictive distributions to learn and extrapolate them.
2 Heteroscedatic nonstationary GP model
- Signal variance, lengthscale, and noise variance are modeled as smoothly varying latent functions with separate GP priors.Logarithmic priors ensure their positivity, and the nonstationary kernel reduces to the standard squared exponential kernel when signal variance and lengthscale are constant.Source: §2 · p. 2
- The framework combines heteroscedastic noise and nonstationary lengthscale while also allowing signal variance to vary over inputs.Restricting parameters to constants or varying only selected components recovers several earlier models.Source: §2 · pp. 2–3
2.1 Maximum a posteriori estimation
- MAP estimation maximizes the marginal likelihood over latent functions using analytical gradients and gradient ascent.It is guaranteed only to reach a local optimum, and the authors say it is adequate when the posterior is close to unimodal.Source: §2.1 · p. 3
2.2 HMC inference
- HMC samples the posterior over the three latent vectors, with the function posterior obtained analytically as a mixture of Gaussians.The sampling uses the same gradients as MAP inference.Source: §2.2 · p. 4
2.3 Posterior whitening
- Posterior whitening samples transformed latent vectors using Cholesky decompositions of the GP prior covariances to address prior-induced correlations.The latent posterior is described as highly correlated, making direct Monte Carlo sampling inefficient.Source: §2.3 · p. 4
2.4 Making predictions
- For prediction, the model extrapolates the latent functions to target inputs before forming the function posterior.The MAP approach can use conditional means for the extrapolated lengthscale and signal variance when observed inputs are sufficiently dense; HMC predictions average over latent samples.Source: §2.4 · pp. 4–5
- Jointly modeling latent functions over observed and target inputs is an alternative, but increases latent-vector sizes to n + n⋆.Source: §2.4 · p. 5
3 Experiments
The experiments assess regression on synthetic datasets and gene-expression time series, alongside recovery of latent functions. Results show that matching input-dependent structure supports strong prediction, while HMC captures posterior modes and the method reconstructs latent processes.
3 Experiments
- The evaluation used eight synthetic datasets and one gene-expression dataset, with half of each dataset used for training and the remainder for testing.The gene-expression data comprise 205 time series measured after irradiation.Source: §3 · p. 5, §3.3 · p. 7
3.1 Regression performance
- On synthetic datasets, matching nonstationarities gave the best or second-best NLPD; stationary GP had slightly better MSE than other models in every dataset but remained worse than the optimal nonstationary method.Table 2 reports test MSE and NLPD across MAP models.Source: §3.1 · pp. 5–6, Table 2 · p. 6
- On dataset J, the lengthscale-GP achieved NLPD −0.72 (optimal), whereas the signal-variance GP had NLPD 1.04; adding heteroscedastic noise weakened the models.Adding unnecessary nonstationarities generally retained or only slightly worsened performance, with J the major exception.Source: §3.1 · p. 6
3.2 HMC performance
- HMC samples captured multiple latent modes in dataset Dℓ, while MAP chose both zigzags; signal variance had the broadest posterior variation.Across datasets and methods, MAP was slightly better on average, although some HMC samples performed better.Source: §3.2 · p. 6, Figure 1 · p. 6, Figure 2 · p. 7
3.4 Latent function reconstruction
- As noisy observations increased, regression error, lengthscale, and noise variance converged to their true values; signal variance showed small bias but remained well estimated.The latent functions were also accurately estimated in comparisons using 150 datapoints.Source: §3.4 · p. 7
3.3 Biological dataset
- Across 205 gene-expression series, heteroscedasticity greatly improved model fits and likelihoods, while full nonstationarity further improved fits and NLPD with similar MSE to the heteroscedastic GP.Figure 3 compares stationary, heteroscedastic, and fully nonstationary models against the heteroscedastic GP.Source: §3.3 · p. 7, Figure 3 · p. 8
4 Discussion
The fully non-stationary GP infers latent functions and improves regression when data are truly nonstationary, while matching stationary-model performance otherwise. Its discussion also highlights how lengthscale and signal variance play distinct roles and notes care needed when extending HMC to hyperparameters.
4 Discussion
- The method improves regression on truly nonstationary datasets and achieves equivalent performance to a stationary model when they are not.It also effectively samples from the exact posterior of the latent functions.Source: §4 · p. 8
- On the jump dataset, signal variance alone cannot model the dynamics, whereas a nonstationary lengthscale produces a good model.Signal variance acts as a linear amplitude, while lengthscale can affect the function model nonlinearly.Source: §4 · p. 8
- Extending HMC to infer hyperparameters θ requires care in setting their priors.The authors note that natural gradients or position-dependent mass matrices could further enhance gradient-based HMC.Source: §4 · p. 8
- With 150 data points, latent lengthscales and noise are estimated correctly, while signal variance is approximately matched.The figure compares posterior and MAP reconstructions with generating functions and the σ, ω-GP model of Tolvanen et al. (2014).Source: Figure 5 · p. 9
Supplemental information
The supplemental material compares the fully nonstationary GP with a vanilla GP, gives a positive-definiteness argument for the kernel, states conditional distributions for latent functions, and notes analytic derivatives of the marginal log likelihood.
Comparison between vanilla and (ω, σ, ℓ)-GP
- The nonstationary, heteroscedastic GP fits the nonstationary dataset better than vanilla GP, whose stationary lengthscale yields overwide confidence intervals and spurious oscillations.Figure 6 compares the nonstationary model on the right with vanilla GP on the left.Source: Figure 6 · p. 10, Comparison between vanilla and (ω, σ,… · p. 10
Kernel SDP proof
- The kernel K_f is stated using a one-dimensional Gaussian SDP kernel, and is positive definite for any function σ(·).Source: Kernel SDP proof · p. 10
Conditional distributions
- The conditional distributions at target timepoints are given for the latent lengthscale, signal variance, and noise functions given their values at observed timepoints.The corresponding kernels are standard Gaussian kernels computed using hyperparameters θ.Source: Conditional distributions · p. 10
Partial derivatives
- The partial derivatives of the unconstrained latent functions with respect to the marginal log likelihood are analytically derived.The displayed likelihood combines the observation likelihood with GP prior terms for the three latent functions.Source: Partial derivatives · p. 11