Source-linked AI summary
Predictively Oriented Gaussian Process Posteriors
Callum Lau, Jeremias Knoblauch, Louis Sharrock
TL;DR
Kernel and likelihood misspecification can impair GP predictive distributions. PrO-GPs target predictive uncertainty through a reduced formulation and practical sampling scheme, and experiments report improved predictive performance under misspecification.
Problem
Choosing a suitable covariance kernel and likelihood is challenging, and misspecification can impair GP predictive distributions and downstream decisions.
Source: §1 · p. 1Method
PrO-GPs choose a predictive-oriented distribution over latent functions, with a finite-dimensional feature reduction and practical sampling scheme for tractable inference.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
PrO-GPs offer a natural robustification when practitioners suspect misspecification, without additional modelling effort, while keeping inference interpretable with respect to the underlying model.
Source: §5 · p. 8The improved robustness requires additional computational effort; for robustifying an existing GP, kernel hyperparameters are treated as external plug-in quantities.
Source: §3.5 · p. 6, §5 · p. 8
Abstract
from arXiv · showhide
Gaussian Processes (GPs) are a powerful tool for modelling and quantifying uncertainty in functional relationships. However, they require practitioners to make a number of design decisions, such as the choice of the kernel and the observation model. Suboptimal choices can produce misspecified models that do not capture the underlying data generating process. We introduce Predictively Oriented Gaussian Processes (PrO-GPs), which treat predictive uncertainty as the primary inferential target and provide a robust alternative to standard GPs. Although direct computation of a PrO posterior for nonparametric models is intractable, we derive a reduced formulation and practical sampling scheme for efficient computation. Through synthetic and real data experiments, we show that PrO-GPs produce better calibrated predictive distributions under model misspecification compared to standard GP approaches.
1 INTRODUCTION
GP predictions depend on kernel and likelihood choices that can be difficult to specify correctly. PrO-GPs instead target predictive distributions, with a finite-dimensional reduction and sampling scheme supporting tractable inference and improved uncertainty quantification under misspecification.
1 INTRODUCTION
- Kernel and likelihood choices are challenging and shape GP predictions; misspecification can make downstream decisions non-robust or poorly approximate the data-generating process.GPs are used to model functional relationships, including when data are scarce or expensive to obtain.Source: §1 · p. 1
- PrO-GPs seek a mixing distribution over latent functions whose induced predictive distribution best describes the data, rather than concentrating on one best-fitting function.On multimodal data, the GP predictive mean can lie between modes, while PrO-GPs assign predictive mass to both branches.Source: §1 · p. 1, Figure 1 · p. 1
- Unlike approaches that restrict induced predictions to Gaussian or finite Gaussian mixtures, PrO-GPs allow unrestricted uncertainty over latent functions.Rather than changing the kernel, likelihood, or loss for a specific misspecification, the approach modifies the inferential procedure.Source: §1 · p. 2
- An exact feature-reduction theorem transforms optimization over probability measures on function space into optimization over a finite-dimensional feature space.This provides a tractable Euclidean representation of PrO-GP posteriors.Source: §1 · p. 2
- A practical sampling scheme supports efficient, scalable optimization of the Euclidean objective.The introduction presents this as an inducing-feature approximation with an efficient sampling scheme.Source: §1 · p. 2
- PrO-GPs significantly improve predictive uncertainty quantification on synthetic misspecification and real-data regression tasks.Source: §1 · p. 2
2 BACKGROUND
The background defines the regression and GP setting, then describes Bayesian prediction through a posterior over functions and its variational characterization. It contrasts that function-wise Bayesian objective with the paper’s focus on predictive distributions.
2 BACKGROUND
- The setup distinguishes the true conditional response distributions from those induced by the assumed model, calling the model well-specified when the true distributions belong to its model family.Source: §2 · p. 2
2.1 Gaussian Processes
- A GP places a probability measure over functions using a mean and covariance kernel; Bayesian regression updates this prior with the likelihood and averages conditional response distributions for prediction.For a homoscedastic Gaussian response model, the posterior predictive is Gaussian.Source: §2.1 · pp. 2–3
2.2 Variational form of Bayesian posteriors
- The variational Bayesian posterior minimizes function-wise predictive log loss plus KL regularization toward the prior, while predictive orientation instead makes the induced predictive distribution the primary concern.Under suitable regularity conditions, the Bayesian posterior concentrates on a single loss-minimizing function, even under prior or observation-model misspecification.Source: §2.2 · p. 3
- The variational formulation sits within Gibbs posteriors with more general scoring rules, though this work primarily focuses on the logarithmic score.Source: §2.2 · p. 3
3 METHODOLOGY
The methodology defines a predictive objective over function-space laws, then develops finite-dimensional reduction, scalable approximation, sampling, and hyperparameter procedures.
3.1 Predictively Oriented GP posteriors
- PrO inference scores the induced predictive mixture directly, unlike Bayesian inference, and can represent uncertainty over latent functional regimes that a single compromise function misses.The paper illustrates this with observations generated by one of two latent functions.Source: §3.1 · p. 3
3.2 Exact feature reduction
- The exact reduction replaces optimization over function-space probability measures with optimization over probability measures on the finite-dimensional feature space R^n.Even with a GP prior and Gaussian observation model, the predictive distribution is generally a non-Gaussian mixture because the optimized mixing law can be non-Gaussian.Source: §3.2 · p. 4
- Unlike explicit mixtures of GPs, PrO-GPs retain a single GP prior and likelihood, with non-Gaussian predictive uncertainty emerging when supported by data.This avoids specifying mixture components, their number, or gating mechanisms in advance.Source: §3.2 · p. 4
3.3 Inducing-point approximation
- The inducing-point approximation restricts the free law to a low-dimensional feature vector, reducing optimization to P(R^m) with m much smaller than n while leaving its distribution unrestricted.Unlike PPGPR's Gaussian approximation to the mixing law, this permits multimodal or asymmetric latent-function explanations.Source: §3.3 · pp. 4–5
3.4 Computation
- Replica–Gibbs alternates allocation and Gaussian particle updates; its particle marginals converge to the objective's minimizing law as K grows.The algorithm is exact for the particle objective and introduces no step-size or time-discretisation error.Source: §3.4 · pp. 5–6
3.5 Hyperparameter Optimisation
- Kernel hyperparameters γ are treated as external plug-in quantities for robustifying an existing GP, though they can instead be optimized as part of a standalone PrO-GP.The plug-in value may come from expert elicitation, leave-one-out predictive performance, or the underlying GP's marginal log-likelihood.Source: §3.5 · p. 6
- Observation variance σ² is updated against held-out predictive density while τ is fixed, and λ_n/n is fixed to 1 for comparison with Bayesian GPs.The data are partitioned into disjoint training and validation sets; jointly optimizing σ² and τ on the same data could drive σ toward zero.Source: §3.5 · p. 6
4 EXPERIMENTS
The experiments compare predictive uncertainty on held-out NLPD across synthetic misspecification, traffic-flow interpolation, and UCI regression. PrO-GPs generally improve predictive performance and calibration under misspecification, with benefits depending on whether the baseline GP is already well specified.
4.1 Synthetic misspecification
- Under misspecification, GPs fit block outliers sharply, spread variance across the domain under heteroskedasticity, and place multimodal predictive density between modes.Pro-GPs instead represent alternative latent-function scenarios, functional uncertainty in heterogeneous regions, and multimodal uncertainty; in the well-specified case, their performance converges to GP as data grows, and a standard GP is preferable.Source: §4.1 · pp. 6–7
- Pro-GP yields lower mean NLPD than GP under block outliers, heteroskedasticity, and multimodality.Figure 2 reports mean NLPD and 95% credible intervals over increasing training-set sizes, randomized over twenty DGPs.Source: §4.1 · p. 6, Figure 2 · p. 7
4.2 Probabilistic interpolation of traffic flow
- Graph pro-GP improves mean NLPD and predictive calibration over Graph GP, with the NLPD gap increasing as training sensors increase.The authors interpret this pattern as evidence that Graph GP is non-trivially misspecified.Source: Table 2 · p. 7, §4.2 · p. 7
- On the traffic network, predictive means are nearly identical, but Graph pro-GP predicts more variable standard deviations, including in regions with more training data.The authors hypothesize that this reflects variation in traffic and congestion patterns across roads that Graph GP cannot represent.Source: §4.2 · p. 7, Figure 3 · p. 7
4.3 UCI regression
- Exact pro-GPs reduce mean NLPD versus GPs except on machine and stock; sparse i-pro-GPs reduce it across all considered datasets.The UCI evaluation covers 16 datasets; the exact machine and stock results are comparable, while the sparse result holds regardless of the hyperparameter objective.Source: Table 3 · p. 8, §4.3 · p. 8
5 CONCLUSION
The paper presents PrO-GPs as a way to improve predictive robustness under GP model misspecification, while noting the additional computational effort. It also describes how generative AI tools were used in the work.
5 CONCLUSION
- PrO-GPs improve predictive performance under model misspecification, supported by theoretical and computational machinery for tractable inference.The conclusion frames them as a robustification approach for GP predictive distributions.Source: §5 · p. 8
- The robustness improvement comes at additional computational effort, while requiring no additional modelling effort.The authors say the resulting inference remains interpretable with respect to the underlying model.Source: §5 · p. 8
- The authors identify alternative scoring rules and observation models, and extensions to multi-output, deep, and spatio-temporal GPs, as future directions.The examples include logistic likelihood for GP classification.Source: §5 · p. 8
AI use statement
- Generative AI tools assisted with synthetic data, proof writing, implementation, and dataset cleaning and reformatting; the authors report reviewing all AI-assisted work.They state that these tools were not used to develop theoretical models or conceptual frameworks, formulate mathematical claims, or design experiments.Source: AI use statement · p. 9
Supplementary Materials
The supplementary material lists core notation and outlines the proof strategy for feature reduction. Its general theorem makes the function-space PrO optimization equivalent to an optimization over finite-dimensional features under a feature-factorization assumption.
A NOTATION
- The notation defines D_n as observed data, J as the PrO-GP objective, S as the function-space PrO objective, and Q as the posterior predictive measure.It also names the space of Borel measures, conditional response laws, true conditional responses, Bayes and predictively oriented posteriors, and KL divergence.Source: A NOTATION · p. 12
B THEORETICAL PROOFS
- The proofs first state general results for nonparametric models with a feature representation assumption, then specialize to GP regression.Theorem proofs are given in Appendices B.2 and B.3, with the GP specialization in B.4.Source: B THEORETICAL PROOFS · p. 13
B.1 Feature reduction for nonparametric models
- Under feature factorization, Theorem 3 makes minimizing the function-space PrO objective equivalent to optimization over probability measures on a finite-dimensional feature space.Theorem 4 introduces an inducing-feature restriction by mapping functions to a finite-dimensional feature vector with an induced prior law.Source: §B.1 · p. 13
B.2 Proof of Theorem 3
The appendices establish finite-feature reductions for the PrO objective and connect them to GP regression, posterior prediction, and parametric GP methods. They also describe the Replica–Gibbs sampler and its computational trade-offs.
B.2 Proof of Theorem 3
- The function-space PrO objective reduces to optimization over the pushforward feature law, with the prior-conditional lift preserving the KL divergence.The proof establishes the reduction by matching the feature-law objective in both directions.Source: §B.2 · p. 15
B.3 Proof of Theorem 4
- Restricting the function-space laws to those determined by inducing features makes the PrO optimization equivalent to optimization over their lower-dimensional feature law.The proof applies the prior-conditional lift to the inducing map.Source: §B.3 · p. 16
B.4 Proofs of Theorems 1 & 2
- For GP regression, pointwise evaluations at training inputs instantiate the exact reduction, while evaluations at inducing points yield the inducing-point result.The appendix presents Theorems 1 and 2 as corollaries of the general feature-map results.Source: §B.4 · p. 16
C.1 Variational characterisation of the Bayesian posterior
- When λn = n, the variational formulation recovers the Bayesian posterior exactly.Source: §C.1 · p. 17
C.2 Posterior Prediction for PrO-GPs
- The PrO posterior predictive is obtained by integrating the conditional GP over the optimized feature law, approximated in practice with particles.The finite-particle predictive is used to evaluate NLPD in the experiments.Source: §C.2 · p. 17
C.3 Connection to Parametric Gaussian Process Regressors
- With a Gaussian restriction on the inducing-feature law, the inducing-point PrO-GP objective is equivalent to the PPGPR objective.This equivalence fixes the inducing points, kernel hyperparameters, and observation noise.Source: §C.3 · p. 18
C.4 Replica–Gibbs algorithm and computational complexity
- Replica–Gibbs has per-sweep time complexity O(Knm^2 + Km^3) and space complexity O(nm).Source: §C.4 · p. 19
- Increasing K permits richer empirical mixing laws but multiplies the dominant O(Knm^2) cost; the experiments use K = 50 as a suitable trade-off.Source: §C.4 · p. 19
C.5 Degenerate σ →0 solutions during hyperparameter optimisation
This section describes a degeneracy in joint noise optimization, a training/validation procedure to address it, practical guidance on λn, and an alternative Wasserstein sampler with additional numerical costs.
C.5 Degenerate σ →0 solutions during hyperparameter optimisation
- Using the same data to fit particles and optimize σ can drive σ toward zero, as particles fit training residuals and predictive variation shifts into the function mixture.Source: §C.5 · pp. 19–20
- Splitting data into training and validation sets prevents particle fitting and noise optimization from reusing the same observations, though alternative regularizers require dataset-dependent tuning.Source: §C.5 · p. 20
C.6 Hyperparameter optimisation
- Hyperparameter optimization alternates particle updates and training-objective kernel updates with validation-score noise updates, recomputing kernel-dependent quantities after kernel changes.The procedure adds repeated Cholesky factorizations at cost O(nm^2) per iteration.Source: §C.6 · p. 20
C.7 Discussion on λn
- For fixed inducing-point counts, cross-validation over the scaling constant is justified; when m ≪ n, λn has negligible impact except in atypical cases needing heavy prior regularization.The authors leave detailed theoretical treatment of this guidance for future work.Source: §C.7 · p. 21
C.8 Sampling via the Wasserstein Gradient Flow
- The Wasserstein Gradient Flow scheme extends sampling to proper scoring rules, but the authors report extra error, slower convergence, and lower stability.Its step size may be difficult to tune, and log-score gradients introduce a term proportional to 1/σ^2.Source: §C.8 · pp. 21–22
D ADDITIONAL EXPERIMENTAL DETAILS
- The reported experiments ran on a desktop with an AMD Ryzen 5 7600 CPU, 24 GB RAM, and eight CPU threads.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 22
D.1 Negative Log Predictive Density
- NLPD is the negative log predictive density; for PrO-GPs it is computed from the finite-particle approximation to the predictive distribution.Source: §D.1 · p. 22
D.2 Synthetic experiments from Section 4.1
The synthetic experiments compare GP and PrO-GP held-out NLPD across four data regimes, using repeated random splits and specified fitting procedures.
D.2 Synthetic experiments from Section 4.1
- The experiments evaluate held-out NLPD in block-outlier, heteroskedastic, multimodal, and well-specified synthetic regimes.Training sizes are n ∈ [100, 200, 400].Source: §D.2 · p. 22
- The misspecification settings include localized training outliers, branch-mixture observations, and position-dependent noise.The heteroskedastic regime locally inflates the noise variance around a sampled region.Source: §D.2 · p. 22
- Held-out NLPD is averaged over 20 splits with 95% credible intervals; PrO-GP uses GP kernel settings and adapts observation noise before final sampling.The GP kernel and noise are optimized with L-BFGS-B, while PrO-GP uses 100 adaptation steps and 30 final sampling steps.Source: §D.2 · p. 22
D.3 Traffic-flow experiment from Section 4.2
The traffic-flow experiment compares graph GP and graph PrO-GP predictions on the San Jose freeway sensor network, including predictive means and standard deviations.
D.3 Traffic-flow experiment from Section 4.2
- The network has 1,016 nodes and 1,173 edges, with traffic-speed measurements at 325 sensors; training uses 200, 225, 250, or 275 sensors.Average held-out NLPD is reported over five-fold random splits.Source: §D.3 · p. 23
- The graph GP uses a graph Matérn kernel, and the PrO-GP keeps the selected GP kernel hyperparameters fixed for posterior prediction.The GP is selected from five random restarts by training marginal log-likelihood.Source: §D.3 · p. 23
- Figure 4 displays graph GP and graph PrO-GP predictive means and standard deviations on the San Jose road network, marking sensor and training nodes.The panels show GP mean and standard deviation above, and PrO-GP mean and standard deviation below.Source: Figure 4 · p. 23
- The plotted standard deviation is for the full posterior predictive distribution, including both function uncertainty and observation noise.Source: §D.3 · p. 23
D.4 UCI regression experiments from Section 4.3
The UCI experiments compare GP variants under a shared regression setup, finding that PrO-GP reduces NLPD except on machine and stock. Residual diagnostics distinguish those datasets from concreteslump, where departures from normality are strongest.
D.4 UCI regression experiments from Section 4.3
- PrO-GP reduces NLPD on all datasets except machine and stock.The comparison uses an RBF kernel and 80/20 train/test partitions with random 10-fold splits.Source: §D.4 · p. 24
- Concreteslump residuals show significant right-skewness and heavier tails, while machine is broadly normal and stock has departures in both tails.The residuals are Cholesky-whitened GP predictions; under the GP prior, they should be normally distributed.Source: §D.4 · p. 24
- When the model is already well-specified, fitting a PrO posterior offers no advantage in predictive performance.Source: §D.4 · p. 24
- Figure 5 shows Q–Q plots with 95% pointwise bands, residual histograms against the standard normal density, and residuals by training index.Source: Figure 5 · p. 24
D.5 Numerical runtime analysis
The runtime analysis compares i-pro-gp Replica–Gibbs sweeps with svgp gradient steps on parkinsons, holding i-pro-gp hyperparameters fixed. Sweeps cost more per iteration, but far fewer are needed, so the added total fitting time is small.
D.5 Numerical runtime analysis
- The benchmark uses parkinsons with 100, 250, 500, and 1000 inducing points, K = 50 particles for i-pro-gp, and batch size 512 for svgp.The analysis holds i-pro-gp hyperparameters fixed and excludes the runtime of additional hyperparameter optimization.Source: §D.5 · p. 25
- An i-pro-gp Replica–Gibbs sweep costs roughly 10 times more than an svgp gradient step, but training score plateaus after approximately 50 sweeps.The authors estimate about O(10) sweeps for convergence, versus O(10^3) to O(10^4) svgp gradient steps; despite the higher per-iteration cost, total added fitting time is small.Source: §D.5 · p. 25
- Figure 6 compares per-iteration costs for i-pro-gp and svgp and plots i-pro-gp average training score against Replica–Gibbs iteration.Source: Figure 6 · p. 25