PaperlayerView exact revision on arXiv

Source-linked AI summary

Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models

Yean Hoon Ong, Paolo Barucca, Wei Pan, Jun Wang

arXiv:2608.29349v1cs.LG

TL;DR

Global GP regression scales poorly, while local GP experts can overestimate posterior variances. The paper proposes GP-pro-c, which uses information gain to calibrate local variances, achieving lower NLL and ENCE while maintaining predictive accuracy and scalability.

  • Problem

    Global GPs incur cubic computational cost, and GP-pro’s disjoint local training subsets can produce overestimated posterior variances.

  • Method

    GP-pro-c uses monotonicity and submodularity of information gain to define a test-time calibration ratio for reducing each local expert’s posterior variance.

  • Results

    GP-pro-c reduces NLL by 2.3% and ENCE by 12.0% on average versus uncalibrated GP-pro, without degrading predictive accuracy.

  • Takeaways & Limitations

    The method preserves GP-pro’s computational advantages and aggregation structure while improving uncertainty calibration and maintaining predictive accuracy.

  • Takeaways & Limitations

    The calibration bound is conservative, and tighter data-dependent bounds would require additional assumptions or adaptive estimation strategies.

Abstract

from arXiv · show

Gaussian process (GP) regression with a single global GP (GP-glo) incurs cubic computational cost, limiting scalability to large datasets. Product-of-experts GP models (GP-pro), which combine local GP models to capture global correlations, alleviate this computational burden. However, training local experts on disjoint data subsets can lead to overestimated posterior variances. We propose GP-pro-c, a product-of-experts GP model that calibrates these variances using an information-based method. The method exploits the monotonicity and submodularity of information gain in GPs to define a calibration ratio that reduces the posterior variance of individual local GP models. We evaluate GP-pro-c using negative log-likelihood (NLL), root mean squared error (RMSE), and expected normalised calibration error (ENCE). Experiments on four synthetic functions and six regression datasets show that GP-pro-c achieves average reductions of 2.3% in NLL and 12.0% in ENCE compared with the uncalibrated GP-pro model. The proposed method mitigates posterior variance overestimation while maintaining predictive accuracy and reducing computational complexity. GP-pro-c provides a promising approach for uncertainty estimation in scalable GP models and may serve as a useful surrogate model for Bayesian optimisation with high-dimensional and large-scale data.

1 Introduction

Global GPs provide uncertainty estimates but scale poorly because their covariance computations grow cubically with dataset size. GP-pro improves scalability through local experts, while GP-pro-c addresses the resulting variance overestimation using information-based calibration.

  • Global GP fitting and inference require O(n^3) computation and O(n^2) memory, hindering large-scale applications.
  • GP-pro partitions data into M subsets and aggregates predictions from local GPs, reducing training complexity to O(M ñ^3).
  • GP-pro offers flexible expert hyperparameters and weighting while capturing local patterns and long-range correlations.
  • Training experts on subsets limits their information, producing overestimated local predictive variances that propagate through aggregation.
  • GP-pro-c adjusts each local expert’s posterior variance using information gain, improving uncertainty estimates while preserving computational efficiency and predictive accuracy.
  • The study compares GP-pro-c with GP-pro, temperature-scaled GP-pro, and a global GP across predictive performance and computational cost.

2 Preliminaries

The preliminaries define global and local Gaussian-process predictions, then explain how local experts are assigned data, trained, weighted, and aggregated. Information-theoretic quantities motivate the later calibration method.

  • Gaussian processes: A Gaussian process defines a distribution over functions using a mean function and covariance function.
  • Gaussian processes: Observed targets equal latent function values plus independent Gaussian noise, with hyperparameters learned from the log marginal likelihood.
  • Product-of-experts GPs: GP-pro trains M local GPs on distinct data subsets and combines their independent predictions through a product-of-experts formulation.
  • Data assignment: Data may be assigned randomly, by k-means clustering, or with balltrees, each producing subsets used to fit local experts.
  • Aggregation: Expert weights represent reliability at a test point and may use entropy-based, variance-based, or uniform schemes.

3 Related Work

Related work addresses GP scalability and uncertainty calibration through sparse, expert-based, and discrepancy-oriented approaches. The paper builds on information gain to directly calibrate local expert variances in GP-pro.

  • GP-pro performance depends on both data assignment and weighting, with prior comparisons covering random, k-means, and balltree assignments.
  • Prior GP scalability methods include sparse inducing-point, expert-based, and dimensionality-reduction approaches.
  • Temperature-scaled calibration adjusts aggregation weights but requires a temperature parameter and does not directly correct individual experts’ overestimated variances.
  • Information gain has supported confidence bounds and variance correction in global GP models for Bayesian optimisation.
  • The proposed method uses Matérn-kernel information-gain bounds to define a pointwise calibration ratio for local experts at test time.

4 Analysing Uncertainty Quantification in Product-of-Experts Gaussian Process Models

GP-pro reduces computational cost by training local experts, but disjoint subsets inflate predictive variances and can degrade uncertainty calibration. This section evaluates calibration and predictive performance using RMSE, NLL, and ENCE across synthetic GP models.

  • Variance overestimation: Local experts trained on data subsets have larger predictive variances because they lack information available to a global GP.GP posterior variance is non-increasing as more training points are incorporated.
  • Variance overestimation: Inflated local variances propagate through expert aggregation, leaving GP-pro’s combined posterior overly conservative.Figure 1 illustrates this effect in a two-expert model using entropy-weighted aggregation.
  • Evaluation metrics: RMSE measures point-prediction accuracy, while NLL jointly evaluates accuracy and uncertainty quality and ENCE specifically assesses calibration.Reliability diagrams compare RMSE and predicted uncertainty through RMV, with calibrated models approaching the identity line.
  • Synthetic evaluation: Across four synthetic functions, GP-pro achieves similar RMSE but higher NLL and ENCE than GP-pro-full and GP-full.The reliability diagrams show GP-pro deviating from the ideal diagonal, indicating overestimated posterior variances.

5 Information-Based Uncertainty Calibration in Product-of-Experts Gaussian Process Models

The method calibrates local GP posterior variances using conditional mutual information and a conservative bound based on information-gain monotonicity and submodularity. Calibration shrinks expert variances at test time while preserving the aggregation rule, computational efficiency, and predictive accuracy.

  • Information-based calibration: Conditional mutual information quantifies the variance reduction needed to correct local experts trained on incomplete data.The calibration ratio relates full-data and local-data posterior standard deviations to information supplied by observations excluded from the local expert.
  • Information-based calibration: Monotonicity and submodularity provide a conservative lower bound on cumulative information gain from unseen observations.Submodularity gives diminishing marginal returns, allowing the cumulative gain to be bounded using the largest marginal contribution without accessing the full dataset.
  • Efficient implementation: The calibration ratio is computed from quantities already available to each local expert at test time, without other experts’ datasets or additional training computation.The local posterior variance serves as the available summary for evaluating the ratio at arbitrary test inputs.
  • Calibration properties: The calibrated variance remains conservative because the proposed ratio does not reduce uncertainty more aggressively than the information-gain lower bound.The authors describe this conservativeness as a safeguard against over-calibration and numerical instability, while tighter data-dependent bounds remain future work.
  • Aggregated predictions: Each expert contributes an information-shrunk posterior variance, while the original product-of-experts aggregation rule remains unchanged.This corrects variance inflation caused by disjoint expert subsets while retaining the computational structure of GP-pro.
  • Aggregated predictions: Calibration reduces predictive variance while maintaining accurate mean predictions in two-expert GP-pro-c models on the one-dimensional Ackley function.Figure 3 compares random and balltree assignment with entropy-weighted aggregation before and after applying the calibration ratio.

6 Experiments

Across assignment methods, GP-pro-c improves uncertainty calibration while preserving predictive accuracy and adding negligible computational overhead. Locality-aware assignments generally improve RMSE and NLL, whereas random assignment can achieve lower absolute ENCE through conservative variances.

  • Overall comparison: GP-pro-c consistently attains lower NLL and ENCE than GP-pro while maintaining comparable RMSE across assignment and weighting schemes.This indicates improved uncertainty calibration without compromising predictive accuracy.
  • Weighting schemes: Among GP-pro-c variants, balltree assignment with entropy weighting produces the lowest RMSE, NLL, and ENCE and the reliability curves closest to the ideal diagonal.
  • Assignment methods: Random assignment achieves lower ENCE but higher RMSE and NLL, whereas balltree and k-means preserve locality and improve predictive accuracy.Random assignment produces conservative variances from weakly correlated experts; locality-aware assignments incur greater information redundancy and more difficult calibration.
  • Interpreting calibration: Lower absolute ENCE does not necessarily indicate larger uncertainty corrections, so ENCE should be interpreted jointly with RMSE and NLL.Random assignment can have lower ENCE despite weaker posterior mean accuracy and smaller apparent corrections.
  • Computational cost: GP-pro-c and GP-pro-sm have similar computational overhead, while GP-full becomes prohibitive on datasets exceeding 5,000 training points.GP-full is omitted in those experiments because of runtime and memory requirements.
  • Assignment and local-GP size: RMSE and NLL decrease monotonically as points per local GP increase, while training time increases accordingly.
  • Assignment methods: k-means performs poorly with small local clusters because high-dimensional clusters can be heterogeneous and poorly conditioned, causing unstable estimates and inflated NLL and ENCE.Performance improves sharply once each expert has sufficient data.
  • Assignment and local-GP size: Balltree assignment with 200-400 data points per local GP provides the best trade-off between predictive accuracy, uncertainty calibration, and computational efficiency.

7 Discussion and Conclusions

GP-pro-c calibrates GP-pro uncertainty using information-based variance correction, preserving predictive accuracy and computational benefits while improving calibration across experiments.

  • Motivation: GP-pro-c targets variance overestimation caused by training local experts on disjoint data subsets.The method is tailored to GP-pro architectures and uses their structural properties.
  • Empirical findings: 2.3% and 12.0% average reductions in NLL and ENCE, respectively, are reported versus uncalibrated GP-pro.These reductions are observed across synthetic functions and real-world regression datasets.
  • Empirical findings: Balltree assignment with entropy-based weighting performs best overall among GP-pro-c variants.Using 200-400 data points per expert provides a reported balance among predictive accuracy, calibration quality, and computational cost.
  • Calibration behavior: The method’s ENCE improvements are consistent but moderate, reflecting conservative correction rather than aggressive reshaping of the posterior distribution.Reliability diagrams show movement toward ideal calibration where GP-pro exhibits inflated variance, though not uniformly at every probability level.
  • Empirical findings: GP-pro-c preserves GP-pro’s aggregation structure and predictive accuracy while improving uncertainty estimates.Stable RMSE values support the claim that calibration does not degrade predictive accuracy.
  • Future work: Future work includes extending the method to alternative kernels, jointly calibrating dependent experts, and establishing theoretical guarantees.The current derivation uses standard normalized-information-gain assumptions, with reference to Matérn kernels.
  • Future work: Integrating GP-pro-c into Bayesian optimisation remains future work because comprehensive BO evaluation is beyond this paper’s scope.The paper identifies uncertainty-sensitive acquisition decisions as a motivation for that extension.

A Information Gain

The information-gain appendix expresses GP information gain through predictive variances, enabling evaluation for hypothetical observations without observing their values.

  • Information-gain expression: Information gain for a point set A can be expressed in terms of predictive variances.This result is presented as the basis for the information-gain calculation.
  • Entropy decomposition: The entropy of observations is decomposed recursively into conditional entropy terms.The decomposition conditions each successive observation on its predecessors.
  • Variance property: The conditional variance of the next function value is independent of previously observed values.The relevant posterior variance depends on preceding inputs and observations structurally, not on their realized values.
  • Gaussian observation model: Observation noise contributes an independent Gaussian variance to each observed function value.The observation is modeled as the sum of the latent function value and independent noise.
  • Computational consequence: Mutual information for hypothetical observations can therefore be computed without observing the data.This property supports evaluating information gain for candidate observation sets in advance.

B Additional Experimental Results

Additional reliability results compare calibrated and uncalibrated product-of-experts GP models across weighting schemes and benchmark datasets.

  • Reliability diagrams: Under entropy-, variance-, and uniform-weighted schemes, GP-pro-c reliability curves lie closer to the ideal diagonal than GP-pro curves.The comparison uses balltree assignment across four synthetic benchmark functions and six regression datasets.
Loading 2608.29349v1…