Source-linked AI summary
Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
Yanshuai Cao, David J. Fleet
TL;DR
Combining multiple learned-model predictions requires a scheme that is scalable, input-adaptive, probabilistically valid, and robust to unreliable experts. The paper develops gPoE for Gaussian processes by weighting expert densities with input-dependent reliability based on entropy change. Experiments and analysis show that GP-gPoE provides these properties while retaining a Gaussian closed form and strong empirical performance.
Problem
Existing prediction-fusion schemes do not simultaneously provide independent training, input-dependent combination, valid probabilistic predictions, and automatic filtering of unreliable experts.
Method
gPoE combines independently learned GP expert distributions by raising each density to an input-dependent reliability power based on prior-to-posterior entropy change.
Results
GP-gPoE combines predictions into a Gaussian model and consistently outperforms bagging, MoE, and PoE on standardized negative log probability and standardized mean square error.
Takeaways & Limitations
GP experts trained independently on small data subsets can be combined into expressive and robust predictions, with parallel training supporting scalability.
Takeaways & Limitations
gPoE cannot capture multimodality, and positive entropy change does not guarantee reliable prediction under model misspecification.
Abstract
from arXiv · showhide
In this work, we propose a generalized product of experts (gPoE) framework for combining the predictions of multiple probabilistic models. We identify four desirable properties that are important for scalability, expressiveness and robustness, when learning and inferring with a combination of multiple models. Through analysis and experiments, we show that gPoE of Gaussian processes (GP) have these qualities, while no other existing combination schemes satisfy all of them at the same time. The resulting GP-gPoE is highly scalable as individual GP experts can be independently learned in parallel; very expressive as the way experts are combined depends on the input rather than fixed; the combined prediction is still a valid probabilistic model with natural interpretation; and finally robust to unreliable predictions from individual experts.
1 Introduction
The paper seeks a prediction-fusion scheme that combines independently trained models adaptively, probabilistically, and robustly to unreliable experts. It proposes gPoE, which preserves PoE’s sharp, analytically Gaussian combination while meeting these four goals.
- Motivation: Four desired properties are independent expert training, input-dependent combination, valid probabilistic predictions, and automatic filtering of unreliable experts.These properties support scalability, expressiveness, uncertainty-aware decisions, and robustness, respectively.
- Proposed approach: gPoE achieves all four desired fusion properties without requiring joint training or a trained meta-model.Its input-dependent weighting provides expressive and robust combination while retaining a valid probabilistic interpretation.
- Existing fusion schemes: Fixed-weight mixtures and bagging fail to adapt predictions to inputs and automatically filter poor experts.Input-dependent mixture gating can address those properties, but still requires joint training and its filtering depends on that training.
- Existing fusion schemes: PoE can let one overconfident expert with a poor prediction damage the combined model because it lacks a gating mechanism to suppress bad experts.Its veto behavior makes low probability from a single expert decisive for the product prediction.
- Proposed approach: Gaussian-process gPoE retains PoE’s ability to combine many weak uncertain predictions into a sharp Gaussian distribution with closed analytical form.The result is another Gaussian distribution rather than an arbitrary unnormalized product.
2 Generalized Product of Expert
gPoE generalizes PoE by raising each expert density to an input-dependent reliability power, controlling how much each prediction influences the product. For Gaussian processes, entropy change supplies this reliability measure and suppresses experts lacking useful local information.
- 2.1 PoE: PoE multiplies expert conditional densities and renormalizes their product into a probability distribution.For Gaussian experts, the product remains Gaussian, with confidence represented through predictive precision.
- 2.1 PoE: PoE gives experts veto power, so an overconfident expert with a biased mean and erroneously low variance can harm the combined prediction.This occurs because predictive precision is not always a reliable measure of prediction quality.
- 2.2 gPoE: gPoE raises each expert density to a nonnegative, input-dependent power α_i(x) that measures the expert’s reliability at x.α_i(x)=1 recovers PoE; values above one sharpen an expert, while values below one broaden it.
- 2.2 gPoE: As α_i(x) approaches zero, an expert’s influence becomes negligible, whereas α_i(x) approaching infinity gives its mode increasingly strong veto power.Thus the reliability power directly controls whether an expert is ignored, softened, or made dominant.
- 2.2 gPoE: For Gaussian experts, gPoE remains Gaussian because each power α_i effectively scales the expert’s precision before combining predictions.The resulting mean and covariance therefore retain the analytical Gaussian form of PoE.
- 2.3 gPoE for Gaussian processes: For GP experts, α_i(x) is chosen as the entropy change from prior to posterior at x, using variances already computed during prediction.Zero entropy change means the expert learned nothing relevant at x, so its weight becomes zero.
- 2.3 gPoE for Gaussian processes: Entropy change is preferred over variance difference because it is unitless and preserves the correct units of the resulting predictive variance.KL divergence is identified as a possible alternative, but entropy change was effective in the reported experiments.
3 Experiment
Experiments compare gPoE with bagging, MoE, and PoE across three datasets and multiple GP-expert constructions. gPoE consistently performs best on SNLP and SMSE, while tree-gPoE provides a further boost and PoE is especially poor on SNLP.
- Experimental setup: Across three datasets and three expert constructions, the study compares gPoE against bagging, MoE, and PoE using 512 experts of 256 training points each.The datasets are KIN40K, SARCOS, and UK-APT; the expert constructions are random subset, local, and tree-based.
- Results: gPoE consistently outperforms bagging, MoE, and PoE by a large margin on standardized negative log probability and standardized mean square error.These are the SNLP and SMSE evaluation metrics used in Tables 1 and 2.
- Results: Tree-gPoE gives a slight further performance boost across the board by retaining only experts on the test point’s root-to-leaf tree path.This is implemented by setting the weights of off-path experts to zero.
- Results: PoE is almost always poor, especially on SNLP, consistent with the analysis that misguided expert over-confidence can damage its combined prediction.The results support entropy-change weighting as an effective correction for this problem.
- Results: GP experts trained on only 256 points can match or exceed sophisticated sparse GP approximations on UK-APT, while training takes less than 30 seconds.The UK-APT comparison uses RMSE, the measure reported by the SVI-GP work.
4 Discussion and Conclusion
The paper concludes that gPoE combines independently learned GP experts into a Gaussian model with greater expressiveness and robustness to poor predictions. It also identifies limits around multimodality and the reliability of entropy change under model misspecification.
- Conclusion: gPoE combines independently learned GP experts without further training, producing a Gaussian prediction with increased expressiveness and robustness to poor experts.The combined model is a generalized product of experts.
- Limitations and future work: gPoE cannot capture multimodality like mixture-of-experts models, motivating generalized products of mixtures as future work.The proposed extension is intended to capture both “or” and “and” constraints.
- Limitations and future work: Zero entropy change indicates an irrelevant prediction, but sufficient entropy change does not necessarily guarantee reliable prediction under model misspecification.Empirical results suggest reliability with RBF kernels, while further theoretical work is needed for the converse case.