Source-linked AI summary
Additive Gaussian Processes
David Duvenaud, Hannes Nickisch, Carl Edward Rasmussen
TL;DR
Regression models must balance interpretable low-dimensional structure against the flexibility of fully interacting kernels, whose complexity can hinder generalization. The paper introduces additive Gaussian processes with efficiently computable interaction kernels and learnable interaction-order structure. Additive GPs improve modeling efficacy and interpretability, achieving state-of-the-art regression performance while recovering flexible models when additive structure is absent.
Problem
Regression models need kernels that represent data structure while balancing GAM interpretability with the flexibility and generalization difficulty of fully interacting squared-exponential GPs.
Method
The paper uses an additive GP kernel that includes interactions from first through Dth order, efficiently evaluates its exponentially many terms, and learns O(D) hyperparameters.
Results
Additive GPs perform best on datasets explained by low-order interactions and approximately match SE-GPs on datasets explained by high-order interactions.
Takeaways & Limitations
Additive GPs provide increased interpretability through order-variance hyperparameters and achieve state-of-the-art regression performance.
Takeaways & Limitations
Performance can suffer from overfitting or local maxima in hyperparameter optimization, and may benefit from integrating out kernel hyperparameters.
Abstract
from arXiv · showhide
We introduce a Gaussian process model of functions which are additive. An additive function is one which decomposes into a sum of low-dimensional functions, each depending on only a subset of the input variables. Additive GPs generalize both Generalized Additive Models, and the standard GP models which use squared-exponential kernels. Hyperparameter learning in this model can be seen as Bayesian Hierarchical Kernel Learning (HKL). We introduce an expressive but tractable parameterization of the kernel function, which allows efficient evaluation of all input interaction terms, whose number is exponential in the input dimension. The additional structure discoverable by this model results in increased interpretability, as well as state-of-the-art predictive power in regression tasks.
1 Introduction
The paper introduces additive Gaussian processes as a tractable model spanning GAM-like first-order effects and fully interacting squared-exponential GP behavior. Its kernel handles exponentially many interaction terms efficiently while improving modeling efficacy and interpretability.
- 1 Introduction: Generalized Additive Models are easy to fit and interpret, but adding many higher-order terms generally becomes intractable and difficult to fit.The paper positions additive GPs as a structured extension of this interpretable model family.
- 1 Introduction: Squared-exponential GP models are more flexible than GAMs because they allow responses to depend on all input variables simultaneously, but they generalize poorly to new combinations.The additive construction targets an intermediate structure between these extremes.
- 1 Introduction: Additive GPs generalize both Generalized Additive Models and squared-exponential Gaussian processes through interactions ranging from first to Dth order.The model can automatically determine which interaction orders matter.
- 1 Introduction: The kernel sums an exponential number of interaction terms but remains efficiently computable with only O(D) hyperparameters.This parameterization makes the model tractable despite the number of possible terms.
- 1 Introduction: HKL explores a similar model class but requires cross-validation for hyperparameters, substantial training machinery, and is outperformed by standard SE-GPs on real datasets.The paper presents additive GPs as a simpler alternative to this related approach.
2 Gaussian Process Models
Gaussian-process behavior is determined largely by the kernel, creating a modeling challenge: flexible kernels capture complex dependencies, while additive structure can represent low-order regularities and support extrapolation. The paper motivates additive kernels as a way to model sums of feature effects and small joint interactions.
- 2 Gaussian Process Models: A GP kernel determines the function structure the model can capture and strongly affects modeling efficacy on small to medium-sized datasets.Choosing a kernel that represents the data's structure is therefore a central modeling difficulty.
- 2 Gaussian Process Models: First-order additive-kernel draws are sums of orthogonal one-dimensional functions and exhibit more long-range structure than higher-order GP draws.This connects first-order additive kernels to Generalized Additive Models.
- 2 Gaussian Process Models: Many natural functions may depend on sums of low-order interactions, such as individual feature effects plus small groups of jointly interacting features.The examples include house prices depending on features such as a sun-roof, size, and building materials.
3 Additive Kernels
Additive kernels combine interactions across orders, from one-dimensional GAM-like terms to the full Dth-order squared-exponential case, while learning order-specific variance. Efficient symmetric-polynomial evaluation makes the exponentially many terms tractable and supports interpretability through low-order functions.
- Kernel construction: Each nth-order additive kernel sums products of n dimension-wise base kernels, with σ2_n controlling the variance assigned to that interaction order.The full additive kernel sums kernels across all orders; the Dth-order term has one product containing every dimension.
- Parameterization: Choosing one-dimensional base kernels per input dimension is sufficient, while their parameters can be learned by maximizing the training-data marginal likelihood.Order-variance hyperparameters additionally indicate how much target-function variance is attributed to each interaction order.
- Model relationships: All first-order variance recovers a GAM, whereas all Dth-order variance recovers an SE-GP, making additive GPs encompass both endpoints.The dominant interaction order can vary across datasets, allowing the model to represent mixtures of low- and high-order structure.
- Interpretability: The additive GP can discover decompositions into low-dimensional functions, supporting both flexible modeling and interpretable plots of low-order components.When most variance is attributed to low-order interactions, the corresponding functions are useful and easy to plot.
- Efficient evaluation: Newton-Girard recurrences evaluate the elementary symmetric polynomials underlying additive kernels in O(D^2), avoiding direct summation over exponentially many interaction terms.The same removal trick computes derivatives with respect to base kernels, enabling marginal-likelihood optimization of their hyperparameters.
- Computation and results: Additive-kernel Gram-matrix evaluation costs O(N^2DR), where R is the maximum interaction order, and experiments indicate that truncating to early orders usually loses little accuracy.The additive GP performed best on low-order datasets and approximately matched SE-GP performance on high-order datasets.
4 Related Work
Prior approaches either restrict interaction orders, use separate weights for exponentially many terms, or rely on difficult hyperparameter selection. Additive GPs retain all interaction orders while learning dimension-specific and order-specific hyperparameters efficiently.
- Hierarchical Kernel Learning: HKL computes a tractable weighted sum over exponentially many kernels but searches restricted hulls and uses cross-validation to set hyperparameters.HKL fixes relative order weighting through a single α-based scheme in the cited formulation.
- Additive GPs: Additive GPs learn individual base-kernel hyperparameters and separate weightings for every interaction order, enabling automatic relevance determination.This contrasts with procedures that select only a tractable subset or constrain order weights.
- Related kernel methods: Support vector ANOVA methods likewise approximate the full sum with one interaction order, largely because hyperparameter setting is difficult.The selected order is described as an appropriate order rather than learned across all orders.
- Related statistical models: SS-ANOVA models usually restrict practice to first- and second-order terms because the number of interaction terms grows exponentially.Each individual interaction term has its own weighting parameter.
- Non-local interactions: Unlike squared-exponential kernels, additive kernels support non-local interactions between points similar in any two input dimensions without spreading kernel mass across the whole space.In three dimensions, third-order kernels remain local while lower-order kernels can connect distant points sharing input values.
5 Experiments
Experiments compare additive GPs with GAM, squared-exponential GP, and HKL variants across diverse datasets. The additive model is never significantly worse than competitors, performs best for low-order structure, and matches squared-exponential GPs for high-order structure.
- Synthetic data: The synthetic experiment uses a sum of two axis-aligned sine functions with an L-shaped training region and a distant test peak.The additive GP recovers both sine functions and identifies first-order interactions as explaining most variance.
- Experimental setup: The experiments compare five models, including additive GP, GP-GAM, squared-exponential ARD GP, and HKL using the all-subsets kernel.For speed, additive kernels are limited to interaction orders up to 10.
- Experimental setup: Tables report mean performance across 10 train-test splits, while HKL is excluded from likelihood comparisons because it lacks a noise model.This constraint applies specifically to likelihood metrics.
- Results: The additive model is never significantly worse than any other model and is sometimes significantly better than all others.Significance is assessed using paired t-tests, with best and statistically indistinguishable models highlighted.
- Results: Additive GPs are reported as a state-of-the-art regression model and benefit relative to HKL from learning individual kernel hyperparameters and interaction-order weights.HKL performs competitively but slightly worse than SE-GP in the cited comparison.
- Results: Additive GPs perform best on datasets explained by low-order interactions and approximately as well as SE-GPs on datasets explained by high-order interactions.The authors attribute slightly worse additive-GP cases to possible overfitting or local maxima in hyperparameter optimization.
6 Conclusion
Additive GPs introduce tractable additive structure that improves interpretability and can improve predictive performance on real datasets. They also support flexible modeling and learn richer kernel structure than HKL.
- Additive GPs provide a simple model family that generalizes two widely used classes of models.
- Order variance hyperparameters indicate which types of structure are present, improving model interpretability.
- Experiments show additive GPs are a state-of-the-art regression model.
- Compared with HKL, additive GPs learn individual kernel hyperparameters and the weightings of different interaction orders.