PaperlayerView exact revision on arXiv

Source-linked AI summary

On Basis Function Selection for Sparse Gaussian Process Regression

Marnix Van Soom, Ivan De Boi

arXiv:2609.26624v2stat.MLcs.LG

TL;DR

Sparse-GP practitioners conventionally spend a fixed compute budget by truncating a basis expansion, and the paper asks whether selecting basis functions can do better. It derives three information-theoretic selection criteria and evaluates them across three sparse-GP methods. No-data selection matches or beats truncation across methods, while data-aware selection adds gains for HSGP.

  • Problem

    Sparse-GP practitioners conventionally truncate a fixed basis to spend their compute budget, while the formalism permits selecting candidates and leaves the choice of criterion open.

    Source: §1 · pp. 1–2

  • Method

    The paper derives information-theoretic criteria that rank basis functions using prior weight, data projection, or both.

    Source: §3 · p. 4

  • Results

    No-data selection ties or beats truncation across HSGP, VFF, and VISH, with substantial VISH gains, while data-aware criteria improve on HSGP.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Which selection criterion helps depends on the method: no-data ranking is a strong default, while data-aware criteria are especially useful for HSGP.

    Source: §8 · pp. 8–9

    The eigenvalue-dependent criteria use a single guessed starting point for hyperparameters, and the experiments cover six UCI regression tasks at five budgets per method.

    Source: §8 · p. 9

Abstract

from arXiv · show

Sparse Gaussian processes achieve $O(N)$ inference by replacing the kernel with an appropriate expansion in a fixed basis $\{φ_j\}$ on the input space. Given a compute budget $M \ll N$, practitioners conventionally truncate the basis to its first $M$ entries. Nothing in the formalism, however, prevents one from selecting only those $M$ basis functions that matter for the data at hand. This would avoid spending budget on basis functions where there is no signal, but it requires a criterion for ranking the candidates. We propose three such criteria derived from an information-theoretic view of the basis-function selection problem. Each criterion matches a different state of knowledge at selection time: a no-data state, a no-prior state, and an in-between state. We then study the performance of truncation versus selection strategies on six UCI regression benchmarks across three basis families: Hilbert-space Gaussian processes (HSGP), variational Fourier features (VFF), and variational inducing spherical harmonics (VISH). We observe that the no-data criterion is a safe default, matching or improving on truncation for HSGP, VFF and VISH, with substantial gains for VISH and improvements over a recently developed selection heuristic for that basis family. The data-aware no-prior and in-between criteria provide substantial gains over truncation specifically for HSGP, which is the most broadly used of the three families in practice.

1 Introduction

Sparse GPs use a fixed basis to reduce inference cost, but the conventional truncation rule spends a fixed budget on the first M basis functions. The paper proposes scoring candidates for selection instead and finds that which criterion helps depends on the basis family.

1 Introduction

  • On the toy example, the in-between criterion performs best by combining low-frequency functions favored by the no-data criterion with higher-energy functions favored by the no-prior criterion.HSGP truncation is restricted to rectangular subsets, while the selection criteria are not.Source: Figure 1 · p. 2, §1 · p. 3
  • A sparse-GP compute budget M is conventionally spent by truncating the expansion to its first M entries, although the formalism permits selecting candidates by another criterion.At fixed M, HSGP implementations conventionally use rectangular allocations, and multiple allocations can satisfy the same budget.Source: §1 · pp. 1–2
  • The paper proposes three information-theoretic scores: prior weight alone, squared data projection alone, or their pointwise product.These correspond to no-data, no-prior, and in-between states of knowledge at selection time.Source: §1 · p. 2
  • Across six UCI benchmarks, no-data selection matches or improves on truncation for HSGP, VFF, and VISH, while data-aware criteria improve specifically on HSGP.No-data selection also improves substantially on VISH and outperforms a recently proposed VISH selection heuristic.Source: §1 · p. 3

2 Background

The paper studies sparse Gaussian-process regression through basis expansions, retaining M functions by truncation or selection from a finite candidate grid. HSGP, VFF, and VISH differ in their basis construction and training procedures.

2 Background

  • The paper uses the standard zero-mean Gaussian-process regression setting with additive Gaussian observation noise, while exact inference scales as O(N^3).Sparse GPs are used for the dataset sizes of interest because exact inference is infeasible there.Source: §2 · p. 3
  • Each method provides an infinite basis expansion, from which inference retains M functions by natural-order truncation or by selecting the top-scoring functions from a finite candidate grid.For large N, the candidate grid can satisfy M ≪ J ≪ N.Source: §2 · p. 3
  • HSGP uses Dirichlet Laplacian eigenfunctions on a box, VFF uses additive per-axis windowed Fourier features, and VISH uses spherical harmonics on inputs lifted to the unit sphere.Their basis and prior constructions differ across methods.Source: §2 · p. 3
  • Experiments follow each method’s recommended training procedure: closed-form marginal likelihood for HSGP and closed-form collapsed Titsias bounds for VFF and VISH.The paper also reproduces published truncation baselines as faithfully as possible.Source: §2 · p. 3
  • The basis-expansion and inducing-variable views agree on the per-basis prior weight and data projection used by the selection criteria.The inducing-variable view specifies a variational distribution.Source: §2 · p. 3

3 The selection criteria

The criteria rank candidate basis functions using information gain under three different states of knowledge at selection time. They provide prior-only, data-only, and combined rankings, but remain heuristics without guarantees on downstream test performance.

3 The selection criteria

  • The per-basis information gain measures how much observing the data changes a coefficient’s belief from its prior to its posterior.A large information gain flags a candidate coefficient as worth keeping.Source: §3 · p. 4
  • Under empirical decoupling, the information-gain score separates into a data-independent variance-shrinkage term and a data-driven term, and requires noise and kernel weights.Those quantities are not yet fitted at selection time.Source: §3 · p. 4
  • The no-data criterion I_j ranks by prior weight alone; for stationary kernels, it favors lower frequencies but does not impose truncation’s structural shape.The ranking is closest to conventional truncation, yet can freely choose candidates across the index set.Source: §3 · p. 4
  • The no-prior criterion P_j ranks by squared data projection, while the weak-observation criterion H̃_j retains both data projection and kernel weight.When kernel hyperparameters and noise are jointly estimated and substituted into the full score, the resulting ranking is rank-equivalent to P_j.Source: §3 · pp. 4–5
  • Selection computes candidate kernel weights and data projections, ranks candidates by the chosen score, and returns the top M in a single greedy pass.The criteria are heuristics, not the canonical information-gain score at fitted hyperparameters, and the ranking has no theoretical guarantee for test performance.Source: §3 · p. 5

4 Experiment I: Hilbert-space Gaussian processes

On six UCI regression benchmarks, HSGP’s data-aware criteria improve on the no-data criterion across most settings, with the no-prior criterion slightly stronger on average. The non-uniform truncation baseline tracks the no-data criterion closely and is slightly worse on average.

4 Experiment I: Hilbert-space Gaussian processes

  • The benchmark sweep used a Matérn-5/2 ARD kernel, five compute budgets from 16 to 256, and 10 random 90:10 train/test splits.The six datasets were concrete, energy, kin8nm, power, yacht, and airfoil.Source: §4 · p. 5
  • The data-aware criteria show their largest improvements at small M, as the budget grows to cover more of the signal.The HSGP candidate grid is anisotropic, and data projections provide an early signal about which axes are active.Source: §4 · p. 5
  • Across six UCI benchmarks, both data-aware criteria improve on I_j on most cells, with P_j slightly stronger on average and non-uniform truncation tracking I_j within noise.Figure 2 plots median test NLL against compute budget M across 10 random 90:10 train/test splits, with interquartile bands.Source: §4 · p. 5, Figure 2 · p. 6

5 Experiment II: Variational Fourier features

For variational Fourier features, the in-between criterion is roughly flat against the no-data criterion across most cells, while the no-prior criterion is reliably worse. The per-axis truncation baseline nearly matches the no-data ordering.

5 Experiment II: Variational Fourier features

  • The VFF experiment used the same six UCI benchmarks and split protocol as HSGP, with a per-axis Matérn-5/2 kernel and globally ranked candidates.The collapsed Titsias bound was fit by L-BFGS-B from the unit starting point.Source: §5 · p. 6
  • For VFF, H̃_j is roughly flat against I_j across most cells, while P_j is reliably worse than both.Figure 3 presents these results.Source: §5 · p. 6
  • The per-axis truncation baseline sits on top of I_j at essentially every cell because both rules pick almost identical basis-function sets.The per-axis eigenvalue order already follows the lowest-frequency-first ordering enforced by truncation.Source: §5 · p. 6

6 Experiment III: Variational inducing spherical harmonics

On six UCI benchmarks, the VISH experiments compare score-based selection with two truncation rules using an order-1 arc-cosine kernel. Score-based criteria consistently outperform phase-truncation, while no-data ranking avoids the cumulative-shell rule’s odd-degree jumps.

6 Experiment III: Variational inducing spherical harmonics

  • The comparison uses the same six UCI benchmarks and split protocol as HSGP, with spherical harmonics on lifted inputs and two literature truncation baselines.The baselines are cumulative-shell truncation and phase-truncation.Source: §6 · p. 6
  • On smooth UCI regressions, data-aware criteria do not improve on I_j, which already ranks spherical-harmonic eigenvalues well because they drop quickly with degree and signal energy is concentrated at low degree.At even-degree cumulative-shell budgets, truncation selects the same basis functions as I_j; odd cutoffs at degrees ℓ≥3 cause upward jumps because the kernel forces those coefficients to zero variance.Source: §6 · p. 6
  • The order-1 arc-cosine kernel’s zero-variance odd shells explain why cumulative-shell truncation jumps upward at odd cutoffs ℓ≥3.The zero pattern is independent of sphere dimension D.Source: §6 · p. 6
  • At small M on energy and power, every score-based criterion beats phase-truncation by up to one to two orders of magnitude in test NLL.The phase-truncation sweep takes about 10.5 seconds per fit versus 1.4 seconds for score-selected VISH, with about 18× as many L-BFGS-B iterations.Source: §6 · p. 7

7 Related work

The related work distinguishes selecting within a fixed sparse-GP basis from choosing inducing inputs or changing the feature construction. It also connects the paper’s scores to correlation screening while emphasizing the GP prior weights and one-pass selection.

7 Related work

  • Unlike active-set methods that select or infer data-indexed inducing inputs during inference, this work ranks a fixed basis candidate set before fitting.Its scores are scalar functions of each candidate’s prior weight λ_j and data projection a_j.Source: §7 · p. 7
  • The no-prior score P_j = |a_j|² matches correlation screening used to initialize matching pursuit, but this selection is a single rank-and-truncate pass rather than an iterative residual update.The I_j and ˜H_j scores also use the GP prior variance, so they are not pure correlation-screening rules.Source: §7 · pp. 7–8
  • For VISH, score-based criteria consistently outperform Eleftheriadis et al. phase-truncation, often by orders of magnitude at small M, while cumulative-shell truncation jumps at odd-degree cutoffs.The phase rule’s single global m⋆ cannot adapt the allocation across degrees and inherits the odd-shell trap.Source: §7 · p. 8
  • Unlike methods that change the feature family or spectral approximation [2], this work keeps a published sparse-GP basis fixed and chooses a subset of its indices.Source: §7 · p. 8

8 Discussion

With a fixed basis budget, selecting candidates instead of truncating can improve sparse-GP fits, but the gains depend on the method. The no-data criterion is a strong default, while data-aware criteria help HSGP; the paper also distinguishes finite basis expansions from the broader sparse-GP construction.

8 Discussion

  • Data-aware criteria are method-specific: P_j and H̃_j improve HSGP, while I_j alone exploits the low-frequency or low-degree ordering of VFF and VISH.The authors note that data-aware criteria are most likely to help when signal structure falls outside what low-frequency truncation captures.Source: §8 · p. 9, Figure 5 · p. 9
  • The criteria I_j and H̃_j use λ_j at one guessed starting point, H_j is exact only under empirical decoupling, and generalisation beyond six UCI regressions remains open.Cross-method differences may also reflect different basis constructions.Source: §8 · p. 9
  • The no-data criterion I_j ties or beats truncation across HSGP, VFF, and VISH, with substantial gains on VISH and gains over its phase-truncation heuristic.The data-aware no-prior and in-between criteria further improve on truncation for HSGP.Source: §8 · pp. 8–9

A GP is already a basis-function expansion

  • General sparse GPs need not be finite basis expansions: conditioning on inducing variables retains infinite-dimensional uncertainty, whereas the Mercer-coefficient construction separates a finite expansion from a residual GP.The criteria are the same in both pictures under the stated empirical-decoupling assumption because the residual affects unmodelled test-input uncertainty, not the marginal posterior on each Mercer coefficient.Source: A GP is already a basis-function… · pp. 11–12

B Derivation of the criteria

The appendix derives the information-based scores and specifies how each basis family is constructed, scored, fit, and compared with truncation. Its assumptions and family-specific structure matter for interpreting both the criteria and their baselines.

B Derivation of the criteria

  • Under empirical decoupling, the per-feature posterior is Gaussian; the resulting KL yields H_j, whose data-averaged form ranks like λ_j, while a fully data-fit H_j ranks like P_j.The scores use prior variance λ_j and data projection a_j; the derivation also extends to scalar linear functionals beyond basis coefficients.Source: B Derivation of the criteria · pp. 12–13

C Hilbert-space Gaussian processes (HSGP)

  • HSGP uses Dirichlet Laplacian eigenfunctions on a box with kernel spectral-density weights; its conventional truncation keeps rectangular per-axis frequency subsets.The appendix constructs a reproducible non-uniform truncation curve by selecting the most-uniform integer factorisation at each target budget.Source: C Hilbert-space Gaussian processes… · pp. 13–14
  • HSGP approximates stationary kernels through box restriction, Dirichlet boundaries, and finite modes; the approximation is asymptotically exact as box lengths grow for inputs away from the boundary with sufficient modes.The experiments set each box length to 1.2 times the largest absolute input value on that axis.Source: C Hilbert-space Gaussian processes… · p. 13

D Variational Fourier features (VFF)

  • VFF uses separate one-dimensional Fourier bases whose candidate count grows linearly with input dimension, and adopts periodic-extension diagonalisation with finite-interval boundary error.For Gaussian likelihood, fitting maximises the collapsed Titsias bound; the approximation becomes asymptotically exact when the interval is large relative to the kernel correlation length.Source: D Variational Fourier features (VFF) · p. 14
  • The VFF truncation baseline keeps frequencies from zero through a shared per-axis cutoff, counting both sine and cosine features for nonzero frequencies.The sweep uses five log-spaced budgets between 16 and 256.Source: D Variational Fourier features (VFF) · p. 14

E Variational inducing spherical harmonics (VISH)

  • VISH lifts Euclidean inputs to a sphere and uses spherical harmonics with variance shared by all orientations in a degree shell; Gaussian-likelihood fitting uses the collapsed Titsias bound.The harmonic formulas use the lifted ambient dimension, one greater than the original input dimension.Source: E Variational inducing spherical… · p. 15
  • For the order-1 arc-cosine kernel, odd shells of degree at least three have zero prior variance, so cumulative-shell truncation can waste budget on them and produce an upward jump.The phase-truncation baseline also inherits the odd-shell trap.Source: E Variational inducing spherical… · p. 15
  • VISH compares cumulative-shell truncation with phase truncation, which keeps a chosen number of orthogonalised phase features per shell through a cutoff degree.The phase count is chosen to approach the target basis budget, and phases are optimised during fitting rather than selection.Source: E Variational inducing spherical… · p. 15

F Toy figure details

The appendix specifies two illustrative setups: a two-dimensional HSGP grid example and a one-dimensional synthetic bandpass example. These figures illustrate candidate selection before the benchmark comparisons.

F Toy figure details

  • The two figures illustrate how ranking rules behave before the UCI benchmark sweeps, rather than supplying additional benchmark evidence.Figure 1 uses two UCI features; Figure 5 uses a synthetic bandpass signal.Source: F Toy figure details · p. 16
  • For Figure 1, the authors select ten of 25 two-dimensional HSGP candidates using four strategies, fit kernel hyperparameters from the same starting point, and report held-out test NLL.The toy setup uses the combined-cycle power-plant dataset's AT and V features with a Matérn-5/2 ARD kernel.Source: F Toy figure details · p. 16
  • Figure 5 uses 512 regularly spaced noisy observations of a synthetic bandpass-sinc signal and compares the I_j and H̃_j score spectra and HSGP fits at M = 10.The signal has bandpass cutoffs ω0 = 30 and ω1 = 100, with Gaussian noise standard deviation 0.3.Source: F Toy figure details · p. 16

G Experimental details for UCI

The UCI experiments use six regression datasets, repeated train/test splits, and a shared optimisation protocol across the three sparse-GP families. Each family has its own basis construction and fitting objective.

G Experimental details for UCI

  • The evaluation covers six UCI regression datasets with 10 random 90:10 train/test splits per cell and training-set standardisation of inputs and targets.The benchmarks range from yacht hydrodynamics (N = 308) to kin8nm (N = 8192).Source: G Experimental details for UCI · p. 16
  • Each family uses a single L-BFGS-B run from the same unit starting point for selection and hyperparameter fitting, without multistart.Variability is reported as median test NLL with interquartile bands across the ten splits.Source: G Experimental details for UCI · p. 16
  • HSGP uses a Matérn-5/2 ARD kernel on a box, a candidate grid capped at 8,000 functions, and a closed-form marginal-likelihood objective.Its truncation baseline is the non-uniform construction described for HSGP.Source: G Experimental details for UCI · p. 16
  • VFF ranks a flat set of per-axis Matérn-5/2 Fourier candidates globally and fits with the closed-form collapsed Titsias bound.Its truncation baseline follows the per-axis frequency rule.Source: G Experimental details for UCI · p. 16
  • VISH projects inputs to a sphere, uses the gpfy spherical-harmonic basis and order-1 arc-cosine spectrum, and fits with the collapsed Titsias bound.The degree limit is set by available precomputed tables; the cumulative-shell and phase-truncation baselines are configured to approach each target budget.Source: G Experimental details for UCI · p. 16

H Test RMSE on UCI for Experiments I, II, III

The appendix RMSE comparisons follow the test-NLL pattern across HSGP, VFF, and VISH, with different selection behavior across the three families.

H Test RMSE on UCI for Experiments I, II, III

Experiment I: HSGP

Experiment II: VFF

Experiment III: VISH

Loading 2609.26624v2…