Non-Stationary Spectral Kernels

1 Introduction

• Stationary, monotonic kernels encode the same similarity notion across inputs and cannot represent input-dependent dynamics or long-range correlations.Non-stationary and non-monotonic functions are encountered in signal processing, time series, bioinformatics, and geostatistics.Source: §1 · p. 1
• Earlier Fourier-derived non-monotonic kernels are stationary, while prior non-stationary approaches generalize monotonic base kernels.The paper positions its family as combining non-stationarity and non-monotonicity.Source: §1 · p. 1
• The proposed kernel family is both non-stationary and non-monotonic, allowing it to infer long-range or periodic relations in an input-dependent manner.Its generalized spectral density is modeled as a mixture of Gaussian process density surfaces.Source: §1 · p. 1
• Figure 1 pairs a spectral density surface with its corresponding kernel for a single bivariate spectral mixture component.The surface has eight permuted peaks; the kernel is shown on inputs x ∈[−1, 1].Source: Figure 1 · p. 2
• The paper presents experiments on time series, image-based pattern recognition and extrapolation, and climate data.The kernel reduces to a stationary kernel with an appropriate parameterization.Source: §1 · p. 2

2 Non-stationary spectral kernels

• The generalised Fourier transform maps a spectral surface to a positive-semidefinite kernel, which is non-stationary unless spectral mass lies only on the diagonal.The resulting parameterisation supports input-dependent characteristics and potentially long-range, non-monotonic correlations.Source: §2 · p. 2

2.1 Bivariate Spectral Mixture kernel

• The bivariate spectral mixture (BSM) kernel uses a mixture of bivariate Gaussian spectral components, with symmetries and diagonal components imposed to obtain a valid spectral density.Its kernel vanishes rapidly away from the origin, so covering a reasonably sized input space would require many components.Source: §2.1 · pp. 2–3

2.2 Generalised Spectral Mixture (GSM) kernel

• GSM parameterizes frequencies, length-scales, and mixture weights as Gaussian processes, yielding a smooth, input-dependent spectrogram and a closed-form kernel.Log and logit transforms constrain weights and length-scales to be non-negative and frequencies to the interval from zero to the Nyquist frequency.Source: §2.2 · p. 3
• The GSM kernel combines input-dependent signal variances, frequencies, and frequency length-scales, and reduces to the stationary Spectral Mixture kernel when its functions are constant.Figure 2 compares kernel matrices, spectral surfaces, and input-dependent frequency amplitudes across Gaussian, sparse spectrum, spectral mixture, and GSM kernels; the GSM example uses Q = 2 surface functions.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
• For grid-structured multivariate inputs, the extension decomposes parameters across dimensions and expresses the kernel matrix as a Kronecker product.The passage notes that missing values and non-grid data can be handled with standard techniques.Source: §2.2 · p. 4

3 Inference

• Kronecker identities are used to compute the predictive GP posterior efficiently for a new input.The passage also notes that the marginalized posterior and its gradients can be evaluated efficiently, but its displayed complexity expressions are incomplete.Source: §3 · p. 5
• MAP inference maximizes the marginalized posterior over kernel parameters after integrating out the function values.The marginalized posterior balances data fit against model complexity.Source: §3 · p. 5
• Whitening reparameterizes the highly self-correlated kernel parameters using the prior-covariance Cholesky factor for gradient-based optimization.The authors say direct optimization is likely to converge very slowly because of these correlations.Source: §3 · p. 5

4 Related Work

• Stationary spectral representations support rich kernels [30] and large-scale inference, but their Fourier basis is a special case of the more general transform.Bochner’s theorem supplies the stationary covariance–Fourier dual [28].Source: §4 · p. 5
• The sparse spectrum kernel learns spectral density as a mixture of Dirac delta functions, while the spectral mixture kernel models it as a mixture of normals [28].These spectral approaches encode long-range, non-monotonic, or periodic kernels but have not handled non-stationarity.Source: §4 · p. 5
• Non-stationary extensions of Matérn and Gaussian kernels, input warping, and local-stationarity products model changing dynamics but are unsuitable for non-monotonic properties [6] [25].The dot-product kernel can assign input-dependent signal variances.Source: §4 · p. 5
• The work is related to wavelets and to Hilbert–Huang transforms and Hilbert spectral analysis, which explore input-dependent frequencies using deterministic transform functions.Figure 3 illustrates longer-range positive and negative correlations near x = −1 that vanish toward x = 1.Source: §4 · p. 5, Figure 3 · p. 6

5 Experiments

• The experiments implement GSM in Matlab and use L-BFGS with ten restarts initialized from the best of 100 randomly sampled hyperparameter values.The comparisons use the GPML Matlab toolbox and its Kronecker inference implementation.Source: §5 · p. 6

5.1 Simulated time series with a decreasing frequency component

• The learned single-component GSM kernel captures the simulated signal’s increasing frequency toward negative x values.The example uses a deterministic frequency pattern and a noisy sampled function.Source: §5.1 · p. 6

5.2 Image data

• The experiments compare GSM with SM, SS, and SE on texture prediction, using Q = 5 components for metal and Q = 10 for wood.The image tasks predict a missing middle cross-section and extrapolate beyond the original borders.Source: §5.2 · p. 6
• GSM gives the most believable texture extrapolations outside image borders, while SM, SS, and SE show specific limitations on the metal and wood patterns.SM fills the missing cross pattern but extrapolates poorly; SS misses structure, and SE overfits with a short length-scale.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

5.3 Spatio-Temporal Analysis of Land Surface Temperatures

• The study uses four years of monthly land temperatures on an 84 × 101 North American grid, imputing water regions with each month’s mean.The training subset contains 407,232 data points and uses a GSM kernel with Q = 5 components.Source: §5.3 · p. 7
• Figure 5 shows non-symmetric spatial covariance slices and predictions for three test months following five months of training data.The analysis targets long-range, non-stationary spatial and temporal covariances in North American land-surface temperatures.Source: §5.3 · p. 7

6 Discussion

• GSM can learn local and global correlations across input domains in an input-dependent manner using a few interpretable components.The authors contrast this with BSM, which they say would require an impractical number of components to cover a sufficiently sized input space.Source: §6 · pp. 7–8
• Figure 5 demonstrates non-stationary spatial covariances in land-surface data and shows reasonably accurate predictions for February–April 2004, following a posterior for the five last training months through January 2004.Source: Figure 5 · p. 8
• The solution to the spectral surface remains unknown because multiple Gaussian process functions occur inside the integral.The authors note that Figure 2h illustrates numerical integration on an example GP frequency surface.Source: §6 · p. 8
• The authors suggest GSM may be dense in the family of non-stationary kernels, potentially reproducing arbitrary non-stationary kernels.This is presented as a suggestion based on theoretical work, not as a proven result.Source: §6 · p. 8

A A tutorial on Gaussian processes

• A Gaussian process defines a distribution over functions, with its mean and positive semi-definite kernel determining function expectations and covariance.Finite collections of function values have a multivariate normal distribution.Source: A A tutorial on Gaussian processes · p. 10
• Kernel-based interpolation treats function values at similar inputs as similar, while the Gaussian kernel encodes monotonic neighborhood similarity.The Gaussian kernel’s parameters are signal variance and lengthscale.Source: A A tutorial on Gaussian processes · p. 10
• The predictive distribution for a new input is closed form, shifting inference to learning the hyperparameters.The tutorial identifies the hyperparameters as signal variance, lengthscale, and noise variance.Source: A A tutorial on Gaussian processes · pp. 10–11
• The marginal log likelihood balances data fit and model complexity and can be maximized with standard gradient ascent.It promotes priors inducing functions that match the data while penalizing complexity.Source: A A tutorial on Gaussian processes · p. 11

B Deriving the bivariate spectral mixture kernel

• A non-stationary kernel can be characterized by a spectral density over frequency pairs, related to the kernel by a generalized Fourier transform.The spectral surface represents the amplitude of frequency pairs and is associated with a positive semi-definite density of bounded variation.Source: B Deriving the bivariate spectral… · p. 11
• The bivariate spectral density is constructed as a mixture of Gaussian components constrained by symmetry and diagonal-component conditions for positive semidefiniteness and a real-valued kernel.The construction iterates over four permutations of the means and their negative-frequency counterparts, yielding eight components.Source: B Deriving the bivariate spectral… · p. 11
• The generalized Fourier transform of the mixture is computed in closed form using Gaussian integral identities.The derivation simplifies the exponential terms and introduces mixture weights.Source: B Deriving the bivariate spectral… · p. 12
• The BSM kernel vanishes rapidly away from the origin, so covering a reasonably sized input space requires an impractical number of components.Replacing its exponential part with a Gaussian can prevent vanishing but still cannot model non-stationary frequencies.Source: B Deriving the bivariate spectral… · p. 12

C Deriving the generalised spectral mixture (GSM) kernel

• GSM defines input-dependent Gaussian process frequencies, lengthscales, and mixture weights, using transformations to constrain their values.Weights and lengthscales are made positive, while frequencies are limited between zero and the Nyquist frequency.Source: C Deriving the generalised spectral… · p. 13
• The construction replaces the BSM exponential part with a Gibbs kernel to accommodate lengthscale functions and replaces its cosine part with a feature-map function.The resulting kernel has a closed form.Source: C Deriving the generalised spectral… · p. 13
• The closed-form GSM kernel is a product of linear, Gibbs, and feature-map cosine kernels, and is positive semidefinite because each factor is positive semidefinite.The cosine kernel is positive semidefinite because it is a dot product.Source: C Deriving the generalised spectral… · p. 13

• The GSM kernel uses Gaussian-process functions for its input-dependent spectral components.The functions are w_i(x), μ_i(x), and ℓ_i(x).Source: §C.1 · p. 13
• With appropriate parameterization, the univariate GSM reduces to the stationary SM kernel.Source: §C.1 · pp. 13–14
• GSM reproduces any kernel reproducible by SM, described as highly flexible [28] [27], while smooth or effectively constant spectral-function priors can simulate stationary kernels.Source: §C.1 · p. 14

D Inference

• For grid-structured multivariate inputs, the kernel decomposes across dimensions and combines dimension-wise kernels with Kronecker products.Missing values and data outside regular grids can be handled with standard techniques.Source: D Inference · p. 14
• Gaussian-likelihood GP regression uses the standard predictive posterior, which can be computed efficiently with Kronecker identities.Source: D Inference · pp. 14–15
• MAP inference uses the marginalized posterior, which balances data fit against model complexity and regularizes against overfitting.Source: D Inference · p. 15
• Whitening addresses slow gradient-ascent convergence from highly self-correlated parameters by optimizing variables transformed with the prior-covariance Cholesky factor.Source: D Inference · p. 15

D.1 Kronecker inference

• Kronecker inference uses eigendecompositions of smaller dimension-wise kernels and extends to non-complete grids and non-Gaussian likelihoods.Source: §D.1 · pp. 15–16