PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Gradient Computation for Warping Gaussian Process Transformations

Emilio Ruiz-Moreno, Konstantinos Slavakis, Baltasar Beferull-Lozano

arXiv:2609.16472v1cs.LGeess.SP

TL;DR

Standard GPs can produce inaccurate predictions and poorly calibrated uncertainty for non-Gaussian observations, while existing streaming warped-GP methods do not jointly update latent moments and warping parameters. This paper derives an exact recursive instantaneous-NLL gradient and builds an online joint-update method; on a warped-GP benchmark, it reports an empirical NLL of −0.28 nats per observation, 0.76 nat better than a standard GP under the stated configuration.

  • Problem

    Standard GPs can produce inaccurate predictions and poorly calibrated uncertainty for non-Gaussian observations, and existing streaming warped-GP methods do not jointly update latent moments and optimize warping parameters.

    Source: §1 · p. 1

  • Method

    The paper derives an exact recursive gradient of the instantaneous warped-GP NLL and uses it in an online method that jointly updates latent GP moments and warping parameters.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    −0.28 nats per observation was the warped GP's empirical NLL, a 0.76 nat improvement over a standard GP under an identical model configuration except for a well-calibrated noise variance.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The recursive gradient supports online joint updating of latent GP moments and warping parameters for warped GPs.

    Source: §1 · p. 1, §6 · p. 4

    The stated GP model assumes a zero-mean GP prior and white Gaussian observation noise.

    Source: §2.1 · p. 1

Abstract

from arXiv · show

Warped Gaussian processes (GPs) handle non-Gaussian observations by mapping them into a latent standard GP via a parametric transformation called warping. Existing streaming variants, however, either optimize the warping parameters periodically or sacrifice analytical tractability for a higher model capacity. To bridge this gap, we show that the gradient of the instantaneous negative log-likelihood of a warped GP admits an exact recursive computation. Based on this result, we propose a novel online method for warped GPs that jointly updates the latent GP moments and optimizes the warping parameters.

1 Introduction

Standard GPs can misrepresent non-Gaussian observations, while warped GPs transform observations into latent targets to retain GP tractability. The paper identifies a gap in online joint updates of latent GP moments and warping parameters.

1 Introduction

  • Standard GPs assume Gaussian observations, which can yield inaccurate predictions and poorly calibrated uncertainty for heavy-tailed, skewed, or constrained data.The paper motivates warped GPs as a response to this limitation.Source: §1 · p. 1
  • Existing streaming warped-GP methods either optimize warping parameters periodically or trade analytical tractability for higher model capacity.The paper identifies the absence, to its knowledge, of an online method that jointly updates latent GP moments and optimizes warping parameters.Source: §1 · p. 1
  • The paper derives an exact recursive gradient for the warped-GP negative log-likelihood and proposes an online method that jointly updates latent GP moments and warping parameters.This addresses the identified gap in online joint optimization.Source: §1 · p. 1

2.1 Gaussian processes

  • A GP model assumes a zero-mean GP prior and white Gaussian observation noise, enabling closed-form predictive distributions under Gaussian likelihoods.The kernel specifies the prior covariance, and the noise terms have standard deviation σ.Source: §2.1 · p. 1, §1 · p. 1

2.1.1 Recursive Gaussian process updates

  • Recursive GP updates maintain α_n and Ω_n as state variables, updating posterior moments as each new observation and input arrive.These updates avoid the n × n matrix inversion bottleneck of direct computation.Source: §2.1.1 · p. 2

2.1.2 Sparse Gaussian process updates

  • Recursive posterior moments still require O(n) computation for the mean and O(n^2) for the variance, motivating sparse approximations such as ALD.The paper relies on ALD because it integrates naturally with recursive updates.Source: §2.1.2 · p. 2

2.2 Warped Gaussian processes

  • Warped GPs apply a monotonic, parametric, differentiable transformation to observations, mapping them to latent targets where standard GP assumptions hold.Their warping parameters can be estimated from observations through a tractable likelihood without explicitly knowing the observation density.Source: §2.2 · pp. 2–3

3 Recursive gradient computation

The instantaneous negative log-likelihood measures how unexpected the latest observation is given earlier observations. Its gradient can be computed recursively, supporting online refinement of warping parameters.

3 Recursive gradient computation

  • The instantaneous NLL quantifies how unexpected the latest observation is given the preceding observations.The joint NLL decomposes additively, allowing warping parameters to be refined online using the instantaneous gradient.Source: §3 · p. 3
  • The instantaneous NLL gradient can be computed recursively even though each instantaneous NLL depends on previous observations.Theorem 3.1 gives the recursive computation with respect to the warping parameters.Source: §3 · p. 3
  • Maintaining α_n, Ω_n, and B_n as state variables supports recursive updates of the instantaneous NLL gradient as each observation arrives.B_n is the additional state variable used for the gradient computation.Source: §3 · p. 3

4 Proposed online method

The proposed online method combines recursive GP updates, recursive instantaneous-NLL gradients, and ALD sparsification to update latent moments and warping parameters.

4 Proposed online method

  • The online algorithm combines recursive GP updates, recursive NLL-gradient computation, and ALD sparsification for joint updates of latent moments and warping parameters.With ALD sparsification, the update rule yields an approximate gradient.Source: §4 · p. 4
  • The method's per-step computational and memory complexity is O(D^2 n + rDn).The paper refers to supplementary material for further complexity details.Source: §4 · p. 4

5 Revisiting the 1D regression task

The section adapts a non-Gaussian 1D regression task to sequential evaluation of the online warped GP. It reports that the warped GP achieves a lower empirical NLL than a standard GP under an otherwise identical configuration except for noise variance.

Experimental setup

  • The online task uses a sinusoidal signal with white Gaussian noise followed by a cubic root transformation, producing non-Gaussian observations with sharp transitions.The task is designed to cause standard GPs to fail, relying on the warped GP's modeling capabilities.Source: §5 · p. 4
  • The evaluation feeds 101,000 input-observation pairs sequentially, from 1,000 realizations of 101 uniformly spaced observations each.The input domain is X = [−π, π].Source: §5 · p. 4

Model configuration

  • The model uses a monotone warping transformation with 10 terms and 30 parameters, alongside the task's specified covariance function and latent observation noise.Monotonicity is imposed through the feasible parameter set; the kernel uses k_a = 2 and k_w ≈ 0.124, and the latent noise standard deviation is σ = 3.Source: §5 · p. 4

Implementation details

  • Implementation uses Adam followed by projection onto the feasible parameter set, with an ALD sparsification threshold of ν = 0.1.The reported dictionary contains 68 atoms.Source: §5 · p. 4

Results

  • The warped GP achieves an empirical NLL of −0.28 nats per observation, a 0.76 nat improvement over a standard GP with an identical model configuration except for well-calibrated noise variance.Its predictions closely align with the true generating distribution, and the learned transformation resembles the true cubic transformation in the region −1 ≲ y ≲ 1, with numerically stable inversion.Source: §5 · p. 4

6 Conclusion and future work

The paper presents an exact recursive gradient evaluation for the instantaneous NLL of warped GPs and an online algorithm that jointly updates latent GP moments and warping parameters. It reports a warped GP benchmark evaluation and identifies kernel, noise-variance, and broader-benchmark work for the future.

6 Conclusion and future work

  • The recursive gradient evaluation requires only one extra state variable over standard recursive implementations, and supports an online algorithm jointly updating latent GP moments and warping parameters.The authors report demonstrating the method's performance on a warped GP benchmark.Source: §6 · p. 4
  • Future work includes online learning of kernel parameters or a suitable kernel, observation-model noise variance, and evaluation on broader benchmarks.The authors state that broader benchmark evaluations are underway.Source: §6 · p. 4

Supplementary material for “Online Gradient

S1 Proof of Theorem 3.1

The proof derives a recursive gradient for the warped GP's instantaneous NLL with respect to warping parameters. It builds on recursive updates for existing terms and maintains an additional state for another gradient term.

S1 Proof of Theorem 3.1

  • The proof obtains the warping-parameter gradient by differentiating the NLL and expanding a term using the recursive representation of Ω_n.It then substitutes the recursive expressions and identifies terms after differentiating the posterior-moment relation.Source: S1 Proof of Theorem 3.1 · pp. 6–7
  • The term e_n already updates recursively because it depends on the current latent target and location, together with prior state terms α_{n−1} and Ω_{n−1}.Source: S1 Proof of Theorem 3.1 · p. 6
  • The method maintains an additional state B_n = Ẑ_n Ω_n to enable recursive updates of b_n.The derivative of each latent target with respect to the warping parameters is represented by ż_i = ∇_θ g(y_i; θ).Source: S1 Proof of Theorem 3.1 · pp. 6–7

S2 ALD sparsification

ALD sparsification builds a dictionary of representative input locations by testing whether each new feature lies near the span of existing dictionary features. The resulting feature approximation is used to approximate the kernel matrix and posterior moments.

S2 ALD sparsification

  • A new input joins the dictionary when its squared feature-space distance from the existing dictionary span exceeds the user-defined threshold ν; otherwise, the dictionary remains unchanged.The squared distance is found by a convex quadratic optimization problem with a closed-form solution.Source: S2 ALD sparsification · p. 7
  • Each input feature is approximated using dictionary features, forming a feature matrix approximation with the dictionary-feature matrix and coefficient matrix A_n.Rows of A_n are formed by padding the corresponding approximation coefficients with zeros.Source: S2 ALD sparsification · p. 8
  • The feature approximation yields an approximation of the posterior moments when substituted into the preceding expressions.Source: S2 ALD sparsification · p. 8

S3 ALD sparse and recursive Gaussian process updates

The recursive update computes approximated posterior moments from the current observation and input, together with the previous state; its form depends on whether the input joins the dictionary.

S3.1 Case Dn = Dn−1 (meaning that δn ≤ν)

For the unchanged-dictionary case, the passage indicates use of a partitioned matrix inverse formula in deriving the update.

S3.2 Case Dn = Dn−1 ∪{xn} (meaning that δn > ν)

When the input joins the dictionary, the derivation applies partitioned matrix inversion; the algorithm’s per-step computational and memory complexity is O(D_n^2 + rD_n).

S3.2 Case Dn = Dn−1 ∪{xn} (meaning that δn > ν)

S3.3 Comments on the computational and memory complexity

Loading 2609.16472v1…