PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Outlier-Robust Gaussian Process Regression with Generative Modeling

Arslan Majal, Aamir Hussain Chughtai

arXiv:2608.16606v1cs.LG

TL;DR

Standard Gaussian-process regression is sensitive to outliers because its Gaussian noise model can let corrupted observations distort estimation. The proposed method jointly models observation-specific contamination and the latent GP using variational generalized-EM, achieving competitive or superior prediction against robust GPR baselines while retaining cubic scaling.

  • Problem

    Standard GPR’s Gaussian observation-noise assumption makes it sensitive to outliers that can substantially distort the estimated latent function.

  • Method

    The method embeds observation-specific precision modeling in a hierarchical generative GPR and jointly learns contamination effects and GP parameters with variational generalized-EM.

  • Results

    ASOR-GPR remains competitive across contamination settings, often achieves the lowest RMSE among non-oracle methods, and shows clear gains under asymmetric contamination.

  • Takeaways & Limitations

    The approach provides enhanced predictive robustness across synthetic and real datasets while preserving the cubic computational scaling of standard GPR.

Abstract

from arXiv · show

Outliers can substantially distort Gaussian process regression (GPR) due to its conventional Gaussian observation likelihood, leading to inaccurate model learning and prediction. To address this limitation, this article introduces a generative GPR model that captures observation-specific contamination and adaptively mitigates the influence of outliers. Subsequently, a variational generalized expectation-maximization procedure is used to learn the latent variables and GPR model parameters. Experiments on synthetic and real datasets under different contamination settings demonstrate that the proposed method remains competitive with-and in several cases outperforms-robust GPR baselines in prediction accuracy. Moreover, the proposed method shares the cubic computational scaling of the compared GPR methods.

I. INTRODUCTION · II. PROPOSED METHODOLOGY

The paper develops an adaptive, observation-specific contamination model for robust Gaussian process regression and learns its latent variables and parameters using variational generalized expectation-maximization. It formulates multi-output problems as independent scalar-output GPR models and evaluates the method against representative baselines under diverse contamination settings.

  • I. INTRODUCTION: Gaussian process regression provides flexible nonlinear modeling and principled uncertainty quantification, including in data-limited settings.
  • I. INTRODUCTION: Standard Gaussian observation noise makes GPR sensitive to outliers, whose large residuals are penalized quadratically and can distort the estimated latent function.
  • I. INTRODUCTION: Existing robust GPR methods modify likelihoods or loss functions, but can depend on prescribed thresholds, tuning parameters, or distributional assumptions.
  • I. INTRODUCTION: The proposed model integrates observation-specific precision modeling with the latent GP function and noise, mean, and kernel parameters in a unified formulation.The formulation is motivated by adaptive outlier treatment that learns contamination characteristics rather than simply discarding suspected observations.
  • I. INTRODUCTION: Because exact Bayesian inference is intractable, a variational generalized expectation-maximization procedure learns the latent variables and model parameters using closed-form updates.
  • I. INTRODUCTION: The method is extensively compared with representative GPR baselines on synthetic and real-data regression problems under diverse contamination settings.
  • II. PROPOSED METHODOLOGY: For n input-output samples, X contains inputs and Y contains corresponding outputs, with rows xi and yi and output columns yj.
  • II. PROPOSED METHODOLOGY: Assuming independent output components and contamination processes, the problem decomposes into dy scalar-output GPR models, each trained using X and yj.

A. Scalar Observation Model

The scalar observation model represents each output as a latent function value plus noise, with a positive latent precision multiplier enabling observation-specific variance adaptation for outliers. A Gaussian-process prior uses an ARD squared-exponential covariance kernel with output-specific signal variances and input-dimension length scales.

  • Scalar Observation Model: Each scalar observation satisfies y_ij = f_ij + ε_ij for output dimension j and input i = 1, . . . , n.Here, f_ij denotes the latent function value f_j(x_i).
  • Scalar Observation Model: A positive latent precision multiplier I_ij > 0 adaptively modifies the nominal observation variance to account for outliers.I_ij = 1 recovers the nominal observation model, whereas I_ij ≠ 1 changes the observation variance.
  • Scalar Observation Model: The latent function vector f_j = [f_1j, . . . , f_nj]^⊤ follows a Gaussian-process prior.The multivariate Gaussian notation uses a covariance matrix Σ.
  • Scalar Observation Model: The covariance matrix uses an automatic relevance determination squared-exponential kernel with output-specific signal variance and input-dimension length scales.The signal variance is denoted σ^2_f,j, and ℓ_jr is the length scale for input dimension r.

B. Hierarchical Priors

The model uses hierarchical priors to distinguish nominal observations from contamination while retaining Bayesian conjugacy through Gamma and inverse-Gamma components. Output-specific hyperparameters govern these prior distributions.

  • Prior for contamination variables: Each contamination variable Iij follows a unit-spike-and-Gamma prior, mixing a Gamma density with a point mass at 1.The mixture weight θij is the prior probability that yij belongs to the nominal component.
  • Hyperpriors: The output-specific Gamma rate bj receives a Gamma hyperprior to retain Bayesian conjugacy.The fixed hyperparameters satisfy A > 1 and B > 0.
  • Noise-variance prior: The nominal observation-noise variance is assigned an inverse-Gamma prior with fixed hyperparameters ν0 > 0 and s0j > 0.The fixed prior hyperparameters associated with output dimension j are collected together.

C. Variational Generalized-EM Inference

The model uses variational generalized EM because the exact joint posterior lacks a closed form. Mean-field coordinate updates define the E-step, while parameter updates in the M-step combine closed-form, MAP, maximum-likelihood, and gradient-based optimization.

  • E-step: Variational generalized EM is employed because the exact joint posterior has no closed-form solution, using flat non-informative priors for m_j and κ_j.The constant prior terms are omitted from the joint posterior.
  • E-step: The E-step uses a mean-field factorization and coordinate-optimal updates for f_j and each contamination variable I_ij.Each update is computed from the expected conditional log joint with respect to the other latent variables.
  • E-step: At each generalized-EM iteration, latent-variable factors are updated sequentially, and their product forms q_j(Z_j).The sequential updates apply the closed-form coordinate updates for f_j and each I_ij.
  • M-step: The M-step optimizes the expected complete-data log posterior, with closed-form updates for m_j, σ_j^2, and b_j, while κ_j uses backtracking gradient descent.The objective is Q_j(Θ_j) = E_qj(Zj)[log p(y_j, Z_j, Θ_j | X, P_j)].
  • M-step: Priors on σ_j^2 and b_j produce MAP updates, whereas uninformative priors on m_j and κ_j produce maximum-likelihood updates.This distinguishes the update type according to the prior assigned to each parameter.

D. Variational E-Step: Latent-Function Update

The variational E-step fixes an output index j and applies equation (12) to update the corresponding latent function f_j.

  • For a fixed output j, the section defines the quantities used in the variational update.
  • The latent-function update is therefore indexed by the fixed output j.
  • Equation (12) is applied to the latent function f_j.

E. Variational E-Step: Precision-Multiplier Update

The variational E-step updates the precision multiplier using moments of q_j(f_j), then applies the resulting expression to I_ij to obtain a robustness weight.

  • The precision-multiplier update is defined using the moments of q_j(f_j).
  • The notation specifies that μ_f,ij is the ith entry of μ_f,j.
  • Applying equation (12) to I_ij yields the update expression.
  • The corresponding robustness weight is then obtained from the update.

F. Generalized M-Step: Parameter Updates

With variational quantities fixed, the generalized M-step separates parameter updates for noise, Gamma-rate, GP mean, and kernel parameters. Kernel hyperparameters are optimized through a positivity-preserving logarithmic parameterization, gradient descent, and backtracking line search.

  • Parameter updates: The fixed-variational-objective Qj(Θj) separates into terms for σ2_j, bj, and (mj, κj), enabling separate parameter updates.MAP updates are obtained for σ2_j and the Gamma rate parameter, while mj is optimized from the GP-dependent terms.
  • Kernel update: The kernel parameterization maps ϕj through exponentials, enforcing positivity of the signal variance and length scales during optimization.The resulting kernel parameters are reconstructed from ϕj after optimization.
  • Kernel update: Kernel optimization minimizes Jj(ϕj), using its gradient, normalized-gradient candidates, and backtracking that halves αj until the Armijo condition is satisfied.The accepted ϕj is then used to update κj and reconstruct Kj.
  • Optimization procedure: The generalized M-step is iterated with variational E-step updates, parameter updates, kernel reconstruction, and convergence checks based on relative change in Sj or an iteration limit.After convergence, a final variational E-step is performed and the converged ASOR-GPR models are stored.

G. Posterior Prediction

The posterior prediction section derives predictive distributions by marginalizing the conditional Gaussian process distribution over the learned model quantities. It then specifies the predictive distribution for a nominal future observation.

  • Posterior prediction marginalizes the conditional Gaussian process distribution over the learned model quantities.
  • For a nominal future observation y∗j, the predictive distribution is obtained from the posterior formulation.
  • The prediction-performance figure covers synthetic data, Air Quality data, and Energy Efficiency data under specified outlier settings.The settings include uniform outliers over [0, 10], uniform outliers over [−5, 15], and zero-mean Gaussian outliers with variance 25.

III. NUMERICAL EXPERIMENTS

The numerical experiments compare ASOR-GPR with standard and robust GPR baselines under synthetic and real-data contamination. Across contamination settings, ASOR-GPR remains competitive and usually achieves the lowest RMSE among non-oracle methods.

  • Experimental setup: Experiments compare ASOR-GPR with standard GPR, RCGPR, Student-t GPR, GMM-GPR, and an Oracle GPR using ground-truth contamination information.All experiments were conducted in MATLAB R2026a on an AMD Ryzen 9 9950X desktop with 64 GB RAM.
  • Experimental setup: Synthetic and real-data predictive experiments use ntrain = ntest = 100, with entry-wise outliers injected only into training targets over 30 Monte Carlo trials.Synthetic RMSE is evaluated against the latent function, whereas real-data RMSE uses unmodified test targets retaining inherent observation noise.
  • Synthetic experiment: The synthetic experiment uses two-dimensional Latin-hypercube inputs over [−3, 3]2, four outputs, standardized responses, nominal Gaussian noise, and contamination probabilities pout ∈{0, 0.1, . . . , 0.8}.The nominal-noise variance is deliberately misspecified at 0.5 versus the true synthetic variance 0.25 for non-oracle methods.
  • Results: For synthetic outliers drawn uniformly from [0, 10], ASOR-GPR remains competitive across contamination levels and has the lowest RMSE among compared non-oracle methods at most levels.The real-data evaluation uses the standardized Air Quality and Energy Efficiency datasets.

IV. CONCLUSION

The paper proposes an outlier-robust GPR model with adaptive observation-specific precision modeling and jointly learns contamination effects and the underlying GP through variational generalized-EM inference. Experiments show competitive predictive performance and enhanced robustness across contamination settings.

  • IV. CONCLUSION: The approach embeds adaptive observation-specific precision modeling within a hierarchical generative framework for outlier-robust GPR.This design models contamination at the observation level.
  • IV. CONCLUSION: Variational generalized-EM inference jointly learns observation-specific contamination effects and the underlying GP model.The inference procedure estimates both contamination-related latent effects and GP components together.
  • IV. CONCLUSION: Experiments on synthetic and real datasets show competitive predictive performance with enhanced robustness across a range of contamination settings.The reported results support robustness under varied contamination conditions.
Loading 2608.16606v1…