PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Correlation as a Diagnostic for Joint-Uncertainty Gains from GP Coregionalisation

Fangqin Zhou, Joaquin Vanschoren

arXiv:2609.30085v2cs.LG

TL;DR

The paper asks when coupling correlated targets through GP coregionalisation helps, since its benefits are inconsistent. It uses residual correlation from independent predictors to diagnose potential joint-uncertainty gains and proposes Residual-ICM. Across the studied settings, coupling chiefly benefits joint uncertainty, and residual correlation predicts those gains better than raw target correlation.

  • Problem

    The benefit of coupling correlated targets through GP-ICM is inconsistent, and raw target correlation does not determine when coupling helps.

    Source: §1 · pp. 1–2

  • Method

    The paper diagnoses potential joint-uncertainty gains using Dlogdet from independent GP residual correlations and proposes Residual-ICM to add that structure to joint covariance.

    Source: §1 · p. 2, §9 · p. 14

  • Results

    Across the separable GP-ICM settings studied, coupling provides little benefit for representation selection or point prediction but can improve joint uncertainty; Dlogdet correlates with observed joint-NLL improvements (ρs = −0.83).

    Source: §1 · p. 2, §9 · p. 14

  • Takeaways & Limitations

    Residual dependence is more informative than raw target correlation for identifying when separable coregionalisation can improve joint uncertainty.

    Source: §9 · p. 14

    Dlogdet captures global Pearson residual dependence, not input-dependent correlations that cancel when pooled or nonlinear dependence with zero Pearson correlation.

    Source: §8 · p. 13

Abstract

from arXiv · show

In multi-target regression, correlated targets are often coupled through multi-output Gaussian processes with an intrinsic model of coregionalisation (GP-ICM), assuming that sharing statistical strength improves overall performance. In practice, the benefits are inconsistent. Across the settings studied, we find that the main benefit of coregionalisation is joint uncertainty quantification rather than point prediction. Raw target correlation does not predict when coupling helps; in the separable GP-ICM settings studied here, residual correlation, the cross-target dependence left unexplained by independent per-target predictors, is the strongest predictor of joint-uncertainty gains. We introduce a lightweight diagnostic, $D_{\rm logdet}=-\frac{1}{2}\log\det R_{\rm res}$, which represents the idealised joint negative log-likelihood (NLL) gain from modelling a full rather than diagonal residual covariance and is computable from independent GPs alone. Across a controlled synthetic study, 16 multi-target benchmarks, and frozen transformer and convolutional neural network representations for keypoint regression, point prediction remains largely unchanged ($\Delta R^2\approx 0$). In contrast, $D_{\rm logdet}$ strongly predicts observed ICM NLL improvements ($\rho_s=-0.83$, $p<0.001$), outperforming heuristics such as the feature-to-sample ratio. We also propose Residual-ICM, which preserves independent marginal variances while adding residual-correlation structure to the joint covariance. Residual-ICM achieves the best average joint NLL among the compared methods, while the diagnostic indicates when covariance coupling is likely to be useful. The diagnostic is specific to global Gaussian residual dependence, the structure captured by separable coregionalisation.

1. Introduction

The paper asks when target coupling helps and argues that residual correlation, not raw target correlation, identifies opportunities for joint-uncertainty gains. It introduces Dlogdet and Residual-ICM, then evaluates coupling across synthetic, real-world, and image-representation settings.

1. Introduction

  • Raw target correlation does not reliably identify when coupling helps: Tecator's mean target correlation is 0.91, yet coregionalisation increases joint NLL.The paper frames the unresolved question as when coupling helps and whether its benefit can be predicted in advance.Source: §1 · p. 2
  • Dlogdet predicts joint-uncertainty gains better than raw target correlation or p/n (ρs = −0.83 versus −0.07 and −0.46).It is computed from independent GPs and represents residual dependence left after fitting independent predictors.Source: §1 · p. 2
  • Across the studied settings, coregionalisation gains concentrate in joint uncertainty estimation and increase only when correlated residual structure remains.The evidence spans a synthetic phase diagram, 16 real datasets, and a sequential target-growth experiment.Source: §1 · p. 2
  • Across the 16 datasets, Residual-ICM achieves the best average joint NLL and mean rank among compared methods.It adds residual correlation to joint predictive covariance while preserving independent GP means and marginal variances.Source: §1 · p. 2
  • Target coupling adds little value for representation selection, while residual-correlation effects also appear in frozen transformer and CNN representations for keypoint regression.The broader analysis examines representation selection, point prediction, and joint uncertainty in a unified GP-readout framework.Source: §1 · p. 2

2. A Unified GP Readout Framework

The framework compares independent and coregionalised Gaussian-process readouts using matched features and held-out splits. It evaluates representation selection, point prediction, and uncertainty with distinct metrics and baselines.

Setup

  • Independent and coregionalised GPs use the same fixed feature matrix and held-out splits to compare representation selection, point prediction, and uncertainty quantification.Φ has n samples and p features; Y contains n samples and T targets.Source: §2 · p. 3

Single-output GP

  • Each target receives an independent RBF-kernel GP fit by maximum marginal likelihood; its log marginal likelihood supports representation selection and its marginal predictions support calibration.The kernel uses variance σ^2_f and lengthscale ℓ.Source: §2 · p. 3

Coregionalised GP (ICM)

  • The ICM extends a scalar kernel to a multi-output joint covariance, while Residual-ICM adds only cross-target covariance and preserves independent means and marginal variances.The supplied framework passage identifies the ICM as the coregionalised GP model.Source: §2 · p. 3

Controlled comparison

  • The controlled comparison uses shared features, fixes the ICM lengthscale to the median optimised independent-GP lengthscale, uses rank-1 coregionalisation, and caps noise variance at 0.5.Relaxing these constraints generally improves ICM, although some failures remain; higher ranks up to T − 1 are also evaluated.Source: §2 · p. 3

Baselines and evaluation

  • Representation selection uses Kendall τ, point prediction uses R^2 and RMSE, and uncertainty uses joint NLL, coverage gap, and sharpness against the stated baselines.Evaluation uses 5-fold cross-validation or leave-one-out when n < 100; negative ΔNLL favours ICM.Source: §2 · p. 3

3. The Residual-Correlation Diagnostic

Residual correlation measures cross-target dependence left after independent prediction, unlike raw target correlation, which may not indicate whether coupling can help. Dlogdet estimates idealised joint-NLL gains from modelling that dependence and is strongly associated with observed ICM gains, but is limited to global Pearson residual structure.

Residual correlation versus target correlation

  • Residual correlation is the cross-target dependence remaining after independent predictors condition on the input representation, whereas raw target correlation does not measure that remaining structure.The feature-to-sample ratio p/n is a coarse proxy: high p/n often leaves larger, more correlated residuals, but it does not measure residual dependence directly.Source: §3 · pp. 4–5

Diagnostic

  • Under an idealised Gaussian setting with fixed marginal predictive variances, the diagnostic is computed from independent GPs alone, and larger values indicate greater potential gains from covariance coupling.The realised ICM improvement may differ from this idealised gain because of finite-sample estimation, hyperparameter fitting, rank constraints, or non-Gaussian residuals.Source: §3 · p. 4

Scope of the diagnostic

  • Because Dlogdet uses one global Pearson correlation matrix, it can miss sign-changing input-dependent and nonlinear Pearson-uncorrelated residual dependence; a small value does not imply independence.The stress tests contrast global covariance coupling with oracle region-wise covariance and nonparametric conditional alternatives.Source: §3 · p. 4, Table 1 · p. 4

Observed ICM gains

  • ρs = −0.83 (p < 0.001): Dlogdet has the strongest association with observed ICM ∆NLL across 16 datasets, versus −0.07 for raw target correlation and −0.46 for p/n.The association remains strong after normalization by target count and after controlling for T, p/n, and independent-GP R2.Source: §3 · p. 5

Residual correlation versus target correlation

  • Tecator shows why raw correlation is insufficient: despite mean target correlation 0.91, independent-GP R2 = 0.945 and ICM worsens joint NLL by ∆NLL = +0.78.Strong target dependence does not imply exploitable residual dependence.Source: §3 · p. 5

4. Empirical Validation

The empirical validation compares independent GPs and ICM across synthetic settings, 16 real datasets, and sequential target growth. Joint-uncertainty gains depend on residual structure, while point-prediction changes are generally small.

4. Empirical Validation

  • The validation tests whether residual correlation predicts coupling benefits, whether gains appear in uncertainty rather than point prediction, and how gains change as targets are added.It uses a synthetic phase diagram, a 16-dataset real-data overlay, and a sequential target-growth experiment.Source: §4 · p. 6

4.1. Datasets

  • The 16 datasets span T = 2–16 and p/n ≈ 0.03–5.0, covering spectroscopy, environmental monitoring, hydrology, software-effort estimation, and robotics.Larger datasets were subsampled to n = 280 per seed for computational comparability.Source: §4.1 · p. 6

4.3. Synthetic Phase Diagram and Real-Data Overlay

  • ∆R2 remains near zero across most synthetic settings, while joint-NLL gains generally increase with target count and p/n; varying target correlation preserves this pattern, and gains are absent when independent predictors leave no correlated residuals.Real datasets broadly follow the synthetic trend, but Tecator worsens under ICM despite strong target correlation and high independent-GP accuracy.Source: Figure 1 · p. 7, §4.3 · pp. 7–8

4.2. The Single-Output GP Is a Strong Calibrated Baseline

  • Independent GPs achieve the lowest NLL and sharpest intervals on Tecator and Energy, and perform comparably to deep ensembles on PHO with similar coverage.They also recover residual cross-target structure better than linear and multilayer-perceptron probes.Source: §4.2 · p. 7

4.4. Sequential T ′-Growth Experiment

  • For atp1d, rf1, and rf2, joint-NLL gains increase as more targets are coupled, whereas wq shows little improvement; point-prediction changes remain |∆R2| ≤ 0.05 across all four datasets.The slower growth for rf2 despite its larger p/n suggests target count and p/n alone do not determine gains.Source: Figure 2 · p. 8, §4.4 · p. 8

5. From Diagnostic to Practical Algorithms

The paper turns the residual-correlation diagnostic into Residual-ICM and a diagnostic-gated coupling alternative. Residual-ICM has the best average joint NLL among the compared methods, while the gate reduces downside risk but misses some gains.

Residual-ICM

  • Residual-ICM estimates Rres from training-only, out-of-fold standardized residuals and forms Σ = diag(s)Rres diag(s), preserving independent-GP means and marginal variances while adding cross-target covariance.Shrinkage toward the identity stabilizes the correlation estimate; when residual samples approach or fall below T, the stated rule can shrink it nearly or entirely to identity.Source: §5 · pp. 8–9

Diagnostic-gated coupling

  • The diagnostic gate couples only when Dlogdet > τ and independent-GP R2 < 0.9; it reduces rank-1 ICM downside but misses the Residual-ICM improvement on rf1.The gate worsens NLL on one dataset by 0.64 nats, versus rank-1 ICM worsening it on three datasets by 1.85 nats; both have mean ∆NLL = −0.61.Source: §5 · p. 9

Downside risk

  • Across T ≥ 3 datasets, hindsight-rank ICM accumulates 0.62 nats of positive ∆NLL versus 0.01 for Residual-ICM; Residual-ICM improves joint NLL on Tecator and Sarcos while preserving independent-GP marginal predictions.Increasing ICM rank largely resolves the rank-1 failure on Sarcos, but not on Tecator.Source: §5 · p. 9

Comparative results

  • −1.13: Residual-ICM achieves the best average ∆NLL across 16 datasets, compared with −0.93 for rank-(T −1) ICM, and has mean rank 1.94.Even hindsight selection of the best ICM rank gives mean ∆NLL = −0.94; on T ≥ 3 datasets, Residual-ICM scores −1.54 versus −1.40 for hindsight-rank ICM.Source: §5 · p. 9, Table 4 · p. 10

Beyond GPs

  • Post-hoc residual covariance improves neural-predictor joint NLL on 13 datasets (mean ∆NLL = −0.80), but the evidence is limited to one neural architecture.Jointly learning a full covariance gives a smaller average improvement, ∆NLL = −0.34, and reduces R2 on 12 datasets.Source: §5 · p. 10

Decision map

  • Within the separable GP-ICM settings studied, p/n is a coarse pre-fit indicator, while Dlogdet and independent-GP R2 can guide coupling decisions after fitting independent GPs.Recommendations reflect average behaviour rather than guarantees; high prediction accuracy does not by itself rule out Residual-ICM gains.Source: §5 · p. 10

6. Further Consequences: Selection and Image Encoders

Target coupling adds little to representation selection in separable GP-ICM, but residual correlation remains strongly associated with joint-NLL gains on frozen image representations. The diagnostic is less informative when residual dependence is weak or nearly singular.

6.1. Representation Selection

  • With B fixed across representations, unequal eigenvalues can change representation rankings; exact agreement with independent-GP evidence holds when B = λI.In the experiments, B is re-optimised for each representation, so ranking invariance is not guaranteed; the result need not extend to non-separable multi-output GPs.Source: §6.1 · p. 11
  • Single-output GP marginal likelihood achieves the highest mean Kendall’s τ (0.706), versus GP-ICM (0.607) and LogME (0.527), for representation selection.The evaluation ranks six representations across six dataset–task pairs.Source: §6.1 · p. 11, Table 6 · p. 11

6.2. Residual Correlation in Frozen Image Encoders

  • Dlogdet is strongly associated with ICM joint-NLL improvement across 50 MPII encoder–subset conditions (ρs = −0.70, p ≈10−8).The conditions combine ten frozen encoders with keypoint subsets of 8–24 targets.Source: §6.2 · p. 11, Figure 3 · p. 12
  • Point-prediction changes remain small across the frozen image conditions.Larger target counts are also associated with greater NLL gains (ρs = −0.60, p < 10−5).Source: §6.2 · p. 12
  • On COCO, independent GPs perform poorly (R2 Indep ≤0.12, median 0.02), leaving little residual dependence that benefits Gaussian covariance coupling.On AFLW2000-3D, residual correlation matrices become nearly singular, so Dlogdet saturates and cannot distinguish conditions.Source: §6.2 · p. 12
  • Across the image experiments, the diagnostic is most informative when independent predictors leave exploitable residual correlation, and less informative when dependence is weak or nearly singular.Source: §6.2 · p. 12

7. Related Work

Prior work studies multi-output Gaussian processes, task relatedness, uncertainty evaluation, and representation selection. This paper uses independent-GP residual correlation to assess when separable coregionalisation may improve joint uncertainty.

  • Multi-output GPs model cross-target dependencies through shared covariance structures, including the linear model of coregionalisation and separable ICM.Extensions include convolution-based covariance models, scalable approximations, heterogeneous output likelihoods, and input-dependent cross-output correlations.Source: §7 · p. 12
  • Task sharing can improve generalisation, while inappropriate sharing may cause negative transfer; correlated prediction errors are also studied in seemingly unrelated regression.This work uses residual correlation from independent GPs to assess potential joint-uncertainty benefits of coregionalisation.Source: §7 · p. 13
  • Prior uncertainty work includes GP predictive uncertainty, deep ensembles, calibration, and proper scoring rules; representation-selection criteria include LogME, H-score, LEEP, and TransRate.The paper asks whether coregionalised GP evidence improves representation rankings beyond independent-GP evidence.Source: §7 · p. 13

8. Limitations

The study isolates cross-target covariance in separable GP-ICM with fixed input representations and marginal predictor families. Its diagnostic is limited to global Pearson residual correlation, and its conclusions about point prediction do not extend to all forms of target coupling.

  • The comparison focuses on separable GP-ICM with B ⊗K and moderate sample sizes that permit exact GP inference.Non-separable, sparse, inducing-point, and deep-kernel GPs may behave differently; broader multi-output methods are outside the controlled comparison.Source: §8 · p. 13
  • Dlogdet uses a single global Pearson correlation matrix, so it misses pooled-away input-dependent correlations and nonlinear dependence with zero Pearson correlation.Residual-correlation estimation can also be noisy or numerically unstable when structure is weak or highly collinear; a small value does not imply statistical independence.Source: §8 · p. 13
  • The point-prediction conclusions apply only to covariance-based coupling, not generally to models that jointly learn representations or parameters.The plant-phenotyping datasets contain only two targets, limiting what they show about coupling larger target sets.Source: §8 · p. 13

9. Conclusion

Across the studied settings, residual dependence is more informative than raw target correlation for identifying when separable coregionalisation may improve joint uncertainty. Residual-ICM preserves independent GP means and marginal variances while adding residual-correlation structure, and achieves the best average joint NLL among compared methods.

  • Residual-ICM achieves the best average joint NLL among compared methods while preserving independent GP means and marginal variances.It incorporates residual correlation into the joint predictive covariance.Source: §9 · p. 14
  • Residual dependence is more informative than raw target correlation for identifying when separable coregionalisation can improve joint uncertainty.Source: §9 · p. 14

Supplementary Material

The supplementary analyses test where the residual-correlation diagnostic works, whether its association is robust, and how numerical stabilization affects it. They also establish independent GPs as a competitive baseline for cross-target uncertainty and recovery tests.

A.1. Scope Stress Tests

  • With opposing regional correlations, Dlogdet fell from 1.76 to 0.08 and global Residual-ICM gained only 0.08 nats, while an oracle region-wise model gained 1.18 nats.The pooled global correlation can cancel local dependence.Source: §A.1 · p. 17, Table 7 · p. 17
  • Under nonlinear dependence with near-zero Pearson correlation (0.01), Dlogdet was approximately zero and Gaussian covariance coupling did not improve joint NLL; a nonparametric conditional model gained 0.30 nats.In the linear-dependence control, Gaussian covariance coupling improved joint NLL by 0.23 nats.Source: §A.1 · p. 18, Table 8 · p. 18

A.2. Robustness of the Diagnostic Association

  • Across 16 datasets, the Dlogdet association remained strong under both residual definitions: Spearman ρs was −0.83 for predictive-standardised and −0.85 for raw residuals.The predictive-standardised association had a bootstrap 95% confidence interval of [−0.99, −0.47]; neither leave-one-dataset-out nor per-seed analyses reversed its sign.Source: §A.2 · pp. 18–19, Table 9 · p. 19

A.3. Stabilising Dlogdet under Residual Collinearity

  • In AFLW, 40 of 50 conditions saturated the empirical Dlogdet ceiling despite observed ICM gains ranging from ΔNLL = −8 to −38.When n < T, rank deficiency can cause saturation; strong collinearity can also cause it when n > T.Source: §A.3 · p. 19
  • Ledoit–Wolf shrinkage restored diagnostic variation in the synthetic collinearity sweep, but its association effects varied: improvement on AFLW, little change on MPII, and degradation across tabular datasets.On AFLW, Spearman ρs changed from −0.69 to −0.83; on tabular datasets it changed from −0.81 to −0.60.Source: §A.3 · p. 20, Figure 4 · p. 20

B.1. Calibration of the Single-Output GP Baseline

  • Independent GPs achieved the lowest NLL and narrowest intervals on Tecator and Energy, with coverage gaps comparable to other methods.On PHO, GP and deep-ensemble NLLs were 0.918 and 0.913, respectively, and both outperformed BLR.Source: §B.1 · p. 21, Table 11 · p. 22

B.2. Cross-Target Recovery: Single-Output GP vs. Probes

  • GPs outperformed linear probes on PHO and Energy, but differences were not statistically distinguishable from zero on the smaller SH and WUR datasets (n ≤ 50).A one-hidden-layer MLP probe performed worse than the linear probe across tested datasets and directions.Source: §B.2 · pp. 21–22, Table 12 · p. 22

Appendix C. Synthetic Phase Diagram: Generator and Sensitivity Analysis

The appendix tests how synthetic phase patterns vary with shared target structure and evaluates ICM configurations, a diagnostic gate, and residual covariance in neural predictors. Across these analyses, Residual-ICM retains better average joint NLL than the compared ICM alternatives, while covariance effects in neural predictors depend on how covariance is modelled.

Appendix C. Synthetic Phase Diagram: Generator and Sensitivity Analysis

  • The synthetic generator holds intrinsic signal dimension at r = 3 while varying target count and feature dimension; cross-target correlation arises from shared latent factors and target loadings.The phase structure was not specific to one level of target correlation, and coupling benefit depended on residual dependence after independent prediction.Source: Appendix C · pp. 22–23
  • At weak shared structure (ρ ≤ 0.3), ICM provided little benefit; at higher ρ, gains concentrated at larger T and p/n, while the mean ΔNLL across the grid was not strictly monotonic.The mean ΔNLL values for increasing ρ were +0.04, +0.09, −0.08, −0.11, and −0.07; residual diagnostic associations persisted at ρ = 0.7 and 0.9.Source: Appendix C · p. 23

D.1. ICM Configuration and Optimisation Details

  • Relaxing ICM restrictions improved mean performance: re-optimising lengthscale helped 15 of 16 datasets, and mean ΔNLL improved from −0.62 at rank 1 to −0.93 at rank T−1.Stronger configurations did not resolve all failures; Tecator and EDM still degraded joint NLL in the reported comparisons.Source: §D.1 · p. 24, Table 13 · p. 24

D.2. Does a Stronger ICM Close the Gap to Residual-ICM?

  • Residual-ICM achieved mean ΔNLL = −1.13 across all datasets and −1.54 for T ≥ 3, outperforming even the hindsight-rank ICM comparison.Positive ΔNLL summed to 0.10 nats for Residual-ICM versus 1.26 for hindsight-rank ICM across all datasets.Source: §D.2 · p. 25, Table 14 · p. 25

D.3. Sensitivity of the Diagnostic Gate to the Accuracy Ceiling

  • The gate’s mean ΔNLL remained −0.641 for accuracy ceilings c ∈ [0.85, 0.926], while Residual-ICM achieved lower mean ΔNLL than the gate at every tested ceiling.The apparent evaluation optimum at c = 0.93 was not a generally optimal threshold; the reported gate range was −0.59 to −0.71 versus −1.13 for Residual-ICM.Source: §D.3 · p. 26, Table 15 · p. 26

D.4. Residual Covariance in Neural Predictors

  • Post-hoc residual covariance improved joint NLL on 13 of 16 neural-prediction datasets, with mean ΔNLL = −0.80 and diagnostic association ρs = −0.88.Jointly learned full covariance yielded mean ΔNLL = −0.34, lacked a significant diagnostic association, and reduced R² on 12 of 16 datasets.Source: §D.4 · p. 27, Table 16 · p. 27

Appendix E. Representation Selection: Evidence Decomposition and Results

For representation selection, separable GP-ICM evidence decomposes into single-output evidence on rotated targets, with exact agreement to independent-GP rankings when the coregionalisation matrix has equal eigenvalues. Across the reported dataset–direction pairs, GP-LML wins or ties GP-ICM.

Evidence decomposition

  • For separable GP-ICM, multi-output log-evidence decomposes exactly into single-output evidences on rotated targets.The rotation is given by the eigendecomposition of the coregionalisation matrix B.Source: Appendix E · pp. 27–28

Interpretation

  • When B is fixed across candidate encoders, its eigenpairs and thus the target rotation and weights are encoder-independent, but unequal eigenvalues can change rankings relative to independent GPs.The fixed weights can make per-direction evidences respond differently to changes in the input kernel.Source: Appendix E · pp. 27–28

Equal eigenvalues

  • When B = λI, multi-output evidence coincides with independent-GP evidence using common signal variance λ, yielding identical representation rankings.This is the equal-eigenvalue case of the decomposition.Source: Appendix E · pp. 27–29

Scope

  • The decomposition applies to separable models, does not generally extend to non-separable multi-output GPs, and includes LogME as a degenerate linear-kernel case.The reported empirical consistency between GP-ICM and single-output GP evidence is not implied by the analysis.Source: Appendix E · p. 28

Per-task selection results

  • GP-LML achieves the highest Kendall τ in five of six dataset–direction pairs, and GP-ICM never outperforms it.The per-task comparison reports agreement for representation selection.Source: Table 17 · p. 29, Appendix E · p. 29

Related papers

Loading 2609.30085v2…