PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross Validation and Maximum Likelihood estimations of hyper-parameters of Gaussian processes with model misspecification

François Bachoc

arXiv:1301.4320v3math.ST

TL;DR

The paper addresses how to estimate Gaussian-process covariance hyper-parameters for Kriging when the assumed covariance family may be misspecified. It uses a two-step analysis, first deriving a predictive-variance criterion and closed-form results for a single variance parameter, then extending the study to estimated correlation hyper-parameters. CV is more robust under misspecification, while ML is more likely to yield the best predictions when the correlation family is ideally specified.

  • Problem

    The paper studies ML and CV covariance-hyper-parameter estimation when the true covariance function may not belong to the assumed parametric family, a setting not fully resolved by asymptotic results.

  • Method

    The paper uses a two-step approach combining closed-form finite-sample analysis for a single variance parameter with numerical experiments that also estimate correlation hyper-parameters.

  • Results

    CV performs better than ML when the correlation model is misspecified, whereas ML is optimal or more likely to yield the best predictions when the correlation family is well specified.

  • Takeaways & Limitations

    CV is suitable when robustness to covariance misspecification has priority, while ML is preferable when the correlation family is ideally specified and best possible performance is prioritized.

  • Takeaways & Limitations

    The study notes that overly regular designs may be unsuitable for CV and that CV may require penalization of large correlation lengths in some cases.

Abstract

from arXiv · show

The Maximum Likelihood (ML) and Cross Validation (CV) methods for estimating covariance hyper-parameters are compared, in the context of Kriging with a misspecified covariance structure. A two-step approach is used. First, the case of the estimation of a single variance hyper-parameter is addressed, for which the fixed correlation function is misspecified. A predictive variance based quality criterion is introduced and a closed-form expression of this criterion is derived. It is shown that when the correlation function is misspecified, the CV does better compared to ML, while ML is optimal when the model is well-specified. In the second step, the results of the first step are extended to the case when the hyper-parameters of the correlation function are also estimated from data.

1. Introduction

The paper examines ML and CV for estimating covariance hyper-parameters when the covariance family may be misspecified, motivated by limits of asymptotic theory and finite-sample practice. It uses a two-step analysis that first studies a single variance parameter and then estimates correlation hyper-parameters as well.

  • Kriging interpolates Gaussian random-field observations and is used for numerical-code approximation, calibration, and global optimization.
  • Misspecified covariance functions can undermine the unbiasedness, minimal variance, and correctness of predictive variances guaranteed under a correctly specified covariance model.
  • Fixed-domain asymptotics distinguishes microergodic hyper-parameters, which can be consistently estimated, from non-microergodic parameters, which cannot be consistently estimated but do not asymptotically affect Kriging predictions.
  • Finite-sample performance remains relevant because asymptotic results may not resolve covariance estimation when the true covariance is outside the assumed parametric family.
  • The two-step approach compares ML and CV first for a single variance hyper-parameter with fixed misspecified correlation, then for jointly estimated variance and correlation hyper-parameters.
  • When the correlation model is misspecified, CV performs better than ML in the paper’s main result.

2. Framework for the variance hyper-parameter estimation

The framework separates the true Gaussian process correlation from the correlation assumed by the Kriging model and evaluates variance estimation through predictive-variance accuracy. A Risk criterion and closed-form analysis support comparison of classical estimators under correlation misspecification.

  • The true process is stationary, centered, and unit-variance, while the Kriging model assumes its covariance belongs to a specified family C.
  • Observations y at n points estimate the variance hyper-parameter σ2, while the resulting Kriging prediction ˆy0 at a new point is unaffected by this variance estimate.
  • Under correlation misspecification, the model uses a classical predictive-variance expression that scales a prediction-error structure by the estimated variance.
  • Risk measures whether the estimated predictive variance correctly predicts the conditional mean square error of ˆy0.
  • When the correlations coincide, the Risk is minimized by ˆσ2 = 1; when they differ, another variance estimate can partly compensate for correlation-function error.
  • For quadratic-form estimators ˆσ2 = y^tMy, Proposition 2.2 provides a closed-form Risk expression applicable to ML and CV variance estimators.

3. ML and CV estimation of the variance hyper-parameter

The paper defines CV through leave-one-out predictive errors and compares it with ML using variance, efficiency, and Risk properties. ML is more efficient under correct specification, motivating numerical analysis of misspecified correlations.

  • CV estimates σ2 by computing leave-one-out predictions and choosing the variance value that makes the normalized leave-one-out criterion equal to 1.
  • The leave-one-out criterion is expected to be close to 1 when the modeled covariance σ2R2 is correct, with mean-square convergence established for regular expanding designs and classical correlation families.
  • Virtual leave-one-out formulas yield a vector-matrix closed form for the CV criterion, including a matrix formed by zeroing the non-diagonal terms of Γ2^-1.
  • Both ML and CV variance estimators are unbiased when the true variance is 1.
  • When R1 = R2, ML is more efficient than CV for estimating the variance parameter.
  • The misspecified-correlation case is analyzed numerically after the correctly specified comparison.

4. Numerical results for the variance hyper-parameter estimation

The numerical study compares ML and CV for predictive-variance assessment under covariance misspecification across designs of experiments and correlation-function errors. CV is generally more robust for SRS and LHS-Maximin designs, whereas regular sparse grids reverse this comparison.

  • Criteria: The study compares ML and CV using RTR, BTR, and their integrated versions to assess predictive-variance estimation.RTR measures relative prediction error, while BTR measures relative bias; for the variance-only setting, these criteria reduce to mean-square error and bias for estimating σ2 = 1.
  • Designs of experiments: Three designs are studied: SRS, LHS-Maximin, and a deterministic sparse regular grid.SRS uses independent uniform points; LHS-Maximin selects the design maximizing minimum pairwise distance; the sparse grid uses Smolyak sparse tensorization.
  • Experimental setting: The experiments vary misspecification of the power-exponential exponent, Matérn regularity, and Matérn correlation length.They use 50 learning samples, n = 70 observations for random designs, n = 71 for the regular grid, and a test sample of size 5000.
  • SRS results: For SRS, CV becomes more efficient than ML under large model errors because its bias increases more slowly with misspecification.The IRTR combines IBTR with an integrated relative-variance term, and the IBTR contribution grows faster than the variance contribution, especially for ML.
  • LHS-Maximin results: LHS-Maximin produces results similar to SRS, with CV achieving lower IRTR than ML at a smaller model error.The comparison is slightly more pronounced for LHS-Maximin than for SRS.
  • Regular sparse-grid results: Regular sparse grids reverse the comparison: CV is less robust than ML because leave-one-out estimates target the grid and generalize poorly outside it.CV bias can become very large, while ML estimates a more general-purpose variance; removing points from structured designs may also produce overpessimistic CV results.
  • Influence of the number of points: As the number of SRS observations increases, BTR approaches a nonzero limit while the relative-variance term decreases toward zero.CV has lower BTR for the tested covariance errors but a slower decrease in relative variance, and its RTR eventually becomes lower than ML's.

5. Study on analytical functions

Analytical-function experiments compare ML and CV for fixed and estimated correlation hyper-parameters, using MSE and predictive-variance accuracy. Across tested functions, ML performs best with adapted covariance models, whereas CV is more robust under misspecification.

  • Numerical considerations: Numerical stability is challenged by ill-conditioned smooth correlation matrices and by CV’s tendency to estimate excessively large correlation lengths.The study uses a nugget and penalizes large estimated variance to obtain appropriate predictive variances.
  • Experimental setup: The experiments use deterministic functions, 100 LHS-Maximin training designs, and Monte Carlo test samples of size 10000.Correlation hyper-parameters are estimated with both ML and CV.
  • Results with enforced correlation lengths: For fixed correlation lengths, Gaussian correlation is more appropriate than exponential correlation for the smooth Ishigami function.Using separate correlation lengths improves prediction quality, especially for the Gaussian model.
  • Results with enforced correlation lengths: When the correlation model is inappropriate, CV yields much smaller predictive-variance errors than ML, while ML and CV have comparable errors for the best-adapted model.The result concerns predictive variance accuracy rather than the shared MSE from using the same correlation function.
  • Results with estimated correlation lengths: For estimated correlation lengths, Gaussian models yield smaller MSEs than exponential models for the smooth Ishigami and Morris functions.Estimating separate lengths generally gives smaller MSE than estimating one isotropic length.
  • Results with estimated correlation lengths: With exponential correlations, CV consistently gives smaller predictive-variance errors than ML, whereas ML is comparable or better for adapted Gaussian and Matérn models.For the Morris function, allowing separate correlation lengths increases ML’s advantage over CV.
  • Case of universal Kriging: For universal Kriging, adding a non-zero mean does not significantly improve performance and can slightly damage Gaussian-model results through over-parameterization.CV is more sensitive to this over-parameterization.

6. Conclusion

The paper analyzes ML and CV for covariance hyper-parameter estimation under covariance-model misspecification. It concludes that CV is more robust to misspecification, whereas ML can yield the best predictions when the correlation family is correctly specified.

  • Conclusion: The analysis uses two steps: closed-form mismatch indices for a misspecified correlation with one free variance parameter, followed by numerical experiments estimating correlation hyper-parameters too.The second step uses analytical functions and confirms and generalizes the first-step results.
  • Conclusion: Under model misspecification, CV performs better than ML in the estimation of covariance hyper-parameters.This conclusion is obtained from the numerical study of the first-step closed-form expressions and supported by the general numerical experiments.
  • Conclusion: CV is more robust to model misspecification and is therefore less likely than ML to yield substantially incorrect predictions.The paper frames this as a robustness advantage rather than a guarantee of the best possible predictions.
  • Conclusion: ML is more likely to yield the best predictions when the correlation function family is ideally specified.The paper therefore presents CV as suitable when robustness has priority over best possible performance.
  • Conclusion: Some overly regular designs of experiments are unsuitable for CV, and large correlation lengths may require penalization of estimated global variance parameters.The paper identifies these issues as topics for further investigation and notes that the penalization motivation was purely numerical in this study.

Appendix A. Proof of Proposition 2.2

Appendix A derives a closed-form predictive-risk expression for the estimated variance by expanding quadratic Gaussian expressions and applying a fourth-moment lemma.

  • Appendix A. Proof of Proposition 2.2: The appendix expresses the predictive risk of the estimated variance in terms of trace and bilinear-form quantities involving M̃0 and M̃1.The displayed expression combines f(M̃0,M̃0), trace terms, cross terms, and f(M̃1,M̃1).
  • Appendix A. Proof of Proposition 2.2: The derivation evaluates Gaussian quadratic-form moments using a lemma for z ∼ N(0, I_n) and symmetric matrices A and B.The lemma states E(z^tAz z^tBz) = f(A,B).
  • Appendix A. Proof of Proposition 2.2: Expanding the preceding expression with the fourth-moment lemma yields the risk formula used in the proposition.The proof explicitly attributes the calculation to fourth-moment computations for Gaussian variables.
  • Appendix A. Proof of Proposition 2.2: The final algebra replaces M̃0 and M̃1 by M0 and M1 using tr(AB) = tr(BA), completing the proof.This trace identity is the stated final simplification.

Appendix B. On the optimality of ˆσ2 ML when R1 = R2

Appendix B examines the well-specified case R2 = R1 and derives the Cramér–Rao bound for unbiased estimation of σ2.

  • Appendix B. On the optimality of σ̂2 ML when R1 = R2: The appendix restricts attention to the case R2 = R1 before applying the Cramér–Rao inequality.It considers σ2 = 1 and an unbiased estimator σ̂2 of σ2.
  • Appendix B. On the optimality of σ̂2 ML when R1 = R2: The likelihood of the observations is used to calculate the Cramér–Rao bound for the statistical model.The derivation uses a Gaussian fourth-moment lemma with A = B = I_n.
  • Appendix B. On the optimality of σ̂2 ML when R1 = R2: The resulting Cramér–Rao bound is 2/n when y ∼ N(0, I_n).This is the bound reported for the statistical model under the stated normalization.

Appendix C. Convergence of (6)

Appendix C proves convergence of CLOO under observations on an infinite regular grid by controlling covariance-matrix eigenvalues and using Fourier-transform conditions.

  • Appendix C. Convergence of (6): Under the stated regular-grid assumptions, CLOO has mean one and converges in mean square to one as n → +∞.The result assumes a summable covariance function with a strictly positive continuous Fourier transform.
  • Appendix C. Convergence of (6): The proof transforms the Gaussian observations using Γ2 and applies formulas from Proposition 3.1 to analyze the criterion.The transformed vector is stated to satisfy a standard normal distribution.
  • Appendix C. Convergence of (6): The argument isolates the diagonal of Γ2^-1 and bounds its eigenvalues using the smallest and largest eigenvalues of a symmetric positive matrix.These bounds support control of the terms appearing in the convergence proof.
  • Appendix C. Convergence of (6): Summability of the covariance on the infinite grid bounds λmax(Γ2), while the positive Fourier-transform infimum supplies a lower bound.Together these properties provide uniform spectral control across n.
  • Appendix C. Convergence of (6): The proof uses distinct observation points on the regular grid and orthogonality properties of grid Fourier terms.These geometric conditions enter the lower-bound argument.
Loading 1301.4320v3…