PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

Tuomas Sivula, Måns Magnusson, Asael Alonzo Matamoros, Aki Vehtari

arXiv:2008.10296v6stat.ME

TL;DR

Bayesian LOO-CV estimates predictive performance, but model comparisons also require uncertainty estimates for the difference, whose calibration needs assessment. The paper studies the estimator and normal and Bayesian-bootstrap uncertainty approximations, finding problematic settings that include similar predictions, misspecification, and small data.

  • Problem

    Predictive performance for unseen data cannot be computed directly, and model comparisons need uncertainty estimates for the difference to assess which model may predict better.

    Source: §1 · p. 1

  • Method

    The paper studies Bayesian LOO-CV difference-estimator properties and normal and Bayesian-bootstrap uncertainty approximations in linear and related model settings.

    Source: §2.1 · p. 4, §6 · p. 20

  • Results

    Uncertainty quantification can be problematic with similar predictions, bad misspecification with outliers, or small data; similar-prediction error skewness may persist as n grows.

    Source: §6 · pp. 20–21

  • Takeaways & Limitations

    Small datasets, such as n<100, call for additional caution, and obtaining more data is the best way to improve reliability and reduce uncertainty.

    Source: §6 · p. 21

    The study focuses on linear models; whether the results extend to other models, including singular models, is left for future research.

    Source: §7 · p. 21

Abstract

from arXiv · show

It is useful to estimate the expected predictive performance of models planned to be used for prediction. We focus on leave-one-out cross-validation (LOO-CV), which has become a popular method for estimating predictive performance of Bayesian models. Given two models, we are interested in comparing the predictive performances and associated uncertainty, which can also be used to compute the probability of one model having better predictive performance than the other model. We study the properties of the Bayesian LOO-CV estimator and the related uncertainty quantification for the predictive performance difference, and analyse when a normal approximation of this uncertainty is well calibrated and whether taking into account higher moments could improve the approximation. We provide new results of the properties both theoretically in the linear regression case and empirically for hierarchical linear, latent linear, and spline models and discuss the challenges. We show that problematic cases include: comparing models with similar predictions, misspecified models, and small data. In these cases, there is a weak connection between the distributions of the LOO-CV estimator and its error. We show that that the problematic skewness of the error distribution for the difference, which occurs when the models make similar predictions, does not fade away when the data size grows to infinity in certain situations. Based on the results, we also provide some practical recommendations for the users of Bayesian LOO-CV for comparing predictive performance of models.

1 Introduction

Bayesian LOO-CV estimates predictive performance, but model comparisons also require uncertainty estimates whose calibration must be assessed. The paper studies this uncertainty theoretically and empirically, identifying settings where it can be unreliable and offering guidance for comparison.

1 Introduction

  • The paper examines whether uncertainty estimates for differences in Bayesian LOO-CV predictive performance are calibrated, since these estimates support probability statements about which model predicts better.Predictive performance on unseen data cannot be computed directly, so cross-validation estimates it and model comparisons must account for uncertainty in the difference.Source: §1 · pp. 1–2

1.1 Our Contributions

  • The authors analyze the error distribution and normal and Bayesian-bootstrap approximations in two-model comparisons, focusing on expected log pointwise predictive density (elpd) in linear models.They study finite-sample behavior and asymptotic properties without assuming either model is true.Source: §1.1 · p. 2
  • Uncertainty estimates can perform poorly when models make similar predictions, models are misspecified with outliers, or data are small.The study derives analytical results for normal linear regression and reports experiments with fixed-covariate, hierarchical linear, Poisson generalized linear, and spline models, as well as Bayesian K-fold-CV.Source: §1.1 · p. 2
  • When models make similar predictions, their predictive performances differ little, so either model can be used for prediction.In this setting, the paper identifies problematic uncertainty behavior even as data size grows in certain situations.Source: §1.1 · p. 2
  • The authors recommend avoiding misspecified model comparisons through model checking and expansion before using LOO-CV.Source: §1.1 · p. 2

2 Problem Setting

The paper examines Bayesian LOO-CV estimates of predictive performance and the uncertainty in comparing two models. It shows that approximations can be poorly calibrated in several settings, including similar model predictions, misspecification, outliers, and small data.

2 Problem Setting

  • The setup evaluates expected log pointwise predictive density for future data from the same data-generating process, using a proper local log score.Source: §2 · pp. 2–3

2.1 Bayesian Cross-Validation

  • LOO-CV estimates expected predictive performance by using each observation as a validation case and fitting on the remaining data; efficient methods can estimate it more cheaply.Source: §2.1 · p. 4

2.2 Uncertainty in Cross-Validation Estimators

  • The paper compares a normal approximation justified through the Dirichlet-process posterior with a Bayesian bootstrap, finding no practical benefit from the latter in experiments.Source: §2.2 · p. 5, §2.3 · p. 6

2.3 Problems in Quantifying the Uncertainty

  • The connection between the available LOO-CV terms and the true error distribution can be weak, so knowing the terms’ distribution may not yield a good error approximation.Source: §2.3 · p. 7
  • When models have asymptotically equal predictive performance, uncertainty about their difference decreases at the same rate as the difference itself.Source: §2.3 · p. 7
  • Misspecification and outliers can affect uncertainty in complex ways, and outliers change moment-term constants so larger samples may be needed for good calibration.Source: §2.3 · p. 7
  • Normal uncertainty approximations work well when predictive-performance differences are relatively large, but can be inaccurate otherwise; similar-performance cases can have skewed distributions.Source: §2.3 · p. 7
  • Outliers can bias the LOO-CV difference estimate, while small samples can produce uncertainty estimates that are too high near the mode or too low in the tails.Source: §2.3 · pp. 7–8

3 Theoretical Analysis using Bayesian Linear Regression

The theoretical analysis derives properties of Bayesian LOO-CV comparison errors for two normal linear regression models. It identifies when finite-sample uncertainty is poorly approximated and shows that some skewness problems can persist asymptotically.

3 Theoretical Analysis using Bayesian Linear Regression

  • The analysis assumes a normal linear-regression data-generating process and compares two normal regression models with fixed noise variance and non-informative uniform parameter priors.Neither comparison model needs to match the data-generating process.Source: §3 · p. 9, §3.1 · p. 9

3.2 Controlling the Similarity of the Predictive Performances

  • The effects of covariates included in only one model, βΔ, control how similar the models’ predictive performances are; near-zero effects can make the comparison problematic.When βΔ = 0, both models include the same covariates with most non-zero effects, while noise in ineffective covariates can still affect predictive performance.Source: §3.2 · p. 9

3.3 Properties for Finite Data

  • The approximation error has a quadratic factorisation that permits efficient computation of its first moments in the linear-regression case.The LOO-CV difference and predictive-performance difference have analogous quadratic forms.Source: §3.3 · p. 10
  • Shared covariate effects do not affect the distributions of the comparison quantities, while error skewness tends to zero as models become more dissimilar.With correctly specified models and no outliers, skewness is most extreme when predictive performances are similar; in nested comparisons, extreme skewness can remain high as sample size grows.Source: §3.3 · p. 10, Figure 2 · p. 11
  • Outliers can affect error bias and variance differently, and reducing skewness does not necessarily improve the approximation because relative bias can increase.With an outlier, problematic skewness may occur at any level of non-shared covariate effects; as sample size grows, skewness can approach zero while bias increases.Source: §3.3 · pp. 11–12

3.4 Asymptotic Behaviour as a Function of the Data Size

  • When βΔ = 0, the error skewness converges to a non-zero value, so uncertainty analysis can remain problematic with large data; for βΔ ≠ 0, it converges to zero.The simpler model performs better asymptotically at βΔ = 0, while a simpler model can also outperform a more complex one in some cases with βΔ ≠ 0.Source: §3.4 · p. 14

4 Simulation Experiments

Simulations test whether the linear-regression analysis generalizes to more commonly used models and examine estimator error and uncertainty calibration. They find that similar predictions, misspecification, and small samples can undermine calibration, while the patterns extend beyond the simplest model.

Negative Correlation and Bias

  • The simulated estimator and its error behave similarly to the theoretical results, including negative correlation when model performances are similar.The negative correlation is more noticeable with larger sample sizes.Source: §4 · pp. 14–15
  • With an outlier, the LOO-CV estimator can be biased, and the relative error’s mean usually deviates from zero.The outlier experiment examines joint distributions of the estimated and true predictive-performance differences.Source: §4 · p. 15

Behaviour of the Uncertainty Estimates

  • Normal and Bayesian-bootstrap uncertainty estimates have similar calibration, but small samples, similar predictions, and outlier misspecification can make calibration problematic.In one setting with n = 512 and βΔ = 0, calibration improves with an outlier because the outlier inflates variance and skewness decreases more than bias increases.Source: §4 · pp. 15–16, Figure 6 · p. 16

Additional simulations

  • Additional simulations show that the theoretical patterns extend to models with more covariates, non-Gaussianity, hierarchy, splines, fixed covariates, and K-fold-CV.The experiments test whether results from a simplified model generalize to more commonly used models.Source: §4 · pp. 14, 16

5 Case studies

Case studies apply LOO-CV uncertainty comparisons to sleep, roaches, and primate milk data. They illustrate that model checks, outliers, sample size, and the size of predictive differences affect how confidently models can be compared.

Primate milk

  • The primate milk covariates are collinear, and their useful predictive information lies along their joint distribution’s second principal component.This explains why models with only one covariate are not better than the intercept-only model.Source: §5 · p. 18

Sleep study

  • For the sleep data, M3 is estimated to predict better than M2, but the probability is 0.9; two outlying observations make the normal approximation likely poorly calibrated.For the better-specified Student’s t models, M3t’s predictive performance is better than M1t and M2t with probability close to 1.Source: §5 · pp. 18–19

Roaches

  • For the roaches data, the zero-inflated negative-binomial model M3 is clearly best, while the spline model M4 is only slightly better than M3.The Poisson model is underdispersed, but its predictive-performance difference from M3 is large; the M4–M3 difference is small and its normal approximation may be imperfectly calibrated.Source: §5 · p. 20

6 Conclusions

Uncertainty estimates can be poorly calibrated when models make similar predictions, when models are badly misspecified with outliers, or when data are small. These cases call for practical caution when comparing predictive performance with LOO-CV.

6 Conclusions

  • Similar predictions, outlier misspecification, and small samples are identified as settings where LOO-CV uncertainty estimates can be problematic.For similar predictions, both normal and Bayesian bootstrap approximations are badly calibrated; bad misspecification can also bias the estimated difference.Source: §6 · p. 20
  • When models make similar predictions, either model can be used for prediction because their predictive performances differ little.For nested models, the authors note a preference for the larger model may provide more information through the posterior of its additional terms.Source: §6 · p. 20
  • With bad model misspecification and outliers, the estimated predictive-performance difference can be significantly biased and its uncertainty miscalibrated.The authors recommend model checking and possible refinement before using cross-validation for comparison.Source: §6 · pp. 20–21
  • For small data, around n < 100, estimating uncertainty in the predictive-performance difference is less reliable and requires additional caution.The authors identify obtaining more data as the best way to improve reliability and reduce uncertainty.Source: §6 · p. 21

7 Discussion

The discussion relates the uncertainty analysis to alternative predictive methods and model-selection practices, while Appendix A distinguishes uncertainty targets and their error distributions. It also describes how estimator variability is framed for the elpd measures.

7 Discussion

  • LOO-CV and WAIC target expected predictive performance and are asymptotically equivalent, so the uncertainty findings are expected to extend to WAIC if its computational approximation does not fail.The authors assume other smooth, strictly proper scoring rules behave similarly, while noting that further research is justified.Source: §7 · p. 21
  • The analysis covers simple, hierarchical, latent, and basis-function linear models; similarity for other models, including singular models, is an assumption left for future research.The discussion also notes lower variance for projection predictive selection and describes leave-group-out and leave-future-out settings for different prediction targets.Source: §7 · pp. 21–22
  • For several models, pairwise comparisons against the best-performing model reduce the comparison count to one less than the number of models; model averaging is an option when no model is clearly best.The discussion cautions against a fixed model-selection probability threshold because the appropriate threshold depends on context, and notes trade-offs in interpreting LOO and stacking weights.Source: §7 · p. 22

Appendix A Difference Between Estimating elpd and e-elpd

  • Appendix A distinguishes predictive performance conditional on observed data (elpd) from its expectation over possible datasets (e-elpd), whose associated estimator uncertainties differ.The analysis is primarily application-oriented, while the appendix formulates uncertainty for both measures.Source: Appendix A · p. 27

A.1 Estimating e-elpd

  • For e-elpd, the data are treated as a random sample, and estimation error is assessed over possible datasets through the estimator’s sampling distribution.Source: §A.1 · p. 27

A.2 Estimating elpd

  • For elpd conditional on observed data, each dataset defines a different target, and uncertainty is assessed over possible datasets in a similar way.The excerpt notes that variability of the error depends on the LOO-CV estimate as well as the target.Source: §A.2 · p. 27

A.3 Error Distributions

Under independent observations, the error distributions for elpd and e-elpd have the same expectation but differ in variability. The appendix also clarifies the contexts in which it calls estimator behavior a sampling distribution.

A.3 Error Distributions

  • Assuming independent observations, the error distributions for elpd and e-elpd have the same expectation but different variability.The passage attributes the variability difference to generally small or negative correlation between the estimator and elpd.Source: §A.3 · pp. 27–28

A.4 Sampling Distributions

  • The appendix calls estimator behavior over possible datasets a sampling distribution, while noting that other assessments of uncertainty about elpd are possible.Source: §A.4 · p. 28

Appendix B Alternative Formulations of the Uncertainty

Appendix B considers alternative targets and formulations for assessing uncertainty, including an estimator using independent test data. Its illustrated comparison shows that connected datasets can yield different uncertainty behavior; Appendix C introduces the normal and Bayesian bootstrap estimators.

Appendix B Alternative Formulations of the Uncertainty

  • Uncertainty about elpd can be assessed through alternatives to stochastic analysis of error over possible datasets, including bounds or Bayesian inference on the error.The appendix notes that alternative uncertainty targets might be feasible, particularly for specific problem settings.Source: Appendix B · p. 28

B.1 LOO-CV Estimate with Independent Test Data

  • One proposed interpretation treats the LOO-CV estimate as a realization of an estimator based on an independent test dataset and a given problem dataset.Source: §B.1 · p. 29
  • In the described LOO-CV application, the same dataset is reused for both estimator arguments, unlike the independent-test-data formulation.The excerpt says the independent-test-data formulation’s error variability is not affected by conditional elpd in the same way.Source: §B.1 · p. 29
  • The connection between datasets affects the estimator’s uncertainty behavior, and the illustrated distributions have close means but different variance and skewness.Source: §B.1 · p. 29
  • The appendix compares sampling distributions with the distribution of elpd, but the supplied passage does not state the outcome of that comparison.Source: §B.1 · p. 29

Appendix C Analysing the Uncertainty Estimates

  • Normal distributions and Bayesian bootstrap are the uncertainty estimators examined in Appendix C.Source: Appendix C · p. 29

C.1 Normal Model for the Uncertainty

The section examines normal and Dirichlet approaches to uncertainty in LOO-CV model comparison. It notes that overlapping training sets challenge a naive standard-error estimate, while the Bayesian bootstrap also has limitations.

C.1 Normal Model for the Uncertainty

  • The normal approximation is a common approach to estimating uncertainty in a LOO-CV estimate.The naive standard-error estimator assumes the LOO contributions are independent, although each observation belongs to n−1 training sets.Source: §C.1 · pp. 29–30
  • For a selected setting, the sampling distributions of d elpdLOO have different shapes.The illustration uses n=128, βΔ=0, and r★=0; the passage does not specify which model's shape is better calibrated.Source: §C.1 · p. 30
  • Propositions 6 and 7 and Corollary 1 characterize bias in the naive variance estimator for model comparison.The supplied passages identify the propositions as establishing the associated bias but do not provide its magnitude.Source: §C.1 · pp. 30–33

C.2 Dirichlet Model for the Uncertainty

  • The Bayesian bootstrap can represent skewness in uncertainty, but it also has problems with higher moments and heavy-tailed distributions.It is presented as an alternative to the normal approximation for modeling the uncertainty distribution.Source: §C.2 · p. 33

C.3 Not Considering All the Terms in the Error

This section examines how omitting an error term can affect sampling-distribution-based uncertainty estimates for LOO-CV, while characterizing the related quantities in normal linear regression. It considers how model structure and covariate effects shape these quantities, including how skewness approaches a constant determined by the explanatory-variable matrix as data variance grows.

C.3 Not Considering All the Terms in the Error

  • In problematic settings, sampling-distribution variability can differ from LOO-CV error because its estimate omits an error term; for e-elpd, the variances correspond.In well-behaved settings, the sampling and error variability can match, but estimating the sampling distribution is sufficient for uncertainty in e-elpd.Source: §C.3 · p. 33

Appendix D Normal Linear Regression Case Study

  • The normal linear regression case study derives quadratic forms for full-data elpd and LOO-CV estimates in a model comparison with a known data-generating mechanism.The derivations cover individual models, their difference, and the LOO-CV error.Source: Appendix D · p. 33, §D.1.1 · p. 38, §D.1.2 · p. 38

D.3 LOO-CV Error

  • Parameters for the elpd difference, LOO-CV estimate difference, and error do not depend on covariate effects shared by the models.These results concern the analytic normal linear regression comparison.Source: §D.1.2 · p. 39, §D.2.2 · p. 42, §D.3 · p. 44

D.2.3 Additional Properties for the Parameters of the LOO-CV Estimate

  • The LOO-CV difference matrix is symmetric, has eigenvalues summing to zero, and is therefore indefinite or the zero matrix.The appendix also establishes symmetry of the matrix from the component products.Source: §D.2.3 · p. 43

D.4 Reparametrisation as a Sum of Independent Variables

  • The quadratic-form variables can be represented as a constant plus a sum of independent scaled non-central chi-square variables with one degree of freedom.The appendix applies this reparameterization to the variables of interest, including the LOO-CV error.Source: §D.4 · pp. 44–45

D.5.2 Effect of the Non-Shared Covariates’ Effects

  • When non-shared covariate effects are scaled, the error moments through the third are quadratic in the scale; under specified no-outlier conditions, skewness is continuous and symmetric.With zero base effects and each covariate included in exactly one model, the absolute skewness reaches a maximum at one or both values specified by Equations (200) and (201).Source: §D.5.2 · pp. 48–49

D.5.3 Effect of Outliers

  • As outlier magnitude is scaled, the second and third moments are either constant or quadratic in that scale.The passage states that the skewness is also affected when those moments are not constant, but does not provide the full result.Source: §D.5.3 · p. 50

D.5.4 Effect of Residual Variance

  • As data variance grows, skewness converges to a constant determined by the explanatory-variable matrix X.Source: §D.5.4 · p. 50

D.5.5 Graphical Illustration of the Moments for an Example Case

The appendix illustrates moment behavior and derives simplified analytic forms for elpd and LOO-CV quantities in a one-covariate case. The example contrasts conditional and random-design analyses, while the derivations identify parameters on which particular moments do not depend.

D.5.5 Graphical Illustration of the Moments for an Example Case

  • With random design X, the skewness of elpd decreases as the non-shared covariate effect grows, unlike the conditional-X case, where it is high across effects.The example estimates moments across simulated X values and uses a Bayesian bootstrap for uncertainty.Source: §D.5.5 · pp. 50–51

D.6 One Covariate Case

  • The one-covariate example compares a null model using only an intercept with a model that also includes a balanced covariate.The covariate column contains half 1s and half −1s; the derivation notes that effects shared by both models do not affect the LOO-CV error.Source: §D.6 · pp. 51–52

D.6.1 Elpd

  • In the one-covariate case, the derived elpd mean does not depend on s★, and its second moment does not depend on β1 or m★.The appendix derives simplified analytic forms for elpd and its moments in this setting.Source: §D.6.1 · pp. 52, 57, 58

D.6.2 LOO-CV

  • For d elpdLOO in the one-covariate case, the first moment does not depend on s★, and the mean-to-standard-deviation limit matches that for elpd.The section derives simplified forms for d elpdLOO and its moments.Source: §D.6.2 · pp. 59, 63, 66

D.6.3 LOO-CV Error

This section derives simplified analytic forms for the LOO-CV error and its moments in the one-covariate case. It reports parameter invariances and asymptotic behavior of the error's skewness.

D.6.3 LOO-CV Error

  • The appendix derives simplified forms for the LOO-CV error parameters and uses them to obtain moments in the one-covariate case.The derivation draws on the one-covariate elpd and LOO-CV results.Source: §D.6.3 · p. 66
  • The error's first moment does not depend on s★, and its skewness does not depend on τ.These parameter invariances are reported for the one-covariate derivation.Source: §D.6.3 · pp. 69, 75
  • The skewness limit approaches −23/2 from below and is reported not to depend on m★ or s★.The appendix analyzes the asymptotic behavior of skewness as n grows.Source: §D.6.3 · pp. 75–76

Appendix E Additional Results for the Simulated Experiment

The simulated experiments show that outliers, similar model predictions, and small samples can distort LOO-CV estimates and uncertainty. Skewness may persist as sample size grows when the models’ predictive performances are equal.

Additional simulation results

  • Without outliers, relative error is near zero; as the non-shared effect grows, LOO-CV is more likely to select the correct model, while at zero it slightly favors the simpler model regardless of sample size.In an extended example, the error is symmetric and its mean and median are near zero.Source: Appendix E · pp. 77, 86, Figure 19 · p. 87, Figure 20 · p. 87

Similar predictions

  • When the non-shared effect is zero, estimator skewness does not fade as sample size grows; when it is nonzero, skewness decreases with sample size.Problematic skewness is more pronounced when model predictions are similar.Source: Appendix E · pp. 77–78

Outlier effects

  • With an outlier, LOO-CV’s estimator is affected more than the predictive-performance estimand, and the effect grows with the non-shared covariate effect.The estimator can be biased; in one setting, it almost surely selects the wrong model.Source: Appendix E · p. 78, Figure 11 · p. 79
  • With an outlier, relative error often shifts the estimated performance difference toward zero or the opposite sign.Whether the estimate instead moves farther from zero depends on the situation.Source: Appendix E · p. 79, Figure 12 · p. 80, Figure 13 · p. 81

Sampling and uncertainty distributions

  • When model performances are similar, the estimator’s sampling distribution can be skewed in the wrong direction even though its mean remains close to the uncertainty distribution’s mean.With outliers, approximation bias can instead make the means clearly different.Source: Appendix E · p. 80

Uncertainty calibration

  • Variance is predominantly underestimated with small non-shared effects or small samples, but with outliers it can also be substantially overestimated.In the misspecified setting, the estimator is biased, making overestimation understandable and acceptable to the authors.Source: Appendix E · p. 84
  • The theoretical approximation is well calibrated only without an outlier and with a large non-shared effect or, to some degree, a large sample.For small samples and similar predictions, sampling and error skewness are weakly connected, so improving the sampling-distribution estimate does not ensure calibrated uncertainty.Source: Appendix E · pp. 82, 86, Figure 18 · p. 86

Appendix F Other model variants

Across additional model variants, the observed behavior matches the theoretical and earlier experimental results. In the hierarchical example, group-level variation increases variation in both the estimate and true value, but not in the error distribution.

Experimental variants

  • The experiments extend the nested-regression comparison to fixed covariates, more covariates, non-Gaussian outcomes, hierarchical models, splines, and 10-fold cross-validation.The setup uses models and data-generating processes based on the same structure, with the more complex model including an additional covariate.Source: Appendix F · p. 88

10-fold cross-validation

  • For 10-fold cross-validation, using fewer observations for each posterior likely makes predictive performance slightly worse than with leave-one-out cross-validation.The authors note that bias correction is rarely used and can increase variance.Source: Appendix F · p. 88

Scope boundary

  • Leave-one-group-out cross-validation behaves much differently from leave-one-out cross-validation and is left for future research..Source: Appendix F · p. 88

Results across variants

  • Across additional variants, the joint-distribution results match the theoretical and previous experimental results; the hierarchical example has greater estimate and true-value variation without a wider error distribution.The authors attribute the hierarchical correlation to variation in how strongly groups differ across data realizations.Source: Figure 21 · p. 89, Figure 22 · p. 89, Appendix F · p. 90
Loading 2008.10296v6…