PaperlayerView exact revision on arXiv

Source-linked AI summary

On the marginal likelihood and cross-validation

Edwin Fong, Chris Holmes

arXiv:1905.08737v2stat.MEstat.ML

TL;DR

Model evaluation contrasts Bayesian marginal likelihood with cross-validation, while exhaustive leave-p-out evaluation can be computationally demanding. The paper establishes a coherence-based uniqueness result and connects marginal likelihood to cumulative cross-validation, then suggests preparatory training to limit prior sensitivity.

  • Problem

    Model evaluation compares Bayesian marginal likelihood with out-of-sample cross-validation, while exhaustive leave-p-out scoring can require many test-set evaluations.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    The paper analyzes coherent Bayesian scoring under exchangeability and proposes cumulative cross-validation after a preparatory training phase.

    Source: §2 · p. 3, §3.2 · p. 5

  • Results

    The marginal likelihood is the unique coherent Bayesian score and is equivalent to cumulative cross-validation over training-test partitions.

    Source: §2 · p. 4, §5 · p. 7

  • Takeaways & Limitations

    The authors suggest using cumulative cross-validation after preparatory training when prior contributions to model evaluation are undesirable.

    Source: §3.2 · p. 5

    With vague priors, marginal-likelihood scores can be highly sensitive to prior dispersion, and diffuse priors can yield arbitrarily large negative scores.

    Source: §1 · p. 2, §3.2 · p. 5

Abstract

from arXiv · show

In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through $k$-fold partitioning or leave-$p$-out subsampling. We show that the marginal likelihood is formally equivalent to exhaustive leave-$p$-out cross-validation averaged over all values of $p$ and all held-out test sets when using the log posterior predictive probability as the scoring rule. Moreover, the log posterior predictive is the only coherent scoring rule under data exchangeability. This offers new insight into the marginal likelihood and cross-validation and highlights the potential sensitivity of the marginal likelihood to the choice of the prior. We suggest an alternative approach using cumulative cross-validation following a preparatory training phase. Our work has connections to prequential analysis and intrinsic Bayes factors but is motivated through a different course.

1 Introduction

The introduction contrasts Bayesian marginal likelihood with cross-validation as approaches to model evaluation and selection. It presents the marginal likelihood as a predictive score while noting its sensitivity to prior choice.

1 Introduction

  • Cross-validation evaluates models out of sample, but exhaustive leave-p-out evaluation requires n-choose-p test-set evaluations and is often computationally unviable.The introduction notes k-fold cross-validation as a commonly used approximation.Source: §1 · p. 1
  • The marginal likelihood is the data’s prior-predictive probability and can inform posterior model probabilities and Bayes factors.The Bayes factor is the ratio of marginal likelihoods between models.Source: §1 · p. 1
  • With vague priors, the marginal likelihood can be highly sensitive to prior dispersion even when the posterior is not.The introduction cites Lindley’s paradox as a known example.Source: §1 · p. 2
  • The log marginal likelihood can be written as a sum of log conditionals, with posterior-predictive terms for later observations and a representation valid under any data ordering.The paper interprets this as a sequential predictive score in an M-open setting.Source: §1 · p. 2

2 Uniqueness of the marginal likelihood under coherent scoring

The section develops coherence as a requirement for Bayesian model evaluation under exchangeability. It identifies the log posterior predictive as the unique coherent scoring rule, including when the model may be misspecified.

2 Uniqueness of the marginal likelihood under coherent scoring

  • Under exchangeability, coherence requires identical models on the same data to receive equal scores regardless of data ordering.The paper extends the idea to partitioning observations as well as ordering them.Source: §2 · pp. 2–3
  • The general Bayesian framework permits updating with a loss function without requiring that the model be true for some parameter value.Traditional Bayesian updating is recovered with w = 1 and l(θ, y) = −log fθ(y).Source: §2 · pp. 2–3
  • For a general Bayesian model, the section defines a posterior-predictive score by transforming the loss with a continuous, monotonically decreasing scoring function.The cumulative prequential score sums these scores across consecutive observations.Source: §2 · p. 3
  • The marginal likelihood is the unique coherent marginal score, and the log posterior predictive is the unique coherent prequential scoring rule.The result holds whether or not the model is true; coherence makes the score invariant to observation permutations under exchangeability.Source: §2 · pp. 3–4

3 The marginal likelihood and cross-validation

The marginal likelihood equals cumulative leave-p-out cross-validation using the log posterior predictive, aggregating predictive scores across held-out set sizes. This representation exposes sensitivity to prior specification and motivates starting evaluation after preparatory training.

3.1 Equivalence of the marginal likelihood and cumulative cross-validation

  • The log posterior predictive is the only coherent scoring rule used in this equivalence.Source: §3.1 · p. 4
  • The marginal likelihood equals the sum of leave-p-out cross-validation scores across held-out sizes when the log posterior predictive is the scoring rule.The per-datum scores are averaged over all held-out sets for each p, then scaled by n.Source: §3.1 · p. 4

3.2 Sensitivity to the prior and preparatory training

  • The cumulative representation exposes a prior-sensitive term with no training data, which scores the model entirely through prior specification.Diffuse priors can produce arbitrarily large negative model scores for real-valued parameters; 10% of terms can arise from predictions using, on average, less than 5% of available training data.Source: §3.2 · p. 5
  • Preparatory training offers an alternative: begin evaluating after training on, for example, 10% or 50% of the data, summing leave-p-out scores for p>P+1 through n.The resulting score retains coherence conditional on the preparatory training set; the authors note its form matches the log geometric intrinsic Bayes factor but follows a different motivation.Source: §3.2 · p. 5

4 Illustration for the normal linear model

The illustration compares marginal likelihood with cumulative cross-validation in polynomial regression as the prior variance changes. Under the vague prior, cumulative cross-validation still selects the true model, while the marginal likelihood favors a simpler model.

4 Illustration for the normal linear model

  • In the example, n = 100 observations come from a true model with r = 1, known σ2 = 1, and coefficient-prior variance s2 varied across values.Source: §4 · p. 6
  • The cumulative cross-validation estimates average over 10 runs of T = 10^6 random training/test splits, and the supplement adds a probit-regression example on the Pima Indian data set.In this illustration, the marginal likelihood is tractable, whereas cumulative cross-validation is estimated by Monte Carlo averaging of tractable log posterior predictives.Source: §4 · p. 6
  • With s2 = 10^4, the marginal likelihood prefers r = 0, while cumulative cross-validation correctly identifies the true model r = 1.For s2 = 10^-1 and s2 = 10^0, the marginal likelihood also identifies r = 1; the reported cumulative cross-validation scores are stable within each tested P column.Source: §4 · p. 6

5 Discussion

The discussion identifies the log posterior predictive probability as the unique coherent Bayesian model-evaluation score and equates marginal likelihood with cumulative cross-validation. For alternative loss functions, it proposes general Bayesian updating paired with the corresponding coherent cumulative cross-validation score.

5 Discussion

  • The log posterior predictive probability is the unique coherent scoring rule for Bayesian model evaluation in both M-open and M-closed settings.Coherence follows because the scoring rule and Bayesian update use the same information: the likelihood function.Source: §5 · p. 7
  • The marginal likelihood equals a cumulative cross-validation score over all training-test data partitions.Source: §5 · p. 7
  • For an alternative loss to the log likelihood, the authors advocate general Bayesian updating to target parameters minimizing expected loss, with models evaluated using the corresponding coherent cumulative cross-validation score.The passage cites Bissiri et al. (2016) and Lyddon et al. (2019) for general Bayesian updating.Source: §5 · p. 7

A.1 Proof of Proposition 1

The proof reduces the condition to a binary, two-observation case to identify its unique non-trivial solution. It then substitutes that choice back into the condition to establish it for general sample sizes, parameter spaces, and priors.

A.1 Proof of Proposition 1

  • Setting p=1 reduces the condition to g(l0)g(h0)=g(l0+h0).Source: §A.1 · p. 10
  • The proof uses continuity and monotone decrease to constrain g, then substitutes the resulting choice into (5) for general n, Θ, and π(θ).Source: §A.1 · p. 10
  • For n=2 and |Θ|=2, the unique non-trivial solution to (5) has λ=w.Source: §A.1 · p. 10

A.2 Proof of Proposition 2

The proof organizes log posterior predictives across all permutations of the observations in a matrix. It uses equal row sums and exchangeability to characterize the distinct training sets and next-observation choices within each column.

A.2 Proof of Proposition 2

  • Each row of Z records the prequential sequence of log posterior predictives for one permutation of the observations.Source: §A.2 · p. 11
  • Conditional probabilities imply that all rows of Z have equal sums.Source: §A.2 · p. 11
  • By exchangeability, each column has n-choose-(i−1) distinct training sets and n−i+1 choices for the next observation given a training set.Source: §A.2 · p. 11

A.3 Alternative proof of Proposition 2

Proposition 3 establishes an equivalence between preparatory cross-validation and an average of log marginal likelihoods over training-set choices. Proposition 2 follows by setting P = 0.

A.3 Alternative proof of Proposition 2

  • Proposition 3 states that the preparatory cross-validation score is the average log marginal likelihood over all choices of the training set.The score sums cross-validation terms from leave-(P + 1)-out through leave-n-out.Source: §A.3 · p. 11
  • The proof establishes the result by induction, starting from P = n−1 and relating adjacent preparatory scores through a leave-P-out term.The relation is SP CV(y1:n; P − 1) = SP CV(y1:n; P) + SCV(y1:n; P).Source: §A.3 · p. 11
  • Permutation invariance of the marginal likelihood is used to account for repetitions among the data partitions.Source: §A.3 · p. 12
  • Proposition 2 follows from Proposition 3 by setting P = 0.Source: §A.3 · p. 12

A.4 Derivation of SCCV for Bayesian models

The corollary derives an equivalence for cumulative cross-validation from the preceding propositions. Its proof decomposes the log marginal likelihood into cumulative and preparatory cross-validation scores, then uses permutation invariance.

A.4 Derivation of SCCV for Bayesian models

  • The corollary states an equivalence relationship for the cumulative cross-validation score.Source: §A.4 · p. 12
  • The proof writes the log marginal likelihood as the sum of the cumulative and preparatory cross-validation scores.Source: §A.4 · p. 12
  • The derivation uses permutation invariance of the marginal likelihood and subtracts the preparatory-score identity from the corresponding expression.Source: §A.4 · p. 12

A.5 Computing SCCV

The probit illustration finds that cumulative cross-validation changes little under a wider prior even when marginal-likelihood model comparison changes its assessment of ped. The appendix also notes bias and possible instability in estimating cumulative cross-validation, and illustrates how cross-validation scores vary with training-set size.

A.5 Computing SCCV

  • The estimated cumulative cross-validation score in (10) is biased; Rischard et al. (2018) provide unbiased estimators using Markov chain Monte Carlo and path sampling.Source: §A.5 · p. 13
  • Arithmetic averaging over training/test splits may be unstable when an ordering yields a training set dominated by one class and poor predictions for the held-out points.The passage suggests the median or a robust trimmed mean over permutations as potentially stabler alternatives.Source: §A.5 · p. 13

A.6 Visualization of cumulative cross-validation

  • In the polynomial regression example, leave-one-out under-penalizes complex models, while the true r = 1 model overtakes r = 0 as the training set grows.The r = 2 versus r = 1 discrepancy decreases for individual leave-p-out scores but remains more noticeable at moderate training sizes for the cumulative score.Source: Figure A.6.1 · p. 13, §A.6 · p. 14

A.7 Illustration for the probit model

  • At g = 10n, marginal likelihood suggests ped is no longer significant, while cumulative cross-validation changes little and the posterior for θped remains similar across prior settings.For g = n, the simpler model without ped performs worse under both scores; the posterior comparison supports the cumulative score's robustness to vague priors.Source: §A.7 · p. 14, Table A.7.1 · p. 15, Figure A.7.1 · p. 15
Loading 1905.08737v2…