PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

Sumio Watanabe

arXiv:1004.2316v2cs.LG

TL;DR

In singular models, the asymptotic behavior of cross-validation was unclear despite its role in estimating generalization error. The paper theoretically compares Bayes cross-validation with WAIC and proves their asymptotic equivalence, while relating their errors to the real log canonical threshold. It also distinguishes both criteria from deviance information criteria.

  • Problem

    Cross-validation’s asymptotic behavior in singular statistical models was unknown, although it estimates generalization error and regular-model leave-one-out cross-validation is equivalent to AIC.

  • Method

    The paper theoretically compares Bayes cross-validation, WAIC, and Bayes generalization error using singular learning theory.

  • Results

    Bayes cross-validation loss is asymptotically equivalent to WAIC as a random variable, and the summed Bayes generalization and cross-validation errors are asymptotically 2λ/n.

  • Takeaways & Limitations

    The relationship between cross-validation and generalization error is determined by the algebraic-geometrical structure of the statistical model, while deviance information criteria differ from both.

  • Takeaways & Limitations

    The equivalence may fail in practice when the posterior distribution is not precisely or sufficiently approximated.

Abstract

from arXiv · show

In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2λ/n$, where $λ$ is the real log canonical threshold and $n$ is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.

1 Introduction

Singular learning machines invalidate familiar regular-model asymptotics, motivating a theoretical comparison of Bayes cross-validation with WAIC. The paper proves equivalence between these criteria and relates cross-validation and generalization errors to the real log canonical threshold.

  • Motivation: Singular models can make maximum-likelihood asymptotics fail, so AIC and BIC no longer retain their usual asymptotic interpretations.The maximum likelihood estimator may diverge or yield large generalization error in singular models.
  • Motivation: Cross-validation estimates generalization error, but its asymptotic behavior in singular models had remained unclear.In regular models, leave-one-out cross-validation is asymptotically equivalent to AIC.
  • Contribution: The paper theoretically compares Bayes cross-validation, WAIC, and Bayes generalization error in singular statistical models.It proves two theorems concerning their asymptotic relationships.
  • Contribution: Bayes cross-validation loss is asymptotically equivalent to WAIC as a random variable.The paper also states that the two criteria consequently support asymptotically equivalent model selection and hyperparameter optimization.
  • Contribution: The sum of Bayes generalization and cross-validation errors is asymptotically 2λ/n, linking their relationship to the model’s algebraic-geometrical structure.Here λ is the real log canonical threshold and n is the number of training samples.

2 Bayes Learning Theory

The paper formulates Bayes learning using probability models, priors, posterior expectations, predictive distributions, and generalization-related losses. Singular learning theory supplies the analytic and probabilistic framework, under compactness and regularity assumptions, for deriving WAIC-based asymptotics.

  • Framework: Bayes learning models data with a density p(x|w), a parameter space W, and a prior distribution ϕ(w).The prior need not represent literal prior knowledge of the parameter.
  • Framework: Posterior expectations use inverse temperature β, with β=1 corresponding to strict Bayes estimation.The Bayes predictive distribution and associated losses are defined from this posterior framework.
  • Quantities: Bayes generalization loss, training loss, functional variance, and WAIC are the central quantities used to characterize learning behavior.The expectation of Bayes generalization loss is asymptotically equal to WAIC under the stated theory.
  • Singular structure: Singular learning theory treats nonunique minimizers as analytic or algebraic sets rather than necessarily single parameter points.This reflects the failure of one-to-one parameterization in singular learning machines.
  • Assumptions: The analysis assumes a compact parameter space, a structured prior, analytic model properties, and additional control conditions.Under these assumptions, the studied observables have finite variances and the stated asymptotics apply.
  • Scope: If the final control assumption is not satisfied for singular unrealizable machines, Bayes errors may have different asymptotic behavior.The paper notes that this assumption holds in some such cases but not others.

3 Bayes Cross-validation

Bayes cross-validation evaluates predictive log loss by repeatedly leaving out each training sample and predicting it from the remaining posterior. Its expected loss is asymptotically aligned with Bayes generalization loss and WAIC.

  • Construction: Bayes cross-validation constructs a posterior and predictive distribution after leaving out each training sample Xi.The left-out observation is then treated as a testing sample.
  • Construction: The cross-validation loss is the empirical average of the log losses obtained from those leave-one-out predictive distributions.This quantity is denoted CvL(n).
  • Expected loss: Cross-validation loss and Bayes generalization loss differ as random variables, although their expectation values are related by sample independence.The paper derives this relationship from the independent training samples.
  • Expected loss: The expectations of CvL(n), BgL(n−1), and WAIC(n−1) are asymptotically equal.The paper identifies clarifying the asymptotic behavior of the corresponding random variables as its primary goal.
  • Practical use: Unlike Bayes generalization loss, both cross-validation loss and WAIC can be calculated using only training samples.Bayes generalization loss requires an expectation over a testing sample from the unknown true distribution.

4 Main Results

The paper represents cross-validation and WAIC through functional cumulants, proving their asymptotic equivalence in singular statistical models. It then relates both errors to birational invariants, especially the real log canonical threshold.

  • 4 Main Results: Functional cumulants provide a common representation for the cross-validation loss and WAIC.The paper defines their generating function and derives functional-cumulant identities used in the main proofs.
  • 4.2 Bayes Cross-validation and Widely Applicable Information Criterion: For arbitrary 0 < β < ∞, the Bayes cross-validation loss and WAIC are asymptotically equivalent.The result is established as a theorem and supplemented by a corollary describing their asymptotic difference.
  • 4.3 Generalization Error and Cross-validation Error: The real log canonical threshold and singular fluctuation are birational invariants determined by the algebraic-geometrical structure of the statistical model.The real log canonical threshold does not depend on β, whereas the singular fluctuation does.
  • 4.3 Generalization Error and Cross-validation Error: The cross-validation error and Bayes generalization error are linked through the real log canonical threshold and sample size.The theorem states that their sum is asymptotically equal to 2λ/n, with λ the real log canonical threshold and n the number of training samples.
  • 4.3 Generalization Error and Cross-validation Error: In the strict Bayes case β = 1, a smaller cross-validation error corresponds asymptotically to a larger Bayes generalization error.The paper also states that the two errors have equal variances in this case and that both scaled errors converge in law.

5 Discussion

The discussion contrasts regular and singular learning theory, examines computational approximations and DIC, and interprets theoretical and experimental evidence for Bayes cross-validation and WAIC. It emphasizes that posterior approximation affects equivalence results, while singular structure distinguishes DIC and constrains cross-validation-based optimization.

  • 5.1 From Regular to Singular: In regular models, Bayes and maximum-likelihood generalization losses share asymptotic behavior, and leave-one-out cross-validation is asymptotically equivalent to AIC.
  • 5.1 From Regular to Singular: In singular models, maximum-likelihood generalization loss exceeds Bayes generalization loss, making maximum likelihood inappropriate for singular learning machines.
  • 5.2 Cross-validation and Importance Sampling: CV1 requires n different posterior distributions and therefore has heavy computational cost, while importance-sampling cross-validation uses one posterior distribution.
  • 5.2 Cross-validation and Importance Sampling: Precisely realized posterior distributions make CV1, CV2, and WAIC asymptotically equivalent, whereas posterior approximation can make their values differ.
  • 5.3 Comparison with the Deviance Information Criteria: The averages of DIC1, DIC2, CV1, CV2, and WAIC can share asymptotic behavior, but DIC averages need not equal the Bayes generalization error in singular or unrealizable settings.
  • 5.4 Experiment: The experiment found approximately matching averages and standard deviations for BG, CV, and WAIC, lower variability for BG + CV, and a CV–WAIC correlation of 0.996.
  • 5.5 Birational Invariant: Minimizing cross-validation reduces generalization loss on average but does not ensure the minimum loss, and WAIC has the same property.

6 Conclusion

The paper shows that, in singular statistical models, leave-one-out Bayes cross-validation is asymptotically equivalent to WAIC, while the combined cross-validation and generalization errors equal twice the real log canonical threshold divided by the sample size.

  • Cross-validation and WAIC are asymptotically equivalent even in singular statistical models.
  • 2λ/n is the asymptotic sum of the cross-validation error and the generalization error.Here, λ is the real log canonical threshold and n is the number of training samples.
  • Cross-validation and WAIC differ from the deviance information criteria.
  • The asymptotic properties of singular models are determined by their algebraic-geometrical structure.
Loading 1004.2316v2…