PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding predictive information criteria for Bayesian models

Andrew Gelman, Jessica Hwang, Aki Vehtari

arXiv:1307.5928v1stat.ME

TL;DR

The paper addresses how Bayesian practitioners can estimate and compare expected out-of-sample predictive accuracy when direct evaluation is unavailable. It reviews AIC, DIC, and WAIC through simple theoretical and applied examples, concluding that cross-validation is preferred while WAIC is a fast alternative with important scope constraints.

  • Problem

    Predictive accuracy must be estimated from training data because true out-of-sample performance is unavailable, while practitioners face competing approximate criteria.

  • Method

    The paper reviews AIC, DIC, and WAIC from a Bayesian predictive perspective and examines them in three simple examples.

  • Results

    Cross-validation is preferred, with WAIC presented as a fast and computationally convenient alternative that uses the posterior distribution.

  • Takeaways & Limitations

    The criteria provide practical but approximate tools for evaluating and comparing Bayesian models on a common predictive scale.

  • Takeaways & Limitations

    WAIC requires partitioning data into n pieces, which is difficult for structured data such as time series, spatial, and network data.

Abstract

from arXiv · show

We review the Akaike, deviance, and Watanabe-Akaike information criteria from a Bayesian perspective, where the goal is to estimate expected out-of-sample-prediction error using a biascorrected adjustment of within-sample error. We focus on the choices involved in setting up these measures, and we compare them in three simple examples, one theoretical and two applied. The contribution of this review is to put all these information criteria into a Bayesian predictive context and to better understand, through small examples, how these methods can apply in practice.

1. Introduction

The paper frames predictive accuracy as a practical model-comparison target, requiring bias correction because evaluating fitted models on their training data is optimistic. It reviews AIC, DIC, and WAIC from a Bayesian perspective through three simple examples.

  • Predictive accuracy helps compare and improve models even when all candidate models mismatch the data.
  • Estimating out-of-sample error requires correcting the bias from evaluating predictions on the data used for fitting.
  • Cross-validation is a natural estimator but requires repeated model fits and can be problematic with sparse data.
  • No general method simultaneously provides unbiased, accurate prediction-error estimates with minimal computation across models.
  • The review explores AIC, DIC, and WAIC using three simple examples rather than deriving the measures from first principles.

2. Log predictive density as a measure of model accuracy

The section defines predictive accuracy through scoring functions and log predictive density, emphasizing expected performance on future data. It develops asymptotic intuition and motivates bias correction of within-sample predictive density.

  • Predictive accuracy evaluates how well a model predicts future data, either for its own performance or for comparing models.
  • The logarithmic score is the unique local and proper scoring rule, making log predictive density central to model comparison.
  • For normal models with constant variance, log predictive density is proportional to mean squared error but is less restricted to point prediction.
  • Asymptotic behavior: As n →∞, the posterior log predictive density approaches a constant minus one-half times a χ2_k random variable.
  • Asymptotic behavior: Singular models invalidate the usual χ2 asymptotics because plug-in estimates are not representative of the posterior and the Fisher information matrix is not positive definite.
  • The observed-data lppd underestimates expected future-data elppd, motivating a bias correction based on posterior simulations.

3. Information criteria and effective number of parameters

Information criteria estimate expected predictive accuracy by correcting optimistic within-sample fit, while cross-validation estimates performance through held-out data. Each approach is approximate and has practical limitations.

  • Within-sample predictive accuracy is optimistic because it evaluates the model on the data used for fitting.
  • AIC, DIC, and WAIC adjust within-sample predictive accuracy using the number or effective number of fitted parameters.
  • These bias corrections can be reasonable in many cases but are guaranteed at best in expectation, not necessarily for each dataset.
  • Cross-validation uses training data for fitting and held-out data for evaluation, avoiding overfitting but requiring potentially many partitions and model fits.
  • AIC: For maximum-likelihood fitting, subtracting k from log predictive density corrects the chance improvement from estimating k parameters, yielding AIC = −2 log p(y|θ̂_mle) + 2k.
  • Effective parameters: Informative priors and hierarchical structures reduce overfitting, so effective parameter counts need not equal the number of nominal parameters.

3.3. Deviance information criterion (DIC) and effective number of parameters

DIC adapts AIC by replacing the maximum-likelihood estimate with the posterior mean and estimating an effective parameter count from the posterior. Its variants differ in stability and positivity.

  • DIC replaces the maximum-likelihood estimate with the posterior mean and replaces k with a data-based effective parameter correction.
  • The effective parameter count pDIC is computed from posterior expectations using simulations from the fitted posterior.
  • Limitations: Negative pDIC can occur when the posterior mean is far from the posterior mode.
  • Both pDIC and pDIC alt are asymptotically correct for fixed models as n grows and reduce to k for linear models with uniform priors.
  • pDIC is more numerically stable, whereas pDIC alt is always positive.
  • DIC is ultimately defined using deviance rather than log predictive density.

3.4. Watanabe-Akaike information criterion (WAIC)

WAIC estimates Bayesian out-of-sample predictive performance by correcting log pointwise posterior predictive density for effective model complexity. The review favors WAIC because it averages over posterior uncertainty and applies to singular models, while illustrating its construction and practical use.

  • Definition and computation: WAIC estimates out-of-sample expectation from log pointwise posterior predictive density with an effective-parameter correction for overfitting.The correction is computed pointwise using either a difference-based measure or summed posterior variances of log predictive-density terms.
  • Definition and computation: pWAIC 2 is recommended because its series expansion more closely resembles LOO-CV and generally produces results closer to LOO-CV.
  • Definition and computation: WAIC is scaled as −2 times lppd − pWAIC, placing it on the deviance scale for comparison with AIC and DIC.Watanabe’s original definition uses the negative average log pointwise predictive density without the factor 2.
  • Bayesian interpretation: Unlike AIC and DIC, WAIC averages over the posterior predictive distribution rather than conditioning on a point estimate.AIC and DIC estimate plug-in predictive-density performance, although Bayesian users still use posterior predictive densities for prediction.
  • Bayesian interpretation: WAIC also applies to singular models, including hierarchical and mixture structures where parameter counts can increase with sample size and point estimates may not make sense.The review therefore finds WAIC more appealing than AIC and DIC and studies the criteria through three simple examples.

3.5. Pointwise vs. joint predictive distribution

WAIC’s pointwise partition creates a practical limitation for structured data, where separating observations into independent pieces may be difficult. Joint-prediction alternatives do not remove all computational or comparability challenges.

  • WAIC relies on partitioning data into n pieces, which is difficult for structured settings such as time series, spatial, and network data.AIC and DIC do not explicitly make this partition, but their derivations assume residual independence conditional on a point estimate.

3.6. Effective number of parameters as a random variable

The effective number of parameters can depend on the observed data, even when the model and true parameters are fixed. A positive-constraint example also motivates why BIC addresses a different goal.

  • For y_i ∼ N(θ, 1) with θ constrained positive, the effective parameter count is approximately 1/2 when observations are near zero.Roughly half the posterior information comes from the data and half from the positivity constraint.
  • When y is positive and large, the positivity constraint is essentially irrelevant and the effective parameter count is approximately 1.
  • The example shows that effective parameter count can depend on observed data despite a fixed model and fixed true parameters.
  • BIC applies a penalty increasing with sample size and can favor simpler models even when a complicated model predicts well and has low AIC, DIC, and WAIC.Because BIC is not intended to predict out-of-sample model performance, the article does not consider it further.

3.8. Leave-one-out cross-validation

Bayesian LOO-CV repeatedly fits models after leaving out data and evaluates held-out predictive density, directly targeting out-of-sample predictive fit. It can be bias-corrected, but partitioning and computation remain important practical constraints.

  • Bayesian LOO-CV partitions data into training and holdout sets, fits the model to training data, and evaluates the holdout log predictive density.The leave-one-out case uses n partitions, each holding out one data point, with posterior simulations summarizing each fit.
  • Each LOO-CV prediction conditions on n−1 observations, which underestimates predictive fit; a first-order correction estimates performance when conditioning on n observations.The bias correction is usually small and is included for completeness.
  • Cross-validation is limited by the need for disjoint, ideally conditionally independent data pieces and can be computationally expensive without shortcuts.With moderate k-fold partitions, such as k = 10, computation is reasonable in most applications.
  • Bayesian cross-validation works with singular models, and Bayesian LOO-CV is asymptotically equal to WAIC, although finite-sample differences can appear for small n or hierarchical models.
  • LOO-CV and WAIC can target different prediction tasks in regression and hierarchical models because LOO-CV conditions on new covariates while WAIC predicts at observed covariate locations.This distinction can matter in flexible models such as Gaussian processes and hierarchical models.

3.9. Comparing different estimates of out-of-sample prediction accuracy

AIC, DIC, and WAIC all adjust within-sample log predictive density to estimate out-of-sample predictive accuracy, but they differ in both their starting points and bias corrections.

  • All three criteria subtract an approximate bias correction from the log predictive density of observed data.
  • AIC conditions on the maximum likelihood estimate, DIC on the posterior mean, and WAIC averages predictive density over the posterior distribution.
  • WAIC is the only one of the three that is fully Bayesian, which motivates the authors’ preference for it when using a bias-correction formula.

4. Theoretical example: normal distribution with unknown mean

The normal-mean example compares AIC, DIC, WAIC, and cross-validation against expected predictive performance, first with a flat prior and then with informative prior structure. It shows how their effective-parameter adjustments behave with sample size, prior information, and data-point definitions.

  • Noninformative prior: WAIC averages pointwise log predictive densities over the posterior, unlike AIC and DIC, which condition on a point estimate.Its effective parameter count is not exactly 1 at finite sample sizes, although it approaches 1 as n grows.
  • Finite-sample behavior: At n = 1, the correct effective-parameter adjustment is 0.5, matching pWAIC 2 and the expected bias correction, while pWAIC 1 gives 0.31.The appropriate adjustment is less than 1 for finite n and tends to 1 in the large-sample limit.
  • Cross-validation: Leave-one-out cross-validation differs from the target because it conditions on n − 1 observations, while bias correction substantially improves its accuracy.The distinction is between predictive performance for future data and evaluation based on the observed data.
  • Noninformative prior: For a noninformative normal prior, AIC and DIC are identical and unbiasedly estimate plug-in predictive log density for new data.In this example, pDIC is exactly 1, so DIC and AIC coincide.
  • Informative prior: AIC remains unchanged when a prior is added because its maximum-likelihood estimate is unaffected, whereas WAIC and cross-validation depend on how observations are grouped as data points.Counting each observation separately gives repeated single-observation results, while counting groups changes the pointwise predictive quantities.
  • Informative prior: Informative priors reduce effective parameter counts: pWAIC approaches 0 under a completely informative prior, while pDIC likewise tends toward 0.With a flat prior, m = 0 and the effective count returns to 1; increasing prior information reduces the amount estimated from the data.

5. Simple applied example: election forecasting

The election-forecasting example uses a Bayesian linear regression to compare predictive-error measures, showing how posterior uncertainty and parameter-fitting corrections shape AIC, DIC, WAIC, and cross-validation.

  • Model setup: The example predicts vote share from economic performance with a linear regression and a noninformative prior, fitting 15 election observations.The posterior mode is (45.9, 3.2, 3.6) for the intercept, slope, and residual standard deviation.
  • Posterior predictive density: Posterior uncertainty produces a log predictive-density distribution with maximum −40.3, mean −42.0, and mean–maximum difference 1.7.The difference is close to the asymptotic value 3/2 for three estimated parameters.
  • Posterior predictive density: The log pointwise predictive density is −40.9, slightly below the predictive density at the maximum likelihood estimate because parameter uncertainty is averaged over.The calculation can use either the analytic posterior predictive distribution or simulation-based formula (5).
  • Information criteria: AIC is 86.6, DIC is 87.0, and WAIC is 86.2 or 87.2 depending on its effective-parameter correction.DIC estimates 3.0 effective parameters, while WAIC uses corrections of 2.2 and 2.7.
  • Cross-validation: Leave-one-out cross-validation gives −43.8 on the expected log predictive-density scale, or 87.6 after multiplying by −2.Its estimated effective number of parameters is 2.9, consistent with the model’s two coefficients and variance parameter.

6. Simple applied example: meta-analysis of educational testing experiments

The educational-testing example compares no pooling, complete pooling, and hierarchical models using predictive densities and information criteria. Although no pooling fits the observed data best, correction for fitted parameters makes complete pooling best for this dataset.

  • Models: The eight-school data are analyzed with no pooling, complete pooling, and hierarchical models representing separate, shared, and partially pooled school effects.The hierarchical model constrains eight school parameters through a common distribution, so its effective parameter count should be below 8.
  • Models: The posterior mass for the hierarchical standard deviation τ is concentrated near 0, indicating little variation in true treatment effects across the eight schools.The model uses θj ∼ N(µ, τ^2) with a uniform hyperprior on µ and τ.
  • Information criteria: No pooling has the highest raw fit, but after parameter correction AIC estimates complete pooling has the best expected log predictive density.The adjustment contrasts eight fitted parameters for no pooling with one for complete pooling.
  • Information criteria: DIC and WAIC give results similar to AIC, while the hierarchical model receives an intermediate fit and an effective parameter count of 2.8.For all three models, pWAIC is much smaller than pDIC because WAIC’s lppd already accounts for much parameter uncertainty.
  • Cross-validation: Cross-validation is impossible for no pooling because predicting a held-out school from the other seven is undefined, whereas it is available for complete pooling and the hierarchical model.WAIC and cross-validation also encode different prediction tasks in hierarchical settings because WAIC uses the local observation as well as other schools’ data.
  • Model comparison: For this dataset, complete pooling wins the expected out-of-sample prediction comparison, although the authors still prefer the hierarchical model because τ may not truly be zero.The data are compatible with zero between-school variation, but school-specific estimates leave substantive uncertainty about that conclusion.

7. Discussion

The discussion frames predictive comparison as useful but imperfect: model complexity, prediction targets, practical significance, selection, and structural assumptions all affect interpretation. The authors prefer cross-validation, with WAIC as a convenient alternative.

  • Model comparison: Model comparison asks whether improved fit justifies added complexity and whether the prior on additional parameters is reasonable.Larger nested models may fit better but be harder to understand and compute.
  • Model comparison: Predictive comparisons must correct for overfitting, especially when candidate models differ greatly in complexity.This correction is central to the value of AIC, DIC, WAIC, and related methods.
  • Interpreting comparisons: Statistical significance reflects uncertainty in comparative prediction accuracy, whereas practical significance depends on the model’s intended use and domain-specific interpretation.Differences in log predictive probability may be difficult to interpret without a familiar application-specific scoring scale.
  • Interpreting comparisons: Average out-of-sample prediction error can miss improvements that are small in magnitude and concentrated in only a few population subsets.A larger model may give more reasonable predictions without appearing dominant in predictive comparisons.
  • Limitations: Selection among many candidate models biases the selected model’s predictive-performance estimate and can strongly overfit when the model set grows rapidly with sample size.An additional level of cross-validation can estimate the selection-induced bias but does not prevent the underlying selection problem.
  • Limitations: AIC, DIC, and WAIC each fail in some settings, while cross-validation is appealing but computationally expensive and prediction tasks can be difficult for structured data.The cited examples include strong prior information, posterior distributions poorly summarized by their means, and spatial or network data partitions.
  • Conclusion: The authors prefer cross-validation and regard WAIC as a fast, computationally convenient, fully Bayesian alternative with an explicit connection to cross-validation.They report that WAIC gives reasonable results in the examples considered.
Loading 1307.5928v1…