PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparison of Bayesian predictive methods for model selection

Juho Piironen, Aki Vehtari

arXiv:1503.08650v4stat.MEcs.LG

TL;DR

The paper compares Bayesian model-selection methods for variable subset selection in regression and classification, focusing on predictive performance and selection-induced overfitting. Across experiments, model averaging generally performs best, while projection robustly simplifies complex encompassing models and cross-validation outside the search helps choose model size and assess performance.

  • Problem

    When datasets are scarce and many candidate models are compared, high variance in LOO-CV and WAIC estimates can cause overfitted selection and biased performance estimates.

  • Method

    The paper compares Bayesian predictive methods using simulated and real-world regression and classification data, including model averaging, projection, cross-validation, and other selection criteria.

  • Results

    Bayesian model averaging generally gives the best predictive results, while projection robustly simplifies the full model and outperforms Median and MAP selection.

  • Takeaways & Limitations

    Use the full encompassing model when feasible, projection when simplification is needed, and cross-validation outside the search to guide model size and evaluate the final model.

  • Takeaways & Limitations

    Relative explanatory power is generally unreliable for predicting submodel performance because the reference model may differ from the true data-generating model.

Abstract

from arXiv · show

The goal of this paper is to compare several widely used Bayesian model selection methods in practical model selection problems, highlight their differences and give recommendations about the preferred approaches. We focus on the variable subset selection for regression and classification and perform several numerical experiments using both simulated and real world data. The results show that the optimization of a utility estimate such as the cross-validation (CV) score is liable to finding overfitted models due to relatively high variance in the utility estimates when the data is scarce. This can also lead to substantial selection induced bias and optimism in the performance evaluation for the selected model. From a predictive viewpoint, best results are obtained by accounting for model uncertainty by forming the full encompassing model, such as the Bayesian model averaging solution over the candidate models. If the encompassing model is too complex, it can be robustly simplified by the projection method, in which the information of the full model is projected onto the submodels. This approach is substantially less prone to overfitting than selection based on CV-score. Overall, the projection method appears to outperform also the maximum a posteriori model and the selection of the most probable variables. The study also demonstrates that the model selection can greatly benefit from using cross-validation outside the searching process both for guiding the model size selection and assessing the predictive performance of the finally selected model.

1. Introduction

The paper quantitatively compares Bayesian predictive model-selection methods in regression and classification, focusing on practical variable-subset selection. It finds that direct optimization of noisy predictive estimates can overfit, whereas model-uncertainty approaches and cross-validation used outside the search are more robust.

  • Study scope: The study compares Bayesian model-selection methods quantitatively in practical regression and classification variable-subset problems.The experiments use simulated and real-world data, while the discussion is intended to apply more broadly.
  • Selection risk: High variance in predictive-utility estimates can make CV or WAIC select overfitted, nonoptimal models and bias their evaluated performance.The risk increases when many candidate models are compared, as in variable selection.
  • Main findings: BMA over candidate models generally gives the best predictive results by accounting for model uncertainty rather than selecting one model.The paper reports that one should not generally expect selection to outperform the full BMA solution.
  • Main findings: The projection method robustly simplifies an encompassing model when its complexity or variable-observation cost is too high.Projection is reported as less vulnerable to selection overfitting and to outperforming MAP and most-probable-variable selection.
  • Practical recommendation: Cross-validation used outside variable searching can guide model size and assess final predictive performance with substantially less overfitting.Selecting only the number of variables reduces the number of models conditioned on during selection.

2. Approaches for Bayesian model selection

The paper organizes Bayesian model-selection approaches around predictive utility and three views of the model space: M-closed, M-completed, and M-open. These views differ in whether a candidate model is treated as true and whether a reference model is constructed.

  • Overview: The section reviews Bayesian model-selection methods relevant to the study rather than providing a comprehensive literature review.It directs readers to Vehtari and Ojanen (2012) for a fuller discussion.
  • M-closed: M-closed assumes that one candidate model is the true data-generating model and permits prior model probabilities followed by Bayesian model averaging.This categorization is presented as part of the paper’s organizing framework.
  • M-open: M-open rejects both the assumption that one candidate model is true and the construction of a reference model.The supplied passage contrasts this view with the reference-model approach.
  • Notation: The notation describes models predicting output y from input x, with θ denoting parameters and D denoting training data.Future observations are denoted by ỹ.

2.1 Predictive ability as an expected utility

The paper evaluates predictive ability through an expected utility because future observations are unknown. It uses the logarithmic score as its main utility, while noting that other loss functions may be appropriate for particular applications.

  • Expected utility: Predictive performance is defined using a utility function that measures the quality of a model’s predictive distribution.The paper uses expected utilities because future observations cannot be evaluated in advance.
  • Logarithmic score: The logarithmic score is the paper’s primary utility because of its information-theoretic basis and broad acceptance.The authors state that other utility functions could also be used.
  • Utility choice: The choice of utility function may be application specific, even though this paper uses the logarithmic score.The paper leaves future input variables out of the notation for simplicity.
  • Generalization utility: The generalization utility is the expected logarithmic predictive score under the true data-generating distribution.Maximizing it is equivalent to minimizing the KL divergence from the true distribution to the candidate predictive distribution.

2.2 Generalization utility estimation

The paper estimates generalization utility with held-out validation or information criteria, emphasizing the bias–variance and computational trade-offs of these methods. Bayesian LOO-CV and WAIC receive theoretical support, while DIC relies on a posterior-mean parameter fit.

  • Cross-validation: Using training data to estimate its own predictive utility is optimistic, so K-fold CV trains on K−1 folds and validates on the held-out fold.The validation sets are used in turn to estimate predictive performance.
  • Cross-validation: K-fold CV has bias from training on fewer observations, while small K increases variance; K=10 is a common compromise when LOO is costly.LOO requires fitting the model n times without computational shortcuts.
  • WAIC: WAIC is a Bayesian information criterion that asymptotically matches Bayesian LOO-CV and generalization utility with error o(1/n).The result includes parameter uncertainty and applies to singular models.
  • DIC: DIC estimates generalization performance with parameters fixed at the posterior mean and is not theoretically justified as an unbiased generalization-utility estimate.The point-estimate treatment is especially questionable for singular models.

2.3 Mixed self and posterior predictive criteria

Mixed self and posterior predictive criteria assess models using replicated measurements, combining fit and predictive uncertainty in different ways. Their variance and tuning choices affect model-selection reliability.

  • The L2-criterion measures squared errors for mean predictions plus predictive variances at the training inputs.
  • The cross-validated L2-criterion avoids using the same data for training and testing but has relatively high variance.This variance may cause significant overfitting in model selection.
  • The Gelfand–Ghosh criterion balances closeness to observed and future data through a free parameter k.
  • When k →∞, the Gelfand–Ghosh criterion equals L2, whereas k = 0 reduces it to the sum of predictive variances.At k = 0, the narrowest predictive distribution is selected without an inherent safeguard against poor fit.

2.4 Reference model approach

The reference model approach evaluates candidate models relative to an encompassing model believed to represent uncertainty about future observations. Projection preserves reference-model predictions while simplifying the model, but reference construction and discrepancy thresholds impose practical limits.

  • 2.4 Reference model approach: The reference model approach replaces the unknown data-generating distribution with the predictive distribution of an encompassing reference model.
  • 2.4.1 Reference predictive method: Reference-utility maximization minimizes predictive KL-divergence between the reference and candidate models at the training inputs.
  • 2.4.1 Reference predictive method: The reference predictive approach requires constructing a reference model, and how to construct it is not obvious.
  • 2.4.2 Projection predictive method: Projection determines candidate-model parameters from the reference-model fit so the candidate predictive distribution remains as close as possible to the reference.
  • 2.4.2 Projection predictive method: The relative explanatory power ranges from 0 to 1, but it is generally an unreliable indicator of submodel predictive performance.This occurs because the reference model can differ from the true data-generating model.
  • 2.4.2 Projection predictive method: Cross-validation outside the searching process repeats model search and reference-model fitting across folds to guide final model-size selection.The approach increases computation because both procedures are repeated K times.

2.5 Model space approach

The model-space approach represents uncertainty over an exhaustive set of candidate models and can either average their predictions or select a single model. Single-model approaches rely on stronger conditions or different decision objectives than BMA.

  • Bayesian model averaging obtains predictions by averaging over the posterior distribution of the candidate models.
  • The MAP model maximizes posterior model probability and, under equal model priors, reduces to maximizing marginal likelihood.
  • The Median model includes variables whose marginal posterior inclusion probabilities exceed 1/2.
  • Under orthogonal predictors, the Median model is optimal for squared-error closeness of mean predictions to the BMA prediction.
  • The Median-model theory assumes Gaussian noise and does not apply, for instance, to classification problems.
  • Figure 1 contrasts utility estimates across datasets, showing how bias and variance shape the models selected by maximizing estimated utility.The caption labels the left method unbiased and the right method biased; the accompanying text says lower variance can favor better models.

3. Overfitting and selection induced bias

Utility estimates can be unbiased for individual models yet still produce overfitted selections when their variance is high. For model selection, ranking competing models with low variability can matter more than unbiasedness.

  • A utility estimate decomposes into a model’s true generalization utility and estimation error.The paper defines g_l = u_l + e_l for each candidate model.
  • Unbiased utility estimates are not required for perfect model selection if higher estimates consistently imply higher generalization performance.
  • High variance can make maximizing an unbiased estimate select a model with nonoptimal expected true utility.The selection procedure fits noise in the utility estimates, producing overfitting and selection-induced optimism.
  • Lower variance can increase the probability of selecting a better-performing model even when estimates are biased.For selection, approximately correct ranking with low variability is more important than unbiasedness.
  • Cross-validation estimates predictive utility nearly without bias, but its variance can still contribute to overfitting during model selection.

4. Numerical experiments

Across simulated and real datasets, predictive model averaging generally outperformed selection, while projection produced simpler submodels that stayed close to the BMA. Utility-based searching, especially with scarce data, was vulnerable to overfitting and selection-induced optimism.

  • Predictive performance: The BMA generally achieved better predictive performance than every selected model, making simplification—not improvement over model averaging—the primary rationale for selection.This pattern held across simulated and real-data experiments.
  • Selection-induced bias: High variance in utility estimates caused CV-based searching to select overfitted models, with CV utility substantially exceeding independent-test utility.The overfitting effect remained visible even at n = 400 and was similar for WAIC, DIC, L2, L2-CV, and L2-k.
  • Reference-model methods: Reference-model methods, especially BMA-proj, selected smaller models with predictive ability close to the BMA and substantially lower variability than CV, WAIC, DIC, and MPP.For the smallest simulated datasets, BMA-proj reached near-BMA predictive ability with about 10–15 variables on average.
  • Projection: Projection improved submodel performance whether variables were ordered by marginal posterior probabilities or by forward search against the BMA.The projected parameterization was compared with standard fitting to the data.
  • Real-world datasets: On real datasets, BMA-proj showed the most robust performance, producing models with predictive ability close to the BMA across all datasets.CV, WAIC, DIC, L2, L2-CV, and L2-k were especially prone to overfitting when datasets were small relative to the number of variables.

5. Conclusions

The experiments show that predictive model selection is vulnerable to overfitting, especially with small datasets and many candidate variables. Bayesian model averaging generally performs best, while projection robustly simplifies the reference model and external cross-validation helps choose model size.

  • Overfitting is especially problematic when datasets are small and many models are compared, because utility estimates have high variance.
  • Bayesian model averaging over candidate models often achieves the best expected predictive performance by accounting for model uncertainty.
  • Projection produces simpler models that retain the full model’s predictive ability while being less vulnerable to selection overfitting and less variable across data realizations.
  • Projection also outperforms median and MAP-style selection in the experiments and can improve prediction even when another method ranks variables.
  • The reference-model approach requires forming a suitable full model, for which no automated construction method is available.
  • Cross-validation outside variable searching can evaluate the size–performance tradeoff and avoid substantial selection bias when selecting among a reduced set of models.

A. Projection for the linear Gaussian model

For linear Gaussian regression, projection derives submodel parameters from posterior samples of a full reference model. It then searches for sparse submodels that minimize discrepancy while preserving predictive distributions similar to the reference model.

  • Projection starts from posterior samples of a full linear Gaussian model and analytically calculates the corresponding projected submodel parameters.
  • The predictor matrix is partitioned so the projection retains only the columns corresponding to the chosen submodel.
  • Projected weights use a maximum-likelihood solution whose response is replaced by the full model’s fitted values.
  • Projected noise variance combines the full model’s noise level with the mismatch between reference and projected predictions.
  • For each posterior sample, the method computes projected parameters and KL divergences, then estimates the discrepancy between full and submodels.
  • Searching variable combinations by discrepancy yields parsimonious models with zero weights for excluded variables and predictive distributions similar to the full model.

B. Projection with hierarchical shrinkage prior

The projection approach is not restricted to the spike-and-slab prior used in the main Bayesian model averaging experiments. It can also operate with hierarchical shrinkage priors.

  • Projection can be used with priors other than the spike-and-slab prior, including hierarchical shrinkage priors.

B.1 Hierarchical shrinkage prior

The hierarchical shrinkage prior assigns local variance parameters to individual weights and a global variance term to overall sparsity. Heavy tails allow important weights to remain weakly shrunk.

  • A hierarchical shrinkage prior assigns local variance parameters to individual regression weights and a global variance term controlling overall sparsity.
  • Local variance parameters are expected to be large for influential inputs and small for inputs with negligible weights.
  • The half-Student-t construction concentrates probability near zero while sufficiently heavy tails allow some weights to remain unshrunk.

B.2 Crime dataset revisited

The Crime dataset experiment applies projection predictive variable selection to a hierarchical-shrinkage full model, then evaluates projected submodels by variable count using cross-validation and held-out test data.

  • The Crime dataset contains 1992 observations and 102 predictor variables, split into 1000 training and 992 test points.
  • The full regression model uses a hierarchical-shrinkage prior for nonconstant-input weights and is fitted with 4000 posterior samples from Stan.Sampling uses four chains with 2000 samples per chain, discarding the first half as warmup.
  • Forward projection predictive selection starts from the empty model and adds the variable that most reduces discrepancy from the full model at each step.
  • The experiment compares 10-fold cross-validated searches within training data against searching on all training data followed by evaluation on the held-out test set.Figure 12 presents the cross-validation average in black and the test-data result in green; grey bars show 95% credible intervals.
  • Projected submodels are evaluated by mean log predictive density and mean squared error as functions of the number of selected variables, up to 50.
Loading 1503.08650v4…