PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Selection Techniques -- An Overview

Jie Ding, Vahid Tarokh, Yuhong Yang

arXiv:1810.09583v1stat.MLcs.ITcs.LGecon.EMphysics.app-ph

TL;DR

Model selection must choose among candidate models while balancing reliable inference against predictive performance. This overview integrates theoretical and practical perspectives on methods including penalized regression and adaptive criteria, illustrating how their behavior depends on the modeling framework, sample size, and analytical objective.

  • Problem

    Selecting among candidate models is difficult because prediction-error estimation and procedure comparison are different targets, while poor selection can produce misleading conclusions or disappointing predictions.

  • Method

    The paper synthesizes model-selection philosophies, asymptotic properties, practical applicability, penalized regression methods, and criteria such as AIC, BIC, and Bridge criterion.

  • Results

    Bridge criterion performs like AIC in practically nonparametric cases and much better than BIC, while BIC and Bridge criterion perform much better than AIC in the parametric case.

  • Takeaways & Limitations

    Model-selection choices should follow the primary objective and setting, because inference favors reliable selection whereas prediction can tolerate instability among similarly performing models.

  • Takeaways & Limitations

    No model-selection method can guarantee consistency in nonparametric regression settings.

Abstract

from arXiv · show

In the era of big data, analysts usually explore various statistical models or machine learning methods for observed data in order to facilitate scientific discoveries or gain predictive power. Whatever data and fitting procedures are employed, a crucial step is to select the most appropriate model or method from a set of candidates. Model selection is a key ingredient in data analysis for reliable and reproducible statistical inference or prediction, and thus central to scientific studies in fields such as ecology, economics, engineering, finance, political science, biology, and epidemiology. There has been a long history of model selection techniques that arise from researches in statistics, information theory, and signal processing. A considerable number of methods have been proposed, following different philosophies and exhibiting varying performances. The purpose of this article is to bring a comprehensive overview of them, in terms of their motivation, large sample performance, and applicability. We provide integrated and practically relevant discussions on theoretical properties of state-of- the-art model selection approaches. We also share our thoughts on some controversial views on the practice of model selection.

I. WHY MODEL SELECTION

Model selection chooses an appropriate model or method from candidates because unsuitable choices can produce noisy discoveries, misleading conclusions, or poor predictions. This section defines the model-selection workflow and distinguishes candidate-model fit from predictive assessment.

  • Motivation and basic setup: Model selection chooses a statistical model from a model class given data.A model class is a collection of candidate models, each with its own parameter space and dimension.
  • Motivation and basic setup: Improper model choices can yield noisy discoveries, misleading conclusions, or disappointing predictive performance.
  • Motivation and basic setup: A typical analysis fits every candidate model to the observed data, then selects one for interpretation or prediction.
  • Motivation and basic setup: Model fitting estimates parameters by minimizing a cumulative loss, with logarithmic loss producing maximum likelihood estimation in parametric models.
  • Motivation and basic setup: The best predictive model is the candidate with the smallest out-sample prediction loss, evaluated using an independent observation and a specified loss function.What counts as best depends on the available data, candidate class, and loss function.
  • Motivation and basic setup: In parametric settings, consistency seeks the true candidate model, whereas in nonparametric settings the best model generally depends on sample size.High dimensionality can make a true candidate too difficult to identify from relatively small data, creating a practically nonparametric setting.

B. Goals of data analysis and model selection

Model selection serves two distinct goals: inference, which seeks a reliable and interpretable model, and prediction, which seeks strong future-data performance. These goals can align in parametric settings but may diverge when predictive performance is judged from a minimax perspective.

  • Inference and prediction: Prediction aims to choose a model with quantitatively strong performance on future observations.
  • Inference and prediction: Inference aims to identify a model that reliably characterizes uncertainty for scientific understanding and interpretation.
  • Inference and prediction: A predictive winner may be unstable among close competitors yet still perform nearly optimally, whereas using it for interpretation can be unreliable and misleading.
  • Inference and prediction: For inference, selection consistency means choosing the best model with probability approaching one as sample size increases.In variable selection, this targets identifying important variables while excluding the rest.
  • Inference and prediction: Asymptotic efficiency requires the selected model’s adjusted prediction loss to approach the smallest loss among candidates.
  • Inference and prediction: Inference and prediction can align under parametric asymptotic efficiency, but a minimax assessment can create an unbridgeable conflict between them.

C. An illustration on fitting and the best model

Illustrations show that better in-sample fit does not guarantee better prediction, and that the predictive optimum depends on the data-generating setting and sample size. The section uses autoregressive candidates to contrast parametric, nonparametric, and practically nonparametric regimes.

  • C. An illustration on fitting and the best model: Better in-sample fitting does not imply better predictive performance.The experiment compares in-sample loss with out-sample prediction loss across candidate autoregressive models.
  • C. An illustration on fitting and the best model: In the autoregressive example, analysts estimate parameters for orders 1 through d_n and then select an order using a model-selection principle.
  • C. An illustration on fitting and the best model: The parametric experiment finds predictive performance optimal at the true autoregressive order.
  • C. An illustration on fitting and the best model: Subtracting σ^2 makes the efficiency ratio distinguish AR(2), which approaches 1, from AR(3), which approaches 2/3.
  • C. An illustration on fitting and the best model: In the nonparametric experiment, the optimal model dimension increases with sample size, making a fixed good model unrealistic for inference.
  • C. An illustration on fitting and the best model: When sample size is small relative to true model dimension, fitting too many parameters increases variance and can make the best candidate differ from the true model.Thus, a parametric framework can behave practically nonparametrically in the small-data regime.

II. PRINCIPLES AND APPROACHES FROM VARIOUS PHILOSOPHIES OR MOTIVATIONS

This section surveys likelihood-based information criteria and their theoretical motivations, including how penalty choices relate to parametric and nonparametric model-selection goals. It also contrasts predictive performance across parametric, nonparametric, and practically nonparametric settings.

  • Information criteria select among likelihood-based parametric models by balancing fit against model complexity.Their penalties increase with model dimension, discouraging more complex candidates.
  • AIC approximates out-sample prediction loss with in-sample loss plus a correction term and uses a penalty proportional to 2k.AICc modifies AIC for small samples, while differences between them are limited unless sample size is small relative to model dimension.
  • Synthetic examples show that predictive performance is optimized at the true order in parametric settings but may depend on sample size in nonparametric settings.In practically nonparametric small-data regimes, including the true model does not ensure optimal prediction because estimating too many parameters increases variance.
  • BIC replaces AIC’s constant penalty with log n and is asymptotically equivalent to selecting the model with the largest marginal likelihood.This connection follows from the marginal likelihood concentrating near the maximum-likelihood estimate with volume of order n^-d/2.
  • HQ achieves strong consistency for autoregressive order selection when the data-generating model has a fixed true order.The selected order converges almost surely to the true order as sample size increases.
  • The Bridge criterion is designed to combine AIC- and BIC-like behavior, performing similarly to AIC nonparametrically and BIC parametrically.Its penalty is approximately c_n log d_m, with the suggested c_n = n^2/3.

B. Methods from other perspectives

This section reviews Bayesian, information-theoretic, and decision-theoretic alternatives to likelihood-based criteria. These methods select models through posterior evidence, coding or compression principles, predictive sequencing, or Bayesian complexity measures.

  • Bayesian model selection can use posterior model probabilities, maximum a posteriori selection, Bayes factors, or marginal likelihoods.With equal prior probabilities on models, the largest marginal likelihood also gives the largest posterior probability.
  • Marginal likelihood favors models through Bayes-factor comparisons and becomes asymptotically equivalent to minimizing BIC.Bayes factors remove prior probabilities on models from the comparison, while parameter priors can still affect non-asymptotic interpretation.
  • MML selects the model yielding the shortest combined message for the model and data, using likelihood, prior, Fisher information, and parameter dimension.Its formulation includes a quantizing lattice constant κ_d, commonly approximated by κ_1 = 1/12.
  • MDL defines the best model through data compression and avoids prior-distribution assumptions, while predictive MDL uses sequential stochastic complexity.The sequential formulation estimates parameters from preceding observations before scoring the next observation.
  • DIC is a Bayesian counterpart of AIC that replaces the MLE and model dimension with the posterior mean and effective number of parameters.Its Bayesian computation can help compare complex models whose likelihoods may not have analytic forms.

C. Methods that do not require parametric assumptions

Cross-validation provides model-selection procedures that do not require parametric candidate models, provided data can be permuted and predictive performance can be assessed. Its variants differ in how data are split and in computational cost.

  • Cross-validation trains each candidate on a training subset, evaluates it on held-out data, and selects the model with the smallest average validation loss.Repeated splits reduce variability caused by a single partition.
  • Leave-one-out cross-validation predicts each observation using a model fitted to the other n−1 observations and averages the resulting losses.Under regularity conditions, LOO is asymptotically equivalent to AIC or TIC.
  • K-fold cross-validation partitions the data into k approximately equal subsets, trains on k−1 folds, and validates on the remaining fold in turn.It is more commonly used than LOO partly because LOO has large computational complexity.
  • The holdout method is a special case of cross-validation that performs data splitting only once.One part is used for training and the remainder for validation.

D. Methods proposed for specific types of applications

This section presents model-selection criteria designed for time-series and regression applications. They include sequential prediction-error criteria and generalized penalties that explicitly trade off fit against model complexity.

  • These criteria address applications including autoregressive order selection, stochastic regression, and linear regression model comparison.The section identifies time series and linear regression as the main application-specific contexts.
  • Predictive least squares selects a stochastic regression model by minimizing accumulated squared prediction errors in a time-series setting.Each prediction uses least-squares estimates based only on data observed before the prediction time.
  • PLS and BIC are asymptotically close and both strongly consistent for selecting the data-generating model in a parametric framework.Extensions with t_0/n → 1 have been shown under some conditions to share AIC’s asymptotic property.
  • Generalized information criteria use penalties linear in model dimension, with λ_n controlling the trade-off between regression fit and model complexity.Replacing the noise-variance estimator yields an equivalent criterion under mild conditions.
  • λ_n = 2 gives AIC, whereas λ_n = log n gives BIC within the generalized information-criterion framework.Mallows’ C_p is also described as a special case of GIC.

E. Theoretical properties of the model selection criteria

Theoretical analysis distinguishes model-selection criteria by whether they target prediction or inference and by whether the setting is parametric or nonparametric. AIC supports predictive optimality broadly, while BIC consistently identifies the smallest true model in parametric settings.

  • Selection consistency targets identifying the best model for scientific understanding or inference, whereas asymptotic efficiency and minimax-rate optimality target prediction.
  • The asymptotic divide between AIC- and BIC-like criteria is whether the penalty is fixed or diverges with sample size.
  • AIC is minimax-rate optimal for several variable-selection tasks, including subset and order selection in linear regression and series-based nonparametric regression.
  • AIC is asymptotically efficient in nonparametric settings, with selected-model prediction asymptotically equivalent to the best candidate-model performance.
  • BIC consistently selects the smallest true data-generating model in parametric settings, with the probability of selecting the correct model approaching one.

III. WAR AND PEACE—CONFLICTS BETWEEN AIC AND BIC, AND THEIR INTEGRATION

AIC and BIC embody different goals: AIC favors predictive performance in nonparametric settings, whereas BIC favors consistency and efficiency when the true model is parametric. The Bridge criterion adapts its penalty to seek both types of behavior across settings, while experiments support this distinction.

  • AIC is efficient and minimax-optimal for nonparametric prediction, whereas BIC is consistent and efficient for parametric model selection.
  • AIC can overselect in parametric settings, while BIC lacks minimax-rate optimality and asymptotic efficiency in nonparametric settings.
  • Consistency and minimax-rate optimality cannot generally be achieved simultaneously by one model-selection procedure.
  • The Bridge criterion applies a heavy BIC-like penalty to small models and relaxes it for larger models when evidence supports an infinitely dimensional truth.
  • In simulations, BIC and BC outperform AIC in the parametric case, while BC performs similarly to AIC and better than BIC in nonparametric cases; PI is near one versus closer to zero, respectively.The experiments use 100 replications with n = 500.
  • The AIC–BIC conflict reflects different objectives: minimizing prediction loss versus selecting a model for inference.

IV. HIGH-DIMENSIONAL VARIABLE SELECTION

High-dimensional variable selection addresses settings where the number of candidate variables is comparable to or larger than the sample size. The review focuses on well-specified linear models and methods for selecting sparse variable subsets.

  • High-dimensional regression allows the candidate-variable dimension d_n to be comparable with or even much larger than n.
  • The analysis commonly assumes a well-specified linear data-generating model in which one candidate variable subset represents the response.
  • The variable-selection problem is also called support recovery or feature selection across different literatures.
  • Methods mainly use penalized regression or iterative variable selection, incorporating structures such as sparsity, multicollinearity, and group behavior.

A. Penalized regression for variable selection

Penalized regression selects variables by combining fit with a penalty that encourages sparse coefficients, while different penalties encode different bias and structural properties. The overview contrasts LASSO, SCAD, MCP, grouped and adaptive penalties, and greedy alternatives.

  • Exhaustive subset search over 2^d_n candidate subsets is usually computationally prohibitive when the variable dimension is large.
  • Penalized regression solves a fit-plus-penalty problem and uses a nondifferentiable penalty at zero to produce sparse solutions.
  • LASSO uses an ℓ1 penalty whose tuning parameter controls shrinkage strength, with larger values selecting fewer variables.
  • SCAD reduces LASSO’s shrinkage bias by suppressing insignificant variables and tapering the penalty for larger coefficients, and it has the stated oracle property.
  • MCP is similar to SCAD and can attain minimax convergence rates in probability for regression-coefficient estimation under some conditions.
  • Elastic net combines LASSO and Ridge effects, while highly correlated covariates tend to receive similar regression coefficients.
  • Group LASSO uses predefined groups so that coefficients within each group are restricted to vanish together.
  • Greedy methods select variables through locally optimal iterations; OMP adds the variable most correlated with the current residual, whereas LARS can adjust previously selected variables.

B. Properties of the penalized regression methods

Penalized regression methods are studied for both prediction and variable-selection goals in high-dimensional settings. Their guarantees depend on assumptions, and methods that improve estimation bias may trade off against other properties.

  • Prediction: High-dimensional prediction targets a vanishing squared-loss bound even when the number of variables greatly exceeds the sample size.Under restricted eigenvalue assumptions, LASSO can achieve tight prediction bounds when log d_n is small relative to n.
  • Prediction: LASSO prediction guarantees permit d_n to be much larger than n, provided log d_n remains small compared with n.The tuning-parameter choice depends on the unknown product c_1σ, which can affect performance in small samples.
  • Variable selection: Selection consistency seeks to identify significant variables, relying on conditions such as sparsity, restricted isometry, mutual coherence, irrepresentability, and restricted eigenvalues.These conditions create theoretical possibilities for distinguishing the true variable subset from other subsets as n grows.
  • Estimation: LASSO is not generally consistent for coefficient estimation, motivating SCAD, MCP, and Adaptive LASSO, which correct its estimation bias and possess the oracle property.The oracle property is emphasized for simultaneous parameter estimation and variable selection in high-dimensional regression.
  • Practical performance: Cross-validation can yield good predictive performance for penalized regression, but different methods may perform very differently across datasets.Method choice therefore remains important even when tuning parameters are selected by proper cross-validation.
  • Inference: Selected variables from penalized regression methods can be severely unstable under tiny data perturbations, making inference based on them unreliable.When the true model is not a stable low-dimensional linear model, the paper recommends stability diagnostics, resampling, model averaging, or related tools.

V. MODELING PROCEDURE SELECTION

Modeling procedure selection uses cross-validation to choose among procedures, with goals of either identifying the best procedure or approaching its predictive performance. The appropriate training-validation split depends on that goal, and increasing both portions does not necessarily improve selection accuracy.

  • Scope and goals: Modeling procedure selection chooses among statistical or machine-learning procedures rather than only among parametric models.Examples include selecting among AIC, BIC, and cross-validation procedures or comparing problem-solving procedures in competitions.
  • Scope and goals: The two goals are selecting the best procedure with high probability and approaching the best out-of-sample prediction loss among candidates.Selection consistency serves the first goal, while predictive performance serves the second.
  • Cross-validation design: Cross-validation splits data into training and validation sets, with the required validation proportion depending on whether the aim is consistency or predictive performance.Consistency requires a sufficiently large evaluation portion, whereas optimal prediction may need less validation data.
  • Consistency: Cross-validation can consistently choose BIC over AIC in parametric settings and AIC otherwise when n_t→∞ and n_v/n_t→∞ under regularity conditions.The result concerns regression variable selection and distinguishes the parametric and nonparametric cases.
  • Cross-validation paradox: More observations in both training and validation do not necessarily improve the accuracy of identifying the better procedure.The cross-validation paradox arises because improved prediction-error estimation can make naturally similar procedures harder to distinguish.
  • Splitting-ratio recommendations: Common 5-fold, 10-fold, and 30%-validation practices lack asymptotic optimality, with optimal splitting depending on the framework and objective.For model selection, asymptotic efficiency may require n_v/n_t to converge to zero or diverge to infinity; procedure comparison may require a large validation share, making LOO least trustworthy.

B. Since all models are wrong, why pursing consistency in selection?

The paper argues that consistency remains a useful theoretical concept despite model misspecification, while acknowledging that nonparametric settings may not permit consistent selection. It also discusses trade-offs surrounding oracle properties and stresses that model selection is exploratory, not confirmatory.

  • Consistency under misspecification: Although all models may be wrong, a stable parametric model can still be treated as a useful theoretical “true model” for studying selection consistency.The paper presents this idealization as potentially having practical implications rather than dismissing consistency outright.
  • Consistency under misspecification: No model-selection method can guarantee consistency in nonparametric regression, helping explain why asymptotic efficiency is more widely used there.Asymptotic efficiency imposes a weaker requirement than selecting a true model.
  • Oracle property: The oracle property does not ensure uniformly good point or interval estimation, because an oracle estimator can have risk whose supremum diverges as n→∞.For consistent selection methods, parameters can be small enough to be omitted while still large enough that omission harms convergence rates.
  • Practical interpretation: Model selection is exploratory and cannot by itself support confirmatory conclusions, which require well-designed follow-up studies.The paper nevertheless regards model-selection tools as capable of providing valuable information for explanation and prediction.
  • Practical interpretation: The objective of selection should determine practice: inference requires consistency and diagnostics, whereas prediction places less demand on declaring the selected procedure best.In high-dimensional settings, penalized regression is described as typically highly uncertain for selection stability.
Loading 1810.09583v1…