PaperlayerView exact revision on arXiv

Source-linked AI summary

A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article)

Jianqing Fan, Jinchi Lv

arXiv:0910.1122v1math.ST

TL;DR

High dimensional variable selection must address computational limits, difficult dimensionality regimes, and the need to identify important variables while estimating their effects. The paper reviews penalized-likelihood methods and screening techniques, emphasizing nonconcave penalties and their computation. It synthesizes results on sparsity, bias reduction, dimensionality, and the contrasting selection behavior of LASSO and folded-concave penalties.

  • Problem

    High dimensional settings make variable selection computationally difficult and raise unresolved questions about the dimensionality limits and optimality of selection procedures.

  • Method

    The article reviews penalized-likelihood theory, methods, and implementations, emphasizing nonconcave penalties, iterative reweighting, and ultra-high-dimensional screening.

  • Results

    The review reports that folded-concave penalties can have more relaxed model-selection conditions than LASSO, while screening methods can address ultra-high dimensionality.

  • Takeaways & Limitations

    Penalty choice determines the balance among sparsity, estimation bias, computational tractability, and the dimensionality that variable-selection procedures can handle.

  • Takeaways & Limitations

    The field remains incomplete, with open issues including optimality, data-driven penalty selection, confidence in selected models, and inference after model selection.

Abstract

from arXiv · show

High dimensional statistical problems arise from diverse fields of scientific research and technological development. Variable selection plays a pivotal role in contemporary statistical learning and scientific discoveries. The traditional idea of best subset selection methods, which can be regarded as a specific form of penalized likelihood, is computationally too expensive for many modern statistical applications. Other forms of penalized likelihood methods have been successfully developed over the last decade to cope with high dimensionality. They have been widely applied for simultaneously selecting important variables and estimating their effects in high dimensional statistical inference. In this article, we present a brief account of the recent developments of theory, methods, and implementations for high dimensional variable selection. What limits of the dimensionality such methods can handle, what the role of penalty functions is, and what the statistical properties are rapidly drive the advances of the field. The properties of non-concave penalized likelihood and its roles in high dimensional statistical modeling are emphasized. We also review some recent advances in ultra-high dimensional variable selection, with emphasis on independence screening and two-scale methods.

1 Introduction

High dimensionality creates statistical, interpretability, computational, and model-selection challenges, especially through spurious collinearity and noise accumulation. The article frames sparsity-based variable selection and penalized likelihood, alongside screening, as responses to these challenges.

  • 1 Introduction: High dimensionality makes inference, theory, interpretability, and computational efficiency difficult when p is comparable to or larger than n.These concerns motivate methods that balance statistical accuracy, model interpretability, and computational complexity.
  • 1 Introduction: Spurious collinearity can make irrelevant variables approximate or replace true variables, causing over-fitting and model mis-identification.In high dimensions, even independent Gaussian predictors can exhibit large sample or multiple correlations.
  • 1 Introduction: Noise accumulation can impair high dimensional prediction and classification as dimensionality increases.The cited literature develops explicit characterizations of dimensionality’s impact on regression and misclassification rates.
  • 1 Introduction: Sparsity assumes that many regression coefficients are zero, enabling variable selection to improve estimation accuracy, interpretability, and sometimes computational cost.The important variables are represented by the nonzero coefficients of a low-dimensional structure.
  • 1 Introduction: The article reviews penalized likelihood for high dimensional modeling and iterative screening methods for ultra-high dimensional feature spaces.Its scope includes theory, methods, implementations, and techniques that combine screening with later variable selection.

2 Classical model selection

Classical model-selection criteria can be expressed within a penalized-likelihood framework, but best subset selection leads to a combinatorial optimization problem. Different penalties and regularization parameters recover criteria such as AIC, BIC, Mallows’ Cp, adjusted R2, GCV, and cross-validation.

  • 2 Classical model selection: Model selection estimates model dimension and compares candidate models when the true density is embedded in a larger parametric family.This formulation applies beyond regression, including applications such as time-series modeling.
  • 2 Classical model selection: AIC and BIC correspond to penalized likelihood with λ = 1 and λ = (log n)/2, respectively.These criteria arise from different interpretations of model fit, including estimated Kullback–Leibler divergence and Bayesian model selection.
  • 2 Classical model selection: Penalized likelihood unifies classical selection by maximizing fit while penalizing model dimension or the number of nonzero parameters.The L0 norm counts nonvanishing components, so fixing model size yields best subset selection.
  • 2 Classical model selection: Best subset selection is computationally intractable in high dimensions because penalized L0 optimization is combinatorial and has NP-complexity.This computational limitation motivates alternative penalty functions.
  • 2 Classical model selection: Mallows’ Cp corresponds to λ = 1, while adjusted R2 is approximately equivalent to penalized maximum likelihood with λ = 1/2.Generalized cross-validation, cross-validation, and RIC are also presented as regularization-based criteria.

3 Penalized likelihood

Penalized likelihood extends model selection with computationally tractable penalties that simultaneously select variables and estimate effects. The review emphasizes folded-concave penalties, their shrinkage and computation properties, and the contrasting selection limitations of LASSO in high dimensions.

  • 3 Penalized likelihood: Penalized likelihood simultaneously estimates regression coefficients and deletes variables whose estimated coefficients are zero.Alternative penalties are introduced because penalized L0 selection is computationally infeasible in high dimensions.
  • 3.1 Canonical regression model: Penalty design targets sparsity, reduced bias for large coefficients, and continuity of the resulting estimator.These properties motivate nondecreasing penalties with a singularity at zero and concavity away from zero.
  • 3.2 Penalty function: L0 and hard-thresholding penalties produce hard thresholding, L1 produces soft thresholding, and SCAD and MCP yield analytical linear-spline solutions.Each increasing penalty generates a shrinkage rule with |β̂(z)| ≤ |z|.
  • 3 Penalized likelihood: LASSO’s model-selection consistency requires stringent conditions, whereas folded-concave penalties such as SCAD have a more relaxed intrinsic condition.The review also notes that LASSO’s bias can force smaller λ values, producing more false-positive variables.
  • 3.3 Computation and implementation: LQA has quadratic convergence, while LLA and LQA generate nondecreasing target values through minorization-maximization updates.The approximations convert nonconvex optimization into sequences of tractable surrogate problems.
  • 3.3 Computation and implementation: Nonconcave penalized likelihood can be solved by iteratively reweighted penalized L1 regression, with adaptive weights that reduce penalization bias for large estimated components.SCAD and MCP can assign no penalty to components estimated with sufficiently large magnitude.

4 Ultra-high dimensional variable selection

Ultra-high dimensional variable selection uses screening to reduce a huge feature space before applying moderate-scale selection. The reviewed results describe when screening retains important variables, how classification benefits from feature selection, and where screening assumptions can fail.

  • Two-scale variable selection: Sure screening first reduces a large feature space to a moderate submodel, enabling established variable-selection methods to operate with lower computational cost.The framework combines large-scale screening with moderate-scale penalized selection.
  • Sure independence screening: Correlation learning ranks features by marginal sample correlation and can reduce a model of size p ≫ n to a submodel smaller than n.For binary classification, the ranking reduces to selecting features using two-sample t-test statistics.
  • Sure independence screening: Under regularity conditions including 2κ + τ < 1, SIS retains the true sparse model with overwhelming probability when the selected size satisfies d_n ∼ n^θ.Here θ lies between 2κ + τ and 1, and the retained model must contain at least s active variables.
  • Feature selection for classification: Feature selection can improve high-dimensional classification because accumulated estimation noise may overwhelm signal when many features are used.The reviewed results report that using only informative features can outperform using all p features, while FAIR is described as having an oracle property.
  • Feature selection for classification: Two-sample t screening can select all important features with probability tending to 1 under regularity conditions, but selecting every detectable feature is not necessarily optimal.FAIR addresses this by selecting the most important features before constructing the independence rule.
  • Limitations: SIS uses only marginal covariate information, so its sure screening property can fail when technical conditions are violated or important predictors are marginally uncorrelated but jointly correlated with the response.This creates a false-negative limitation for marginal screening.
  • Generalized linear models: For generalized linear models, sure screening can converge exponentially fast and accommodate dimensionality as high as log p = o(n^(1−2κ)) for logistic regression with bounded covariates.The stated dimensionality rate depends on the model and covariate assumptions.

5 Sampling properties of penalized least squares

The reviewed results characterize sampling behavior, estimator equivalence, model-selection conditions, and oracle properties for penalized least-squares methods. They emphasize both the strengths of folded-concave penalties such as SCAD and the restrictive conditions required by LASSO.

  • 5.1 Dantzig selector and its asymptotic equivalence to LASSO: Under restricted eigenvalue conditions, Dantzig selector and LASSO prediction losses achieve sparsity oracle inequalities.The bounds include the prediction error rate for s parameters and additional factors reflecting the number of regressors.
  • 5.1 Dantzig selector and its asymptotic equivalence to LASSO: The Dantzig selector and LASSO can be identical under suitable conditions, including conditions that allow p to exceed n.The Dantzig constraint uses the negative gradient of the penalized least-squares loss, and equivalence depends on the design and common regularization parameter.
  • 5.2 Model selection consistency of LASSO: LASSO sign consistency requires restrictive irrepresentable conditions, and violating related conditions can produce inconsistent models with many false positives.The strong irrepresentable condition is sufficient for sign consistency, while the corresponding necessary condition can be difficult to satisfy in high dimensions.
  • 5.4 Additional properties of SCAD estimator: For SCAD, suitable tuning can make the penalized estimator asymptotically as efficient as the oracle estimator and, in stronger results, equal to it.The reviewed results also give conditions for uniqueness and global optimality, including settings where the minimum eigenvalue of the full correlation matrix tends to zero.
  • 5.5 Sparse recovery and compressed sensing: Concave penalties are advantageous for noiseless sparse recovery, while the computational difficulty of L0 motivates tractable penalty relaxations such as L1.The sparse-recovery connection provides a simplified view of penalty roles in high-dimensional variable selection as noise approaches zero.

6 Oracle property of penalized likelihood with ultra-high dimensionality

The review examines weak and classical oracle properties of penalized likelihood estimators in ultra-high dimensions, emphasizing how dimensionality, sparsity, signal strength, and penalty concavity shape variable-selection guarantees. Non-concave penalties such as SCAD can support broader conditions and stronger oracle behavior than the L1 penalty under suitable assumptions.

  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: Weak oracle properties combine uniform consistency with model-selection consistency, requiring estimated coefficients outside the true support to vanish with probability tending to one.This property is weaker than the oracle property and is derived under L∞ loss.
  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: The considered penalty class is increasing and concave with a continuous derivative, including SCAD and MCP, while L1 appears at the class boundary.The derivative at zero is positive, and the derivative increases with the regularization parameter.
  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: For L1, model-selection consistency is limited by an intrinsic upper-bound condition, whereas SCAD-type penalties impose no such limit when signals are sufficiently strong.For folded-concave penalties, the intrinsic condition becomes more relaxed when the minimal signal dominates the regularization level.
  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: The dimensionality p may grow at a non-polynomial rate under non-concave penalized likelihood, while weaker minimal signals require slower dimensionality growth for successful selection.The reviewed results also note that concave penalties can reduce estimation bias.
  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: Under regularity conditions, concave penalized likelihood estimators can achieve sparsity, the rate OP(√(s)n^-1/2), asymptotic normality, and the oracle property.The conditions are less restrictive for concave penalties such as SCAD; the L1 estimator generally cannot achieve the same consistent rate when p diverges with n.
  • 6 Oracle property of penalized likelihood with ultra-high dimensionality: SCAD penalized likelihood can be identical to the oracle estimator under regularity conditions, a feature not shared by the L1 penalty.The review characterizes this through global and restricted global optimality.

7 Concluding remarks

The review concludes that high-dimensional variable selection is better understood, but its theoretical and practical development remains incomplete. It calls for new techniques and critical analyses of optimality, data-driven penalty choices, and confidence in selected models and estimated parameters.

  • 7 Concluding remarks: High-dimensional statistical learning still requires innovative techniques and critical comparisons of the relative merits of existing methods.The review identifies optimality characterization, data-driven penalty and parameter selection, and confidence in selected models as continuing issues.
  • 7 Concluding remarks: Open problems include selecting data-driven penalty functions and parameters and establishing confidence in selected models and estimated parameters.
Loading 0910.1122v1…