PaperlayerView exact revision on arXiv

Source-linked AI summary

A Selective Review of Group Selection in High-Dimensional Models

Jian Huang, Patrick Breheny, Shuangge Ma

arXiv:1204.6491v2math.STstat.ME

TL;DR

High-dimensional models often contain naturally grouped predictors, but selecting groups differs from selecting meaningful variables within groups. This paper selectively reviews group and bi-level selection methods, emphasizing concave penalties, their theory, algorithms, and applications. It synthesizes established developments while identifying unresolved issues, including data-driven tuning and theoretical understanding of local solutions.

  • Problem

    High-dimensional variable selection must accommodate naturally grouped predictors and distinguish group-level from individual-level scientific selection goals.

  • Method

    The paper provides a selective review of group and bi-level selection methods, emphasizing concave penalties, methodological developments, theoretical properties, computational algorithms, and applications.

  • Results

    The review describes group LASSO and concave group selection methods, including threshold-operator behavior for group MCP and group SCAD in orthogonal groups.

  • Takeaways & Limitations

    Group-selection practice must account for whether within-group variables are scientifically meaningful, while the reviewed methods span multiple regression, genomic, and genetic applications.

  • Takeaways & Limitations

    The review identifies unresolved questions about data-driven penalty selection and the theoretical properties of local solutions for concave selection methods.

Abstract

from arXiv · show

Grouping structures arise naturally in many statistical modeling problems. Several methods have been proposed for variable selection that respect grouping structure in variables. Examples include the group LASSO and several concave group selection methods. In this article, we give a selective review of group selection concerning methodological developments, theoretical properties and computational algorithms. We pay particular attention to group selection methods involving concave penalties. We address both group selection and bi-level selection methods. We describe several applications of these methods in nonparametric additive models, semiparametric regression, seemingly unrelated regressions, genomic data analysis and genome wide association studies. We also highlight some issues that require further study.

1. INTRODUCTION

The review examines statistical methods that exploit naturally occurring variable groups, covering methodological, theoretical, computational, and application perspectives. It distinguishes group selection from bi-level selection according to whether individual variables within groups are scientifically meaningful.

  • Existing methods: Group LASSO extends LASSO by penalizing each group through the ℓ2 norm of its associated coefficients.The method and related algorithms were developed for selecting groups of variables as units.
  • Motivation: Grouping structures arise from categorical indicators, basis functions, biological pathways, and genetic markers, motivating methods that respect group organization.These structures may reflect modeling representation or scientifically meaningful prior knowledge.
  • Selection goals: Bi-level selection jointly selects important groups and individual variables when the variables within groups have scientific meaning.When individual variables are artificial, such as basis-function components, group selection alone is typically the relevant goal.
  • Selection goals: The distinction between group and bi-level selection affects both the statistical methods used and the derivation of theory and algorithms.Predictors can be made orthonormal in settings where bi-level selection is not required.
  • Review scope: The review covers group and bi-level selection methods, including concave penalties, across methodological developments, theoretical properties, computational algorithms, and applications.Applications include nonparametric additive models, semiparametric regression, seemingly unrelated regressions, genomic data analysis, and genome-wide association studies.

2. GROUP SELECTION METHODS

Group selection methods use penalties on group-level coefficient norms, with concave penalties offering thresholding behavior that differs from the group LASSO. The review covers their properties, solution paths, and coordinate-descent computation, while noting unresolved tuning and convergence issues.

  • Group LASSO: The group LASSO penalizes group coefficient norms and can be formulated with positive definite matrices that adjust for group sizes and predictor scaling.Using the group Gram matrix makes the penalty proportional to the fitted group contribution and corresponds to group-level standardization.
  • Theoretical properties: Group LASSO selection consistency depends on an irrepresentable condition that is difficult to satisfy, especially when p ≫ n.The review contrasts this selection limitation with the group LASSO's favorable prediction and ℓ2 estimation-error properties.
  • Open issues: The value of γ has a substantial effect on estimates, yet determining it remains an inadequately studied problem.The group MCP connects γ ≈ 2.7 to a commonly suggested choice and γ → ∞ to the group LASSO.
  • Solution paths: In a two-nonzero-group example, group MCP estimates can approach the true parameters over part of the solution path, whereas group LASSO estimates remain biased toward zero unless λ = 0.The example also shows that changing γ substantially alters the solution paths.
  • Concave group selection: Concave group penalties produce threshold operators ranging from multivariate hard thresholding at γ = 1 to soft thresholding as γ → ∞ for group MCP.Group SCAD instead approaches a different threshold operator at γ = 2 and excludes the hard-threshold operator from its family.
  • Computation via group coordinate descent: Closed-form single-group solutions provide the building blocks for group coordinate descent, whose steps are fast and guaranteed to decrease the objective function.The algorithm converges to a global minimum for group LASSO and may converge to a local minimum for group MCP and SCAD.

3. BI-LEVEL SELECTION

Bi-level selection methods select important groups and individual variables within them, addressing settings where group-level sparsity alone is insufficient. The review covers concave penalties, their theoretical properties, and a genetic association application.

  • Motivation: Group-level methods select either all variables in a group or none, which is inappropriate when individual predictors remain scientifically meaningful.Genetic association studies may use grouping to increase search power while still targeting individual variants.
  • Bi-level selection: Bi-level selection targets sparsity at both the group and individual-variable levels, but has received less methodological study than group LASSO.The review identifies unanswered questions surrounding these methods.
  • Concave 1-Norm Group Penalties: Concave penalties applied to a group ℓ1 norm produce grouping properties, including the group bridge and broader composite-penalty frameworks.The group bridge uses an outer bridge penalty and an inner LASSO penalty; composite penalties combine outer group-level and inner variable-level penalties.
  • Composite Penalties: Composite MCP applies MCP at both group and individual levels, with tuning chosen so the group penalty reaches its maximum only when every component reaches its maximum.This construction supports internal group sparsity while limiting how strongly one variable can dominate the group penalty.
  • Composite Penalties: Unlike group LASSO and group bridge, composite MCP combines capped penalties to reduce overshrinkage while permitting sparse groups internally.The group bridge lets a single strong predictor continually lower other variables’ entry threshold, preventing consistency for individual-variable selection.
  • Example: Genetic Association: In the genetic association example, group penalization achieved lower test error than one-at-a-time analysis while selecting fewer genes, with composite MCP identifying 11 promising SNPs.Group bridge identified 3 promising genes, whereas group LASSO selected 17 genes and 435 markers.

4. ORACLE PROPERTY OF THE 2-NORM GROUP MCP

The review develops sufficient conditions for the 2-norm group MCP estimator to equal or behave like the oracle least squares estimator. These results cover convex and high-dimensional settings, while local-solution properties remain largely unknown.

  • Oracle property: The analysis provides sufficient conditions under which the 2-norm group MCP estimator equals the oracle least squares estimator.The stated goal is preliminary theoretical justification for this concave group-selection method rather than a systematic treatment of all its properties.
  • Convex case: Under Gaussian errors and γ > 1/cmin, the criterion is strictly convex, yielding a uniquely characterized estimator through the Karush–Kuhn–Tucker conditions.The additional condition nλ2 > σ2 prevents λ from being too small.
  • Convex case: The 2-norm group MCP estimator behaves like the oracle least squares estimator with high probability and is therefore group selection consistent.The result also avoids the need for a separate irrepresentable condition because the MCP form makes the analogous condition hold automatically.
  • High-dimensional case: For J ≫ n, theorems establish sufficient conditions for selection consistency under a sparse Riesz condition and additional tuning and signal-strength requirements.These results permit settings such as J − |S| = exp{o(n/(c*ds))}, but impose stronger conditions than the lower-dimensional theorem.
  • Limitations: The high-dimensional theory concerns global solutions, whereas the selection consistency of local solutions from group coordinate descent remains largely unknown.Open questions include when local solutions equal or sufficiently approximate global solutions and retain selection consistency.

5. APPLICATIONS

Group selection methods are applied across flexible regression, multitask, and genetic settings, with the appropriate selection target determined by whether individual variables are scientifically meaningful.

  • Applications: Group selection methods are reviewed for nonparametric additive, semiparametric partially linear, varying coefficient, seemingly unrelated regression, multitask, genomic, and genetic association models.The applications span function selection, mixed linear and nonlinear effects, correlated tasks, pathways, genes, and SNPs.
  • Nonparametric additive models: COSSO treats additive component selection and estimation as a group LASSO procedure in a reproducing kernel Hilbert space.For tensor product designs, it correctly selects nonzero additive components with high probability.
  • Semiparametric regression: Partially linear models pose a selection problem because covariates may have either linear or nonlinear effects, although existing methods commonly assume this division is known.Applied work rarely knows in advance which covariates belong to each effect type.
  • Seemingly unrelated regressions and multitask learning: In seemingly unrelated regressions and multitask learning, groups can link coefficients for the same variable across regressions so that the variable is selected or dropped jointly.This structure reflects the assumption that a variable plays a similar role across tasks.
  • Genomic and genetic data: Genomic analyses group genes by pathways and SNPs by genes, while bi-level selection can identify both important groups and important individual variables.Selecting SNPs and genes simultaneously is described as more powerful than selecting them separately.

6. DISCUSSION

The review identifies unresolved practical and theoretical issues, especially penalty tuning, data-driven guarantees, local concave-penalty behavior, and overlapping groups.

  • 6.1 Penalty Parameter Selection: Penalty parameters are difficult to determine, and group selection complicates AIC/BIC because these criteria require error-variance and degrees-of-freedom estimates.For group LASSO, the proposed degrees-of-freedom estimate is infeasible when p ≫ n.
  • 6.1 Penalty Parameter Selection: Cross-validation avoids variance and degrees-of-freedom estimation, but its group-selection properties have not been rigorously analyzed.Stability selection is presented as a resampling-based alternative that also avoids those estimates.
  • 6.2 Theoretical Properties: Existing p ≫ n theory mainly concerns group LASSO linear regression for a given penalty range, leaving data-driven penalty selection such as cross-validation unresolved.The review calls for confirmation of estimation and prediction properties under such procedures.
  • 6.2 Theoretical Properties: For concave group selection, global 2-norm group MCP results exist, but local solutions from group coordinate descent require further theoretical study and comparison with group LASSO.The review emphasizes local rather than only global solution properties as more practically relevant.
  • 6.3 Overlapping Groups: The review excludes overlapping groups, although genes can belong to multiple pathways and general concave group-selection theory for overlap remains unaddressed.Existing overlapping-group results cover some group LASSO and group bridge settings under specific conditions.

APPENDIX

The appendix supplies proof components for theoretical results, including projection identities, chi-square bounds, event intersections, and reductions to smaller group models.

  • Probability bounds: The appendix invokes chi-square tail bounds and a Laurent–Massart exponential inequality in its probability arguments.These bounds support control of random quantities in the proofs.
  • Concave-penalty proof: A strict-convexity argument and KKT conditions are used to establish equality between the estimated coefficients and the target coefficient vector on intersecting events.The argument assumes c_min > 1/γ.
  • Proof completion: The appendix completes its probability calculations by combining intermediate bounds and concludes the corresponding lemmas and theorems.Several displayed steps are presented as continuations of earlier inequalities and event bounds.
  • Projection arguments: Projection arguments show that, for supersets A of S, the difference between projections annihilates the signal component and acts on the error.The identity (P_A − P_S)y = (P_A − P_S)ε is used in the proof.
  • Theorem proofs: The proofs combine events and lemmas to establish bounds on candidate sets and reduce the original model to at most (K* + 1)|S| groups.This reduction allows conditions from one theorem to imply those of another.
Loading 1204.6491v2…