Source-linked AI summary
Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
Patrick Breheny, Jian Huang
TL;DR
Grouped variable selection requires methods that account for predictor structure, while existing group lasso and nonconvex alternatives have different statistical and computational limitations. The paper develops stable, efficient algorithms for group SCAD and group MCP, evaluates them in simulations and real data, and reports advantages over group lasso in relevant examples.
Problem
Grouped predictors require selecting groups rather than individual variables, but group SCAD and group MCP have seen limited use because efficient algorithms and public software were lacking.
Method
The paper extends coordinate descent with group descent algorithms, orthonormalization, and majorization-minimization for fitting group SCAD and group MCP in linear and logistic regression.
Results
Group MCP and group SCAD outperform group lasso in prediction accuracy and false discoveries; in one genetic simulation, group MCP produced four false discoveries versus 12.5 for group SCAD.
Takeaways & Limitations
Group MCP and group SCAD are presented as powerful alternatives to group lasso for grouped variable selection, with group MCP offering greater flexibility in supported examples.
Takeaways & Limitations
Group MCP's flexibility makes proper tuning-parameter selection important, while logistic regression lacks the simple closed-form solutions available after orthonormalization.
Abstract
from arXiv · showhide
Penalized regression is an attractive framework for variable selection problems. Often, variables possess a grouping structure, and the relevant selection problem is that of selecting groups, not individual variables. The group lasso has been proposed as a way of extending the ideas of the lasso to the problem of group selection. Nonconvex penalties such as SCAD and MCP have been proposed and shown to have several advantages over the lasso; these penalties may also be extended to the group selection problem, giving rise to group SCAD and group MCP methods. Here, we describe algorithms for fitting these models stably and efficiently. In addition, we present simulation results and real data examples comparing and contrasting the statistical properties of these methods.
1 Introduction
Grouped predictors motivate methods that select groups rather than individual variables, especially in high-dimensional settings. This paper extends coordinate descent to group SCAD and group MCP, addressing efficiency and availability limitations while comparing them with group lasso.
- Grouping predictors can improve model interpretability and accuracy, particularly when sparsity and variable selection matter in high-dimensional settings.
- The group lasso selects grouped covariates by penalizing norms of coefficient groups rather than individual covariates.
- Because its penalization rate does not change with group-coefficient magnitude, group lasso over-shrinks large coefficients and may include spurious coefficients.
- Group SCAD and group MCP extend nonconvex penalties designed to achieve selection consistency and asymptotic unbiasedness, known jointly as the oracle property.
- Existing group SCAD fitting methods based on local quadratic approximation were inefficient for large regressions because they lacked natural sparsity and repeatedly factored large matrices.
- The paper extends coordinate descent to group SCAD and group MCP, presenting a fast, stable algorithm, a public grpreg implementation, and simulated and real-data comparisons with group lasso.
2 Group descent algorithms
The paper develops group descent algorithms for group lasso, group MCP, and group SCAD models by orthonormalizing predictor groups and using closed-form or majorization-based updates. The methods support linear and logistic regression, with descent and stationary-point convergence guarantees under stated conditions.
- Model formulation: The model minimizes a loss plus penalties applied to groupwise L2 norms, with λ controlling sparsity and allowing unpenalized groups when λj = 0.Groups may contain predictors such as indicator or basis-function expansions, and covariates outside groups can be treated as singleton groups.
- Orthonormalization: Orthonormalization removes the need for initially orthonormal groups because fitted solutions can be transformed back to the original coefficient scale.The approach also handles non-full-rank groups by fitting in a lower-dimensional parameter space and transforming back.
- Group lasso: For linear regression, multivariate soft-thresholding provides the single-group group-lasso solution and yields simple coordinate-descent updates.With orthonormalized groups, the algorithm requires only simple arithmetic rather than complicated numerical optimization, matrix factorizations, or inversions.
- Group MCP and group SCAD: Group MCP and group SCAD use multivariate firm-thresholding operators that select or exclude entire groups and reduce shrinkage of sufficiently large coefficients.Both penalties become unpenalized when the relevant norm exceeds γλ, while MCP and SCAD differ in how quickly their penalization rate relaxes.
- Logistic regression: In logistic regression, IRLS destroys the simple orthonormal closed forms, so an MM majorization step preserves a spherical likelihood surface before applying groupwise minimization.The resulting algorithms retain the descent property, and, provided coefficients do not diverge, every limit point is stationary.
- Convergence and limitations: The group lasso converges to a global minimum, whereas group MCP and group SCAD may converge only to local minima.For logistic regression, complete separation can produce infinite maximum-likelihood estimates; the implementation stops path fitting when saturation is detected.
3 Algorithm efficiency
The algorithms scale linearly with p because each full pass costs O(2np), while runtime also depends on iterations and the number of nonzero groups. In timing comparisons, grpreg appeared uniformly more efficient, with group SCAD estimation taking 0.35 seconds versus 32 minutes in a comparable setting.
- O(2np) operations are required for one full pass over all groups, enabling linear scaling in p.The dominant computations are inner products requiring O(nK_j) operations.
- Iteration counts depend mainly on the number of nonzero groups, so fitting takes disproportionately longer at small λ.Groups fixed at zero require no iteration, whereas less-sparse paths activate more groups.
- grpreg appears uniformly more efficient than grplasso and standGL across increasingly large group-lasso data sets.The comparison reports median times over 100 data sets and a 100-value λ grid.
- Group MCP and group SCAD tend to be slightly faster than group lasso, presumably because their solution paths are sparser.
- Timing comparisons were restricted to problems where the entire solution path could be computed to avoid different termination points.
- 32 minutes versus 0.35 seconds: the proposed approach was much faster for group SCAD when n = 100 and p = 500.The comparison is with the algorithm reported by She (2012).
4 Simulation studies
Simulations compare group lasso, group MCP, and group SCAD in basic, semiparametric, and genetic-association settings. Across these studies, nonconvex group penalties generally improve prediction and sparsity, with group MCP showing stronger finite-sample selection in the reported comparisons.
- Design: The simulations compare group lasso, group MCP, and group SCAD in basic, semiparametric, and genetic-association settings.Five-fold cross-validation selected λ; γ was fixed at 3 for group MCP and 4 for group SCAD, and results averaged over 1,000 data sets.
- 4.1 Basic: As coefficient magnitudes increase, group MCP and group SCAD approach theoretically optimal RMSE, whereas group lasso performs increasingly poorly.For small coefficients, all three methods perform similarly; the comparison uses n=100 and p=400 with five active groups.
- 4.1 Basic: Group MCP and group SCAD select smaller models and approach the true model size faster than group lasso, which selects too many variables.The basic simulation varies |β| from 0 to 1.5 while retaining five nonzero groups.
- Comparison: Group MCP and group SCAD have nearly identical estimation accuracy, but group MCP generally has better finite-sample variable-selection properties.This pattern is reported in both the basic and semiparametric comparisons, and group MCP is more sparse in the basic study.
- 4.2 Semiparametric regression: In semiparametric regression, all group-selection methods greatly outperform the lasso, while group MCP and group SCAD combine superior prediction accuracy with more parsimonious models.Each of 100 continuous variables is represented by a six-term B-spline basis, yielding n=200 and p=600; six variables have effects.
- 4.3 Genetic association study: In the genetic-association simulation, group methods outperform ungrouped or ad-hoc grouped alternatives; group MCP and SCAD outperform group lasso in prediction and false discoveries.Among 500 SNPs, three have effects; group MCP produces four false discoveries versus 12.5 for group SCAD.
5 Real data
The real-data examples apply grouped selection to gene-expression and genetic-association studies, showing different trade-offs among parsimony, prediction, and correlated predictors. Group MCP, SCAD, and lasso improve or match predictive performance while producing distinct model sizes.
- Data and methods: The studies apply grouped variable selection to rat gene-expression data and an age-related macular degeneration case-control study.The gene-expression analysis uses n = 120 and p = 15,000; the genetic study includes 400 cases, 400 controls, and 1,064 grouped predictors.
- Bardet-Biedl syndrome gene expression study: Group lasso and group SCAD select many correlated genes, whereas group MCP selects one gene with a fit nearly equivalent to that gene alone.The MCP model explains 65% of TRIM32-expression variance, while the other approaches shrink each selected gene’s coefficient group substantially.
- Bardet-Biedl syndrome gene expression study: A large TRIM32 outlier, almost 10 standard deviations below the remaining points, strongly influences the fitted relationships across genes.The median absolute correlation for genes in the associated table is 0.62, and many scatterplots resemble Figure 5.
- Bardet-Biedl syndrome gene expression study: 0.092 ± 0.04 versus 0.099 ± 0.04 root cross-validation error favors group SCAD/lasso over group MCP, although all three approaches are within random variability.The SCAD/lasso models provide somewhat better predictive ability while avoiding an arbitrary choice among highly correlated genes.
- Reported comparisons: Table 4 reports selected genes and coefficient-group Euclidean norms, while Table 5 reports selected SNP counts and cross-validated misclassification errors against a baseline.These summaries expose both model size and predictive performance for the real-data comparisons.
- Age-related macular degeneration genetic association study: Group MCP and group SCAD produce more parsimonious genetic-association models without losing prediction accuracy, and all three methods improve substantially over the intercept-only baseline.Group MCP is slightly superior in predictive accuracy, but its difference from the other methods is comparable to the standard error.
6 Conclusion
The conclusion presents group MCP and group SCAD as alternatives to group lasso for grouped variable selection. The paper addresses their limited use by developing efficient algorithms and a publicly available R implementation.
- Group MCP and group SCAD are presented as powerful alternatives to group lasso for grouped variable-selection problems.
- The paper develops efficient algorithms and provides an implementation through the R package grpreg.These contributions target the prior lack of efficient algorithms and publicly available software, especially for high-dimensional problems.
Appendix
The appendix establishes conditions under which group-descent updates are strictly convex within each group, descend monotonically, and converge to stationary points. The results cover linear and logistic regression with specified penalty-parameter ranges.
- Linear regression: For linear regression, groupwise strict convexity holds for group lasso, group SCAD with γ > 2, and group MCP with γ > 1.
- Linear regression: Each linear-regression update minimizes the objective over one group, yielding descent and making every limit point stationary under Tseng’s convergence conditions.
- Linear regression: The linear group-descent sequence is guaranteed to converge to a unique limit point because transitions between distinct limit points are ruled out by the unique groupwise minimizer.
- Logistic regression: For logistic regression, the majorizing approximation is strictly convex within each group for group lasso, group SCAD with γ > 5, and group MCP with γ > 4.
- Logistic regression: The MM-based logistic updates descend at every iteration and converge to stationary points of the original objective when the sequence remains bounded.