Source-linked AI summary
Elastic Net Regularization Paths for All Generalized Linear Models
J. Kenneth Tay, Balasubramanian Narasimhan, Trevor Hastie
TL;DR
The paper addresses the limited coverage of earlier elastic net software, which handled selected GLMs and right-censored Cox models. It extends the framework to all GLM families, more flexible Cox data, and a simplified relaxed lasso, with these capabilities available in glmnet 4.1 and later. The paper also discusses model-fit assessment and practical use of the relaxed fit for large-p variable selection.
Problem
Earlier glmnet versions provided specialized routines for only selected GLMs and right-censored Cox models, motivating broader model-family and survival-data support.
Method
The paper extends elastic net regularization to all GLM families, Cox models with (start, stop] data and strata, and a simplified relaxed lasso implemented in glmnet.
Results
The resulting capabilities are available in glmnet version 4.1 and later, including functions for assessing the fit of elastic net models.
Takeaways & Limitations
The relaxed fit can provide a faster alternative to forward stepwise regression when the number of variables is large.
Takeaways & Limitations
Cross-validation standard-error estimates are crude and generally too small because of correlations across folds.
Abstract
from arXiv · showhide
The lasso and elastic net are popular regularized regression models for supervised learning. Friedman, Hastie, and Tibshirani (2010) introduced a computationally efficient algorithm for computing the elastic net regularization path for ordinary least squares regression, logistic regression and multinomial logistic regression, while Simon, Friedman, Hastie, and Tibshirani (2011) extended this work to Cox models for right-censored data. We further extend the reach of the elastic net-regularized regression to all generalized linear model families, Cox models with (start, stop] data and strata, and a simplified version of the relaxed lasso. We also discuss convenient utility functions for measuring the performance of these fitted models.
1. Introduction
Elastic net regularization combines ℓ1 and ℓ2 penalties and computes a path of models across λ values, extending beyond ordinary regression to generalized linear and Cox models. Version 4.1 of glmnet adds support for all GLM families, Cox models with (start, stop] data and strata, and a simplified relaxed lasso.
- Elastic net framework: The elastic net minimizes residual error plus a mixture of ℓ1 and ℓ2 penalties controlled by λ and α.λ controls regularization strength, while α selects the penalty mixture.
- Elastic net framework: Setting α = 0 gives ridge regression, α = 1 gives the lasso, and values near 1 retain sparsity while addressing correlated features.The final λ is usually selected by cross-validation using the smallest cross-validated error.
- Generalized models: For GLMs and Cox models, the residual sum of squares is replaced by a negative log-likelihood or negative log partial likelihood.This extends the same regularization framework beyond ordinary least squares.
- Software implementation: glmnet computes elastic net solutions over a decreasing path of λ values using cyclic coordinate descent, with core routines implemented in FORTRAN.The package also supports cross-validation, coefficient-path plots, and cross-validation-error plots.
- Paper contribution: Version 4.1 and later supports regularization paths for all GLMs, Cox models with (start, stop] data and strata, and a simplified relaxed lasso.The paper builds on earlier algorithms for ordinary, logistic, multinomial logistic, and right-censored Cox models.
2. Regularized generalized linear models
The section extends elastic-net fitting from ordinary regression to generalized linear models by replacing least-squares optimization with penalized likelihood optimization. It presents an iterative algorithm and two implementations that trade broad family support against computational efficiency.
- Overview of generalized linear models: GLMs are specified through a link function and variance function, with examples represented by R family objects.For one-dimensional exponential families, the family determines the variance function; quasi-likelihood models specify both functions directly.
- Extending the elastic net to all GLM families: Elastic-net GLM fitting replaces the residual sum of squares with a negative log-likelihood and solves a penalized weighted least-squares problem at each iteration.The algorithm computes working responses and weights before solving the penalized WLS subproblem.
- Algorithm: The regularization path evaluates decreasing λ values, using the previous solution as a warm start and initializing the first solution at zero.Warm starts exploit the expected similarity between solutions at adjacent λ values to reduce iterations.
- Implementation: The generalized implementation computes family-specific working quantities in R while solving the penalized WLS bottleneck in FORTRAN, enabling bespoke family objects.This avoids requiring a new FORTRAN subroutine for every GLM family.
- Implementation: 10 to 15 times slower fitting is reported for the general implementation than for specialized routines in Gaussian and logistic comparisons.The difference is attributed to R being slower than FORTRAN and to repeated switching between the two environments.
- Implementation: Users are advised to use specialized implementations for popular families such as Gaussian, logistic, and Poisson regression, and the general implementation for other GLM families.Equivalent character-string and family-object calls return the same regularized Poisson result, but the family-object route takes longer.
3. Regularized Cox proportional hazards models
The paper extends elastic-net regularization to Cox models with right-censored, (start, stop] and stratified data, using modified partial-likelihood algorithms and efficient risk-set computations.
- Regularized Cox models: The Cox model relates predictors to survival through a semi-parametric hazard and estimates coefficients by maximizing the partial likelihood.The partial likelihood conditions failure events on observations at risk, eliminating the need to estimate the baseline hazard for coefficient inference.
- Regularized Cox models: Elastic-net Cox fitting follows a λ path and uses iterative working responses, weights, and penalized weighted least-squares updates.The algorithm selects α and decreasing λ values, then iterates outer and inner optimization loops until convergence.
- Regularized Cox models: For right-censored Cox models, glmnet reuses the penalized weighted-least-squares machinery with Cox-specific working quantities and diagonal Hessian approximations.This enables efficient fitting, including when the design matrix is sparse; tied event times use the Breslow approximation.
- Extending to (start, stop] data: (start, stop] responses define risk-set membership by whether each event time lies within an observation’s interval, while retaining the same partial-likelihood form.This representation supports time-dependent covariates, repeated events, correlated-data models, and case-cohort designs.
- Extending to (start, stop] data: O(n log n) computation is possible for (start, stop] data because only sorting requires O(n log n), while subsequent working-quantity updates require O(n).The sorting result can be cached because observation ordering does not change.
- Stratified Cox models: Stratified Cox models assign each stratum its own baseline hazard while sharing coefficient values, and their negative log partial likelihood decomposes across strata.This decomposition permits regularized fitting with a modified Cox algorithm; glmnet specifies strata through a response attribute.
4. The relaxed lasso
The relaxed lasso combines lasso-selected variables with an unpenalized refit to reduce coefficient shrinkage, and glmnet supports fitting and cross-validating this model.
- The relaxed lasso: The relaxed lasso addresses lasso shrinkage by combining the lasso estimator with an unpenalized fit on its active set.The combination is controlled by γ, which ranges from 0 to 1.
- Implementation: glmnet obtains relaxed fits by first computing lasso estimates and active sets across λ, then refitting the unpenalized model on each active set.Repeated active sets can reuse previously computed refits.
- Cross-validation: Cross-validation can select both λ and γ, with default γ values of 0, 0.25, 0.5, 0.75, and 1.The implementation allows users to consider any desired set of γ values.
- Implementation: For non-Gaussian families, glmnet fits the unpenalized model for the selected family on the lasso active set rather than always using OLS.The relaxed-fit framework is available for model families beyond Gaussian regression.
- The relaxed lasso: Relaxation is typically used with α = 1 because α values below 1 produce larger active sets than the lasso.The authors state that relaxation is intended to produce sparser models by undoing active-set coefficient shrinkage.
- Application to forward stepwise regression: The relaxed fit can provide a faster alternative to forward stepwise regression by using lasso paths to identify candidate variable sets.This is especially useful when p is in the tens of thousands, where lasso paths can identify candidate sets over many λ values.
5. Assessing models
The paper provides glmnet utilities for cross-validation, test-set assessment, and family-specific performance measures across fitted regularized models.
- Assessing models: Model assessment requires predictions and performance measures for each fitted model indexed by λ and, for relaxed fits, γ.The framework applies to evaluation or test data after choosing an appropriate performance measure.
- Cross-validation: During cross-validation, cv.glmnet stores pre-validated held-out predictions and uses them to compute performance measures recorded in cvm.These values are also used to generate the cross-validation error plot.
- Performance measures: glmnet supports multiple family-dependent measures, including AUC for binomial models, and lets users select the measure through type.measure.For binomial data, roc.glmnet and confusion.glmnet provide ROC curves and confusion matrices on test data.
- assess.glmnet: assess.glmnet evaluates predictions supplied as a matrix, glmnet object, or cv.glmnet object against true responses.For a prediction matrix, the user must specify the model family because it cannot be inferred from the matrix alone.
- assess.glmnet: Keeping pre-validated fits allows assess.glmnet to compute additional performance measures without rerunning cross-validation.The keep = TRUE option returns the pre-validated matrix for later assessment.
6. Discussion
The paper extends glmnet to broader GLM and Cox-model settings, relaxed lasso fitting, and model-assessment utilities available from version 4.1 onward.
- Discussion: glmnet 4.1 and later supports elastic-net paths for all GLM families, Cox models with (start, stop] data and strata, and a simplified relaxed lasso.The paper also presents utilities for assessing fitted elastic-net models.
Affiliation:
Balasubramanian Narasimhan is affiliated with the Department of Biomedical Data Sciences and the Department of Statistics.
- Balasubramanian Narasimhan is affiliated with the Department of Biomedical Data Sciences and the Department of Statistics.