Elastic Net Regularization Paths for All Generalized Linear Models

1. Introduction

• The elastic net combines ℓ1 and ℓ2 penalties with the residual sum of squares (RSS) [Zou and Hastie 2005].Its tuning parameter λ is nonnegative, and α ∈ [0, 1] controls the mixture of penalties.Source: §1 · p. 1
• The choice of α determines the penalty type: α = 0 gives ridge regression [Hoerl and Kennard 1970], while α = 1 gives the lasso [Tibshirani 1996].For sparse models with correlated features, α may be set close to but not equal to 1; λ is usually selected by cross-validation.Source: §1 · p. 2
• For GLMs and Cox models, the elastic-net objective replaces RSS with a negative log-likelihood or negative log partial likelihood, respectively.The package’s earlier versions had specialized routines for selected GLMs and right-censored Cox models.Source: §1 · p. 2
• glmnet computes paths across decreasing λ values using cyclic coordinate descent, with core routines programmed in FORTRAN.It also supports cross-validation, coefficient-path and error plots, prediction, and sparse predictor matrices.Source: §1 · p. 2
• glmnet extends elastic-net paths to all GLMs, Cox models with (start, stop] data and strata, and a simplified relaxed lasso.The paper builds on earlier glmnet work for OLS, logistic and multinomial logistic regression, and right-censored Cox models.Source: §1 · p. 2

2.1. Overview of generalized linear models

• For one-dimensional exponential families, the family determines variance and link plus variance specify the GLM; quasi-likelihood allows these functions to be specified more generally, and OLS is a special case.Source: §2.1 · p. 3

2.2. Extending the elastic net to all GLM families

• The GLM extension replaces residual sum of squares with negative log-likelihood and solves a penalized weighted least-squares problem at each iteration, warm-starting successive λ values.Initializing each λ fit at the solution for the previous λ is expected to reduce the iterations needed.Source: §2.2 · p. 4

2.3. Implementation details

• 10 to 15 times slower than the first approach, the second implementation supports bespoke GLM families by computing working weights and responses from user-supplied family objects in R.The first implementation uses family-specific FORTRAN routines; the second keeps the penalized WLS bottleneck in FORTRAN.Source: §2.3 · pp. 4, 6, Figure 1 · p. 7

2.4. Details on the penalized WLS subroutine

• The FORTRAN solver uses cyclic coordinate descent; strong rules limit updates to a predicted active set, and KKT checks add violators to ensure the exact solution.Users can vary coefficient penalty weights, set coefficient bounds, and use a sparse-matrix solver when the design matrix is sparse.Source: §2.4 · pp. 7–8

2.5. Other useful functionality

• glmnet can fit a log-spaced λ path, and cv.glmnet selects λ by cross-validation; predictions default to lambda.1se, while lambda.min requests minimum-error predictions.CV-error standard errors are crude and generally too small because of correlations across folds. bigGlm fits a single unpenalized GLM with coefficient bounds or sparse input.Source: §2.5 · pp. 8–10

3. Regularized Cox proportional hazards models

• Cox elastic-net paths minimize negative log partial likelihood via Algorithm 3, reusing penalized WLS; for tied observed times, glmnet uses the Breslow approximation.The working-response and weight formulas also support sparse design matrices.Source: §3 · pp. 10–12

3.1. Extending regularized Cox models to (start, stop] data

• For (start, stop] data, an observation is at risk at failure time t_i exactly when t_i falls in its interval; cached ordering enables O(n log n) initial and O(n) later computation.This counting-process extension supports time-dependent covariates, multiple events per subject, and models for correlated data.Source: §3.1 · pp. 12–13

3.2. Stratified Cox models

• Stratified Cox models give each group its own baseline hazard while sharing β, and their stratum-specific partial likelihoods allow fitting with a modified Algorithm 3.In glmnet, users specify strata by adding a strata attribute to the response with stratifySurv.Source: §3.2 · pp. 13, 15

3.3. Plotting survival curves

• For regularized Cox fits, glmnet's survfit method reconstructs a coxph object from original x and y and returns one survival curve per newx individual.For cv.glmnet fits, the default is lambda.1se; users can request lambda.min instead.Source: §3.3 · p. 15, Figure 3 · p. 15

4. The relaxed lasso

• Hastie et al. (2020) found the relaxed lasso performs like the lasso at low SNR, nearly like best subset selection at high SNR, and has an advantage over best subset and forward stepwise when p is large.The relaxed lasso was introduced by Meinshausen (2007) to undo shrinkage inherent in the lasso estimator.Source: §4 · p. 16
• For a given λ, the simplified relaxed lasso is a convex combination of the lasso estimate and an unpenalized OLS refit on the lasso’s active set, controlled by γ ∈ [0, 1].The exposition defines the method for Gaussian OLS and α = 1; the active-set OLS coefficients are padded with zeros to match the lasso solution.Source: §4 · p. 16
• glmnet fits the relaxed path by refitting unpenalized models on lasso active sets, efficiently avoiding a repeated OLS fit when active sets coincide.cv.glmnet selects both λ and γ; by default it cross-validates γ = c(0, 0.25, 0.5, 0.75, 1), and additional γ values can be considered.Source: §4 · pp. 16–17
• Relaxed fits are available for other model families, where glmnet fits the corresponding unpenalized model on the lasso active set instead of an OLS model.The relaxed option is enabled with relax = TRUE.Source: §4 · pp. 16–17
• Although relaxation can be used with α < 1, the section discourages it because the larger active set works against the goal of obtaining sparser models.Undoing shrinkage gives active-set coefficients more freedom to fit the response, and cross-validation over λ and γ often yields a sparser model than the lasso.Source: §4 · p. 17

4.1. Application to forward stepwise regression

• When p is large, a lasso path can identify candidate variable sets across over 100 λ values, after which cv.glmnet can fit OLS models to selected sets as a faster alternative to forward stepwise regression.Forward stepwise adds one variable at a time and must test all predictors not already included at each step.Source: §4.1 · p. 17

5. Assessing models

• Assess fitted models by building predictions for each λ and, for relaxed fits, γ, then computing a chosen performance measure on evaluation or test data.cv.glmnet also evaluates pre-validated predictions on held-out folds and records the measures in cvm for its CV plot.Source: §5 · p. 17
• glmnet supports family-specific performance measures, and type.measure lets users choose the CV measure, such as AUC instead of binomial deviance.The full list is available through glmnet.measures().Source: §5 · p. 17
• assess.glmnet accepts prediction matrices, glmnet fits, or cv.glmnet fits with true responses, and returns measures for a fit’s λ sequence or at lambda.1se for a CV fit.For prediction matrices, specify family because assess.glmnet cannot infer it; s and gamma select other hyperparameter values.Source: §5 · p. 18
• With keep=TRUE, cv.glmnet retains pre-validated fits that assess.glmnet can evaluate for multiple measures, avoiding repeated cross-validation runs.keep is FALSE by default because retaining the matrix can enlarge the returned object when there are many training observations.Source: §5 · p. 18
• For binomial test data, roc.glmnet produces an ROC curve and confusion.glmnet produces a confusion matrix.These functions provide test-performance assessments specific to binomial data.Source: §5 · p. 18

• Elastic net regularization is extended to all GLM families and Cox models with (start, stop] data and strata.Source: §6 · p. 19
• The paper also extends elastic net regularization to a simplified version of the relaxed lasso.Source: §6 · p. 19
• The paper discusses glmnet utilities for assessing the fit of these models, available in package version 4.1 and later on CRAN.Source: §6 · p. 19