PaperlayerView exact revision on arXiv

Source-linked AI summary

lassopack: Model selection and prediction with regularized regression in Stata

Achim Ahrens, Christian B. Hansen, Mark E. Schaffer

arXiv:1901.05397v1econ.EM

TL;DR

Stata has offered limited machine-learning tools for regularized regression in high-dimensional settings. This paper introduces lassopack and compares tuning approaches, finding that EBIC and rigorous methods generally perform well when p > n, whereas AIC and BIC are not appropriate.

  • Problem

    Stata offers limited machine-learning tools, motivating a flexible and accessible suite for regularized regression.

  • Method

    The paper develops lassopack, implementing regularized regression methods with theory-driven penalization for high-dimensional prediction and estimation.

  • Results

    EBIC and rigorous methods generally recover the true structure when p > n, while AIC and BIC are not appropriate in that setting.

  • Takeaways & Limitations

    For high-dimensional model selection, EBIC and rigorous methods offer generally strong structure recovery, while rigorous methods trade slightly higher prediction error for lower false-positive rates.

  • Takeaways & Limitations

    AIC and BIC are unsuitable in large-p-small-n settings because they tend to select too many variables.

Abstract

from arXiv · show

This article introduces lassopack, a suite of programs for regularized regression in Stata. lassopack implements lasso, square-root lasso, elastic net, ridge regression, adaptive lasso and post-estimation OLS. The methods are suitable for the high-dimensional setting where the number of predictors $p$ may be large and possibly greater than the number of observations, $n$. We offer three different approaches for selecting the penalization (`tuning') parameters: information criteria (implemented in lasso2), $K$-fold cross-validation and $h$-step ahead rolling cross-validation for cross-section, panel and time-series data (cvlasso), and theory-driven (`rigorous') penalization for the lasso and square-root lasso for cross-section and panel data (rlasso). We discuss the theoretical framework and practical considerations for each approach. We also present Monte Carlo results to compare the performance of the penalization approaches.

1 Introduction

lassopack addresses Stata’s limited machine-learning toolkit by providing accessible, flexible regularized-regression methods. The introduction emphasizes their prediction-oriented strengths, model-selection usefulness, inferential limitations, and alternative tuning strategies.

  • Motivation: lassopack fills a gap in Stata by offering easy-to-use, flexible methods for regularized regression.The package is motivated by growing interest in machine learning across scientific disciplines and applied econometrics.
  • Methodological background: Regularized linear regression extends OLS by adding a penalty that limits model complexity.It minimizes squared deviations between observed and predicted values while imposing regularization.
  • Prediction and inference: Regularized regression primarily targets prediction, while coefficient interpretation as causal and statistical inference remain difficult.Selecting the true model generally requires strong assumptions, although regularized regression can aid causal inference.
  • Model selection: Except for ridge regression, lassopack methods produce sparse solutions that can support model selection with many potential predictors.The introduction contrasts this with iterative testing procedures that may induce pre-testing bias and false positives.
  • Tuning parameters: lassopack selects tuning parameters through cross-validation or information criteria, with cross-validation optimizing out-of-sample prediction but requiring substantial computation.The introduction describes cross-validation as broadly applicable and generally effective for prediction tasks.

2 Regularized regression

This section introduces lassopack’s regularized regression methods for high-dimensional linear models, where predictors may outnumber observations under a sparsity assumption. It explains penalization parameters and contrasts lasso-based methods with ridge regression and post-estimation OLS.

  • Regularized regression: lassopack targets high-dimensional models in which p may exceed n, assuming only s ≪ n predictors belong to the true model.The section defines exact sparsity and the active set of non-zero coefficients, while noting that small non-zero coefficients can later be allowed.
  • Regularized regression: Penalized estimation uses λ to control overall penalization and α to determine the relative contribution of ℓ1 versus ℓ2 penalties.lassopack introduces information criteria, cross-validation, and rigorous penalization as approaches for selecting tuning parameters.
  • Lasso: The lasso combines prediction-oriented regularization with variable selection by setting some coefficient estimates exactly to zero.Its coefficient path is piecewise linear in λ, with λ = 0 yielding OLS and λ →∞ yielding an empty model.
  • Ridge regression: Ridge replaces the lasso’s ℓ1 penalty with an ℓ2 penalty, does not perform variable selection, and does not require sparsity.This makes ridge attractive for dense signals, although dense high-dimensional problems are more challenging than sparse ones.
  • Elastic net: Elastic net mixes ℓ1 and ℓ2 penalization, producing sparse solutions for some α > 0 while retaining advantages for groups of correlated regressors.The lasso typically selects one variable from a correlated group, whereas ridge tends to assign similar estimates across the group.
  • Other methods: Square-root lasso simplifies theoretically grounded λ selection because its optimal penalty level is independent of unknown error variance under homoskedasticity.The section also introduces adaptive lasso and post-estimation OLS, which addresses attenuation bias using variables selected in the first stage.

3 Tuning parameter selection using information criteria

This section frames information criteria as penalized-likelihood tools for selecting regularization parameters, contrasting their loss efficiency and model-selection consistency. It also highlights that conventional AIC and BIC can perform poorly in large-p-small-n settings, motivating AICc and EBIC adjustments.

  • Information criteria and regularization: AIC adds a penalty of 2p to −2×log-likelihood, but finding its minimum requires estimating all possible models.This ℓ0-penalty becomes impractical as the number of candidate predictors grows.
  • Information criteria and regularization: Regularized regression reduces information-criterion model selection to choosing λ, or λ and α for elastic net.Information criteria are theoretically well understood and easy to compute after coefficient estimates are obtained.
  • Criterion properties: AIC is loss efficient, whereas BIC is model-selection consistent only when the true model is among the candidates; AIC is inconsistent.Loss efficiency targets the smallest attainable averaged squared error, while consistency means selecting the true model with probability approaching 1 as n →∞.
  • High-dimensional limitations: Both AIC and BIC tend to select too many variables in large-p-small-n settings.The paper points to its Section 8 Monte Carlo simulations as evidence for this limitation.
  • High-dimensional limitations: AICc should be favored when n is small or the data are high-dimensional because AIC bias can be severe when df is large relative to n.AICc is introduced as a bias-corrected alternative to AIC.
  • High-dimensional adjustments: EBICξ adds a model-size penalty that addresses BIC’s prior-probability problem and outperforms traditional BIC in simulations when p is large relative to n.Its additional penalty is controlled by ξ, with the default ξ = 1 −log(n)/(2 log(p)) constrained to [0,1].

4 Tuning parameter selection using cross-validation

Cross-validation selects regularization tuning parameters by evaluating predictive performance on held-out data. K-fold CV minimizes estimated out-of-sample MSPE, while practical choices of K balance statistical properties against computational cost and dependence concerns.

  • Cross-validation objective: Cross-validation repeatedly divides data into training and validation sets to select tuning parameters yielding the best out-of-sample predictive performance.The validation data assess models fit on the training data, commonly using mean squared prediction error.
  • K-fold cross-validation: K-fold CV partitions observations into K approximately equal folds, using each fold once for validation and the remaining folds for training at given λ and α.The resulting predictions form the basis of the cross-validation estimate of MSPE.
  • Tuning-parameter choice: Selecting λ and α to minimize estimated cross-validation loss targets predictive performance, whereas the largest λ within one standard deviation of the minimum yields a more parsimonious model.The one-standard-deviation rule is an alternative to choosing the exact minimum.
  • Computational considerations: Cross-validation cost is approximately proportional to K because models must be estimated K times at every λ or (λ, α) grid point.Elastic net tuning requires a two-dimensional grid, and standardization adds further computational cost.
  • Choice of K: K between 5 and 10 is often recommended because performance rarely improves for K larger than 10, while larger K reduces the variance and bias of the CV estimate.Variance is minimal for leave-one-out CV, K = n, but computational constraints motivate smaller K.
  • Dependent data: Serial dependence violates training-validation independence, motivating rolling h-step-ahead CV with expanding or fixed training windows for dependent data.With an expanding window, the training sample grows incrementally; fixed-window rolling CV is also presented.

5 Rigorous penalization

The rigorous penalization framework chooses theory-based penalty parameters to support consistent prediction and parameter estimation in high-dimensional models. The section develops its assumptions, feasible algorithms for lasso and square-root lasso, and a related joint-significance test.

  • 5 Rigorous penalization: Rigorous penalization selects penalty parameters to guarantee consistent prediction and parameter estimation, supporting causal-inference methods with many instruments or controls.The framework is grounded in theory and is especially relevant to high-dimensional causal inference.
  • 5.1 Conditions: The lasso consistency framework requires sparsity, a restricted sparse eigenvalue condition, and a regularization event controlling the penalty level and loadings.Approximate sparsity allows small true coefficients to be omitted, while the eigenvalue condition replaces full rank when p > n.
  • 5.1 Conditions: The regularization event controls the random score component with high probability through the penalty level λ, penalty loadings ψ_j, and confidence level γ.In the no-regressor example, choosing λ relative to q_Λ(1 − γ) identifies the correct empty model with probability at least 1 − γ.
  • 5.2–5.5 Feasible algorithms: The unknown quantile function q_Λ(·) is the chief practical obstacle, addressed in rlasso through asymptotic X-independent and alternative feasible approaches.The default asymptotic approach is theoretically grounded and feasible.
  • 5.2–5.5 Feasible algorithms: Under the stated assumptions and algorithmic penalty estimation, lasso and post-lasso achieve near-oracle prediction and target-parameter convergence rates.The square-root lasso additionally permits one-step estimation under homoskedasticity because its ideal penalty parameters are data-calculable and independent of the unobserved error.
  • 5.6 Joint significance testing: rlasso also implements a sup-score test for the joint significance of regressors, analogous to conventional F or χ2 regression tests.The test is proposed for high-dimensional inference, where lasso-based inference remains challenging.

6 The commands · 6.1 lasso2: Base command and information criteria

lassopack provides three commands—lasso2, cvlasso, and rlasso—with alternative methods for selecting tuning parameters. The lasso2 command estimates several regularized regression models over specified penalty levels and supports information-criterion-based model selection.

  • 6 The commands: lassopack consists of lasso2, cvlasso, and rlasso, each offering an alternative method for selecting tuning parameters λ and α.The paper discusses each command’s syntax and principal options.
  • 6.1 lasso2: Base command and information criteria: lasso2 obtains adaptive lasso, elastic net, and square-root lasso solutions for one or multiple penalty levels λ1, ..., λr, ..., λR.The number of penalty levels is denoted R.
  • 6.1 lasso2: Base command and information criteria: The alpha(real), sqrt, adaptive, and ols options select elastic net, square-root lasso, adaptive lasso, and post-estimation OLS, respectively.The default estimator is lasso, corresponding to alpha(1), while alpha(0) yields ridge regression.
  • 6.1 lasso2: Base command and information criteria: With multiple λ values, lasso2 stores coefficient estimates in e(betas), with rows for penalty levels and columns for predictors; with one value, it stores estimates in e(b).The default behavior uses a list of 100 penalty values.
  • 6.1 lasso2: Base command and information criteria: lasso2 calculates AIC, AICc, BIC, and EBIC, enabling model selection through the λr that minimizes a chosen criterion.The ic(string) option controls which criterion is shown, while lic(string) displays the corresponding selected model.
  • 6.1 lasso2: Base command and information criteria: alpha(real) controls the elastic-net balance between ℓ1-norm and ℓ2-norm penalization, with alpha values restricted to [0,1].alpha(1) corresponds to lasso and alpha(0) to ridge regression.
  • 6.1 lasso2: Base command and information criteria: The default penalty sequence contains 100 values ranging from lmax to lminratio*lmax, equally spaced on the logarithmic scale.The lambda() option accepts a scalar or descending list of positive penalty levels.
  • 6.1 lasso2: Base command and information criteria: The lic(string) option estimates and displays the model corresponding to the minimum information criterion, with EBIC as the default criterion.The postresults option stores the selected model’s estimation results in e().

6.2 Cross-validation with cvlasso

cvlasso performs K-fold and h-step-ahead rolling cross-validation for selecting regularization parameters in lasso models. It supports tuning over lambda and alpha grids, prediction-error plotting, and post-cross-validation estimation using either the minimizing or one-standard-deviation rule.

  • Cross-validation methods: cvlasso implements both K-fold and h-step-ahead rolling cross-validation.The rolling procedure allows the forecasting horizon, initial training-sample size, and training-window size to be specified.
  • Tuning grids: With alpha() supplied as a numlist, cvlasso cross-validates over λr and αm combinations.The options support λr for r = 1, . . . , R and αm for m = 1, . . . , M.
  • Diagnostics and estimation: plotcv plots estimated mean-squared prediction error against λr, while saveest() stores intermediate lasso2 results.Internally, cvlasso repeatedly calls lasso2.
  • Post-cross-validation selection: After cross-validation, lopt selects the λr minimizing mean-squared prediction error, whereas lse selects the largest λr within one standard deviation of that minimum.Both options cause lasso2 to estimate the corresponding post-cross-validation model.

6.3 rlasso: Rigorous penalization

rlasso implements theory-driven penalization for lasso and square-root lasso, including settings with heteroskedastic, clustered, and non-Gaussian errors. Unlike lasso2 and cvlasso, it estimates the penalty level λ iteratively and supports alternative loading, testing, and prediction options.

  • Core method: rlasso implements theory-driven penalization for lasso and square-root lasso under heteroskedastic, cluster-dependent, and non-Gaussian errors.Unlike lasso2 and cvlasso, rlasso estimates λ using iterative algorithms.
  • Penalty loadings: robust and cluster(varname) make penalty loadings account for heteroskedasticity and clustering, while center centers moments in robust loadings.lassopsi instead uses lasso or square-root lasso residuals to obtain penalty loadings; post-estimation OLS is the default.
  • Penalty selection: xdependent selects the X-dependent penalty level, using 5,000 simulations by default; lalternative provides a less sharp penalty level.The rigorous penalty defaults are γ = 1/log(n) and c(1.1).
  • Inference: supscore reports a sup-score significance test, with a conservative critical value at the default 5% significance level.testonly runs the sup-score test without lasso estimation, and ssnumsim(integer) defaults to 500 multiplier-bootstrap simulations.

7 Demonstrations

The demonstrations apply lasso2, cvlasso, and rlasso to cross-section and time-series examples, including house-price prediction with the Boston Housing Dataset. The lasso2 example traces model selection across penalty levels and selects lambda=16.21799867742649 using EBIC.

  • The section demonstrates lasso2, cvlasso, and rlasso using one cross-section example and one time-series example.
  • 7.1 Cross-section example: The cross-section demonstration uses 506 observations and 14 predictors from the Boston Housing Dataset to predict house prices.
  • 7.1 Cross-section example: The lasso2 coefficient path adds predictors as the penalty decreases, reaching R-sq 0.6956 after nine model additions.
  • 7.1 Cross-section example: At the largest penalty level, 6858.99, the lasso model includes only the constant.
  • 7.1 Cross-section example: lambda=16.21799867742649 is selected by EBIC in the lasso2 demonstration.

Lasso

The section demonstrates lasso model selection using post-estimation OLS, cross-validation, information criteria, and rigorous penalization, including applications to time-series lag selection. It also explains how tuning choices and rolling-validation settings affect selected models.

  • Lasso estimation: lassopack reports separate lasso and post-estimation OLS results, with OLS applied to the model selected by lasso.
  • K-fold cross-validation: cvlasso reports λr, its value, estimated mean squared prediction error, and its standard deviation, marking λlopt and λlse in the output.λlopt minimizes MSPE, while λlse is the largest λ within one standard error of the minimum.
  • Rigorous penalization: rlasso estimates penalty levels and loadings iteratively, reports the selected model directly, and supports a sup-score joint-significance test.The reported sup-score statistic is 16.59, compared with a 5% critical value of 3.18 and a bootstrap p-value of 0.000.

8 Monte Carlo Simulation

The Monte Carlo simulations compare penalization-tuning approaches for in-sample fit, out-of-sample prediction, model selection, and sparsity under sparse designs with p = 100 and p = 220. Results show that information criteria and rigorous methods differ substantially in overfitting, selection accuracy, and computational cost.

  • Simulation design: Simulations use n = 200, exact sparsity with s = 20, correlated Gaussian predictors, and noise levels σ = {0.5, 1, 2, 3, 5}.The design reports results for p = 100 and p = 220, with βj = 1{j ≤s}.
  • Results for p = 100: For p = 100, AIC and stepwise regression falsely include around 18-20 predictors on average and combine the lowest RMSE with among the worst out-of-sample prediction.The authors interpret this pattern as over-fitting; stepwise regression is reported only for p = 100.
  • Results for p = 100: For p = 100, rigorous lasso, rigorous square-root lasso, and EBIC exhibit overall the lowest false-positive performance, while BIC-type criteria outperform AICc in selection.EBIC is expected to gain relative advantage over BIC as p increases relative to n, whereas 5-fold cross-validation behaves similarly to AICc.
  • High-dimensional results: For p = 220, EBIC and rigorous methods keep false-positive frequency below 1 across all noise levels, with zero false-negative rate when σ is 1 or smaller.AIC includes between 164 and 195 predictors on average, and its RMSPE exceeds RMSE by a factor of 5 or more.
  • Computational cost: rlasso with X-independent penalty is the fastest method, while increasing p from 100 to 220 raises the computational cost of lasso and square-root lasso only slightly.For p = 100, lasso and square-root lasso take 0.1s and 0.4s, respectively; X-dependent rigorous lasso takes 5.9s for p = 100.

9 Technical notes

The technical implementation relies on coordinate descent, with standardization and warm starts supporting estimation across penalized regression methods. lassopack also provides equivalent ways to retain theoretically important predictors without penalization, including options for partialling out and fixed effects.

  • Coordinate descent: lassopack implements elastic net and square-root lasso using coordinate descent algorithms that cycle through predictors and update one coefficient at a time until convergence.The approach uses univariate regression of current partial residuals under standardized predictors and unit penalty loadings.
  • Coordinate descent: The soft-thresholding operator sets some coefficient estimates equal to zero during coordinate descent.For square-root lasso, the coordinate descent algorithm is detailed in Belloni et al. (2014b, Supplementary Material).
  • Standardization: lassopack standardizes regressors on the fly by default because it is faster, while prestd can improve numerical precision or stability in difficult problems.The prestd option incurs typically small additional computation time, and unitloadings avoids standardizing the data.
  • Unpenalized predictors: Zero-penalization and partialling out are numerically equivalent ways to ensure selected predictors remain in the model without penalization.The equivalence holds for lasso, ridge, and elastic net, although practical numerical results can differ because the calculation methods differ.
  • Fixed effects: The fe option implements fixed effects through a within transformation that is computationally faster and more accurate than specifying unpenalized panel-specific dummies.The transformation removes the constant and fixed effects, while the panel variable is taken from xtset.

About the authors

The authors hold research and academic positions in economics, econometrics, and statistics across institutions in Ireland, the UK, Germany, and the United States.

  • Achim Ahrens is a post-doctoral research fellow at The Economic and Social Research Institute in Dublin, Ireland.
  • Mark E. Schaffer is Professor of Economics at Heriot-Watt University and a research fellow at CEPR and IZA.His affiliations are in Edinburgh, London, and Bonn.
  • Christian B. Hansen is the Wallace W. Booth Professor of Econometrics and Statistics at the University of Chicago Booth School of Business.

A Additional Monte Carlo results

An alternating-sign sparse design causes model-selection performance to deteriorate sharply, with high false-negative rates and divergent selection behavior across methods. Despite this, out-of-sample prediction can remain satisfactory even when few predictors are correctly selected.

  • A Additional Monte Carlo results: The additional design alternates nonzero coefficients between +1 and -1, with s = 20 and p = 100.The sparse parameter vector is defined as β_j = (−1)^j·1{j ≤ s}.
  • A Additional Monte Carlo results: Model-selection performance deteriorates drastically relative to the base specification, and false-negative rates are high across all methods.These results are reported for the alternating-coefficient design.
  • A Additional Monte Carlo results: When σ is 2 or larger, BIC-type information criteria and rigorous methods often select no variables, whereas AIC and stepwise regression tend to overselect.The methods therefore exhibit contrasting underselection and overselection behavior.
  • A Additional Monte Carlo results: 9.0% above the RMSPE of the oracle estimator is the RMSPE of cross-validation at σ = 2, measuring 2.3 compared to 2.11 despite only 4.2 correctly selected predictors.Thus, satisfactory prediction can be achieved without perfect model selection.
Loading 1901.05397v1…