lassopack: Model selection and prediction with regularized regression in Stata

• lassopack aims to fill Stata’s machine-learning gap with easy-to-use, flexible methods for regularized regression.Source: §1 · p. 1
• Primarily predictive rather than causal, regularized regression can aid causal estimation via post-double-selection controls (Belloni et al. 2014a) and lasso/ridge instrument selection (Belloni et al. 2012; Carrasco 2012; Hansen and Kozbur 2014).Its coefficients are not generally causal, and inference on them is complicated; the cited selection approaches are implemented in the sister package pdslasso.Source: §1 · p. 2
• Shrinkage trades bias for lower predictor variance, often improving out-of-sample prediction over OLS; ℓ1 methods except ridge also produce sparse models, including when p>n if truth is sparse or approximable.Source: §1 · p. 2
• lassopack offers cross-validation, information criteria, and rigorous penalization; their computational cost, assumption robustness, and overfitting priorities differ, so suitability depends on data and analytic purpose.The article discusses their theoretical foundations and compares their relative performance in Monte Carlo results.Source: §1 · p. 3

2 Regularized regression

• In high-dimensional linear models, p may exceed n; the methods are introduced under exact sparsity, with s ≪ n nonzero predictors, an assumption later relaxed.Source: §2 · pp. 3–4

2.2 Choice of the tuning parameters

• λ controls overall penalization and α balances ℓ1 and ℓ2; lassopack offers information criteria, cross-validation, and theory-driven penalties, with the choice depending on goals and data setting.Source: §2 · p. 4, §2.2 · pp. 8–9

Lasso

• Lasso’s ℓ1 penalty can set coefficients to zero for model selection and interpretation, may improve prediction over OLS, and yields a piecewise-linear coefficient path from OLS to an empty model as λ rises.Source: Lasso · pp. 4–5

Ridge regression

• Ridge does not select variables or require sparsity, making it attractive for dense signals; when p/n approaches a finite constant as p and n grow, it tends to outperform sparsity-based methods.Source: Ridge regression · p. 5

Elastic net

• Elastic net mixes ℓ1 and ℓ2 penalties, allowing sparse solutions that retain or drop correlated predictors jointly, unlike lasso’s typical selection of one variable per correlated group.Source: Elastic net · p. 7

Adaptive lasso

• Adaptive lasso uses data-dependent penalty loadings and can achieve variable-selection consistency under weaker fixed-p assumptions; square-root lasso’s score-based penalty is independent of unknown error variance under homoskedasticity.Source: Adaptive lasso · pp. 7–8

Post-estimation OLS

• Post-estimation OLS refits the variables selected in a first step to alleviate attenuation bias; with theory-driven penalization, post-lasso performs at least as well as lasso under mild additional assumptions.Source: Post-estimation OLS · p. 8

3 Tuning parameter selection using information criteria

• Regularized regression reduces model selection to choosing λ, or λ and α for elastic net, making information criteria practical once coefficient estimates are available.By contrast, evaluating every possible model can be infeasible; with 20 predictors there are more than 1 million specifications.Source: §3 · p. 9
• Information criteria differ in their aims: AIC is loss efficient but model-selection inconsistent, whereas BIC is consistent if the true model is among the candidates but is not loss efficient.The choice depends on whether the analysis prioritizes prediction or identifying the true model.Source: §3 · pp. 9–10
• Both AIC and BIC tend to select too many variables when p is large relative to n.AICc corrects AIC's small-sample bias and is favored when n is small or the data are high-dimensional.Source: §3 · p. 10
• BIC can over-select variables when p is large because equal prior probability for each model gives greater combined probability to model sizes with more possible specifications.For p = 1000, there are 1000 models with one nonzero parameter and 1000 × 999/2 models with two.Source: §3 · p. 10
• Extended BIC adds a model-size penalty designed to offset that shortcoming, and Chen and Chen (2008) report that it outperforms traditional BIC in simulations when p is large relative to n.Their simulations found traditional BIC had a higher false discovery rate; ξ controls the additional penalty.Source: §3 · p. 10

4.1 K-fold cross-validation

• K-fold CV selects λ and α by minimizing estimated MSPE; the largest λ within one standard deviation of the minimum is an alternative that yields a more parsimonious model.Each fold is held out once while the model is fit on the remaining folds.Source: §4.1 · pp. 11–12
• K-fold CV can be costly because it repeats estimation across each grid point and fold, and training-data transformations must exclude validation data.Re-centering and re-standardizing, or recalculating penalty loadings, should use only the training data; leave-one-out CV for linear models has a closed-form MSPE that requires only one model estimate.Source: §4.1 · p. 12, §4.2 · p. 12
• K values between 5 and 10 are often recommended, but K-fold CV is ill-suited to selecting the true model because its training sample is approximately n/K.For model identification, theory requires a small training sample and an evaluation sample close to n, a tension called the cross-validation paradox (Shao 1993, 1997; Yang 2006).Source: §4.1 · p. 12

4.2 Cross-validation with time-series data

• Serial dependence violates the independence principle behind CV, although standard K-fold CV can remain appropriate in certain circumstances; rolling h-step-ahead CV uses expanding or fixed training windows.The passage notes that Bergmeir et al. (2018) show validity for pure autoregressive models under an assumption, but the supplied text truncates that condition.Source: §4.2 · pp. 12, 14, Figure 3 · p. 13, Figure 4 · p. 13

4.3 Comparison with information criteria

• AIC and leave-one-out CV are asymptotically equivalent for fixed p, while CV adapts better when information-criteria assumptions such as homoskedasticity fail; BIC and EBIC better suit true-model identification.Information criteria require estimating the model once and are computationally more attractive; BIC and EBIC are model-selection consistent under strong but well-understood conditions.Source: §4.3 · p. 14

5 Rigorous penalization

• Rigorous penalization grounds penalty choices in theory to ensure consistent prediction and parameter estimation, supporting causal-inference methods such as IV-Lasso, PDS, and CHS.These methods are implemented in the sister package pdslasso.Source: §5 · p. 14

5.1 Theory of the lasso

• Lasso consistency relies on approximate sparsity, a restricted sparse eigenvalue condition, and penalty choices that make the regularization event hold with high probability.Approximate sparsity allows a high-dimensional dictionary to approximate the target function using a small number of nonzero coefficients, even when some true coefficients are small rather than exactly zero.Source: §5.1 · pp. 15–16

5.2 Rigorous lasso

• Under the SEC and ASM conditions, penalty levels and loadings estimated by Algorithms A or B yield near-oracle prediction and parameter-estimation rates for lasso and post-lasso.The extra log(p ∨ n) term in the prediction rate reflects the cost of not knowing the true model; asymptotic penalty choices are provided for homoskedastic and heteroskedastic errors.Source: §5.2 · pp. 17–19

5.3 Rigorous square-root lasso

• For i.i.d. errors, the rigorous square-root lasso’s optimal penalty is independent of the noise level σ, making it a one-step estimator under homoskedasticity.Under heteroskedasticity, its penalty loadings require estimation, but remain pivotal with respect to the error term up to scale.Source: §5.3 · pp. 19–20

5.4 Rigorous penalization with panel data

• For clustered data, the rigorous lasso aggregates observations within clusters into super-observations treated as independent, extending the approach to balanced and unbalanced panels.The penalty level uses the heteroskedastic-case form, with the default γ based on the number of clusters rather than observations.Source: §5.4 · pp. 20–21

5.5 X-dependent lambda

• The X-dependent penalty estimates the conditional score-maximum distribution by simulation and uses its (1 − γ)-quantile given observed X, rather than the asymptotic X-independent upper bound.The X-independent choice is the rlasso default; the X-dependent alternative is introduced as a sharper choice.Source: §5.1 · pp. 16–17, §5.5 · p. 21
• X-dependent penalties adapt to regressor correlation and are sharper than the baseline choice (Belloni and Chernozhukov 2011).For heteroskedasticity they use a multiplier bootstrap; the cluster-robust version defines super-observations, with the normal multiplier varying across but not within clusters.Source: §5.5 · p. 22, Table 2 · p. 22
• Square-root lasso X-dependent penalties use quantiles of simulated Λ distributions for homoskedastic, heteroskedastic, and clustered cases.The corresponding definitions are given in Table 2.Source: §5.5 · p. 22, Table 2 · p. 22

5.6 Significance testing with the rigorous lasso

• The sup-score test assesses joint regressor significance, following Belloni et al. (2012) and Chernozhukov et al. (2013, Appendix M).Its p-value is obtained by a multiplier bootstrap; the procedure is valid under homoskedasticity and heteroskedasticity but requires independence, while a cluster-robust version accommodates within-group dependence.Source: §5.6 · p. 23
• rlasso reports conservative sup-score critical values using CV = cΦ−1(1 −γSS/(2p)); defaults are c = 1.1 and γSS = 0.05.Because simulated p-values can be computationally intensive, users can request only conservative critical values by setting ssnumsim(int) to zero.Source: §5.6 · p. 24

• lassopack consists of three commands: lasso2, cvlasso, and rlasso.Source: §6 · p. 24
• Each command offers an alternative method for selecting the tuning parameters λ and α.Source: §6 · p. 24
• The section presents main syntax and important options, with full syntax and options available in the help files.Some technical options are omitted for brevity.Source: §6 · p. 24

6.1 lasso2: Base command and information criteria

• lasso2 estimates adaptive lasso, elastic net, and square-root lasso for one or more penalty levels, producing a coefficient vector or path and defaulting to 100 penalty values.For a path, estimates are stored by penalty level and can be plotted; replay syntax can change display options or estimate at an information-criterion-selected penalty without rerunning the model.Source: §6.1 · pp. 24–25, Replay syntax · p. 27

Information criteria

• lasso2 calculates AIC, AICc, BIC, and EBIC, and selects the penalty minimizing the chosen criterion; lic displays the selected model, while postresults stores its estimates.The information-criterion option controls which criterion is shown in output.Source: §6.1 · p. 25, Information criteria · p. 26

Estimation methods

• The alpha parameter ranges from lasso at 1 to ridge at 0; sqrt selects square-root lasso and is incompatible with alpha, while adaptive and ols request adaptive lasso and post-estimation OLS.Adaptive lasso loadings use the absolute initial estimate raised to −θ; the initial estimate may be OLS or, when p > n, univariate OLS.Source: Estimation methods · p. 25

Options relating to lambda

• The default penalty list contains 100 positive values descending from lmax to lminratio times lmax, equally spaced on the logarithmic scale.Users may instead supply a scalar or descending list of penalty levels.Source: Options relating to lambda · pp. 25–26

Penalty loadings and standardisation

• notpen keeps specified predictors in the model without penalizing them, partial removes specified variables before estimation, and ploadings overrides default penalty loadings.prestd standardizes the outcome and predictors before estimation rather than using penalty loadings on the fly; coefficients are returned in original units by default, and fe applies a within-transformation before estimation.Source: Penalty loadings and standardisation · pp. 26–27

Prediction syntax

• Prediction can return fitted values or residuals and, for a prior multi-penalty fit, can select the penalty by value or index; linear approximation is faster but exact only for a piece-wise linear coefficient path.Source: Prediction syntax · p. 27

6.2 Cross-validation with cvlasso

• cvlasso implements K-fold and h-step-ahead rolling cross-validation, and can cross-validate over lists of λ and α values when given multiple values.Unlike lasso2, cvlasso accepts a list for alpha(); it cross-validates over λ_r and α_m when the list has more than one value.Source: §6.2 · p. 28

Replay syntax; Predict syntax

• cvlasso repeatedly calls lasso2 and can store intermediate results; replay avoids re-estimation, supports plots and fits at λ_lopt or λ_lse, and prediction syntax includes xb and residuals.plotcv plots estimated mean-squared prediction error against λ_r.Source: §6.2 · p. 28, Replay syntax · p. 29, Predict syntax · p. 29

Options for K-fold cross-validation

• K-fold CV defaults to 10 folds; users can set and save fold assignments and a random seed, while unspecified folds are generated at approximately equal sizes.foldvar() takes integer values from 1 to K.Source: Options for K-fold cross-validation · p. 28

Options for h-step ahead rolling cross-validation

• Rolling CV requires ordered data; h() sets the forecasting horizon, origin() sets the first training-sample size, and fixedwindow keeps training size constant.The default horizon is h(1).Source: Options for h-step ahead rolling… · pp. 28–29

Options for selection of lambda

• After CV, lopt estimates the model at the λ minimizing mean-squared prediction error, while lse uses the largest λ within one standard deviation of that minimum.postresults stores lasso2 estimation results in e() when used with lopt or lse.Source: Options for selection of lambda · p. 29

6.3 rlasso: Rigorous penalization

• rlasso estimates theory-driven penalty levels iteratively for lasso and square-root lasso, allowing heteroskedastic, cluster-dependent, and non-Gaussian errors.Source: §6.3 · p. 29
• Robust and cluster options adapt penalty loadings to heteroskedasticity or clustering, while center centers moments in those loadings.Source: §6.3 · p. 30
• The lassopsi option uses lasso or square-root lasso residuals to obtain penalty loadings, and post-estimation OLS is the default.Source: §6.3 · p. 30

Options relating to lambda

• For lambda selection, xdependent uses an X-dependent penalty level with 5,000 simulations by default; gamma defaults to 1/log(n), and c to 1.1.Source: Options relating to lambda · p. 30

Sup-score test

• The supscore option reports a sup-score significance test; testonly runs that test without lasso estimation, with a default test level of 0.05.Source: Sup-score test · p. 30

7.1 Cross-section

• The cross-section demonstration predicts Boston house prices from 506 observations and 14 predictors in the Boston Housing Dataset.The dataset comprises census-level characteristics and house prices.Source: §7.1 · p. 31

Estimation with lasso2

• EBIC selected the lasso model at λ=16.21799867742649.The displayed lasso path starts with a constant-only model at λ=6858.99 and ends with 14 predictors at λ=3.66043.Source: Estimation with lasso2 · pp. 32–33

Lasso

• The lasso output presents separate columns for the lasso estimator and post-estimation OLS applied to the model selected by lasso.Source: Lasso · p. 33

K-fold cross-validation with cvlasso

• cvlasso reports λ values, estimated mean squared prediction errors, and their standard deviations, marking the minimum-MSPE λ as lopt and the largest λ within one standard error as lse.The lopt and lse options estimate the model corresponding to the respective penalty.Source: K-fold cross-validation with cvlasso · p. 34

Rigorous penalization with rlasso

• The sup-score test reports statistic 16.59, p-value 0.000, and an asymptotic 5% critical value of 3.18; rlasso iteratively estimates the penalty and loadings and reports the selected model directly.The supscore option performs a joint-significance test, with its p-value obtained by multiplier bootstrap.Source: Rigorous penalization with rlasso · p. 35

7.2 Time-series data

• The time-series example uses quarterly West German consumption, investment, and income data from 1960 Q1 to 1982 Q4, and omits rlasso because its independence assumption seems too restrictive.The example demonstrates lag selection using information criteria and h-step-ahead rolling cross-validation.Source: §7.2 · pp. 35–36

Information criteria

• With lic(aicc), lasso2 estimates the AICc-selected model, which the example identifies as λ11 = 0.207.The output reports the selected penalty as lambda=.2071920751852477.Source: Information criteria · p. 36

h-step ahead rolling cross-validation

• Rolling cross-validation partitions observations into training data and validation points; its h, origin, and fixedwindow options control the forecasts and partitioning, but default settings have no particular econometric justification.The default is h(1); fixedwindow keeps the training-set size constant across steps.Source: h-step ahead rolling cross-validation · pp. 37–38

8 Monte Carlo Simulation

• The design used n=200, p=100 or 220, correlated predictors, five noise levels, and exact sparsity with 20 nonzero coefficients.All simulations used at least 1,000 iterations; the design assessed false inclusions and omissions when relevant and irrelevant regressors were correlated.Source: §8 · p. 38
• For p=100, AIC and stepwise regression had the weakest selection, while BIC-type criteria, EBIC, and rigorous methods better controlled false inclusions.AIC and stepwise regression averaged 18–20 false positives and had the lowest RMSE but among the worst RMSPE; corrected AIC improved bias and prediction. Rigorous methods, EBIC, and cross-validation had similar or better selection or prediction properties, and post-estimation OLS reduced rigorous methods’ shrinkage bias.Source: §8 · p. 39, Table 3 · p. 40
• For p=220 and n=200, EBIC and rigorous methods had fewer than 1 false positive across noise levels and zero false negatives when σ ≤ 1.BIC performed similarly to EBIC at σ = 0.5 and σ = 1, but resembled AIC at higher noise levels.Source: Table 4 · p. 41, §8 · p. 42
• X-independent rigorous lasso was fastest, while K-fold cross-validation was slowest among the compared methods.X-independent rigorous lasso and square-root lasso took 0.1s and 0.4s at p=100; X-dependent rigorous lasso took 5.9s at p=100 and 12.7s at p=220. lasso2 averaged 3.1 seconds and was slower than X-dependent rigorous lasso in the high-dimensional setup.Source: Table 5 · p. 42, §8 · p. 42

9.1 Pathwise coordinate descent algorithms

• Coordinate descent cycles over predictors, updates coefficients using partial residuals, and applies soft-thresholding that can set coefficients to zero.lasso2 initializes with the ridge estimate and uses previous estimates as warm starts when computing a coefficient path over λ values.Source: §9.1 · p. 43

9.2 Standardization

• Because penalized regression is scale-dependent, lassopack standardizes regressors on the fly by default; prestd can improve precision or stability but adds computation time.The results are theoretically equivalent; unitloadings allows estimation without standardizing the data.Source: §9.2 · pp. 43–44

9.3 Zero-penalization and partialling out

• Zero-penalization and partialling out are theoretically equivalent, but partialling out can improve numerical stability on difficult problems at the cost of computation time.lasso2 and cvlasso use notpen(varlist), while rlasso uses pnotpen(varlist); partial(varlist) is available across the three programs. For rlasso, pnotpen ensures penalty loadings match those obtained after partialling out unpenalized variables.Source: §9.3 · pp. 44–45

9.4 Treatment of the constant

• By default the constant is unpenalized, equivalent to mean-centering; partial() and fe also mean-center, making noconstant redundant and incompatible with those options.If noconstant is specified, no intercept is included, but penalty loadings are still estimated using mean-centered regressors.Source: §9.4 · p. 45

A Additional Monte Carlo results

• At σ = 2, cross-validation RMSPE was 9.0% above the oracle estimator (2.3 compared to 2.11), despite only 4.2 predictors correctly selected on average.The results illustrate that satisfactory prediction can be achieved without perfect model selection.Source: Table 6 · p. 51, A Additional Monte Carlo results · p. 52
• Compared with the base specification, model selection deteriorated drastically: false negative rates were high, and at σ ≥ 2, BIC-type and rigorous methods often selected no variables while AIC and stepwise regression tended to overselect.The supplementary design uses p = 100 and 20 nonzero coefficients alternating between +1 and −1.Source: A Additional Monte Carlo results · p. 51