PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimation Stability with Cross Validation (ESCV)

Chinghway Lim, Bin Yu

arXiv:1303.3128v2stat.MEstat.ML

TL;DR

Selecting a regularization parameter in high-dimensional settings is difficult, and prediction-oriented CV does not necessarily optimize estimation or variable selection. ESCV combines CV with estimation stability to select a solution. The authors report improved model selection with comparable prediction performance, including smaller models on both real data sets.

  • Problem

    Finding a regularization parameter with practical performance guarantees is difficult, and optimizing prediction with CV does not necessarily yield strong parameter-estimation or variable-selection performance.

    Source: §1.2 · p. 3

  • Method

    ESCV combines CV with estimation stability to select a solution from the regularization path based on stability and fit.

    Source: §2.1 · p. 7, §4 · p. 26

  • Results

    ESCV models were less than half the size of CV models on both real data sets while preserving prediction accuracy and outperforming CV in model selection.

    Source: §4 · pp. 26, 28

  • Takeaways & Limitations

    The smaller ESCV models from neuroscience and cell biology were more parsimonious and corroborated by subject knowledge.

    Source: §4 · p. 28

    Extending ESCV to classification through generalized linear models is left for future work.

    Source: §4 · p. 28

Abstract

from arXiv · show

Cross-validation (CV) is often used to select the regularization parameter in high dimensional problems. However, when applied to the sparse modeling method Lasso, CV leads to models that are unstable in high-dimensions, and consequently not suited for reliable interpretation. In this paper, we propose a model-free criterion ESCV based on a new estimation stability (ES) metric and CV. Our proposed ESCV finds a locally ES-optimal model smaller than the CV choice so that the it fits the data and also enjoys estimation stability property. We demonstrate that ESCV is an effective alternative to CV at a similar easily parallelizable computational cost. In particular, we compare the two approaches with respect to several performance measures when applied to the Lasso on both simulated and real data sets. For dependent predictors common in practice, our main finding is that, ESCV cuts down false positive rates often by a large margin, while sacrificing little of true positive rates. ESCV usually outperforms CV in terms of parameter estimation while giving similar performance as CV in terms of prediction. For the two real data sets from neuroscience and cell biology, the models found by ESCV are less than half of the model sizes by CV. Judged based on subject knowledge, they are more plausible than those by CV as well. We also discuss some regularization parameter alignment issues that come up in both approaches.

1 Introduction

Regularization methods provide solution paths indexed by a parameter, but choosing that parameter in high dimensions remains difficult. ESCV combines estimation stability with cross-validation (CV) to select a more regularized solution, aiming to balance fit and stability while improving on CV in model selection and often parameter estimation.

1.2 Selecting the Regularization Parameter λ

  • Regularization methods yield families of estimators indexed by a parameter, but theoretical guarantees that a suitable λ exists offer little practical guidance for finding it.Traditional AIC/BIC variants rely on parametric assumptions and asymptotic results, which may limit performance in finite samples.Source: §1.1 · p. 2, §1.2 · p. 3

1.1 Regularization Methods

  • The Lasso and ridge add L1 and L2 penalties, respectively, while early-stopping methods index their solution paths by iteration count; the paper uses λ as a general parameter notation.The paper focuses on the Lasso while suggesting its method may apply more broadly to regularization methods.Source: §1.1 · p. 2

1.2 Selecting the Regularization Parameter λ

  • CV and bootstrap assess candidate solutions through resampling and prediction error, but optimizing prediction does not necessarily optimize parameter-estimation or variable-selection performance.The paper notes the practical importance of these other performance measures.Source: §1.2 · p. 3

1.3 Estimation Stability

  • Estimation stability describes how close estimates remain across samples; it is related to variance under L2 distance but can be considered using other metrics.The paper motivates stability because an estimate that varies considerably from sample to sample is not meaningful.Source: §1.3 · p. 4
  • ESCV combines a new estimation-stability metric with CV, selecting a local minimum of ES(λ) smaller than the CV choice; its computational cost is similar and parallelizable.ES(λ) is the reciprocal of a test statistic for the null that the regression function is zero, standardized by an approximate delete-d jackknife standard error based on CV pseudo-data.Source: §1.3 · pp. 4–5

1.4 Goal for ESCV

  • Choosing a regularization parameter is difficult in practice: CV can predict well but perform poorly in model selection, while BIC can perform poorly under high noise.The paper compares ESCV, CV, and (extended) BIC on simulated and real Lasso data using estimation, prediction, and model-selection measures.Source: §1.4 · p. 5
  • ESCV is presented as a compromise across criteria: it substantially improves model selection over CV and often improves parameter estimation for correlated predictors while retaining comparable prediction error.The authors note that one solution is unlikely to be optimal on every performance measure.Source: §1.4 · p. 5
  • The paper distinguishes its estimation-focused stability measure from prior stability methods focused on model selection, and adds no further tuning parameters.Unlike some related stability procedures, CV and ESCV commonly use 5–10 model-fitting runs rather than hundreds or thousands.Source: §1.4 · pp. 5–6

2 Methodology

The Lasso supplies a path of coefficient estimates indexed by λ, from which the method selects a solution based on estimation stability and fit. CV folds provide pseudo-data sets for generating the pseudo-solutions needed to assess stability.

2.1 Lasso and Pseudo Solutions

  • The Lasso solution path is a family of coefficient estimates indexed by λ, and the selection task is to choose one solution from that path.The paper seeks a choice informed by both estimation stability and fit.Source: §2.1 · pp. 6–7
  • To assess estimation stability, the method needs multiple solution paths, which it obtains by perturbing the existing data rather than collecting additional samples.Each pseudo-data set is derived from the observed data.Source: §2.1 · p. 7
  • In the described setup, the data are divided into V folds, with each pseudo-data set formed by leaving out one fold before obtaining pseudo-solutions.The pseudo-solutions are generated for k = 1 through V.Source: §2.1 · p. 7

2.2 Alignment

Solution paths must be aligned meaningfully across resampled data, especially when n < p and predictors are correlated. The paper reports poorer CV performance with L1-fraction alignment and uses λ alignment for ESCV.

2.2 Alignment

  • When n < p, the unregularized solution's L1-norm can vary substantially across samples, making L1-fraction a poor alignment index, especially with correlated features.In the stated simulation, the upper decile of bootstrap maximum L1-norms exceeds the lower decile by over 20%.Source: §2.2 · pp. 7–8, Figure 1 · p. 8
  • L1-fraction alignment performs worst for cross-validation in the reported comparison; the table compares λ, L1-norm, and L1-fraction alignment.The table summarizes 1,000 simulations of the base-case Gaussian setup; the prose reports the relative performance comparison for low-noise scenarios.Source: §2.2 · p. 8, Table 1 · p. 9
  • The authors use λ alignment because its performance is similar to L1-norm alignment and λ is the canonical Lasso parameterization.They also note this choice permits use of glmnet for faster Lasso computation.Source: §2.2 · p. 8

2.3 Convergence of Pseudo Solutions

The section measures similarity among pseudo-solutions through fitted values and uses their changing dispersion along the solution path to identify convergence. Because the variance dip can be unclear and can favor small-norm solutions, normalization is needed.

2.3 Convergence of Pseudo Solutions

  • Pairwise coefficient distances can vary too wildly under dependent predictors, so the method compares fitted values through Xβ instead.The estimates are combined linearly through Xβ to estimate the regression function.Source: §2.3 · p. 9
  • The sample-variance metric is proportional to the average pairwise squared Euclidean distance between the pseudo-solutions’ fitted values.The metric assesses stability by comparing the V estimates at each λ.Source: §2.3 · p. 9
  • A dip in average pairwise fitted-value distance marks pseudo-solutions that converge before diverging again along the solution path.The section treats this dip as a heuristic for choosing the regularization parameter associated with a potentially correct amount of regularization.Source: §2.3 · pp. 9–10
  • The sample-variance metric’s global minimum can occur when solutions are close to zero, where agreement is trivial.The section instead seeks a dip after an initial decline, rather than simply the global minimum.Source: §2.3 · p. 11
  • The dip is not always clear, and raw sample variance can skew selection toward smaller-norm solutions because larger solutions naturally vary more.The section motivates normalization to account for this dependence on solution size.Source: §2.3 · p. 11

2.4 Hypothesis Testing and the Estimation Stability Metric

The estimation stability (ES) metric normalizes sample variance by the squared mean size of the fitted solution. This preserves a clear variance dip and can introduce a local minimum when the raw metric’s dip is muted.

2.4 Hypothesis Testing and the Estimation Stability Metric

  • ES is the normalized sample-variance metric, making dispersion relative to the squared mean size of the corresponding solution.The normalization addresses the effect of solution size on raw variance.Source: §2.4 · p. 11
  • The ES metric preserves the sample-variance local minimum in one example and introduces a pronounced minimum in the example without one.The figure compares ES with the original sample-variance metric.Source: §2.4 · p. 11, Figure 3 · p. 12
  • Unlike the related measure of Yuan and Yang (2005), ES is normalized by solution size rather than estimated noise and selects one solution from many λ candidates.Yuan and Yang’s measure is used for model averaging across a small number of models.Source: §2.4 · p. 11
  • The reciprocal of ES has test-statistic form; selecting λ by ES is equivalent to choosing a locally minimal normalized variance, without a specified theoretical distribution.The analogy treats each fitted solution as a test of whether its fit differs from the null fit E(Y)=0.Source: §2.4 · p. 12

2.5 ESCV : Incorporating Cross-Validation

ESCV uses cross-validation to constrain ES selection, addressing spurious local minima while retaining a smaller model than the CV choice. Its computation is similar in cost to CV and is easily parallelizable.

2.5 ESCV : Incorporating Cross-Validation

  • ESCV selects a local ES minimum only when it gives a smaller solution than the CV choice, guarding against spurious minima when Y bears no relation to X.The criterion also follows CV’s trivial solution when the signal is absent or overwhelmed by noise.Source: §2.5 · pp. 12–13
  • ESCV’s grid search has computational cost similar to CV, and both methods are easily parallelizable.The bulk of the computation is in computing the multiple solution paths.Source: §2.5 · p. 13
  • When ES has a meaningful local minimum, CV is likely to overselect and ESCV behaves like ES; when signal is absent or overwhelmed, ESCV follows CV to the trivial solution.The authors state that this uses the CV choice at negligible additional computation cost.Source: §2.5 · p. 13

2.6 ESCV the Method

ESCV chooses a local minimum of ES subject to a smaller solution than the CV choice, selecting the minimum ES among eligible local minima. The method assumes no intercept, or that data are centered first.

2.6 ESCV the Method

  • The paper describes ES as a metric for measuring estimation stability.Source: §2.6 · p. 13
  • ESCV chooses a local minimum of ES that gives a smaller solution than the CV choice.Among multiple local minima, it selects the one with the smallest ES value.Source: §2.6 · p. 13
  • If no local minimum exists, ESCV drops that condition and minimizes ES over λ values at least as large as λ_CV.The passage states this as the rare case where the set of local minima is empty.Source: §2.6 · p. 14
  • The method assumes a linear model without an intercept; if that assumption is unreasonable, the data should first be centered.Source: §2.6 · p. 14

2.7 Discussion on ESCV

ESCV assesses stability of fitted values rather than coefficient estimates, addressing finite-sample ambiguity among correlated predictors. The discussion reports robustness to reasonable perturbation choices and computation comparable to CV.

2.7 Discussion on ESCV

  • Finite-sample correlated predictors can produce different feature combinations with approximately equivalent fits, motivating stability assessment on fitted values rather than coefficients.The paper notes that coefficient-based metrics can therefore be statistically unstable.Source: §2.7 · p. 14
  • Because ESCV and CV select from the same Lasso solution path, both remain exposed to failures of the original Lasso.Source: §2.7 · p. 14
  • Reasonable perturbation schemes preserve ESCV’s convergence effect and yield a reasonable pick across performance metrics.The paper considers CV folds, bootstrap sampling, penalty perturbations, and direct data noise.Source: §2.7 · pp. 14–15
  • Selecting the regularization parameter with ES costs about as much computation as CV because both rely mainly on solution paths from V perturbed datasets.The paper notes that V can be small, unlike related approaches requiring a much larger V.Source: §2.7 · p. 15

2.8 Discussion on Choice of V in CV

The paper finds that increasing V from 2 to 20 reduces solution variance for ESCV, with the reduction asymptoting quickly. Smaller V reduces CV over-selection in simulations, but does not eliminate it.

2.8 Discussion on Choice of V in CV

  • For ESCV, increasing V from 2 to 20 reduces final-solution variance, with the reduction asymptoting quickly.The paper reports the same effect for CV and notes that it concerns variance of the final solution, not the pseudo-data fits.Source: §2.8 · p. 15
  • In simulations, CV overselects less with smaller V, but still overselects; cited work also indicates fixed-V CV is inconsistent.The paper presents results using V = 10 for both ESCV and CV.Source: §2.8 · p. 15
  • The paper uses V = 10 for its Section 3 results and compares CV choices of V with ESCV in the fMRI example.Source: §2.8 · p. 15

3 Experimental Results for Lasso

The experiments compare ESCV with CV and BIC for Lasso across simulations and two real data sets, assessing estimation, prediction, and model selection. Across these settings, ESCV often improves estimation and selection while retaining comparable prediction, with smaller models in the real-data examples.

3 Experimental Results for Lasso

  • The evaluation applies Lasso to simulated and real data, comparing ESCV and CV on estimation, prediction, and model-selection measures using shared data splits and solution paths.The real-data examples are from neuroscience and bioinformatics.Source: §3 · p. 16

3.1 Gaussian Simulation

  • The Gaussian simulations vary predictor correlation, signal-to-noise conditions, dimension, and correlation structure, with performance aggregated over 1,000 repetitions.The base case uses n = 100 and p = 300, varying ρ and σ.Source: §3 · p. 16, §3.1 · p. 17, §3.1.1 · p. 17

3.1.1 A Base Case

  • ESCV often reduces false positives with limited loss of true positives, and outperforms CV in estimation and model selection while prediction is often comparable.In the base case, ESCV has lower estimation error once predictors are dependent; extended BIC performs well in model selection but poorly in estimation and prediction, especially as noise rises.Source: §3.1.1 · pp. 17–18, Table 2 · p. 19

3.1.2 Effect Of Ambient Dimension

  • With p = 50 and p = 500, CV does well in prediction, especially for independent predictors, while ESCV performs better in dependent settings for estimation and model selection.The comparison keeps n = 100 and the number of nonzero coefficients at 10 while changing the number of irrelevant features.Source: §3.1.2 · p. 18

3.1.3 Other Correlation Structures

  • Across block-correlated and Toeplitz designs, ESCV gains on CV in estimation as correlation rises and usually has higher F-measure, with lower false-positive rates and relatively little loss in true-positive rates.CV almost always outperforms ESCV in prediction, though their predictive performance can be close when predictors are correlated.Source: §3.1.3 · p. 21

3.2 fMRI Data

  • On fMRI data, ESCV selects less than half as many features as CV across voxels, with little or no loss in prediction; its selected features cluster while CV-only features scatter.The authors interpret the extra CV-selected features as likely not meaningful given the expected localized visual receptive field.Source: Table 4 · p. 22, §3.2 · pp. 22–23

3.2.1 A Comparison with CV for other choices of V

  • Even with V = 2, CV model sizes generally exceed ESCV’s, while V = 5 and V = 10 yield relatively little change in model size.The exception is the 100 voxels with the poorest predictive performance.Source: §3.2.1 · p. 25, Table 5 · p. 25

3.3 Cytokine Data

  • For cytokine data, CV selects the most features for every cytokine, whereas ESCV selects about half as many as PCR and usually retains PCR-selected features that survive in the minimal model.ESCV selects the same or slightly more features than the minimal model and has fewer false positives relative to it; the minimal model uses additional independent data.Source: §3.3 · p. 26

4 Conclusion

ESCV is presented as an effective alternative to CV for Lasso regularization-parameter selection, with comparable prediction and stronger performance on other statistical measures. In the real-data examples, its models were much sparser and corroborated by subject knowledge.

4 Conclusion

  • ESCV delivers comparable prediction to CV while outperforming it in parameter estimation and model selection for dependent predictors.The conclusion describes this as an overall advantage in the practical setting of dependent predictors.Source: §4 · pp. 26, 28
  • On cytokine data, ESCV selected the pathways chosen by the minimal model in all but one case, with much smaller models.The figure caption identifies the minimal model as the ground truth for this comparison.Source: Figure 6 · p. 27
  • ESCV models were less than half the size of CV models in both real datasets, preserved prediction accuracy, and were corroborated by subject knowledge.The datasets are from neuroscience and cell biology.Source: §4 · p. 28
  • Extending ESCV to classification through generalized linear models is left for future work.Source: §4 · p. 28

Supplementary Materials

Supplementary materials provide R code for the simulations and data for the problems described in Sections 3.2 and 3.3.

Supplementary Materials

Related papers

Loading 1303.3128v2…