Source-linked AI summary
Controlling false discoveries in high-dimensional situations: Boosting with stability selection
Benjamin Hofner, Luigi Boccuto, Markus Göker
TL;DR
High-dimensional data make variable selection difficult because there can be many more variables than observations. This paper studies stability selection combined with boosting, finding that selection probabilities can increase true positive variables while generally retaining error control, but that tight error control can reduce true-positive rates and may not benefit prediction models.
Problem
High-dimensional data with many more variables than observations make variable selection a central statistical challenge.
Method
The paper combines resampling-based stability selection with boosting to add error control to high-dimensional variable selection.
Results
Selection probabilities increase the number of true positive variables while keeping error control in most settings, including complex log-linear interaction models in the case study.
Takeaways & Limitations
Boosting with stability selection provides a flexible high-dimensional selection approach whose error bounds and tuning choices require practical interpretation.
Takeaways & Limitations
Stability selection controls the per-family error rate, and tight error control usually yields a smaller true positive rate than cross-validation.
Abstract
from arXiv · showhide
Modern biotechnologies often result in high-dimensional data sets with much more variables than observations (n $\ll$ p). These data sets pose new challenges to statistical analysis: Variable selection becomes one of the most important tasks in this setting. We assess the recently proposed flexible framework for variable selection called stability selection. By the use of resampling procedures, stability selection adds a finite sample error control to high-dimensional variable selection procedures such as Lasso or boosting. We consider the combination of boosting and stability selection and present results from a detailed simulation study that provides insights into the usefulness of this combination. Limitations are discussed and guidance on the specification and tuning of stability selection is given. The interpretation of the used error bounds is elaborated and insights for practical data analysis are given. The results will be used to detect differentially expressed phenotype measurements in patients with autism spectrum disorders. All methods are implemented in the freely available R package stabs.
1. Introduction
High-dimensional data make variable selection difficult because the number of variables can greatly exceed the number of observations. The paper examines combining stability selection with boosting to add error control while retaining flexible modeling, with an autism-spectrum-disorder case study.
- Motivation: High-throughput experiments often have far more variables than observations, making variable selection a central analytical task.The setting includes genomics, transcriptomics, proteomics, metabolomics, metabonomics, and phenomics.
- Motivation: Traditional stepwise regression is known to be unstable even when relatively few variables are available.This motivates methods designed for high-dimensional and more stable selection.
- Existing approaches: Penalized regression methods such as lasso, elastic net, and boosting, together with tree-based methods such as random forests, address high-dimensional selection.These approaches are presented as alternatives for settings with n ≪ p.
- Paper focus: Stability selection uses data resampling and can be combined with many selection procedures, particularly in high-dimensional settings.The paper studies its combination with component-wise functional gradient descent boosting.
- Paper focus: Boosting permits more flexible specification of competing effects subject to selection than the commonly used lasso combination.The paper evaluates this combination through an empirical study and an autism-spectrum-disorder case study.
- Case study: The case study compares autism spectrum disorder patients with healthy controls to identify differentially expressed phenotype measurements and differing amino acid pathways.The authors state that the methods are used to assess which amino acid pathways differ between the groups.
2. A Short Introduction to Boosting
Boosting minimizes an empirical loss by repeatedly fitting the variable that best explains the current residuals and updating the model incrementally. Early stopping determines the selected model, while linear or smooth base-learners allow different effect specifications.
- Loss and objective: Boosting targets an expected loss function, but in practice optimizes the empirical loss over observed outcomes and covariates.The loss function depends on the fitting problem, including squared loss for Gaussian regression and negative log-likelihood for generalized linear models.
- Boosting algorithm: Component-wise functional gradient descent boosting can handle arbitrary loss functions and has intrinsic variable selection properties.It is introduced as the algorithm used to optimize the empirical objective.
- Boosting algorithm: Starting from a constant model, boosting computes loss-gradient residuals and fits each variable separately to those residuals.The residuals are evaluated at the fit from the previous iteration.
- Boosting algorithm: At each iteration, only the variable that best describes the residuals is updated by adding a small fraction of its fitted effect.The update fraction is denoted ν, with ν = 10% given as an example.
- Stopping and effects: The procedure recomputes residuals and iterates until a fixed stopping iteration, after which the final model sums the fitted component models.Early stopping selects variables by ending boosting after an appropriate number of iterations, usually determined by cross-validation.
- Base-learners: Boosting can use linear base-learners or smooth penalized base-learners, allowing linear and smooth effects for the covariates.The choice of base-learner changes how each variable is fitted to the gradient residuals.
3. Stability Selection
Stability selection combines resampling with high-dimensional variable-selection procedures to identify stable signal variables while controlling false selections. With boosting, it repeatedly fits models on half-samples, aggregates selection frequencies, and uses thresholds and error bounds whose interpretation depends on assumptions and design choices.
- Stability selection addresses high-dimensional variable selection by controlling the expected number of falsely selected noise variables while identifying signal variables.
- Boosting-based stability selection repeatedly fits boosting models on random subsets of size floor(n/2), selecting q base-learners per run.The procedure repeats this process B times and computes each variable's relative selection frequency.
- Choice of parameters: Variables selected in at least πthr of fitted models are retained as stable, with πthr generally chosen above 0.5 and often within (0.6, 0.9).The number q should be large enough to allow all signal variables to enter the stable set; otherwise only a subset may be selected.
- Error control: The original PFER bound depends on q, p, and πthr and requires exchangeability among noise variables plus selection better than random guessing.Exchangeability means noise variables should have equal selection probabilities, while the second assumption requires signal variables to be selected more often than noise variables.
- Limitations and interpretation: The half-sample size is essential for the stated error bound, and PFER control is conservative because FDR ≤ FWER ≤ PFER.Using other subsample sizes would require a different error-bound derivation; PFER can therefore be stricter than the other error rates.
- Improved stability selection: Complementary-pair stability selection provides worst-case and tighter error bounds, including bounds that do not require the original exchangeability and selection assumptions.The modified bounds instead control the expected number of selected variables with low selection probability, which can coincide with false-positive control under suitable conditions.
4. Empirical Evaluation
The simulation study evaluates boosting with stability selection across high-dimensional data settings, correlation structures, error bounds, and tuning choices. Error control was generally effective, while power and false-positive behavior depended on sample size, correlation, assumptions, and tuning.
- Simulation design: The study varied predictor dimensionality, sample size, influential-variable count, predictor correlation, stability-selection assumptions, PFERmax, and threshold settings.Simulations used p ∈ {100, 500, 1000}, n ∈ {50, 100, 500}, independent or Toeplitz predictors, three error bounds, πthr ∈ {0.6, 0.75, 0.9}, and PFERmax ∈ {0.05, 1, 2, 5}.
- True-positive rates: The true positive rate increased with stronger assumptions for the error bound and with larger sample size, but decreased as the number of influential variables increased.With n = 50, TPR was generally very small; effects were more extreme for uncorrelated predictors.
- True-positive rates: TPR was lower for correlated predictors, and threshold dependence was stronger under correlation.Threshold choice mattered less when it was sufficiently large to select enough variables but not so large that each run selected too many variables.
- False positives: The false-positive rate was generally well controlled, although the less conservative bounds showed some violations in median settings.Violations occurred in 1.2% (4 cases) under unimodality and 4.0% (13 cases) under r-concavity; the standard bound E1 was especially conservative.
- False positives: Average false positives increased with PFERmax and stronger distributional assumptions, while increasing n reduced their average but increased variability.The reported PFER bounds corresponded to per-comparison error rates between 0.05 and 0.00005.
- Tuning trade-offs: Larger thresholds reduced false positives, but both large and small q values could yield low false-positive counts, making TPR preferable when error control is maintained.The authors note that stability selection is conservative because it controls the PFER.
5. Case Study: Differential Phenotype Expression for ASD patients versus controls
The case study applies boosting and stability selection to phenotype measurements from ASD patients and controls to identify differentially expressed amino-acid pathways. The analysis selected several amino acids consistently and indicated abnormal metabolism of large amino acids in ASD cells.
- Data and model: The data comprised 35 biological replicates and 384 wells, yielding 13,439 observations after one missing value, with 27 amino-acid occurrence annotations.The measurements came from Phenotype Microarrays and represented relative absorbance per well.
- Data and model: The model used group, amino-acid, interaction, and replicate-specific effects to assess differential phenotype expression between ASD patients and controls.The interaction coefficients represented deviations of group effects from the global effect for each amino acid.
- Selection results: Tryptophan, leucine, and arginine each had a 100% selection frequency, while valine was selected in 97% of models and became stable under the unimodality assumption.Without assumptions, only amino acids with 100% selection frequency were considered stable.
- Phenotypic findings: The stability-selection analysis confirmed previously reported abnormal tryptophan metabolism in ASD cells.The selected amino-acid findings were concordant with additional patient and metabolic-panel observations.
- Phenotypic findings: ASD patients showed decreased utilization of tryptophan (−0.273), tyrosine (−0.135), and valine (−0.054), and increased utilization of arginine (+0.084) and leucine (+0.081).The authors interpreted these findings as suggesting abnormal metabolism of large amino acids, potentially related to impaired transport across the cellular membrane.
6. Discussion
The discussion presents stability selection with boosting as a high-dimensional variable-selection framework that adds error control, while emphasizing trade-offs, assumptions, and practical tuning choices.
- Method and scope: Stability selection extends boosting and other high-dimensional selection procedures by adding error control for falsely selected noise variables.The framework is intended for settings with more predictors than observations and can also be combined with complex learners.
- Method and scope: Assumptions about the distribution of simultaneous selection probabilities can increase true positives while maintaining error control in most settings.The discussion considers no additional assumption, unimodality, and r-concavity conditions for computing the error bound.
- Applications and extensions: Complex log-linear interaction models and other complex learners can be used with stability selection, including in the autism-spectrum-disorder case study.The paper also notes compatibility with generalized additive and structured additive models.
- Limitations and tuning: Stability selection controls the per-family error rate, which can be very conservative compared with less stringent error-control choices.The discussion places PFERmax = α and PFERmax = mα at opposite extremes and suggests intermediate choices may be sensible.
- Limitations and tuning: When PFERmax is small, true positive rates are usually lower than in cross-validated prediction models without stability selection, and prediction accuracy can suffer.The paper stresses that prediction and variable selection are different goals.
- Implementation: In the R implementation, users apply stability selection to boosting models by specifying two of PFER, cutoff, and q, after which the missing parameter is computed.The stabs package supports extensions to other fitting approaches through functions returning selected-variable indices.
A. Additional Figures
The additional figures summarize simulation true-positive and false-positive behavior across sample size, signal complexity, cutoff values, predictor correlation, PFER, and error-bound assumptions.
- True-positive rates: True positive rates are displayed across observation counts, predictor-correlation settings, and assumptions used to compute the error bound.Each boxplot observation averages 50 simulation replicates, with open red circles marking average true positive rates.
- True-positive rates: True positive rates are also stratified by the number of influential variables, predictor correlation, and error-bound assumptions.The plots use boxplots based on averages over 50 simulation replicates.
- True-positive rates: Cutoff-value figures compare true positive rates across πthr, predictor-correlation settings, and assumptions used for the error bound.The open red circles represent average true positive rates over 50 simulation replicates.
- True-positive rates: When q exceeds the number of influential variables, scatter plots show true positive rates by signal count, correlation setting, and error-bound assumption, with smoothed group trends.Shaded confidence regions accompany the group-specific smoothers.
- False positives: False positives are summarized across observation counts, predictor-correlation settings, PFER values, and assumptions used to compute the error bound.Each boxplot observation averages 50 simulation replicates, and open red circles mark average false positives.
- False positives: Additional false-positive boxplots stratify results by influential-variable count and cutoff value alongside correlation settings, PFER, and error-bound assumptions.The figures report averages over 50 simulation replicates.
- False positives: For settings where q exceeds the number of influential variables, false positives are plotted against signal count, correlation setting, and error-bound assumptions.The scatter plot includes a smoother for each group.