Source-linked AI summary
The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures
Anne-Claire Haury, Pierre Gestraud, Jean-Philippe Vert
TL;DR
Biomarker discovery from high-dimensional molecular data requires signatures that are accurate, stable, and interpretable, yet the relative strengths of feature-selection methods remain insufficiently studied. The paper compares 32 methods across four breast cancer gene-expression datasets and finds that simple filters generally outperform more complex methods, while ensemble selection generally adds no benefit. Overall, the Student’s t-test provides the strongest reported combination of performance and stability.
Problem
The paper addresses limited comparative evidence about the relative accuracy, stability, and interpretability of molecular feature-selection methods, a question important for biomarker discovery.
Method
The study compares 32 feature-selection methods on four public breast cancer gene-expression datasets using predictive performance, stability, and functional interpretability of resulting signatures.
Results
Simple filter methods generally outperform more complex wrapper or embedded methods, ensemble feature selection generally has no positive effect, and the Student’s t-test gives the best overall results.
Takeaways & Limitations
Feature-selection choice substantially affects molecular-signature accuracy, stability, and interpretability, with the simple t-test emerging as the strongest overall option in this comparison.
Abstract
from arXiv · showhide
Motivation: Biomarker discovery from high-dimensional data is a crucial problem with enormous applications in biology and medicine. It is also extremely challenging from a statistical viewpoint, but surprisingly few studies have investigated the relative strengths and weaknesses of the plethora of existing feature selection methods. Methods: We compare 32 feature selection methods on 4 public gene expression datasets for breast cancer prognosis, in terms of predictive performance, stability and functional interpretability of the signatures they produce. Results: We observe that the feature selection method has a significant influence on the accuracy, stability and interpretability of signatures. Simple filter methods generally outperform more complex embedded or wrapper methods, and ensemble feature selection has generally no positive effect. Overall a simple Student's t-test seems to provide the best results. Availability: Code and data are publicly available at http://cbio.ensmp.fr/~ahaury/.
1 Introduction
Biomarker discovery uses high-dimensional molecular data to support diagnosis, prognosis, patient stratification, and treatment-response prediction. The paper compares feature-selection techniques because signatures can have similar accuracy yet differ in gene overlap, stability, and biological interpretability.
- Molecular signatures use small sets of expressed genes to stratify early breast cancer patients by relapse risk.These signatures can guide decisions about adjuvant therapy.
- Restricting signatures to few genes can reduce over-fitting, improve prediction on new samples, and facilitate biological interpretation and cheaper prognostic chips.
- Published signatures share few genes, while many different signatures can achieve similar predictive accuracy and selection is sensitive to the samples used.
- The study empirically compares feature-selection techniques across accuracy and stability at both gene and functional levels using four breast cancer datasets.
2.1 Feature selection methods
The paper compares filter, wrapper, and embedded feature-selection methods that reduce gene-expression data to user-defined signatures. These methods differ in whether they rank variables individually, search jointly for predictive sets, or select variables during model training.
- Feature-selection methods take gene-expression data from two sample categories and return a user-defined set of s genes for classification.
- Filter methods: Univariate filters rank genes by relevance scores, including t-test, Wilcoxon, Bhattacharyya distance, and relative entropy.The top s ranked genes form the signature.
- Wrapper methods: Wrapper methods jointly seek predictive feature sets through searches such as SVM recursive feature elimination and greedy forward selection.RFE removes less-contributing variables iteratively, whereas GFS adds variables sequentially using internal cross-validation.
- Embedded methods: Embedded methods perform feature selection while training a learning algorithm, including Lasso and elastic net regression.
- Embedded methods: Elastic net combines ℓ1 and ℓ2 regularization to allow selection of correlated predictive variables while retaining sparsity.
2.2 Ensemble feature selection
The ensemble procedure reruns each feature-selection method on 50 bootstrap samples, ranks genes in each sample, and aggregates those rankings. Three aggregation functions determine how bootstrap ranks contribute to the final signature.
- Ensemble feature selection reruns a method on random subsamples and merges the selected variable lists to seek a more stable subset.
- The study bootstraps training samples B = 50 times and obtains one feature ranking from each resampled dataset.
- Each gene receives an aggregated score based on its rank across bootstrap experiments.
- Ensemble-mean averages ranks, ensemble-stability selection counts the percentage of samples ranking a gene in the top s, and ensemble-exponential averages an exponentially decreasing rank function.
- The final ensemble signature contains the s genes with the largest aggregated scores.
2.3 Accuracy of a signature
The study evaluates predictive accuracy by training classifiers on selected signatures and measuring their AUC. It uses both within-dataset cross-validation and cross-dataset assessment to examine signature performance.
- Accuracy is assessed by applying supervised classifiers to data restricted to the genes in each selected signature.
- The five tested classifiers are nearest centroids, k-nearest neighbors with k = 9, linear SVM with C = 1, LDA, and naive Bayes.
- Within each dataset, 10-fold cross-validation performs feature selection and classifier training on 90% of the data and computes AUC on the remaining 10%.
- Cross-dataset accuracy is assessed by estimating a signature on one dataset and evaluating it on other datasets.
2.4 Stability of a signature
Stability is assessed by comparing gene signatures estimated from samples that differ in the extent of overlap, including across independent datasets.
- Soft perturbation uses paired subsets with 80% sample overlap and averages overlap across 20 repeated samplings.
- Stability is measured as the fraction of shared genes between two signatures, |S1 ∩ S2|/s.The procedure follows the figure of merit defined by Ein-Dor et al. (2006).
- Hard perturbation uses non-overlapping subsets of size N/2 within each dataset to assess stability under stronger sample changes.
- Between-datasets stability compares signatures estimated independently on each dataset using all available samples.
2.5 Functional interpretability and stability of a signature
Functional interpretability is evaluated by testing whether selected signatures are enriched for biological processes represented by Gene Ontology terms.
- The analysis tests each of 5,830 Gene Ontology biological-process terms associated with at least one dataset gene.A hypergeometric test is applied to identify over-represented terms.
- Resulting enrichment p-values are corrected for multiple testing.
3 Data
The study uses four breast cancer gene-expression datasets addressing metastatic-relapse prediction across different cohorts.
- All four datasets use Affymetrix HG-U133A technology and concern prediction of metastatic relapse in breast cancer.
- Expression levels were estimated for 12,065 genes using EntrezGene identifiers and normalized with Robust Multi-array Average.
- The datasets were collected from Gene Expression Omnibus.
4 Results
The results compare feature-selection methods across accuracy, gene-level stability, functional interpretability, and functional stability. Filter methods generally perform strongly across these dimensions, while ensemble aggregation usually adds little benefit.
- Accuracy: 100-gene signatures are evaluated with five classifiers using 10-fold cross-validation and mean AUC across datasets.The comparison separates gene selection from classifier training to make evaluation common across methods.
- Accuracy: The t-test performs significantly better than most methods in the single-run setting, while Lasso and Elastic Net outperform some alternatives.The t-test is significant against random, entropy, Bhattacharyya, Wilcoxon, and GFS; Lasso and Elastic Net outperform GFS and Entropy.
- Accuracy: Ensemble feature selection generally does not improve accuracy; only Bhattacharyya and GFS benefit from ensemble-mean.The reported significance levels are p < 0.05 for Bhattacharyya and p < 0.1 for GFS.
- Accuracy: Fewer than 100 genes may sometimes achieve maximal performance, although 100-gene signatures significantly outperform lists of fewer than 10 features.The latter comparison is significant at p < 0.05 regardless of method or setting.
- Accuracy: Across datasets, entropy is significantly less accurate than other methods, while the t-test outperforms other filters and some wrapper methods.The t-test and SVM RFE benefit from ensemble-mean, but no method significantly benefits from ensemble-exponential or ensemble-stability selection.
- Gene-list stability: Filter methods produce more stable gene lists than other tested methods across soft-perturbation, hard-perturbation, and between-datasets settings.
- Gene-list stability: SVM-RFE benefits from ensemble aggregation under soft perturbation, but this effect vanishes under hard perturbation and between-dataset comparisons.
- Functional interpretability: Filter methods appear more interpretable than wrappers and embedded methods, while ensemble methods do not enhance interpretability.The number of significant GO terms is often zero, producing large error bars.
5 Discussion
Across the benchmark, feature-selection choice shaped accuracy and stability, with simple methods often outperforming more complex alternatives. The authors identify t-test as the strongest overall choice, while cautioning that stability estimates depend on perturbation design and sample size.
- Filter methods generally produced more stable signatures than wrapper and embedded methods.This pattern was observed for gene-level stability across soft-perturbation, hard-perturbation, and between-datasets settings.
- Batthacharyya distance and relative entropy were more stable but less accurate than random selection.They tended to select low-expression genes independently of sample labels, a behavior that pre-filtering low-variation genes can fix.
- T-test achieved the best overall performance and stability among the compared methods.Only elastic net, Lasso, and t-test clearly exceeded random selection in accuracy, while only t-test also exceeded it in stability.
- Ensemble aggregation sometimes increased stability but did not clearly improve the best methods.Ensemble-stability selection and ensemble-exponential performed better than ensemble-average, but the overall benefit remained inconsistent.
- The nearest centroids classifier achieved the best accuracy among the tested classifiers.Its lack of parameter tuning makes computation fast and less prone to overfitting.
- Soft-perturbation evaluations may produce untrustworthy stability and interpretability estimates.Hard-perturbation experiments appeared to provide the best estimate of behavior between datasets, with small sample sizes identified as a major source of instability.