Source-linked AI summary
Robustness of Random Forest-based gene selection methods
Miron B. Kursa
TL;DR
Microarray gene selection is difficult because p ≫n data contain redundant features and can produce unstable false associations, while minimal-optimal and all-relevant selection pursue different goals. The paper compares four Random Forest-based methods using repeated bootstrap validation and self-consistency analysis. Post-selection accuracy was equivalent across methods, whereas Boruta selected the most genes and achieved the highest self-consistency; Random Ferns reduced its computational demands to comparable levels.
Problem
Microarray data contain many more genes than measurements, redundant features, and random false associations, while minimal-optimal and all-relevant selection require different approaches.
Method
The study compares four Random Forest-based selection methods using 30 bootstrap replications, post-selection accuracy, self-consistency stability analysis, and Random Ferns importance scores when possible.
Results
Post-selection classification accuracy was effectively equivalent across methods, while Boruta found the most potentially important genes and the highest ratio of self-consistent selections.
Takeaways & Limitations
Selection stability and the number of consistently selected genes provide more informative evidence than post-selection classifier accuracy alone, with Boruta producing promising candidates for further exploration.
Takeaways & Limitations
The tests used four standard preprocessed microarray datasets without additional temporal, gene-ontology, or microarray-calibration information.
Abstract
from arXiv · showhide
Gene selection is an important part of microarray data analysis because it provides information that can lead to a better mechanistic understanding of an investigated phenomenon. At the same time, gene selection is very difficult because of the noisy nature of microarray data. As a consequence, gene selection is often performed with machine learning methods. The Random Forest method is particularly well suited for this purpose. In this work, four state-of-the-art Random Forest-based feature selection methods were compared in a gene selection context. The analysis focused on the stability of selection because, although it is necessary for determining the significance of results, it is often ignored in similar studies. The comparison of post-selection accuracy in the validation of Random Forest classifiers revealed that all investigated methods were equivalent in this context. However, the methods substantially differed with respect to the number of selected genes and the stability of selection. Of the analysed methods, the Boruta algorithm predicted the most genes as potentially important. The post-selection classifier error rate, which is a frequently used measure, was found to be a potentially deceptive measure of gene selection quality. When the number of consistently selected genes was considered, the Boruta algorithm was clearly the best. Although it was also the most computationally intensive method, the Boruta algorithm's computational demands could be reduced to levels comparable to those of other algorithms by replacing the Random Forest importance with a comparable measure from Random Ferns (a similar but simplified classifier). Despite their design assumptions, the minimal optimal selection methods, were found to select a high fraction of false positives.
1 Background
Microarray gene selection must address extremely high dimensionality, redundant genes, and instability while distinguishing minimal-optimal from all-relevant goals. Random Forest-based methods are attractive because they support noisy p ≫n data and provide feature-importance measures, but this study validates selections through repeated resampling and stability analysis.
- Microarray gene-selection challenges: p ≫n microarray data create stability, statistical-power, and feasibility challenges while containing many redundant genes.The number of genes is much larger than the number of measurements, and measured genes are rarely targeted to one decision.
- Microarray gene-selection challenges: Feature selection reduces dimensionality while preserving a direct relationship between selected features and genes.This direct relationship is why dimensionality reduction by feature selection is commonly called gene selection in microarray analysis.
- Selection objectives: Minimal-optimal selection seeks the smallest classifier-accurate subset, whereas all-relevant selection seeks every feature relevant to the analyzed phenomenon.These aims require different approaches and tools.
- Selection objectives: In p ≫n data, random false associations can displace relevant genes during minimal-optimal selection, lowering stability and recall.Minimal-optimal methods may also favor genes with particular expression-level characteristics because they optimize a classifier.
- Random Forest rationale: Random Forest is promising for gene selection because it handles p ≫n data and noise, needs little tuning, and produces interaction-aware feature importance.Its importance measure reflects weak and multivariate interactions used by the model.
- Study design: Four Random Forest-based approaches were analyzed: RF-ACE and Boruta as all-relevant methods, and RFE and RRF as minimal-optimal methods.The study also re-evaluated methods with Random Forest importance measures and Random Ferns scores when possible.
- Study design: Thirty bootstrap replications supported validation beyond post-selection accuracy by enabling a self-consistency-based assessment of selection stability and robustness.The datasets were four standard preprocessed microarray datasets, without additional temporal, ontology, or calibration information.
Results and Discussion
Post-selection error rates made the four methods appear broadly equivalent, but selection stability and gene-set size revealed substantial differences. Boruta produced the strongest self-consistency, while Random Ferns substantially reduced its computational cost.
- Post-selection classification accuracy: Post-selection errors were nearly indistinguishable across methods except for RRF, whose SRBCT and Prostate results were not significantly worse than the best method.Validation models used genes selected in each bootstrap iteration and were tested on held-out objects.
- Post-selection classification accuracy: Post-selection error has too little resolution and too much variability to reliably assess gene-selection quality.RFE on Colon produced error values spanning almost 50%, from random guesses to perfect classification.
- Self-consistency: Genes selected in more bootstrap iterations than expected by chance were classified as significantly self-consistent selections (SCSs).SCSs were derived by comparing selections across 30 bootstrap iterations.
- Self-consistency: RF-ACE selected 62%–99% of genes, but SCSs were negligible for Colon and SRBCT and reached only approximately 20% for Leukemia and Prostate.The authors interpret this pattern as a large number of false positives overwhelming the signal.
- Self-consistency: Boruta produced the highest number of SCSs, covering 56–64% of selections in the best cases and approximately 55% on average.Its SCS performance remained stable across importance sources, although Random Ferns increased speed by 20–200 times.
2 Conclusions
The investigated methods were equivalent by post-selection classification accuracy, but differed substantially in selection size, stability, and computational cost. Boruta provided the strongest stability-related results, while Random Ferns reduced its computational burden.
- All investigated methods were effectively equivalent in post-selection classification accuracy.The study therefore cautions that classifier accuracy alone may be deceptive or inconclusive for assessing gene-selection quality.
- Boruta identified the most potentially important genes and achieved the highest ratio of self-consistent selections.The biological relevance of all newly identified genes remains unknown, but the selections were considered promising candidates for further study.
- Random Ferns importance reduced Boruta’s running time to levels comparable with other methods without changing selection quality.Boruta remained computationally intensive in its default configuration.
- RFE and RRF selected much smaller gene subsets than the all-relevant methods but still generated substantial false positives.Their similar selection stability indicates that selecting fewer genes did not eliminate the effects of p ≫n data.
3 Methods
The study compared Random Forest-based gene-selection algorithms and importance sources on four microarray datasets using repeated bootstrap analyses. Selection consistency, validation error, and runtime were evaluated under standardized computational conditions.
- Feature selection algorithms: RF-ACE and Boruta use permuted shadow features to assess whether real-feature importance exceeds randomized baselines.RF-ACE compares mean real-feature importance with mean shadow importance using Student’s t-tests, whereas Boruta compares features with the best shadow using sequential significance tests.
- Feature selection algorithms: RFE iteratively removes the least important features until four remain, returning the feature set with minimal bootstrap-validation error.The implementation used a 50000-tree Random Forest and reduced feature counts to the highest lower power of 2 at each step.
- Feature selection algorithms: RRF regularizes tree growth by penalizing features not previously used, favoring information that is not redundant with existing splits.The features used in the resulting ensemble form the selected subset.
- Importance sources: The study evaluated three Random Forest importance measures—RF Gini, RF Raw, and RF Norm—and a Random Ferns importance score.RF Gini is based on impurity decrease; RF Raw uses out-of-bag accuracy changes after permutation; RF Norm standardizes the raw importance.
- Importance sources: Random Ferns is a simplified Random Forest variant using fixed-depth ferns, shared split criteria by level, probability vectors, and randomized splits.Its implementation provides an importance score based on out-of-bag probability differences, evaluated here across fern depths 1–7.
- Experimental design: Large ensembles were required so randomly scanned features would have equal participation opportunities and produce stable importance scores.All runtime comparisons used homogeneous single-threaded hardware and software conditions.
- Experimental design: Four standard preprocessed datasets—Colon, Leukemia, SRBCT, and Prostate—were analyzed using 30 bootstrap resamples per dataset.Each selection method was run on every resample; significant selection consistency was assessed against a binomial null model with Holm–Bonferroni correction, followed by validation-error and runtime measurements.