Source-linked AI summary
Gene selection with guided regularized random forest
Houtao Deng, George Runger
TL;DR
RRF can select weakly relevant features when small tree nodes give many variables identical information gains. The paper proposes GRRF, which guides RRF with ordinary RF importance scores, and reports robust, compact, and competitive feature selection across gene datasets.
Problem
RRF evaluates features using few node-level instances, so many variables can share information gains and weakly relevant features may be selected.
Method
GRRF uses importance scores aggregated by an ordinary RF to guide the regularized feature selection process of RRF.
Results
GRRF is computationally efficient, selects compact subsets, and has competitive accuracy performance; GRRF selects 0.75 irrelevant or redundant variables on average in simulated datasets.
Takeaways & Limitations
RF applied to the least regularized RRF subset produced better accuracy than using all features on most considered datasets, despite selecting fewer features.
Takeaways & Limitations
RRF and GRRF are designed for feature selection rather than classification, and their correlated trees may yield higher classifier variance than RF.
Abstract
from arXiv · showhide
The regularized random forest (RRF) was recently proposed for feature selection by building only one ensemble. In RRF the features are evaluated on a part of the training data at each tree node. We derive an upper bound for the number of distinct Gini information gain values in a node, and show that many features can share the same information gain at a node with a small number of instances and a large number of features. Therefore, in a node with a small number of instances, RRF is likely to select a feature not strongly relevant. Here an enhanced RRF, referred to as the guided RRF (GRRF), is proposed. In GRRF, the importance scores from an ordinary random forest (RF) are used to guide the feature selection process in RRF. Experiments on 10 gene data sets show that the accuracy performance of GRRF is, in general, more robust than RRF when their parameters change. GRRF is computationally efficient, can select compact feature subsets, and has competitive accuracy performance, compared to RRF, varSelRF and LASSO logistic regression (with evaluations from an RF classifier). Also, RF applied to the features selected by RRF with the minimal regularization outperforms RF applied to all the features for most of the data sets considered here. Therefore, if accuracy is considered more important than the size of the feature subset, RRF with the minimal regularization may be considered. We use the accuracy performance of RF, a strong classifier, to evaluate feature selection methods, and illustrate that weak classifiers are less capable of capturing the information contained in a feature subset. Both RRF and GRRF were implemented in the "RRF" R package available at CRAN, the official R package archive.
1. Introduction
Feature selection seeks compact, interpretable subsets, but existing methods face limitations from restricted association modeling, linear assumptions, iterative computation, or greedy node-level evaluation. GRRF addresses a node-level feature evaluation issue in RRF by using ordinary RF importance scores to guide selection.
- Feature selection selects a compact subset of original variables while retaining predictive information and can be more interpretable than feature extraction.
- Pairwise information-theoretic measures do not capture high-order interactions such as Y = XOR(X1, X2).
- LASSO logistic regression and linear SVM-RFE assume linear relationships and may require preprocessing such as binary encoding or normalization.
- varSelRF can eliminate useful low-importance features when removing a fraction of variables from a large feature set.
- RRF uses one ensemble for feature selection, but evaluates features on node-specific training subsets and may therefore be greedy.
- GRRF uses ordinary RF importance scores to guide RRF selection after analyzing feature evaluation problems at nodes with few training instances.
2. Background
Random forests select node splits using Gini information gain and aggregate split-based importance across trees. RRF regularizes this gain to favor compact feature subsets while retaining previously selected features more readily.
- Random forest: At each node, RF evaluates mtry randomly selected features and splits on the feature with maximum Gini information gain.
- Variable importance scores: RF variable importance aggregates contributions from the nodes split by each variable across the forest and evaluates feature contributions for class prediction.
- Regularized random forest: RRF applies regularized information gain and adds a new feature to F only when it contributes enough predictive information relative to selected features.
- Regularized random forest: With λ = 1, RRF uses minimum regularization, but a new feature must still be more informative at a node than features already selected.
- Feature selection procedure: RRF visits forest nodes sequentially, adding splitting-variable indices to F as features are selected across trees.
3. The Node Sparsity Issue
Node sparsity arises because recursive splitting leaves few instances at later tree nodes, limiting the number of distinguishable Gini information gains. This can cause RRF to choose less relevant or redundant features when many variables tie.
- Node sparsity commonly occurs in tree-based models because recursive splitting decreases the number of instances reaching each node.
- Gini information gain calculated from few node instances may not distinguish features with different predictive information.
- For binary classification, the number of distinct information gain values is bounded above by N(N + 2)/4 − 1.
- With 10 instances, at most 29 distinct Gini information gain values exist, so among 1000 genes at least 971 share an information gain with other genes.
- The upper-bound analysis motivates using an additional metric to distinguish tied features during RRF selection.
- RRF randomly selects among features tied for maximum regularized gain, so small nodes may produce less relevant or redundant selections.
4. Guided Regularized Random Forest
GRRF guides RRF feature selection with importance scores from a preliminary ordinary RF, assigning feature-specific penalties rather than one shared coefficient. Its regularization is controlled primarily through γ, while larger RF importance yields less penalization.
- GRRF uses importance scores from a preliminary RF to guide RRF feature selection and may address node sparsity through aggregation across trees and training data.The RF scores are aggregated from all trees using all training data.
- GRRF assigns each feature its own penalty coefficient instead of applying the same coefficient to every feature.
- Each feature coefficient λi is calculated from that feature's importance score in an ordinary RF.The coefficient satisfies λi ∈ (0, 1].
- The base coefficient λ0 controls regularization, while γ controls the weight assigned to normalized importance; RRF is GRRF with γ = 0.For fixed λ0 and γ, more important features receive larger λi values and are penalized less.
- The feature-subset size can be controlled by λ0 or γ, but changing γ often improves classification accuracy, so λ0 is fixed at 1.With λ0 = 1, γ is the only GRRF parameter considered.
- For features below the maximum importance, increasing γ decreases λi and increases their penalty when they were not previously used in a node.GRRF with γ = 0 is equivalent to RRF(1), which uses minimal regularization.
5. Experiments
Experiments on simulated and 10 gene-expression data sets evaluate GRRF, RRF, varSelRF, and LASSO using RF and C4.5 classifiers. GRRF selects compact, stronger feature subsets with competitive accuracy, while parameter changes control subset size and affect error rates differently.
- Simulated Data Sets: GRRF selected only 0.75 irrelevant or redundant variables on average, compared with 4.95 for varSelRF and RRF in the simulated experiment.The methods identified almost all feature groups, while LASSO identified an average of 4 groups.
- Gene Data Sets: RRF(1) used many fewer features than all features and improved RF accuracy on 7 of 10 gene data sets, with significant differences on 4.The least-regularized subset may still exceed 200 features for the breast.2.class data set.
- Feature selection and classification: GRRF(0.1) outperformed GRRF(0.2) with RF on 9 of 10 data sets, including 5 significant differences, while all selected subsets contained fewer than 80 features.GRRF(0.1) selected more features than GRRF(0.2), but remained reasonably compact.
- Feature selection and classification: GRRF(0.1) with RF outperformed varSelRF with RF on 7 data sets, including 6 significant differences, and outperformed LASSO logistic regression with RF on 7 data sets, including 3.Tree models also naturally handle mixed categorical and numerical features and multiple classes.
- Parameter Sensitivity and Computational Time: Increasing λ for RRF or decreasing γ for GRRF increased selected-feature counts while generally decreasing RF error rates; GRRF error rates were reasonably robust to γ.The parameters therefore provide control over feature-subset size, with different accuracy sensitivity for RRF and GRRF.
6. Conclusions
The paper addresses node sparsity in RRF by introducing GRRF, which uses preliminary RF importance scores to guide feature selection. Experiments support GRRF’s efficiency and compactness, while emphasizing RF for evaluating selected subsets because RRF and GRRF may have higher classifier variance.
- Conclusions: GRRF uses preliminary RF importance scores to guide RRF selection when many features share the same maximal Gini information gain.This targets the node sparsity problem caused by evaluating features on few instances at a tree node.
- Conclusions: GRRF is computationally efficient, selects a small feature set, and achieves competitive accuracy performance in the reported experiments.
- Conclusions: The minimally regularized RRF subset produces better accuracy than the complete feature set, although feature selection can remove features with small importance.The paper presents this setting as preferable when accuracy matters more than subset size.
- Conclusions: RF should evaluate selected subsets because RRF and GRRF trees are correlated, potentially giving those methods higher variance when used directly as classifiers.The study therefore applies RF after feature selection for classification.