Source-linked AI summary
Correcting Variable Importance Scored by Random Forests
Guancheng Zhou, Haiping Xu, Jason Liu, Donghui Yan
TL;DR
RF variable importance is widely useful but can be masked by correlated variables because its calculation does not account for their relationships. The paper corrects this by removing or clustering conditionally correlated variables before measuring importance, and experiments report sensible corrections across benchmark datasets. The approach can overestimate importance by removing information beyond that entailed by the variable, and Method 2 can leave some correlations insufficiently covered.
Problem
RF variable importance does not account for correlations among variables, so important variables may receive low or near-zero importance when correlated variables mask them.
Method
The paper detects conditional correlations and uses either individual correlated-variable removal or spectral clustering into groups before recomputing predictive-accuracy-based importance.
Results
Experiments on UC Irvine benchmark datasets show that both proposed options achieve the expected, sensible corrections to RF variable importance.
Takeaways & Limitations
Removing correlated-variable influence provides a correction framework motivated by RF that the paper notes also applies to general predictive methods.
Takeaways & Limitations
The individual-removal method can slightly overestimate importance by dropping predictive information beyond that entailed by the variable, while Method 2 may miss some correlations.
Abstract
from arXiv · showhide
Variable importance produced by Random Forests (RF) is used widely in statistical data analysis, and has played an important role in a variety of tasks such as assisting model interpretation, model selection and diagnosis, and cost-bounded learning etc. However, the calculation of variable importance in RF does not take into account of the correlations among variables, and variables that are correlated to many other variables tend to receive a lower importance index or being completely masked (i.e., with an importance index near zero) by other strongly correlated variables. To prevent influence from unwanted correlated variables in calculating variable importance, we propose to group variables by their conditional correlations (conditional on the response variable). We explore two computationally efficient options, with one grouping variables individually, and then separates the variable of interest from all correlated variables, while the other uses clustering to group variables according to their pair-wise conditional correlations. Our experiments show that both lead to sensible corrections to the importance of variables.
1 Introduction
Random Forests are widely used because they offer strong empirical performance and provide variable-importance measures useful for interpreting and selecting variables. However, permutation importance can be seriously distorted when correlated variables mask one another, motivating a conditional-correlation-based correction.
- Random Forests: Random Forests combine decision trees through recursive feature-based partitioning and often achieve strong predictive performance with few tuning parameters.At each node, RF samples features and selects a split that optimizes a metric such as Gini-purity improvement.
- Variable importance: RF variable importance supports model interpretation, selection, diagnosis, and cost-bounded learning by quantifying relative variable contributions.The paper emphasizes the practical value of identifying which variables matter when many variables are used.
- Permutation importance: Permutation importance breaks a feature’s association with the response by randomly permuting that feature before measuring the resulting accuracy difference.The paper considers permutation importance because it is easier to calculate and avoids some biases associated with Gini importance.
- Problem: Correlated variables can mask an important variable, producing a small permutation-importance value despite a substantial relationship with the response.If correlated variables remain available after permutation, predictive accuracy may change little, creating a discrepancy from the variable’s underlying importance.
- Proposed direction: The proposed correction detects conditional correlations, removes correlated variables during importance calculation, and evaluates the individual variable without their unwanted predictive influence.The paper presents this as a way to prevent correlated variables from contributing to predictive accuracy during assessment.
2 The method
The paper corrects Random Forest variable importance by grouping variables using conditional correlations, then computing importance after excluding correlated variables. It presents an individual-grouping method and a spectral-clustering method, with Seeds experiments illustrating their adjustments and limitations.
- Conditional correlation: Conditional correlation measures dependence between variables after conditioning on the response and forms the basis for grouping variables.The pairwise conditional correlation matrix contains each pair's conditional correlation and induces a similarity matrix for clustering.
- Method 1: Method 1 removes variables conditionally correlated with each variable, then assigns importance from the predictive-accuracy increase when that variable is added back.The remaining variables are conditionally uncorrelated with the variable of interest, reducing masking; the estimated reduction can be slightly overestimated.
- Method 2: Method 2 partitions all variables into disjoint spectral-clustering groups and measures each variable's importance after removing its entire group and adding the variable back.Variables outside the selected group are not or only weakly correlated with the variable being assessed.
- Seeds example: In the Seeds example, Method 1 substantially increases importance for V1, V2, and V7, while V3, V4, and V5 receive smaller increases.The adjustment reflects conditional correlations and the expected roles of area, perimeter, and kernel-groove length in distinguishing seed types.
- Seeds example: Method 2 also increases V1 and V2 substantially but fails to adjust V7 and provides limited adjustment for V3, V4, and V5 because spectral clusters do not capture all affinities.The authors attribute these differences to imperfect clustering, including strong affinities involving V4, V7, V1, and V2.
- Seeds example: V6 retains the same importance under all three approaches because it is nearly uncorrelated with the other variables and therefore is not masked.V6 is omitted from the conditional-correlation table for this reason.
- Removal instead of permutation: For multiple-variable exclusion, the methods remove correlated variables rather than permute them because permutation can reduce accuracy more than removal when several noise variables are present.The performance gap becomes non-negligible when permuting three or more variables.
3 Background and related work
The paper introduces spectral clustering as a similarity-based method that finds minimal graph cuts to group variables. It also situates its approach among related methods for variable-importance correction and model interpretation.
- Spectral clustering: Spectral clustering groups points using pairwise similarity rather than directly using distances.It operates on an affinity graph and seeks a minimal graph cut.
- Spectral clustering: The affinity graph represents data points as vertices connected by weighted edges encoding pairwise similarity.The resulting matrix is called the similarity or affinity matrix.
- Spectral clustering: Normalized cuts partition the graph by minimizing a cut objective over a partition of the vertices.The integer optimization is relaxed to an eigenvalue problem for the Laplacian matrix.
- Spectral clustering: The normalized-cut procedure uses the second-smallest Laplacian eigenvector and recursively rounds its components to obtain graph bipartitions.Recursion continues until a stopping criterion is met.
- Related work: Related work addresses other variable-importance biases or model interpretation, whereas this paper focuses on bias from multicollinearity.The comparison includes corrections for continuous or high-cardinality variables, SHAP, and LIME.
4 Experiments
Experiments across benchmark datasets show that correcting RF importance for conditional correlations often raises scores for variables whose original importance was near zero or understated. Method 1 and Method 2 generally agree, but clustering can miss correlations and leave some variables partially masked.
- Indian liver patients and Hearts datasets: Method 1 substantially increases the importance of several medically relevant variables that original RF scores near zero, including AlkPhos, total proteins, age, gender, and V9.These adjustments are described as more consistent with the variables’ diagnostic or biological relevance.
- Cross-dataset comparison: Method 2 produces importance values similar to Method 1 for most variables in the Indian liver patients, Hearts, Wine, and Maternal health datasets.For Maternal health, the two methods use exactly the same grouping and yield the same importance for every variable.
- Hearts dataset: Method 2 adjusts age and gender importance upward in the Hearts dataset, while V6 remains very low under both RF and Method 1 and V12 receives a higher score under Method 2.The passage attributes the V12 adjustment to its diagnostic relevance for heart disease.
- Indian liver patients dataset: Method 2 can fail to capture correlations that Method 1 detects, leaving V5, V4, and V11 partially masked in the Indian liver patients dataset.The reported missed relationships are V5–V4, V4–V1, and V11–V8.
- Wine dataset: Method 2 gives stronger upward adjustments to the Wine dataset’s top three variables, V1, V10, and V13, while preserving the same top-three ranking as Method 1.The spectral-clustering groups are reported as very similar to those from Method 1.
5 Conclusions
The paper develops two approaches to correct Random Forest variable importance by removing influences from correlated variables. Both approaches use conditional correlations, with one operating per variable and the other grouping variables by clustering.
- Two options correct Random Forest variable importance by removing influences from correlated variables.