Source-linked AI summary
Correlation and variable importance in random forests
Baptiste Gregorutti, Bertrand Michel, Philippe Saint-Pierre
TL;DR
Variable selection with random forests becomes difficult and unstable when predictors are highly correlated, while the effect of correlation on permutation importance lacks theoretical consensus. The paper analyzes permutation importance theoretically for an additive regression model and compares recursive with non-recursive elimination. Its simulations indicate that RFE is more efficient, with smaller prediction errors for small models, while the paper notes scope limitations and an open consistency issue.
Problem
Variable selection is difficult and unstable in high-dimensional settings with highly correlated predictors, and the effect of correlation on permutation importance lacks theoretical consensus.
Method
The paper derives permutation-importance properties for an additive regression model and compares recursive and non-recursive random-forest wrapper algorithms through simulations.
Results
RFE is more efficient than NRFE in correlated-predictor settings, with smaller prediction errors when small-size models are considered.
Takeaways & Limitations
Updating the ranking during recursive elimination reduces the effect of predictor correlation on permutation importance and supports RFE for variable selection in this setting.
Takeaways & Limitations
Updating the ranking is especially crucial near the final elimination steps, and empirical permutation-importance consistency for Breiman’s random forests remains an open problem.
Abstract
from arXiv · showhide
This paper is about variable selection with the random forests algorithm in presence of correlated predictors. In high-dimensional regression or classification frameworks, variable selection is a difficult task, that becomes even more challenging in the presence of highly correlated predictors. Firstly we provide a theoretical study of the permutation importance measure for an additive regression model. This allows us to describe how the correlation between predictors impacts the permutation importance. Our results motivate the use of the Recursive Feature Elimination (RFE) algorithm for variable selection in this context. This algorithm recursively eliminates the variables using permutation importance measure as a ranking criterion. Next various simulation experiments illustrate the efficiency of the RFE algorithm for selecting a small number of variables together with a good prediction error. Finally, this selection algorithm is tested on the Landsat Satellite data from the UCI Machine Learning Repository.
1 Introduction
The paper examines variable selection with random forests when predictors are correlated, focusing on how correlation affects permutation importance and motivates recursive elimination. It combines theoretical analysis with simulations comparing recursive and non-recursive strategies.
- Motivation: Highly correlated predictors make variable selection unstable, so small training-sample perturbations can substantially change the selected variables.This instability complicates selecting a stable subset in high-dimensional settings.
- Background: Random forests aggregate many tree-based estimators and are used for both regression and classification in high-dimensional problems.The aggregation is described as improving the bias-variance trade-off relative to a single random tree.
- Research gap: Permutation importance is widely used for variable selection, but its interpretation under correlated predictors lacks theoretical consensus.The paper identifies the absence of a theoretical description of correlation’s effect as a key gap.
- Contributions: The paper theoretically expresses permutation importance in an additive regression model as a function of predictor correlations and the number of correlated variables.The theoretical results are validated through simulation.
- Contributions: Recursive Feature Elimination updates the permutation-importance ranking after each backward-elimination step, unlike the non-recursive strategy’s fixed ranking.The comparison targets wrapper variable-selection algorithms for random forests with correlated predictors.
- Simulation evidence: Simulations comparing recursive and non-recursive strategies indicate that the recursive algorithm provides better results across correlated-data designs.The reported comparison is based on several simulated designs from the literature.
2 Random forests and variable importance measures
This section defines prediction error and permutation importance for random-forest analyses, then explains why correlated predictors require deeper study. It also notes that empirical importance depends on the estimation algorithm and that existing consistency results assume independent predictors.
- Random forests: Random forests build many fully grown trees from bootstrap samples, randomly restricting candidate splitting variables at each node, then average their predictions.The method is presented as an aggregation of tree-based estimators.
- Permutation importance: Permutation importance assesses a predictor by measuring how prediction error changes after breaking its link with the outcome.The random-forest implementation uses permutations of the predictor values in out-of-bag samples.
- Permutation importance: Permutation importance mimics replacing a predictor with an independent copy by permuting its observed values.The empirical permutation is described as the sample analogue of the theoretical construction.
- Permutation importance: The theoretical permutation importance depends only on the joint distribution of predictors and outcome, whereas empirical importance also depends strongly on the regression algorithm.Thus, consistency of the empirical measure is relative to the chosen estimator.
- Consistency: Existing consistency results for importance measures assume independent predictors, limiting their direct applicability to the paper’s correlation-focused setting.The text distinguishes consistency of random forests from consistency of permutation importance and identifies the latter as an open problem for Breiman’s algorithm.
- Motivation: The permutation-importance measure can rank or select predictors, but its dependence on correlations is especially important when predictors are highly correlated.The section investigates this dependence to clarify the measure’s properties.
3 Permutation importance measure of correlated variables
The paper develops theoretical expressions for permutation importance under correlated predictors, showing how correlation and predictor-group structure alter variable rankings. These results are supported by simulations and motivate recomputing importance during backward elimination.
- Additive regression framework: Theoretical analysis is conducted under an additive regression model, where permutation importance admits a correlation-dependent expression.The analysis assumes E[ε|X] = 0 and finite conditional error variance, with measurable component functions.
- Additive regression framework: Permutation importance equals twice the variance of the component fj(Xj) when fj(Xj) is centered.This simple expression depends strongly on the additive structure of the regression function.
- Correlation configurations: For two positively correlated predictors, the permutation importance of both variables decreases as their correlation increases.The remaining correlated predictor carries similar information after one predictor is permuted, reducing the resulting error increase.
- Correlation configurations: With multiple equally correlated predictors, increasing their number makes permutation importance decrease faster toward zero.This result is stated for the correlation matrix C = (1 − c)I_p + c11^t with equal outcome correlations.
- Correlation configurations: Independent variables can receive higher importance than less-correlated relevant variables, even when their association with the outcome is weaker.This occurs in configurations containing correlated and independent predictor groups.
- Classification: The theoretical analysis is limited in classification because the relevant importance expression cannot be easily written as a function of predictor correlations.The paper notes that the classification calculations are therefore difficult to adapt from the regression framework.
4 Wrapper algorithms for variable selection based on importance measures
The paper compares backward wrapper strategies that use random-forest permutation importance, distinguishing static ranking from recursive recomputation. Because importance depends on the remaining correlated predictors, RFE is presented as better suited to selecting compact predictive models.
- Motivation: Exhaustively evaluating all variable subsets is computationally infeasible when the number of predictors is large.Backward ranking and elimination provide a practical alternative for wrapper-based selection.
- Non Recursive Feature Elimination: NRFE computes a permutation-importance ranking once, then repeatedly trains forests and eliminates the least relevant variables using that static ranking.Its procedure ranks variables before the first elimination and does not update the ranking afterward.
- Recursive Feature Elimination: RFE recomputes permutation importance after each elimination step before removing the next least relevant variable.The algorithm repeats training, importance computation, and elimination until no variables remain.
- Comparison: Because importance depends on the predictors still present, RFE can be more reliable than NRFE when predictors are highly correlated.The ranking may change at each step as correlated variables are removed.
- Illustration: In the illustrated example, RFE selects a variable first that NRFE selects in fifth position.The example contains two correlated relevant variables, four independent less-correlated relevant variables, and six irrelevant variables.
- Comparison: RFE is reported to find small models that remain efficient in prediction by recomputing importances throughout backward elimination.The paper focuses on reducing correlation effects at the end of the backward procedure.
- Alternative importance measure: Conditional importance can perform well with few predictors but is computationally demanding for problems with several hundred predictors.It permutes variables conditionally on correlated predictors.
5 Numerical experiments
Across five experiments, the study compares RFE and NRFE under correlated predictors using empirical permutation importance, error estimates, and variable-screening behavior. RFE generally yields more efficient screening, especially when correlations are high, while empirical importance follows the theoretical results in the studied regression settings.
- Experimental framework: The experiments compare RFE and NRFE for classification and regression, using out-of-bag and independently simulated validation errors.The model error is misclassification rate for classification and mean square error for regression.
- Theoretical validation: The empirical permutation importance matches the theoretical measure as predictor correlation and the number of correlated predictors vary.This agreement is observed for both two-predictor and multi-predictor settings, although consistency has not been established theoretically for correlated predictors.
- Experiment 1: With small correlations among six relevant variables, RFE and NRFE behave similarly overall, but RFE achieves validation error 0.042 versus 0.23 for NRFE with two variables.RFE recomputes permutation importance at each elimination step, reducing ranking distortions caused by correlation.
- Experiment 3: In a 500-variable classification design with four highly correlated blocks, RFE reduces validation and out-of-bag errors faster than NRFE and reaches comparable performance with 20 variables.The design contains four blocks of 15 correlated variables, 10 independent variables, and 430 irrelevant variables.
- Experiment 4: In a difficult 100-observation, 800-variable classification problem, relevant variables may not have higher importances than other variables within their correlated groups.RFE and NRFE have similar validation-error screening, while RFE appears more efficient according to out-of-bag error.
- Overall findings: Across experiments, correlation increases empirical permutation-importance instability, and RFE tends to outperform NRFE in screening during later elimination steps.The reported comparison identifies RFE as particularly useful for reducing correlation effects near the end of variable selection.
6 Application to the Landsat Satellite dataset
The Landsat application compares RFE and NRFE on multispectral pixel data with correlated variables. RFE achieves lower errors and more stable selection, especially for small models.
- Data: The dataset contains 4,435 samples with 36 spectral variables from 3 × 3 pixel neighborhoods, used to classify the central pixel into seven classes.Variables 17–20 correspond to central-pixel spectral values, and variables sharing a spectral band form four correlated blocks.
- Evaluation: RFE and NRFE were run 100 times, with out-of-bag and validation errors averaged across runs.Each validation set contained one third of the observations.
- Results: With five variables, the OOB error was 0.13 for RFE versus 0.48 for NRFE.The reported standard deviations were 0.001 for RFE and 0.003 for NRFE.
- Results: With five variables, the validation error was 0.13 for RFE versus 0.48 for NRFE.The reported standard deviations were 0.008 for RFE and 0.01 for NRFE.
- Selection stability: RFE selected variable 19 in 54 of 100 runs, whereas NRFE never selected it and selected no variable from group G19 among its first ten.Variables 17, 18, and 20 were always selected by both algorithms.
7 Conclusion
The paper links predictor correlation to permutation-importance behavior and evaluates recursive versus non-recursive elimination. Across simulations, RFE performs better for small models, while ranking updates remain most important near the end of elimination.
- Landsat evidence: Figures 14–17 compare error estimates, selection frequencies, initial importance measures, and selected-variable groups across Landsat runs.The Landsat error curves are averaged over 100 variable-selection runs.
- Theoretical contribution: The paper derives an additive-regression expression for permutation importance that depends on predictor correlations and the number of correlated variables.The authors identify extending this analysis beyond the considered setting as a challenging problem.
- Algorithmic comparison: Across classification and regression simulations, RFE has smaller prediction errors than NRFE when small-size models are considered.The conclusion attributes this pattern to RFE reducing the effect of correlation on importance measures.
- Scope and future work: Updating the importance ranking is especially crucial in RFE’s final steps, after most irrelevant variables have been eliminated.The authors suggest combining non-recursive elimination early with recursive elimination later as future work.
A.1 Proof of Proposition 1
The proof uses independence and centering properties of the additive regression components to establish the required expectation identities.
- Expectation identities: Independence and centering make the relevant error-component expectations equal to zero.The proof applies these identities to terms involving f(X) and the permuted vector X^(j).
- Additive structure: Because the model is additive, the proof decomposes the regression function into component contributions.The argument also uses that the component variables are identically distributed.
A.2 Proof of Proposition 2
The proof specializes the preceding proposition to a multivariate normal setting. The conditional mean is linear, and the coefficient vector solves a covariance-matrix equation.
- Conclusion: Invertibility of C yields the stated solution α = C^-1τ.The proof identifies this as the expected result under the proposition’s assumptions.
- Gaussian model: The multivariate normal assumption implies that the conditional distribution of Y given X is normal with a linear conditional mean.The proof writes the conditional mean as a linear combination of the predictors.
- Covariance equation: The coefficients satisfy τ = Cα, where C is the covariance matrix.The coordinate expansion expresses each component of τ as a covariance-weighted sum of the coefficients.
A.3 Proof of Proposition 3
The proof assumes an equicorrelation matrix and constructs its inverse in the same matrix form, determining the coefficients through algebraic conditions.
- Matrix inversion: The correlation matrix is assumed to have the form C = (1 −c)I_p + c11^t.The proof then seeks an inverse representation M = aI_p + b11^t.
- Matrix inversion: The inverse is constructed as M = aI_p + b11^t, with a and b chosen to satisfy the inversion conditions.The passage states that the inverse of C can be decomposed in the same way.
- Coefficient calculation: The coefficient b is obtained as b = −c / ((1 −c)(1 −c + pc)).
- Coefficient calculation: For every j ∈ {1, …, p}, the diagonal-related quantity is expressed as j_jj = a + b.
- Conclusion: The proof concludes with an expression involving τ_0 and the denominator 1 −c + pc, then notes that the second point follows from Proposition 2.