PaperlayerView exact revision on arXiv

Source-linked AI summary

Grouped variable importance with random forests and application to multiple functional data analysis

Baptiste Gregorutti, Bertrand Michel, Philippe Saint-Pierre

arXiv:1411.4170v2stat.ME

TL;DR

The paper addresses selecting groups of predictors with random forests and develops a grouped permutation importance measure, including theoretical results for additive models. It uses wavelet-coefficient groups to select functional variables, illustrating the approach in simulations and an aviation-safety application; the normalization choice remains open for further work.

  • Problem

    Random-forest permutation importance selects individual variables, while the authors identify group selection in random forests as a problem not previously studied.

    Source: §1 · p. 1

  • Method

    The paper defines grouped permutation importance and uses it to guide a backward selection procedure over groups of wavelet coefficients representing functional predictors.

    Source: §1 · pp. 1–2, §5 · p. 17

  • Results

    Theoretical analysis gives exact decompositions for specific models such as additive regression, while simulations illustrate the measure and the aviation application obtains encouraging results.

    Source: §5 · p. 17

  • Takeaways & Limitations

    Grouped importance can be applied to selecting functional variables and was used to study flight parameters related to long-landing risk.

    Source: §1 · p. 2, §5 · p. 17

    The normalization factor for rescaled grouped importance is chosen heuristically, and selecting a good value remains an open question.

    Source: §5 · p. 17

Abstract

from arXiv · show

The selection of grouped variables using the random forest algorithm is considered. First a new importance measure adapted for groups of variables is proposed. Theoretical insights into this criterion are given for additive regression models. Second, an original method for selecting functional variables based on the grouped variable importance measure is developed. Using a wavelet basis, it is proposed to regroup all of the wavelet coefficients for a given functional variable and use a wrapper selection algorithm with these groups. Various other groupings which take advantage of the frequency and time localization of the wavelet basis are proposed. An extensive simulation study is performed to illustrate the use of the grouped importance measure in this context. The method is applied to a real life problem coming from aviation safety.

1. Introduction

The paper extends random-forest permutation importance to groups, addressing settings where variables are naturally considered together. It analyzes the measure theoretically and uses it to guide grouped feature selection, including for functional data.

1. Introduction

  • The proposed grouped permutation importance supports selecting groups in random forests, a setting the authors say had not previously been studied.Grouping may improve interpretation and prediction and stabilize selection when based on prior knowledge about the data.Source: §1 · p. 1

2.1. Decomposition of the grouped variable importance

  • In general, a group’s importance need not equal the sum of its variables’ individual importances; for additive models, the paper derives exact decompositions.With independent variables in an additive model, group importance equals the sum of individual importances; correlation removes that property.Source: §1 · p. 1, §2.1 · p. 3

2. The grouped variable importance measure

  • The group permutation importance evaluates a selected sub-vector of predictors as a group rather than assessing each variable separately.The measure is defined for a subgroup XJ and depends on which variables it contains.Source: §2 · p. 3

2.1. Decomposition of the grouped variable importance

  • For a specified additive model, group importance equals twice the variance of the group’s component function.The result is stated under the additive model assumptions in Proposition 1.Source: §2.1 · p. 3

2.2. Grouped variable importance and random forests

  • The random-forest estimate permutes a group jointly in out-of-bag samples, preserving its empirical joint distribution while breaking its links to the response and other predictors.The rescaled estimate is then used as a feature-selection criterion.Source: §2.2 · pp. 4–5

3. Multiple functional data analysis using grouped variable importance

The paper applies grouped variable importance to multiple functional predictors by representing curves with wavelet coefficients. This approach uses a common wavelet basis in the presented setup, while allowing adaptation to variable-specific bases.

3. Multiple functional data analysis using grouped variable importance

  • Functional predictors are projected onto a finite-dimensional basis, and wavelets provide coefficients localized in both frequency and time.The presented setup uses the same basis for all functional predictors, though the method could be adapted to use a specific basis for each.Source: §3 · p. 5, §3.1 · p. 6
  • A wavelet representation separates a curve into a smooth approximation at level j0 and a detail component.The covariates are assumed observed on a fine sampling grid before decomposition.Source: §3 · p. 5

3.1. Grouped variable importance for functional variables

  • The method treats the wavelet coefficients of each functional predictor as a vector that can be evaluated as a group.This provides the coefficient-based representation used to compare functional predictors through grouped importance.Source: §3.1 · p. 6

Groups of wavelet coefficients

Wavelet coefficients can be grouped according to the functional variable, frequency level, or time location. The paper notes that group size varies and recommends rescaled importance when comparing groups of different sizes.

Groups of wavelet coefficients

  • Coefficient groups can represent one functional variable, one frequency level within a variable, or a frequency level across variables.These groupings use the coefficients’ frequency and variable identities.Source: Groups of wavelet coefficients · pp. 6–7
  • Time-based groups collect wavelet coefficients active at a time point or over an interval, with activity determined by wavelet support.The size of a time-point group depends on the supports of the scaling and wavelet functions.Source: Groups of wavelet coefficients · p. 7
  • Because frequency- and time-based groups can differ in size, the paper recommends rescaled grouped importance to compensate for group size.Other possible groupings include correlated variables or coefficients in a frequency interval.Source: Groups of wavelet coefficients · p. 7

Grouped variable selection

The paper adapts recursive feature elimination to random forests using grouped variable importance, selecting nested group subsets by validation error. The groups can represent whole functional variables or wavelet levels, although time-location groups do not fit the elimination procedure easily.

Grouped variable selection

  • Grouped recursive feature elimination repeatedly trains a random forest, computes validation error and grouped importance, then removes the least important group; validation error selects among nested subsets.The procedure repeats these steps until no groups remain.Source: Grouped variable selection · p. 7
  • Grouping all coefficients for each functional variable lets the procedure identify the most relevant functional variables.The groups are G(1), ..., G(p).Source: Grouped variable selection · p. 8
  • Selecting groups G(j, u) for a fixed functional variable identifies wavelet frequency levels that yield predictive information.Source: Grouped variable selection · p. 8
  • Time-location groups overlap in their wavelet coefficients, so eliminating a whole group may be inefficient; the smooth-part coefficient is shared across times and is usually a good predictor.Source: Grouped variable selection · p. 8

3.2. Experiments

  • Section 3.2 introduces numerical experiments illustrating grouped variable importance for functional-data analysis.Source: §3.2 · p. 8

Presentation of the general simulation design

The simulations generate noisy functional covariates using wavelet-based designs, with covariates often constructed as functions of the outcome. In Experiment 1, grouped importance detected the target time interval in both designs, though detection was harder when the outcome–coefficient link was complex.

Presentation of the general simulation design

  • The general design simulates discrete functional processes as a signal depending on latent Z_i plus Gaussian noise, with Z_i correlated with Y_i and usually equal to it.Except for Simulation 2 of Experiment 1, the functional covariates are defined as functions of Y.Source: Presentation of the general simulation… · p. 8
  • In the wavelet-domain model, signal coefficients combine a shared random coefficient and a link function, with noise added through the highest signal level j⋆.Coefficients above j⋆ are set to zero; the noise standard deviation decreases with wavelet level.Source: Presentation of the general simulation… · p. 8
  • The simulated curves are inverse-transformed using a Daubechies wavelet filter with four vanishing moments, also used to project the functional observations.Source: Presentation of the general simulation… · p. 9

Experiment 1: Detection of important time intervals

  • Experiment 1 uses one functional covariate to test detection of T* under two designs in which Y is correlated with the signal on that interval.Source: Experiment 1: Detection of important… · p. 9
  • Simulation 1 links Y linearly to wavelet coefficients in the support of T*, while coefficients outside that support lack the Y term.It uses n = 1000, σ = 0.01, N = 28, and j⋆ = 7.Source: Experiment 1: Detection of important… · p. 9
  • In Simulation 2, Y measures curve oscillations over T*, while X is simulated first and Y is then generated as a function of X.The functional variable uses the general wavelet-domain design with link functions set to zero.Source: Experiment 1: Detection of important… · p. 9
  • In both Experiment 1 designs, grouped importance detected the target interval T*; estimates averaged 100 iterations, with first and third quartiles showing variability.Importance was evaluated at 50 equally spaced time points.Source: Experiment 1: Detection of important… · p. 9
  • In the second design, the complex link between Y and wavelet coefficients produced low importance estimates, making important intervals harder to detect.Source: Experiment 1: Detection of important… · p. 9

Experiment 2: Selection of wavelet levels

The experiments assess wavelet-level selection and functional-variable selection under strong predictor correlation. Recursive elimination reaches lower prediction error with fewer variables than the non-recursive strategy in the correlated setting.

Experiment 2: Selection of wavelet levels

  • The experiment groups wavelet coefficients by level and applies Algorithm 1 with rescaled grouped importance because group sizes differ.The aim is to identify wavelet levels most relevant to predicting Y.Source: Experiment 2: Selection of wavelet… · p. 10
  • Under both links, first-step importance ranks match θ_j; wavelet orthogonality makes correlation nearly null, so backward elimination adds no information over non-recursive selection.This comparison is specific to the wavelet-level experiments.Source: Experiment 2: Selection of wavelet… · p. 11
  • For linear links, MSE is minimized at four groups (five is close); for logistic links, it selects five, with G(3) selected more often.The figures show grouped-importance boxplots, MSE by number of groups, and selection frequencies for each link.Source: Experiment 2: Selection of wavelet… · p. 11, Figure 4 · p. 12, Figure 5 · p. 13

Experiment 3: Selection of functional variables in the presence of strong correlation

4. A case study: variable selection for aviation safety

The aviation case study targets prediction of landing distance and long-landing risk from flight data. It uses 23 expert-preselected functional variables from 1,868 flights, projected onto a Daubechies wavelet basis.

4. A case study: variable selection for aviation safety

  • The study focuses on long-landing risk and predicts landing distance from 512 seconds of data before touchdown.The stated aim is a sparse model with good predictive performance.Source: §4 · p. 14
  • The sample comprises 1,868 flights from one airport and company, with 23 variables preselected following aviation experts.Flight recorders can capture up to 1,000 parameters each second.Source: §4 · p. 14, Table 1 · p. 14
  • The functional variables are projected onto a Daubechies wavelet basis with four vanishing moments, chosen because the flight data contain time- and frequency-scale information.Source: §4 · p. 14

Preliminary dimension reduction

The aviation application uses grouped wavelet coefficients to select flight parameters and identify informative wavelet levels and time intervals for predicting long landings. The paper concludes with the grouped-importance method’s theoretical scope and an open question about its rescaling.

Preliminary dimension reduction

  • The 23 flight curves yield 11,776 wavelet coefficients, so the method first shrinks coefficients simultaneously across curves to reduce dimension.This avoids the large block of mostly zero coefficients that independent curve-wise shrinkage would produce.Source: Preliminary dimension reduction · pp. 14–15

Selection of flight parameters

  • The five most frequent parameters were always selected, and the first ten appeared in more than 60 of 100 runs, including several aircraft-trajectory measures.Regrouping correlated flight parameters into N1, N2, and ELEV left the ranking unchanged.Source: Selection of flight parameters · p. 15, Figure 8 · p. 17

Selection of wavelet levels

  • Gross weight selection favored the first approximation levels, whereas angle of attack drew predictive information from both approximation and detail levels.For gross weight, the last levels were selected in fewer than 40 of 100 trials; all angle-of-attack levels were selected more than 50 times.Source: Selection of wavelet levels · p. 15, Figure 9 · p. 18

Detection of important time intervals

  • For altitude, informative intervals were [−460, −400] and [−60, 0]; for angle of attack, the detected interval was [−200, −100].The paper relates the altitude intervals to leveling off before final approach and the seconds before touchdown, and the angle-of-attack interval to the flare.Source: Detection of important time intervals · pp. 15, 17, Figure 10 · p. 19

Selection of flight parameters

  • Five flight parameters were significantly relevant, and five selected variables achieved risk close to the minimum despite an average selection of ten.Figure 7 summarizes grouped importance, prediction error, and selection frequencies across repeated runs.Source: Selection of flight parameters · p. 15, Figure 7 · p. 16

5. Conclusion

  • The paper finds that grouped importance generally differs from summed individual importances, but decomposes exactly into that sum in specific models such as additive regression.It also develops wavelet-based groupings and a backward selection method for functional variables, with encouraging aviation results.Source: §5 · pp. 17, 19
  • The rescaled grouped importance uses a heuristic normalization factor, and choosing a good factor remains an open question for further mathematical work.Source: §5 · p. 17

Appendix A. Proof of Proposition 1

The supplied proof fragments refer to variables defined earlier, assume an iid condition, and use expectation identities involving conditional independence.

Appendix A. Proof of Proposition 1

  • The proof refers to J and X(J) as defined in Section 2.Source: Appendix A · p. 19
  • The proof sets expectation terms to zero using conditional expectation and independence, then applies a decomposition of f into J and non-J components.Source: Appendix A · p. 19
  • It invokes that the variables indexed by J are independent and identically distributed.Source: Appendix A · p. 19

Appendix B. Additional experiments on the grouped variable importance

The appendix compares grouped importance with summed individual importances across additive, non-additive, and sparse simulation settings. The results show that equality depends on model structure and predictor correlation.

Appendix B. Additional experiments on the grouped variable importance

  • Across experiments, each of 500 repetitions used 1,000 simulated samples to compare grouped, rescaled grouped, and summed individual importances for group sizes 1–16.The framework includes variables in W with some components related to Y and variables in Z independent of Y.Source: Appendix B · pp. 19–20
  • In additive experiments with independent predictors, grouped importance equaled summed individual importances; with correlated predictors, it was larger as group size increased.In correlated cases, summed individual importances decreased with group size; the experiments are shown in Figures B.11 and B.12.Source: Results · p. 21, Figure B.11 · p. 21, Figure B.12 · p. 22
  • With independent predictors but a non-additive regression function, grouped importance differed from summed individual importances.This result is shown for Experiment 3 in Figure B.13.Source: Results · p. 21, Figure B.13 · p. 22
  • In sparse Experiment 1c, estimated grouped and summed importances agreed; in correlated sparse Experiment 1d, the summed estimate was significantly higher.In both sparse experiments, estimated grouped importance remained unchanged as group size varied from 2 to 16.Source: Results · pp. 21, 23, Figure B.14 · p. 23

Appendix C. Curve dimension reduction with wavelets

For the flight-data analysis, the paper adapts wavelet shrinkage from one random signal to n independent random processes to reduce the dimension of the wavelet decomposition.

Appendix C. Curve dimension reduction with wavelets

  • The flight-data analysis requires preliminary dimension reduction of the wavelet decomposition for computational reasons.Source: Appendix C · p. 23
  • The paper adapts the hard-thresholding estimator of Donoho and Johnstone (1994) from one random signal to n independent random signals.Source: Appendix C · p. 23
  • The extension is designed for independent random processes, which need not be identically distributed.Source: Appendix C · p. 23

Signal denoising via wavelet shrinkage

The section extends wavelet hard-thresholding to functional observations whose mean can depend on a latent variable, and thresholds wavelet-coefficient vectors jointly. It gives an elementary consistency result while noting that exact common wavelet support is a demanding assumption.

Identically distributed case

  • For repeated noisy observations of one deterministic signal, the section estimates its wavelet coefficients by hard-thresholding, using a universal threshold proposed by Donoho and Johnstone (1994).The observations are modeled as the signal plus independent Gaussian noise.Source: Signal denoising via wavelet shrinkage · pp. 23–24

Non-identically distributed case

  • In the non-identically distributed setting, a latent variable Z represents effects on the mean signal, and supervised prediction can use Z = Y.Conditionally on Z_i = z_i, the signal mean is s(t, z_i).Source: Consistent wavelet thresholding for… · p. 24
  • For each wavelet level and location, the method thresholds the vector of coefficients jointly, retaining it only when its norm exceeds δN,n.The threshold depends on N, n, and σ.Source: Consistent wavelet thresholding for… · p. 25

Consistency result

  • For an exactly zero signal, choosing x = 2 log(N) permits recovery with high probability under the proposed thresholding bound.The text notes that x must be much larger than log(N) for this high-probability recovery.Source: Consistent wavelet thresholding for… · p. 25

Scope

  • The non-identically distributed method assumes some wavelet coefficients vanish almost surely across Z, a strong assumption that is difficult to meet in practice.It may still apply when support varies little with Z, including when a common index set approximates each conditional signal in L2.Source: Consistent wavelet thresholding for… · p. 25

Related papers

Loading 1411.4170v2…