PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Random Forests: From Theory to Practice

Gilles Louppe

arXiv:1407.7502v3stat.ML

TL;DR

This thesis addresses limited theoretical understanding, implementation guidance, and interpretability of decision trees and random forests. It analyzes their mechanisms, computational behavior, and variable-importance measures, and proposes Random Patches for large datasets and strong memory constraints. The work reports competitive accuracy with substantially lower memory requirements, with further gains under strong memory constraints, while cautioning that variable-importance magnitudes can be misleading.

  • Problem

    The thesis targets gray areas in random forests, including heuristic theory, overlooked implementation effects, and difficulty interpreting variable importances.

  • Method

    It combines theoretical analysis, algorithm and implementation study, extensive experiments, and characterization of Mean Decrease of Impurity variable importance.

  • Results

    Random Patches matches other randomization schemes in accuracy, significantly reduces per-model memory requirements, and can significantly improve accuracy under strong memory constraints.

  • Takeaways & Limitations

    Building ensembles from random subsets of both samples and features can provide competitive performance without training each model on the entire dataset.

  • Takeaways & Limitations

    Variable-importance magnitudes can be misleading when variables have different numbers of categories because binary threshold selection introduces combinatorial effects.

Abstract

from arXiv · show

Data analysis and machine learning have become an integrative part of the modern scientific methodology, offering automated procedures for the prediction of a phenomenon based on past observations, unraveling underlying patterns in data and providing insights about the problem. Yet, caution should avoid using machine learning as a black-box tool, but rather consider it as a methodology, with a rational thought process that is entirely dependent on the problem under study. In particular, the use of algorithms should ideally require a reasonable understanding of their mechanisms, properties and limitations, in order to better apprehend and interpret their results. Accordingly, the goal of this thesis is to provide an in-depth analysis of random forests, consistently calling into question each and every part of the algorithm, in order to shed new light on its learning capabilities, inner workings and interpretability. The first part of this work studies the induction of decision trees and the construction of ensembles of randomized trees, motivating their design and purpose whenever possible. Our contributions follow with an original complexity analysis of random forests, showing their good computational performance and scalability, along with an in-depth discussion of their implementation details, as contributed within Scikit-Learn. In the second part of this work, we analyse and discuss the interpretability of random forests in the eyes of variable importance measures. The core of our contributions rests in the theoretical characterization of the Mean Decrease of Impurity variable importance measure, from which we prove and derive some of its properties in the case of multiway totally randomized trees and in asymptotic conditions. In consequence of this work, our analysis demonstrates that variable importances [...].

2.2.2 Bayes model and residual error

The Bayes model defines the lowest achievable generalization error, while consistency asks whether learned models approach it as sample size grows. Because the data distribution is unknown and direct probability estimation is infeasible in high dimensions, practical learning relies on restricted model families and approximate evaluation.

  • The Bayes model has no greater generalization error than any learned model, and its residual error is irreducible noise.
  • For classification, the Bayes model predicts the most likely class given X; for squared-error regression, it predicts the conditional mean of Y.
  • Weak consistency concerns convergence of expected generalization error to the Bayes error, whereas strong consistency requires almost-sure convergence as N tends to infinity.
  • Consistency is distribution-dependent; universal strong consistency requires proof for every distribution P(X, Y).
  • Accurately estimating P(Y|X) can require a learning set that grows exponentially with the number p of input variables, motivating restricted hypothesis families.

3.4.1 Classification

Decision-tree classification estimates local class probabilities from node sample proportions and assigns each terminal node its plurality class. Although splitting continually lowers training error, unrestricted growth can overfit, making stopping and pruning essential for selecting tree complexity.

  • The plurality rule assigns each terminal node the class with the largest observed count among its training samples.
  • Using node class proportions to approximate local generalization error reduces the objective to the tree's resubstitution estimate.
  • Every non-empty split cannot increase the resubstitution estimate, so fully developed trees can achieve zero training error when leaves contain one object each.
  • Deeper trees lower training error but may capture noise, causing test error to diverge as complexity increases.
  • Stopping rules halt growth when nodes are too small, too deep, insufficiently impurity-reducing, or unable to produce adequately sized children.
  • Post-pruning usually outperforms pre-pruning for single trees, whereas pruning is not required for good generalization in tree ensembles.

3.6.1 Families Q of splitting rules

Decision-tree splitting partitions a node’s input space into non-empty child regions, then selects a restricted candidate split that maximizes impurity decrease. Common rules use binary, univariate partitions, while multiway and oblique alternatives change the search space and computational trade-offs.

  • A split partitions a node into disjoint, non-empty subsets, producing one child node per subset.
  • Because all possible partitions can be exponentially numerous, induction restricts the search to a structured candidate family Q.For binary partitions, the number is 2^N_t−1 −1 under distinct node-sample values.
  • The usual candidate family consists of binary splits defined on one variable and yielding non-empty subsets of the node samples.
  • Multiway splits create one child per category, while oblique splits allow arbitrarily oriented hyperplanes but make finding the best split more computationally intensive.
  • Impurity decrease can prefer structurally simpler splits even when training error is unchanged, as splitting on X1 yields a terminal child earlier than alternatives.
  • Standard Shannon entropy and Gini impurity are reliable but can favor unbalanced splits and variables with many possible outcomes.

3.6.3 Finding the best binary split

The best binary split is found by optimizing each variable’s candidate partitions and comparing their impurity decreases. For ordered variables, iterative statistics make exhaustive threshold evaluation linear in the number of node samples, while categorical searches require special reductions or approximations.

  • The split-search procedure finds the best binary split for each input variable and then selects the candidate with the greatest impurity decrease.
  • For ordered variables, only thresholds between consecutive observed values need consideration because thresholds within an interval induce identical sample partitions.
  • Mid-cut-point thresholds are commonly chosen because they preserve the observed partition while providing a practical generalization heuristic.
  • Updating child-node class or regression statistics as samples move across thresholds enables exhaustive evaluation in time linear in N_t.The necessary statistics are initialized once and updated iteratively between neighboring splits.
  • The ordered-variable procedure is implemented by sorting node samples, initializing statistics, and iteratively evaluating updated candidate splits.
  • Since sorting bounds the algorithm’s complexity, subsampling node samples or discretizing variables can reduce computation with little accuracy impact in randomized trees.
  • Binary classification reduces categorical search from 2^L−1 −1 to L−1 partitions by ordering categories by class probability.
  • This reduction does not extend to multiclass classification, where exhaustive search can become infeasible as the number of categories grows.

4.1.1 Regression

The section decomposes prediction error into residual noise, bias, and variance, then explains how randomized ensembles reduce variance while preserving bias under stated conditions.

  • Bias-variance decomposition: The expected generalization error decomposes into irreducible noise, squared bias from the Bayes model, and prediction variance across learning sets.Noise provides a theoretical lower bound; bias measures average-prediction discrepancy, while variance measures prediction spread.
  • Classification: For binary classification, the decomposition links variance in estimated class probabilities to the resulting misclassification error.When the true majority probability exceeds 0.5, reducing estimate variance decreases total misclassification error; excessive variance reduction can instead increase it.
  • Randomized ensembles: An ensemble of randomized models retains the individual models’ bias while reducing variance when randomization makes their predictions less correlated.As ensemble size grows, variance approaches ρ(x)σ², and improvements arise solely from variance reduction when ρ(x)<1.
  • Randomized ensembles: The ensemble principle is to introduce perturbations that decorrelate predictions, while controlling the bias increase caused by randomization.Only variance attributable to random effects can be reduced by averaging; stronger random effects permit greater variance reduction.
  • Classification: In classification, ensembling reduces class-probability-estimate variance and decreases misclassification error when the expected estimate remains above 0.5.This conclusion follows from the classification and randomized-ensemble decompositions.

4.3.1 Randomized induction algorithms

Randomized induction algorithms create diverse decision trees by perturbing split selection, variable selection, or training samples. Averaging these trees can reduce variance, but the resulting trade-off depends on the randomization strategy and its strength.

  • Bagging: Bagging approximates the average model by combining trees trained on bootstrap samples, exploiting instability so averaging can reduce variance.Bootstrap replicates contain the same number of cases as the learning set, but omit about 37% of original cases on average.
  • Variable-selection randomization: Randomized variable selection considers only K≤p randomly chosen variables at each node when finding the best split.This produces structurally different but individually good trees; K controls the bias-variance trade-off.
  • Random Subspace: Random Subspace builds each tree using a random subset of input variables selected once before tree construction and can achieve near state-of-the-art performance.The subset size controls the trade-off between randomization variance and increased bias.
  • Random Forests: Random Forests combine Bagging with random variable selection at each node, yielding an effective general-purpose method competitive with boosting and arcing algorithms.The combined randomization strategies are presented as complementary mechanisms for constructing the forest.
  • PERT: PERT avoids impurity evaluation during splitting and is often nearly as accurate as Random Forests, although its random trees are typically larger.Its simplicity also supports theoretical analysis of randomized-tree forests.
  • Trade-offs: For Random Forests, the reported optimal trade-off occurs at K=8, while for extremely randomized trees it occurs at K=10.The Random Forest result suggests Bagging and random variable selection are complementary at the tested settings.
  • Trade-offs: Smaller K strengthens random effects, lowers prediction correlation, and increases the variance reduction obtainable through averaging; extremely randomized trees are less correlated than Random Forest trees.The correlation comparison is illustrated using predictions from trees grown on the same learning set.
  • Ensemble size: Averaging reduces the variance component attributable to random effects as the number of trees increases, while learning-set variance remains constant.At the limit M→∞, ensemble variance approaches the correlation-scaled individual-model variance.

4.4.2 Variable importances

The thesis examines random-forest learning, computational complexity, consistency, and interpretability through proximity measures and variable importances. It characterizes theoretical properties of randomized trees and analyzes implementation-related performance.

  • Interpretability: Random forests provide mechanisms for assessing input-variable importance, supporting model interpretability.These measures are studied in Chapter 6 to improve understanding of random-forest models.
  • Theoretical properties: Variable importance can be analyzed through consistency results for randomized-tree ensembles and their voting combinations.Consistency is established for several randomized-tree settings, including ensembles whose base models are consistent.
  • Theoretical properties: Random-forest performance depends on the number r of relevant variables rather than the total number p of variables in the analyzed result.This result is presented as explaining robustness to many noise variables.
  • Complexity: Building a decision tree has average-case complexity Θ(KN log N) under linear split-evaluation cost C(N)=Θ(KN).The corresponding lower bound is Ω(N log^2 N) for the alternative cost model described by Theorem 5.2.
  • Complexity: Ensemble construction remains polynomial, with average complexity following best-case behavior despite quadratic worst-case dependencies on effectively used samples.The best cases are linear in K and linearithmic or quasilinear in N, whereas worst cases can be O(N^2) or O(N^2 log N).
  • Complexity: Forest prediction costs Θ(M log N) in the best case and Θ(MN) in the very worst case, while average behavior is logarithmic in N.The average-case analysis indicates that pathological leaf depths are not dominant.

5.3.1 Scikit-Learn

Scikit-Learn offers a unified, accessible machine-learning framework built around efficient numerical data representations and estimator interfaces. Its design separates parameter initialization from fitting while supporting composition and batch processing.

  • Library design: Scikit-Learn provides machine-learning algorithms, preprocessing, model selection, and workflow composition in Python.The library is intended to make established algorithms efficient, accessible to non-experts, and reusable across scientific domains.
  • Data representation: The library relies on NumPy and SciPy arrays or sparse matrices to represent datasets and target vectors.This representation supports efficient vector operations while remaining close to the standard matrix formulation.
  • Data representation: Scikit-Learn processes batches of samples to reduce Python-call and per-element dynamic-typing overhead.Online-learning algorithms instead operate on minibatches.
  • Estimator interface: The estimator interface standardizes model construction through hyper-parameters and a fit method.Initialization attaches named parameters without accessing data; fit learns model-specific parameters from training arrays.
  • Estimator interface: A common estimator object serves both as the configurable estimator and the learned model, simplifying usability and maintenance.The design avoids parallel estimator and model class hierarchies.
  • Estimator interface: Decision-tree and random-forest workflows use the same interface, requiring only a constructor change to switch algorithms.A decision-tree estimator is initialized with hyper-parameters and trained by calling fit on X_train and y_train.

5.3.2 Internal data structures

Scikit-Learn represents decision trees with compact contiguous arrays and modular induction components. The implementation emphasizes memory efficiency, cache locality, split-search costs, and alternatives to exact presorting.

  • Array representation: Decision trees use contiguous arrays rather than per-node objects to store node relationships, split metadata, impurities, sample counts, and values.The representation includes child identifiers, splitting features and thresholds, node impurities, weighted sample counts, and class or regression values.
  • Array representation: Array storage enables O(1) amortized node insertion, fewer allocations, and improved CPU-cache use during repeated node access.These properties can improve performance when fast predictions are important.
  • Induction components: Tree induction is organized into Builder, Splitter, and Criterion components that construct nodes, find splits, and evaluate split quality.The Builder recursively partitions nodes using splits supplied by the Splitter and scored by the Criterion.
  • Induction components: Depth-first, best-first, and breadth-first construction provide alternative node-processing strategies for different computational settings.Breadth-first induction can be more efficient when data access is expensive or data must be streamed from disk.
  • Split search: Randomized-tree splitters search K≤p randomly selected variables, while Random Forests and Extremely Randomized Trees use different split-search procedures.The implementation combines random variable selection with either optimized or randomized cut-point selection.
  • Split search: Presorting is not used in Scikit-Learn random forests because its O(p/K log N) comparison with the alternative does not necessarily reduce build time.Sorting sample indices by feature values drives induction complexity, while alternative memory-layout and buffering strategies improve access locality.

5.3.5 Criteria

Experiments evaluate how forest size, sample size, variable selection, noise, and bootstrap sampling affect computation and accuracy, alongside benchmarks of implementations. The chapter also develops theoretical results for MDI variable importance under asymptotic randomized-tree conditions.

  • Experimental criteria: Increasing the number M of trees raises induction time as O(M), while leaf depth remains constant and mean squared error decreases inversely with M.Extremely Randomized Trees build faster than Random Forests but have slightly deeper trees in the reported experiment.
  • Experimental criteria: Building time grows slightly faster than linearly with N, while average leaf depth grows as O(log N).The observed trends agree with the theoretical O(N log^2 N) and O(N log N) dependencies for Random Forests and Extremely Randomized Trees.
  • Experimental criteria: Adding irrelevant variables reduces forest accuracy, and Random Forests are more affected by noisy variables than Extremely Randomized Trees.The text attributes this difference to end-cut preference in classical impurity criteria.
  • Experimental criteria: For random variable selection, build time scales as O(K), while the reported lowest errors occur at K=5 for Random Forests and K=7 for Extremely Randomized Trees.Increasing K slightly reduces tree depth by enabling better split selection, after which stronger perturbation becomes less harmful.
  • Experimental criteria: Bootstrap sampling makes Random Forest tree construction about 1.5× faster and reduces average tree depth by roughly 0.5.The speedup is weaker for Extremely Randomized Trees but remains non-negligible.
  • Implementation benchmarks: Across 29 datasets, Scikit-Learn implementations are benchmarked against other libraries using fixed training, testing, tree-count, and variable-selection settings.Datasets range from N=208 to 70000 samples and p=6 to 24496 variables, with averages over 10 runs.
  • Implementation benchmarks: Scikit-Learn is fastest on average for building and predicting with Random Forests, while Extremely Randomized Trees are empirically 1.41× faster to build than Random Forests.The benchmark conclusion is qualified by selection bias from the 29 chosen datasets.
  • Variable importance: Under asymptotic conditions, MDI importances from fully developed totally randomized trees decompose information among variables, interaction degrees, and variable combinations.The sum of variable importances equals I(X1,...,Xp;Y), and irrelevant variables have zero importance without altering relevant-variable importances.

6.4.1 Generalization to other impurity measures

The framework generalizes beyond Shannon entropy by defining impurity decreases and irrelevance for generic impurity measures. Under nonnegative impurity decreases, MDI retains key relevance properties, while pruning and random subspaces preserve variable-identification behavior under stated asymptotic conditions.

  • General impurity framework: The framework defines node impurity decrease G(Y; Xj|t) and rewrites variable importance using these generic impurity reductions.At infinite sample size, node conditioning corresponds to conditioning on the variables and values along the branch.
  • Irrelevance and invariance: MDI importance is invariant to adding or removing irrelevant variables under the generalized impurity formulation.All irrelevant variables receive zero MDI importance, although relevant variables need not receive positive importance without additional nonnegativity conditions.
  • Irrelevance and invariance: When impurity decreases are nonnegative, the relevance characterization extends to the generalized impurity measure; with Shannon entropy, irrelevance is equivalent to conditional independence.The same framework applies to Gini impurity for classification and can extend to variance for regression.
  • Pruning and random subspaces: For trees pruned at depth q, MDI includes only the first q terms of the fully developed-tree decomposition.This truncation follows because branches contain at most q−1 conditioning variables.
  • Pruning and random subspaces: Pruned trees of depth q have the same asymptotic importances as fully developed randomized trees built on random subspaces containing q variables.The equivalence follows after replacing p with q and accounting for the probability that variables and conditioning variables are selected.
  • Pruning and random subspaces: Relevant variables retain strictly positive importance when q is at least the number r of relevant variables, although their values generally differ from fully grown-tree importances.This condition supports identifying relevant variables asymptotically when q is a suitable upper bound on r.

6.4.3 Non-totally randomized trees

Guided split selection changes how random-tree importances sample conditioning sets: masking favors strong variables near the root and weaker variables near the leaves. Consequently, importance can be biased and lose the clean relevance guarantees of totally randomized trees.

  • Masking effects: For K > 1, guided split selection creates masking effects because variables with larger impurity decreases prevent other variables from being selected.The resulting trees omit some conditioning sets, so importance no longer sums all mutual-information terms and may over- or underestimate relevance.
  • Masking effects: With K = 2, X1 is always selected at the root and masks X2, yielding ImpK=2(X1) = I(X1; Y) and ImpK=2(X2) = I(X2; Y|X1) = 0.Although X2 is nearly as informative as X1, its contribution is hidden because X1 makes subsequent nodes pure.
  • Consequences for interpretation: For K > 1, a relevant variable can have zero importance and become indistinguishable from an irrelevant variable.Adding an irrelevant variable can increase the probability that X2 is selected at the root, demonstrating that importance also depends on the input-variable set.
  • Consequences for interpretation: Totally randomized trees have importance equal to zero if and only if a variable is irrelevant, whereas this guarantee does not extend to Random Forests or Extremely Randomized Trees.The distinction arises from the guided structure of trees with K > 1, which excludes some conditioning sets.
  • Redundant variables: Adding a totally redundant copy decreases the original variable’s importance, while its effects on other variables combine reduced weights for some terms with increased weights for equivalent conditioning sets.The net change for another variable depends on the relative contributions of terms that include the duplicated variable.
  • Redundant variables: Random forests’ importance scores depend on the output and on the other input variables, so low importance does not necessarily mean that a variable is uninformative.Redundancy can make information conveyed by a variable appear less important.

7.2.1 Bias due to masking effects

Variable-importance bias arises from both guided masking and finite-sample impurity estimation, especially with binary splits and heterogeneous variable cardinalities. These effects can obscure relevant variables and make importance magnitudes difficult to interpret.

  • Masking effects: Tuning K > 1 can improve predictive accuracy by balancing bias and variance while simultaneously biasing variable importances through masking.Some conditioning sets are never represented, and relevant variables may receive null importance or be over- or underestimated.
  • Empirical impurity estimation: Finite-sample impurity estimates are increasingly overestimated for high-cardinality variables and for deeper nodes with fewer samples.The resulting importance bias can make variables with many categories appear more important than they are.
  • Empirical impurity estimation: For relevance = 0.1 and max_depth=1, X2’s importance is nearly 6 times larger than the other variables’ importances, but deeper trees can make irrelevant variables appear larger.Stronger signal allows deeper trees before X2 becomes indistinguishable from irrelevant variables.
  • Empirical impurity estimation: Variable selection with K = 5 adds a further cardinality-related effect, making detection of X2 as relevant more difficult in deeper trees.Limiting tree depth reduces misestimation bias, but larger-cardinality variables still appear significantly more important.
  • Binary-split effects: Binary splits introduce conditioning sets involving multiple values of the same variable, unlike multiway exhaustive splits.Threshold selection therefore controls which additional impurity terms enter the importance calculation; ETs consider all intermediate thresholds, whereas RF selects a local best threshold.
  • Binary-split effects: Classical random-forest importances remain difficult to interpret with variables of different cardinalities, although they may still identify the most relevant variables.Importance amplitudes can be misleadingly low or high because of combinatorial effects from binary splits and thresholds.

7.3.1 Feature selection

Random forests support feature selection, but variable importance is shaped by masking and interaction effects. Random-patch ensembles can preserve competitive accuracy while reducing memory requirements, with performance depending on the problem and base estimator.

  • Random-forest variable importances provide an effective basis for ranking and selecting relevant variables.Their usefulness follows from random forests' prediction performance, robustness to noise, and ability to model complex interactions.
  • Finite-sample bias from masking and impurity misestimation can make relevant variables appear less important than irrelevant ones.Suggested controls include statistically reliable stopping, artificial contrasts, and permutation tests.
  • Variable importances may capture indirect or combined effects, so they require caution when used to infer direct network interactions.The paper suggests inducing masking effects or using alternative strategies when direct interactions are the target.
  • Random-patch ensembles: Random-patch ensembles learn models from random subsets of samples and features, preserving or improving comparable accuracy while lowering memory and computing requirements.The approach avoids requiring each individual model to use the whole dataset.
  • Empirical comparison: Randomized-tree ensembles generally outperform standard decision-tree ensembles, while Random Patches remains competitive with the best performers.The study reports no strong statistical evidence that Random Patches performs worse, but also no conclusive evidence that it significantly improves performance.

8.3.2 Memory reduction, without significant loss

The study examines how to reduce random-patch sizes while preserving competitive accuracy, finding that optimal memory use is problem-specific and often substantially below the full dataset.

  • Memory allocation: Optimal patch sizes vary across datasets: some favor sampling more samples, while others favor sampling more features.Sensitivity to αs and αf is problem-specific.
  • Memory allocation: Even without memory constraints, optimal patches rarely use the whole dataset; most consume less than half the available memory.Only a couple of datasets exceed µ′ = 0.75.
  • Accuracy preservation: For more than half of the datasets, using only 10% or 20% of the original data reaches competitive accuracy without a significant decrease.The threshold µ′_max is chosen so the constrained and unconstrained average accuracies are not statistically distinguishable at α = 0.05.
  • Method comparison: RP-DT generally permits greater memory reduction than RP-ET, although RP-ET is somewhat more competitive in accuracy.The authors attribute this difference to optimized versus randomized split thresholds.

8.3.3 Memory reduction, with loss

Under severe memory constraints, the study evaluates how accuracy changes as random patches shrink and finds that balancing sample and feature subsampling can preserve strong performance.

  • Accuracy under severe constraints: With very low memory budgets, RP-based ensembles often achieve the best accuracy, although RS performs better on arcene.On the other five representative datasets, RP is equivalent or better than RS and P at low µ′_max.
  • Accuracy under severe constraints: RP-DT outperforms RP-ET at small memory budgets on several datasets because it is more resistant to the strong randomization induced by very low µ′_max.RP-DT does not randomize split thresholds, unlike RP-ET.
  • Comparison with instance sub-sampling: Building trees from re-sampled random patches clearly outperforms straightforward instance sub-sampling when the memory budget is low.The comparison uses ET and RF trees built on the same training sample with all features.
  • Comparison with instance sub-sampling: Random-patch ensembles improve performance even when dataset learning curves suggest that RF and ET have not yet converged.The authors report this pattern across the evaluated problems.
  • Dataset-size effects: No conclusive correlation was found between dataset size Np and the minimum µ′_max required for good performance.The conclusion concerns the evaluated datasets and their memory-reduction thresholds.

8.3.4 Conclusions

The conclusions summarize random-forest methodology, interpretability, and scalable learning under memory constraints. They emphasize both established findings and unresolved theoretical and practical questions.

  • 8.3.4 Conclusions: Sampling-based ensembles have intrinsically low memory requirements and can reduce memory substantially without significant accuracy loss.When datasets exceed available memory, sampling both samples and features can improve accuracy over instance-only subsampling.
  • 8.3.4 Conclusions: Random Patches builds each ensemble model from random subsets of samples and features, matching popular randomization schemes in accuracy.The method is designed for very large datasets or strong memory constraints and independently built models are straightforward to parallelize.
  • 8.3.4 Conclusions: The first part analyzes tree induction and random forests, including variance reduction through decorrelation, computational complexity, scalability, and implementation details.The work also discusses software contributions within Scikit-Learn and emphasizes considerations affecting computational performance.
  • 8.3.4 Conclusions: The interpretability analysis characterizes Mean Decrease of Impurity importances and derives a three-level decomposition covering interaction terms.Under the studied setting, relevant variables determine importances while irrelevant variables have importance strictly equal to zero.
  • 8.3.4 Conclusions: Variable-importance analysis indicates defects in importances computed from non-totally randomized trees, including masking, impurity misestimation, and binary-tree structure.The authors call for systematic decomposition and further characterization, especially for finite settings and binary trees.
  • 8.3.4 Conclusions: The Random Patches analysis remains mostly empirical, and its conclusions depend on parameters tuned through exhaustive validation-set grid search without accounting for tuning costs.Future work includes theoretical analysis, larger-scale experiments, smarter sampling, and efficient parameter selection under global memory constraints.
  • 8.3.4 Conclusions: The thesis concludes that machine-learning algorithms should be examined through their mechanisms, properties, and limitations rather than treated as black boxes.Its broader methodological position is to connect algorithmic choices with the specific problem being solved.

T. Mitchell. Machine learning. McGraw-Hill, New York, 1997.

This supplied section consists of bibliographic entries rather than substantive discussion. It records references spanning decision trees, random forests, ensemble methods, machine learning, and related computational techniques.

  • References: The references include works on decision-tree induction, split selection, discretization, and uncertainty measures.Examples include entries by Quinlan, Miyakawa, Wehenkel, and related authors.
  • References: The bibliography includes research on random forests, their consistency, variable importance, and applications.Referenced topics include random-forest consistency, bias in variable importance, conditional importance, and scientific applications.
  • References: The section also records literature on ensemble methods, sampling, parallel computation, and scientific software.Entries cover bagging, random subspaces, parallel stochastic gradient descent, NumPy, IPython, and Scikit-Learn.
Loading 1407.7502v3…