PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperparameters and Tuning Strategies for Random Forest

Philipp Probst, Marvin Wright, Anne-Laure Boulesteix

arXiv:1804.03515v2stat.MLcs.LG

TL;DR

Random forest hyperparameter choices lack sufficiently systematic guidance, despite affecting prediction performance and variable-importance measures. The paper reviews this evidence, presents model-based optimization through tuneRanger, and reports that tuning achieved better average performance than standard RF and other software implementations. Its conclusions are bounded by the literature’s scarcity of large-scale comparisons, especially for variable importance.

  • Problem

    Systematic evidence and clear guidance on random-forest hyperparameter choices remain limited, particularly regarding effects on variable-importance measures.

  • Method

    The paper reviews hyperparameter effects, discusses tuning strategies, and demonstrates model-based optimization implemented in the tuneRanger R package.

  • Results

    Tuning random forest improved performance, with tuneRanger achieving better average performance than standard random forest and other software implementations.

  • Takeaways & Limitations

    mtry is among the most influential parameters, while sample size and node size have smaller performance effects but are often worth tuning.

  • Takeaways & Limitations

    Large-scale systematic comparisons of random-forest variants and hyperparameters remain scarce, especially for variable-importance effects.

Abstract

from arXiv · show

The random forest algorithm (RF) has several hyperparameters that have to be set by the user, e.g., the number of observations drawn randomly for each tree and whether they are drawn with or without replacement, the number of variables drawn randomly for each split, the splitting rule, the minimum number of samples that a node must contain and the number of trees. In this paper, we first provide a literature review on the parameters' influence on the prediction performance and on variable importance measures. It is well known that in most cases RF works reasonably well with the default values of the hyperparameters specified in software packages. Nevertheless, tuning the hyperparameters can improve the performance of RF. In the second part of this paper, after a brief overview of tuning strategies we demonstrate the application of one of the most established tuning strategies, model-based optimization (MBO). To make it easier to use, we provide the tuneRanger R package that tunes RF with MBO automatically. In a benchmark study on several datasets, we compare the prediction performance and runtime of tuneRanger with other tuning implementations in R and RF with default hyperparameters.

1 Introduction

The paper addresses how to choose random-forest hyperparameters, reviewing their effects and presenting tuning strategies compared in a benchmark study.

  • The review addresses hyperparameter effects on prediction performance and variable importance, where systematic guidance remains difficult to find.
  • RF applications may prioritize predictive accuracy or assessing predictor relevance, so parameter effects must be interpreted relative to the objective.
  • Most hyperparameters are tuning parameters whose optimal values depend on the dataset and chosen performance measure.
  • Tuning can overfit training data, making test data, cross-validation, or out-of-bag observations useful for parameter selection.
  • The paper considers mtry, sampling settings, node size, number of trees, and splitting rule, while noting that hyperparameters and algorithm variants can overlap.
  • Its two-part design combines a literature review with tuning-strategy and software comparisons in a benchmark study.

2 Literature Review

The literature review examines hyperparameter effects on prediction performance and runtime, then organizes the parameters alongside typical software defaults.

  • The review separates evidence about prediction performance and runtime from evidence about variable importance.
  • Table 1 summarizes random-forest hyperparameters, their descriptions, and typical default values, with n denoting observations and p variables.

2.1 Influence on performance

Hyperparameters govern randomness, tree structure, and splitting, with tuning involving trade-offs among tree diversity, individual-tree accuracy, runtime, and bias.

  • Randomness in RF aims to balance low tree correlation with reasonable tree strength through mtry, sample size, and node size.
  • mtry: mtry controls candidate split variables: lower values can increase tree diversity and computation efficiency, whereas higher values can help locate sparse relevant variables.
  • mtry: The optimal mtry depends on the number of relevant predictors; √p is a convenient baseline but higher values can perform better in high-dimensional data.
  • Sampling scheme: Sample size creates a similar diversity–accuracy trade-off, and its optimal value is problem dependent and can be estimated using out-of-bag predictions.
  • Sampling scheme: Sampling with replacement may induce slight variable-selection bias for categorical predictors with differing numbers of categories.
  • Node size: Larger node sizes can substantially reduce runtime, often without substantial prediction loss, while node-size constraints can create selection bias for categorical variables.
  • Splitting rule: Extremely randomized trees are fastest, followed by classical RF, while conditional inference forests have the largest runtime.

2.2 Influence on variable importance

The review finds that hyperparameters affect variable-importance measures, but evidence is uneven and some measures are intrinsically biased.

  • RF variable importance can capture nonlinear and interaction effects, with Gini and permutation measures representing distinct variants.
  • The review focuses on permutation importance because Gini importance favors variables with many categories or continuous values even without outcome influence.
  • The literature is relatively extensive for tree number but remains scarce on sample size and node size effects on variable importance.
  • Number of trees: Multiple thousands of trees may be required for stable variable-importance estimates in settings with many noisy variables.
  • mtry: Higher mtry values lower the importance assigned to weak regressors, while standard RF importance is less mtry-dependent than conditional-inference-forest importance.
  • mtry and node size: Increasing mtry raises importance magnitudes, whereas small mtry combined with larger node size makes importances more equal across variables.

3 Tuning random forest

Tuning random forest means selecting hyperparameters for a dataset and performance measure, but prior literature offers general trends rather than clear guidance. The section reviews tunability, evaluation choices, search strategies, R implementations, and benchmark comparisons.

  • Tuning selects hyperparameters for a learning algorithm and dataset, with optimality potentially defined by performance measures or runtime.The paper mainly focuses on performance-based optimality.
  • Users often do not know whether values different from software defaults will improve random-forest performance.Random forest is known to provide good results with default settings, but the literature does not offer clear-cut guidance.
  • The paper defines tunability as the performance gain achievable by tuning one hyperparameter or all hyperparameters relative to default values.The distinction is between a hyperparameter’s tunability and the algorithm’s tunability.
  • The section reviews evaluation strategies, evaluation measures, tuning search strategies, R software implementations, and a large-scale benchmark of implementations.The benchmark compares different implementations after the literature and strategy reviews.

3.1 Tunability of random forest

Random forest is relatively robust to hyperparameter choices, yet tuning can yield moderate average gains and larger improvements on some datasets. The strongest average effects reported concern mtry and sample size.

  • Random forest is reported as far less tunable than algorithms such as support vector machines.This comparison accompanies the finding that tuning can nevertheless improve RF performance.
  • 0.010 average AUC increase was achieved through tuning across 38 datasets compared with default software-package hyperparameter values.The reported average gain is characterized as small or moderate.
  • Around 0.03 performance gains occurred for some datasets, although the average improvement was much smaller.Larger gains may matter when each wrongly classified observation has high costs.
  • 0.002 positive effect was observed when replace changed from sampling with replacement to sampling without replacement.The node-size effect was similarly small at 0.001.
  • Higher numbers of trees are generally preferable for performance, so the number of trees is not treated as a tuning parameter.This distinguishes increasing forest size from selecting among competing hyperparameter values.

3.2 Evaluation strategies and evaluation measures

The section compares cross-validation and out-of-bag evaluation for tuning random forests and reviews common classification measures. Out-of-bag tuning is recommended for most datasets because it is generally reliable and faster than k-fold cross-validation.

  • Repeated k-fold cross-validation reduces estimation variance by averaging results across repetitions of the procedure.The number of folds is usually chosen between 2 and 10.
  • Out-of-bag evaluation generally approximates random-forest performance on independent data reasonably well.A bias can arise in specific data situations.
  • Out-of-bag estimates can be biased for very small datasets with n < 20, many predictor variables, and balanced classes.These problems are described as specific to particular and relatively rare situations.
  • Out-of-bag tuning has much smaller runtime than procedures such as k-fold cross-validation, especially on big datasets.The authors recommend the out-of-bag approach as appropriate for most datasets.
  • Classification evaluation may use error rate, AUC, Brier score, or logarithmic loss.The appropriate evaluation measure depends on the learning problem.

3.3 Tuning search strategies

Tuning search strategies differ in whether candidate hyperparameters are fixed in advance or selected iteratively from prior evaluations. The paper introduces SMBO, which models evaluated settings and proposes new ones using expected outcomes and unexplored regions.

  • SMBO iteratively chooses future hyperparameters using results from previously evaluated settings.This contrasts with strategies that specify all candidate values from the beginning.
  • Grid search evaluates all combinations in a discrete parameter space, while random search samples hyperparameter values from a specified space.Continuous parameters must be discretized before grid search.
  • Random search was more efficient than grid search for finding good hyperparameter specifications in neural networks.This result is attributed to Bergstra and Bengio (2012).
  • SMBO is implemented in mlrMBO and used by tuneRanger for random-forest tuning.The described setup specifies an evaluation measure, strategy, and constrained hyperparameter space.
  • SMBO begins by evaluating random initial design points, then repeatedly fits a surrogate model, proposes a new point, and evaluates it.The surrogate uses the evaluation measure as the dependent variable and hyperparameters as predictors.
  • The SMBO infill criterion favors points with good expected outcomes and points in sparsely evaluated regions of the hyperparameter space.The selected point is evaluated and added to the existing design points.

3.4 Existing software implementations

Existing R implementations tune random forests with different search strategies, parameter scopes, evaluation procedures, and stopping rules. The reviewed implementations range from grid or sequential searches over selected hyperparameters to iterative out-of-bag tuning of mtry.

  • mlrHyperopt: mlrHyperopt applies sequential model-based optimization to ranger, tuning mtry and node size for 25 iterations.Its default evaluation uses 10-fold cross-validation and mean missclassification error.
  • Comparison: The implementations expose different tuning parameters and evaluation choices, allowing users to modify spaces or measures in some cases.The described platform supports user-defined tuning parameters and spaces, while caret and tuneRF use narrower default procedures.
  • caret: caret performs a grid search over mtry using three parameter values and 25 bootstrap iterations.It selects the setting with the lowest error rate for classification or lowest mean squared error for regression.
  • tuneRF: tuneRF tunes only mtry by comparing out-of-bag error while progressively testing smaller and larger values.The search continues when the relative improvement reaches at least 0.05 and returns the best mtry value.

3.5 The tuneRanger package

tuneRanger packages model-based optimization for automatic random-forest tuning in a single-line interface. By default it tunes mtry, sample size, and node size using out-of-bag evaluation, while supporting customizable parameters, measures, and classification or regression.

  • Package overview: tuneRanger provides automatic random-forest tuning through a single line of code and is designed for users unfamiliar with tuning strategies.The package is based on ranger, mlrMBO, and mlr.
  • Optimization: Sequential model-based optimization uses 30 random initial points and 70 iterative optimization steps by default.The numbers of warm-up and optimization steps can be changed by the user.
  • Tuned parameters: The default procedure simultaneously tunes mtry, sample size, and node size over configurable parameter spaces.Users can add parameters such as sampling strategy or unordered-factor handling through tune.parameters.
  • Evaluation: Out-of-bag predictions are used for evaluation, making tuneRanger faster than procedures based on cross-validation.The package supports both classification and regression.
  • Measures: The default optimized measures are the Brier score for classification and mean squared error for regression, with alternative mlr measures available.The classification default is intended to provide finer evaluation than the commonly used error rate.
  • Recommendation: The final recommended setting averages each hyperparameter across the best 5 percent of SMBO iterations.The resulting mtry and node-size values are rounded, and the setting is used to train the final random-forest model.
  • Example: On the monks-problem-1 example, tuning achieved an improvement of 0.014 in error rate and 0.120 in Brier score.The benchmark output reports a multiclass Brier score of 0.006925637 for the tuned procedure.

3.6 Benchmark study

The benchmark compares tuned and default random forests across 39 binary-classification datasets using multiple performance measures, ranks, and runtime analyses. Tuning generally improves performance over ranger defaults, while tuneRanger is especially advantageous on larger datasets because it uses out-of-bag evaluation.

  • Design: The benchmark evaluates tuneRanger variants, mlrHyperopt, caret, tuneRF, and default ranger across 39 OpenML datasets.Datasets are binary-target, missing-value-free classification tasks evaluated with MMCE, AUC, Brier score, and logarithmic loss.
  • Average performance: All algorithms perform better on average than ranger default, although the differences are small.The best algorithm is around 0.013 better in MMCE and 0.007 better in AUC.
  • Average performance: The tuneRanger methods outperform ranger default on average for all evaluated measures.Optimizing the specific measure generally gives the best result among the tuneRanger variants for that measure.
  • Runtime: tuneRF is fastest for most datasets, while mlrHyperopt becomes increasingly slower than tuneRanger as dataset runtime grows.For larger datasets, mlrHyperopt uses 10-fold cross-validation rather than tuneRanger's out-of-bag evaluation, which takes around 10 times longer.
  • Runtime: The authors claim tuneRanger is preferable especially for bigger datasets, where runtime matters more.This conclusion follows the runtime comparison and tuneRanger's use of out-of-bag evaluation.

4 Conclusion and Discussion

The paper finds that RF hyperparameters affect performance and variable-importance measures, but systematic comparative evidence remains limited. Tuning can improve performance, with model-based optimization and out-of-bag predictions offering a practical approach implemented in tuneRanger.

  • Hyperparameter effects: The number of trees should be set high because more trees improve performance and variable-importance precision, although gains diminish as trees are added.The required number depends on the dataset’s convergence behavior.
  • Hyperparameter effects: mtry is the most influential randomness parameter, and its best value depends on how many variables relate to the outcome.Sample size and node size usually have smaller effects but are often still worth tuning.
  • Evidence gaps: The literature lacks systematic large-scale comparisons of RF variants and hyperparameter values, especially regarding variable-importance measures.This gap is important because many RF users attend to variable importances as much as prediction.
  • Evidence gaps: The review calls for more neutral studies comparing RF variants and hyperparameter choices rather than primarily developing new methods.Such comparisons are described as important but rare amid publication bias favoring novel methods.
  • Tuning strategy: Sequential model-based optimization can tune mtry, sample size, and node size using out-of-bag predictions faster than cross-validation.The procedure is implemented in tuneRanger, which lets users select the measure to minimize.
  • Benchmark conclusions: In the benchmark, tuneRanger achieved better average performance than standard RF and other RF-tuning software, while fast tuneRF remains preferable when computational speed matters.The paper reports that tuning improves performance, though the effect is smaller than for some other machine-learning algorithms.
Loading 1804.03515v2…