PaperlayerView exact revision on arXiv

Source-linked AI summary

To tune or not to tune the number of trees in random forest?

Philipp Probst, Anne-Laure Boulesteix

arXiv:1705.05654v1stat.MLcs.LG

TL;DR

The paper asks whether random forests should use the largest feasible number of trees or tune a smaller number. Through theoretical analysis and empirical study, it finds that classification error and AUC can be non-monotonic in tree count, whereas several probability-based measures are not, supporting large feasible forests.

  • Problem

    Whether to tune or maximize the computationally feasible number of random-forest trees has not been systematically studied theoretically and empirically.

  • Method

    The paper combines theoretical monotonicity analyses with repeated out-of-bag evaluations across 306 OpenML datasets.

  • Results

    Expected classification error and AUC may be non-monotonic in tree count, while Brier score, logarithmic loss, and mean squared error cannot show this pattern.

  • Takeaways & Limitations

    The authors argue against tuning tree count in practice and favor a computationally feasible large number chosen according to the desired measure’s convergence properties.

  • Takeaways & Limitations

    The empirical study examined only datasets with fewer than 1000 observations and features, so generalizability to larger datasets was not investigated.

Abstract

from arXiv · show

The number of trees T in the random forest (RF) algorithm for supervised learning has to be set by the user. It is controversial whether T should simply be set to the largest computationally manageable value or whether a smaller T may in some cases be better. While the principle underlying bagging is that "more trees are better", in practice the classification error rate sometimes reaches a minimum before increasing again for increasing number of trees. The goal of this paper is four-fold: (i) providing theoretical results showing that the expected error rate may be a non-monotonous function of the number of trees and explaining under which circumstances this happens; (ii) providing theoretical results showing that such non-monotonous patterns cannot be observed for other performance measures such as the Brier score and the logarithmic loss (for classification) and the mean squared error (for regression); (iii) illustrating the extent of the problem through an application to a large number (n = 306) of datasets from the public database OpenML; (iv) finally arguing in favor of setting it to a computationally feasible large number, depending on convergence properties of the desired performance measure.

1. Introduction

The paper examines whether the random forest tree count T should be tuned or set as high as computationally feasible. It combines theoretical analysis with experiments on 306 OpenML datasets to explain non-monotonic error curves and guide this choice.

  • Motivation: The paper challenges the assumption that adding trees always improves random-forest classification error, while noting that convergence results do not resolve whether T is a tuning parameter.Each tree is trained independently, so the overfitting risk associated with adding trees in boosting is not relevant here.
  • Theoretical contributions: Theoretical results show that expected error and AUC may vary non-monotonically with T, whereas Brier score, logarithmic loss, and regression mean squared error cannot exhibit this pattern.The paper also aims to explain the circumstances producing non-monotonic expected error.
  • Empirical motivation: Repeated experiments average 1000 independent random-forest runs per dataset to estimate out-of-bag error curves more stably across OpenML datasets.Illustrative curves show error dropping steeply, increasing after a certain tree count, and eventually reaching a plateau.
  • Conclusion: The paper ultimately argues that T should not be treated as a tuning parameter because adding trees has no disadvantage beyond additional computational cost.The recommendation is framed in relation to the convergence properties of the desired performance measure.
  • Empirical study: The empirical study covers 306 public OpenML datasets and validates the theoretical model for error as a function of tree count.The paper reports these results in Section 4.

2. Background: random forest and measures of performance

This section introduces random forests as ensembles of bootstrapped decision trees and defines how their predictions and performance are evaluated. It focuses mainly on binary classification and regression, using loss-based measures, probability-based measures, AUC, test-set averages, and out-of-bag error.

  • Random forest: Random forests aggregate T decision trees, reducing variance relative to individual trees; this paper considers Breiman’s original RF while acknowledging alternative variants.Each tree is built from a bootstrap sample or subsample, with random candidate variables considered at each split.
  • Random forest: Regression forests average tree predictions, whereas classification forests usually assign the class receiving the most votes.For binary classification, averaging tree outputs gives a probability estimate, which is thresholded at 0.5 to obtain the predicted class.
  • Performance measures: With an independent test dataset, performance measures are averaged over test observations, including classification error rate and regression mean squared error.Regression can also use mean absolute error or median-based counterparts.
  • Performance measures: Performance is quantified by loss functions comparing true responses with predicted values, or binary labels with predicted probabilities.Squared loss yields squared error in regression and the 0-or-1 classification error; absolute loss is another regression measure.
  • Performance measures: Binary classification also uses the Brier score and logarithmic loss based on predicted probabilities, while AUC evaluates class discrimination by ranking those probabilities.AUC is interpretable as the probability that a randomly selected positive observation is ranked above a randomly selected negative observation.
  • Performance measures: Out-of-bag error assesses performance on training observations using only trees whose bootstrap samples excluded the observation being predicted.OOB estimates can predict classes, probabilities, or real-valued responses, depending on the task.

3. Theoretical results

Theoretical analysis shows that expected RF error and AUC can vary non-monotonically with tree count, whereas the Brier score, logarithmic loss, and regression squared error decrease with more trees. Error-curve convergence depends on the distribution of per-observation tree error probabilities rather than directly on dataset size or feature count.

  • AUC: Expected AUC is not always increasing in T because it depends on pairwise observation comparisons rather than a sum of individual contributions.A two-observation example demonstrates that increasing the number of trees can reduce expected AUC.
  • Error rate: Expected error can be non-monotonic because observations with εi > 0.5 contribute increasing error as T grows, while those with εi < 0.5 contribute decreasing error.For εi values near 0.5, convergence is slower and increasing contributions can eventually outweigh rapidly decreasing ones.
  • Error rate: The expected error curve initially decreases but can later increase when many observations have εi ≈0 and some have εi ≥0.5 close to 0.5.By contrast, observations with εi > 0.5 can be compensated by observations with εi < 0.5, often producing a monotonically decreasing curve.
  • Error rate: Error-curve convergence depends on the distribution of the εi values, not directly on the number of observations n or features.Dataset size and feature count may still affect convergence indirectly by changing the empirical εi distribution.
  • Brier score and squared error: The expected Brier score and regression squared error are strictly decreasing in T, with E(bi(T)) = E(eit)^2 + Var(eit)/T.For binary classification, E(eit) = εi and Var(eit) = εi(1 − εi).
  • Logarithmic loss: The expected logarithmic loss decreases with T, and its Taylor approximation improves as prediction variance shrinks with increasing tree count.The approximation is E(li(T)) ≈ −ln(1 − εi + a) + εi(1 − εi)/(2T(1 − εi + a)^2).

4. Empirical results · 4.1 Study design

The empirical study uses 306 OpenML tasks to examine non-monotonic random-forest error-rate patterns and related performance measures. It evaluates repeated 2,000-tree forests across classification and regression datasets under computationally feasible selection criteria.

  • 4. Empirical results: 306 OpenML tasks comprise the study: 193 classification and 113 regression tasks.The study aims to assess the frequency of non-monotonic error-rate curves in real-data settings.
  • 4.1 Study design: The selected datasets have predefined OpenML tasks, fewer than 1000 observations, and fewer than 1000 features.Cleaning procedures, including deletion of duplicated datasets, are also applied.
  • 4.1 Study design: Each dataset is evaluated with random forests containing T = 2000 trees, repeated 1000 times with different seeds.The R package randomForest is used with default parameter settings.
  • 4.1 Study design: The choice of 2000 trees follows a preliminary study in which OOB-curve convergence was reached within 2000 trees.
  • 4.1 Study design: Classification OOB curves are calculated for error rate, balanced error rate, multiclass Brier score, logarithmic loss, and multiclass AUC.The calculations use the OOBCurve package.
  • 4.1 Study design: Regression OOB curves use mean squared error, mean absolute error, median squared error, and median of absolute error.Computations are parallelized with batchtools, and each final curve averages 1000 RF runs for each measure and dataset.

4.2 The R package OOBCurve

The OOBCurve R package computes out-of-bag estimates across performance measures from random forests built with randomForest or ranger. It supports any measure available in mlr and includes application examples, with CRAN availability forthcoming.

  • 4.2 The R package OOBCurve: OOBCurve calculates out-of-bag estimates for different performance measures.The package implements out-of-bag curves for random forests.
  • 4.2 The R package OOBCurve: The package accepts random forests constructed with randomForest or ranger.These are the supported input implementations named in the passage.
  • 4.2 The R package OOBCurve: OOBCurve computes curves for any performance measure available from mlr, with application examples provided in the R code.The package is planned for the CRAN R package repository and is already available in an unspecified current form.

4.3 Results for binary classification

Across binary-classification datasets, increasing the forest from 11 to 2000 trees generally improves performance, but the OOB error rate is non-monotonic in approximately 10% of datasets. The Brier score and logarithmic loss decrease monotonically, whereas AUC usually increases but can eventually decline.

  • Overall performance gains: −0.0324, −0.0683, −2.383, and 0.0553 are the average performance gains for 2000 instead of 11 trees in error rate, Brier score, logarithmic loss, and AUC, respectively.The gains are measured on out-of-bag performance.
  • OOB error rate: 15 datasets, approximately 10%, had an OOB error rate at 2000 trees at least 0.005 above the curve’s minimum for T ∈[10, 150].Most curves decreased quickly toward a dataset-specific plateau; the increases occurred mainly for smaller datasets.
  • OOB error rate: Most observation-specific errors were below 0.5 in typical datasets, producing monotonically decreasing error-rate curves.Non-monotonic curves occurred when a non-negligible share of observations had errors larger than but close to 0.5, matching the theoretical explanation.
  • Other performance measures: The Brier score and logarithmic loss decreased monotonically for all datasets, while AUC usually grew but sometimes fell after reaching a maximum.One such AUC example was the third binary-classification dataset, OpenML ID 905.

4.4 Results for multiclass classification

For multiclass classification, increasing the forest from 11 to 2000 trees yields metric-specific average out-of-bag performance changes, including a decrease in error rate and increases in Brier score, logarithmic loss, and AUC. These gains exceed those observed for binary classification, while the visual patterns are similar across classification settings.

  • Multiclass classification results: -0.0753 is the average out-of-bag performance gain in error rate for 2000 instead of 11 trees.The corresponding multiclass gains are -0.1282 for the Brier score, -5.3486 for logarithmic loss, and 0.0723 for AUC.
  • Multiclass classification results: The multiclass average performance changes are higher than those observed for binary classification.
  • Multiclass classification results: The visual patterns observed for binary classification also hold for multiclass classification.The results for correlation are described as quite similar across the two settings.

4.5 Results for regression

For regression, increasing the forest size generally improves performance, with mean-based measures decreasing monotonically and median-based measures showing occasional increases. The measures cluster by whether they average or take the median of observation-level losses.

  • 0.1249 average performance gain in out-of-bag R2 occurred for 2000 trees compared with 11 trees.
  • Mean squared error decreases monotonically with the number of trees, as theory predicts, while mean absolute error also appears strictly decreasing.
  • Median squared error and median absolute error sometimes increase with tree count in either the small-T or large-T region, although more trees usually improve results.
  • Measures averaging losses across observations correlate highly with one another, as do measures taking medians.
  • 0.5 linear correlation and 0.2 rank correlation characterize the relationship between mean-based and median-based measure groups.

4.6 Convergence

Increasing the number of trees substantially improves performance in most datasets, with most gains occurring within the first 250 trees. The authors recommend assessing convergence with probability-sensitive measures such as the Brier score or logarithmic loss rather than error rate alone.

  • Convergence: Increasing trees from 10 to 500 yields substantial performance gains in most cases, with the biggest gains usually reached within the first 250 trees.At 250 trees, curves in all datasets and for all measures reach values that adding more trees cannot improve substantially.
  • Convergence: At 250 trees, performance curves in all datasets and for all measures have reached values that cannot be improved a lot by adding more trees.
  • Convergence: OOB convergence understates future independent-data convergence by exp(1) ≈2.718 because the OOB error curve at T uses only exp(−1) · T trees.This agrees with findings that growing beyond 128 trees yields no big gain in AUC performance.
  • Convergence: The authors recommend assessing convergence with the Brier score or logarithmic loss because their rates are less dependent on observations with εi close to 0.5.These measures indicate the general stability of probability estimates and are especially useful when the classification threshold is not fixed at 0.5.

5. Discussion and extensions

The paper argues against tuning the number of random-forest trees: non-monotonic error-rate behavior is measure- and data-specific, usually yields negligible gains, and does not undermine using many trees. Extensions include convergence-based stopping and generalization beyond binary random forests, though further research is needed.

  • Discussion and extensions: Tuning the number of trees is not recommendable despite occasional non-monotonic expected error-rate curves.The authors argue that these curves do not justify tuning T.
  • Discussion and extensions: Non-monotonic patterns occur for measures such as error rate and AUC, but not for the Brier score or logarithmic loss.The latter measures are probability-based and were shown theoretically and empirically not to produce non-monotonic patterns.
  • Discussion and extensions: For independent future data, non-monotonic error curves arise only under a rare combination of training-data ε_i values, making large T preferable in many cases.This issue is especially relevant with small training datasets, where the ε_i distribution may differ between training and test data.
  • Discussion and extensions: The minimum error rate is usually less than 0.01 below its convergence value, so the small gain may not offset advantages of more trees.Those advantages include other performance measures and greater precision of variable-importance measures.
  • Discussion and extensions: The theoretical results may generalize to most bagging methods and possibly to multiclass classification, but more research is needed.The proofs do not depend on the base learners being trees, and the multiclass possibility is supported by results from 44 multiclass datasets.
  • Discussion and extensions: Most performance gain occurs in the first 100 trees, while convergence depends on other random-forest hyperparameters and could motivate automatic stopping criteria.Lower bootstrap sample sizes, stronger depth constraints, or more variables can reduce tree correlation and require more trees for convergence; OOB curves and variable-importance convergence could guide stopping.

Supplementary File · Online discussions

The supplementary material lists online discussions addressing how to choose and evaluate the number of trees in random forests. Topics include out-of-bag error, tuning practice, model comparison, overfitting, and interactions with other forest parameters.

  • Online discussions: Online discussions ask whether out-of-bag error is the optimal criterion for selecting the number of trees.
  • Online discussions: One discussion examines whether the optimal number of trees depends on the number of predictors.
  • Online discussions: Another asks whether forests trained on the same data with different random seeds should be compared.
  • Online discussions: Several discussions consider whether the number of trees should be treated as a tuning parameter.
  • Online discussions: Online discussions address whether random forests overfit and whether n_tree should be tuned.
  • Online discussions: A discussion compares evaluating random forests with out-of-bag estimates versus cross-validation.
  • Online discussions: Another discussion asks whether out-of-bag error increases as the number of trees grows.
  • Online discussions: Additional topics concern the relationship between tree depth and tree count and methods for determining how many trees to generate.
Loading 1705.05654v1…