PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding overfitting in random forest for probability estimation: a visualization and simulation study

Lasai Barreñada, Paula Dhiman, Dirk Timmerman, Anne-Laure Boulesteix, Ben Van Calster

arXiv:2402.18612v2stat.MEcs.CYcs.LG

TL;DR

Evidence on RF as a probability-estimation method was limited, despite its relevance to clinical decision-making. This study examined RF probability predictions and found that local peaks around training events could yield near-perfect training discrimination without strongly affecting test discrimination; its simulation findings challenge the recommendation to use fully grown trees for probability estimation.

  • Problem

    Literature on RF performance as a probability-estimation tree was scarce, although probability estimation is important for optimal clinical decision-making.

    Source: §1 · p. 2, §5 · p. 20

  • Method

    The paper examined RF probability estimation and visualized its behavior, including local prediction peaks around training events.

    Source: §2 · p. 3, §5 · p. 17

  • Results

    RF learned local probability peaks around training events, often producing near-perfect training discrimination without strongly affecting test-data discrimination.

    Source: §5 · p. 17

  • Takeaways & Limitations

    For probability estimation, the simulation results go against the common recommendation to use fully grown trees in RF models.

    Source: §5 · p. 21

    Further simulation studies should examine hyperparameters beyond min.node.size.

    Source: §5 · p. 20

Abstract

from arXiv · show

Random forests have become popular for clinical risk prediction modelling. In a case study on predicting ovarian malignancy, we observed training c-statistics close to 1. Although this suggests overfitting, performance was competitive on test data. We aimed to understand the behaviour of random forests by (1) visualizing data space in three real world case studies and (2) a simulation study. For the case studies, risk estimates were visualised using heatmaps in a 2-dimensional subspace. The simulation study included 48 logistic data generating mechanisms (DGM), varying the predictor distribution, the number of predictors, the correlation between predictors, the true c-statistic and the strength of true predictors. For each DGM, 1000 training datasets of size 200 or 4000 were simulated and RF models trained with minimum node size 2 or 20 using ranger package, resulting in 192 scenarios in total. The visualizations suggested that the model learned spikes of probability around events in the training set. A cluster of events created a bigger peak, isolated events local peaks. In the simulation study, median training c-statistics were between 0.97 and 1 unless there were 4 or 16 binary predictors with minimum node size 20. Median test c-statistics were higher with higher events per variable, higher minimum node size, and binary predictors. Median training slopes were always above 1, and were not correlated with median test slopes across scenarios (correlation -0.11). Median test slopes were higher with higher true c-statistic, higher minimum node size, and higher sample size. Random forests learn local probability peaks that often yield near perfect training c-statistics without strongly affecting c-statistics on test data. When the aim is probability estimation, the simulation results go against the common recommendation to use fully grown trees in random forest models.

1 Background

Random forests (RF) showed near-perfect training discrimination in ovarian-tumor prediction yet remained competitive on external validation, prompting an investigation of their probability-estimation behavior. The study examined real-world prediction visualizations and conducted a simulation study.

1 Background

  • Although RF is popular in clinical prediction and has been described as robust to overfitting, evidence about its performance as a probability-estimation tree was scarce.The background also notes reports that overfitted individual trees can perform better in RF ensembles.Source: §1 · p. 2
  • RF achieved training PDI 0.93 versus 0.47–0.70 for other models, and external-validation PDI 0.54 versus 0.41–0.55 (n=3199).The ovarian-tumor study compared six modeling approaches; with five outcome categories, PDI ranges from 1/5 for an uninformative model to 1 for perfect discrimination.Source: §1 · p. 2
  • Near-perfect training discrimination seemed inconsistent with RF’s competitive external-validation performance and raised concern about overfitting.The authors expected overfitting from modeling noise to reduce performance on new data more strongly.Source: §1 · p. 2
  • The study aimed to understand RF probability estimation by visualizing data space in three real-world case studies and conducting a simulation study.The paper also examined tree depth, training sample size, and data-generating mechanisms in the simulation.Source: §1 · pp. 2–3

2 Random forest for probability estimation

The section describes RF as a tree ensemble for probability estimation, outlining bootstrap sampling, randomized predictor selection, splitting, and aggregation. It contrasts common advice favoring fully grown trees with the study’s focus on probability estimation.

2 Random forest for probability estimation

  • Each tree is grown on a bootstrap sample using recursive binary splits, with a random subset of predictors considered at each split to reduce correlation between trees.The selected split optimizes a criterion such as the Gini index.Source: §2 · p. 3
  • Splitting continues while child nodes meet the minimum node-size condition; each terminal leaf can supply class proportions or a majority-vote class.Other stopping criteria are also possible.Source: §2 · p. 3
  • RF probability estimates for a new case aggregate tree-level predictions, either by majority-vote proportions or by averaging class proportions in terminal leaves.The paper uses the latter approach, averaging class proportions across trees.Source: §2 · pp. 3–4
  • Hastie and colleagues suggest setting min.node.size very low, even 1; they identify mtry as a more important tuning parameter, while full-grown trees seldom cost much and remove one tuning choice.The authors indicate that increasing ntree does not cause overfitting.Source: §2 · p. 4

3 Case studies

Across the case studies, RF predictions formed local probability peaks around training events, while discrimination and calibration varied between training and test data. The section also describes the visualization approach and reports results from ovarian-cancer, traumatic-brain-injury, and stroke case studies.

3.1 Methods

  • The visualization method plotted two continuous predictors on the axes, estimated probabilities as heatmaps, and overlaid individual training or test cases.Categorical predictors were held fixed to show a two-dimensional subset of data space; model-specific probability ranges set heatmap scales, with shared-scale plots also provided in supplementary material.Source: §3.1 · p. 4

3.2 Ovarian cancer diagnosis

  • Ovarian-cancer RF PDI was 0.97 in training and 0.56 in testing, while MLR PDI was 0.52 in both sets.RF probabilities peaked around training events; nearby events formed larger high-probability areas, while isolated peaks were idiosyncratic relative to test cases.Source: §3.2 · pp. 5–6
  • In ovarian-cancer probability plots, regression estimates changed smoothly with continuous predictors, whereas RF estimates peaked where events occurred; RF calibration was very poor in training and much better in testing.In training, high probabilities were underestimated and low probabilities overestimated.Source: §3.2 · p. 6, Figure 1 · p. 7

3.3 CRASH3: traumatic brain injury prognosis

  • CRASH3 RF PDI was 0.96 in training and 0.54 in testing, compared with MLR PDI of 0.61 and 0.60, respectively.RF showed probability peaks and poor training calibration, but decent test calibration for the two most common outcomes; MLR probabilities changed smoothly.Source: §3.3 · pp. 9–10

3.4 IST: Type of stroke diagnosis

  • In IST, RF training and test PDI were 0.89 and 0.35, respectively; its data space showed very local peaks around training cases and poor calibration in both sets.The reported MLR training PDI was 0.39.Source: §3.4 · p. 10

4 Simulation study

The simulation study examined how data and modeling choices relate to exaggerated training discrimination and test performance. Across scenarios, training discrimination was often near perfect, while test discrimination and calibration varied with the simulation factors and hyperparameters.

4.2 Data Generating Mechanism (DGM)

  • The 48 logistic data-generating mechanisms varied predictor distribution and count, predictor correlation, and true AUC, set at 0.75 or 0.90.Predictors were either continuous with a multivariate normal distribution or binary with 50% prevalence.Source: §4.2 · pp. 10–11

4.2.1 Estimands

  • The study assessed discrimination, risk-estimate spread, prediction error, and—on test data—discrimination loss and bias/variance contributions; test results represent internal validation.The training and test samples were generated from the same DGM.Source: §4.2.1 · p. 11

4.3 Methods

  • Researchers fitted random forests with minimum node sizes 2 or 20 across 192 scenarios, running 1,000 training datasets per scenario and validating against one large test dataset per DGM.Training sample sizes were 200 or 4,000; each test dataset had N=100,000.Source: §4.3 · pp. 11–12

4.4 Performance

  • Performance was measured using AUC for discrimination, calibration slope for risk-estimate confidence, and mean squared error for probability prediction error.Calibration slopes below 1 indicate overconfidence; slopes above 1 indicate underconfidence.Source: §4.4 · p. 12

4.5.1 Discrimination

  • Training AUC was 0.97–1 in most scenarios, with exceptions for 4 binary predictors or 16 binary predictors paired with minimum node size 20.Higher minimum node size produced less extreme training AUC.Source: §4.5.1 · p. 12, Figure 4 · p. 13
  • Median test AUC was generally higher with larger training samples, higher minimum node size, greater predictor correlation, binary predictors, and fewer predictors, except for correlated continuous predictors.The reported pattern compares each factor’s higher versus lower setting, or binary versus continuous predictors.Source: §4.5.1 · p. 12, Figure 5 · p. 14

4.5.2 Calibration

  • Training calibration slopes ranged from 1.10 to 19.4; perfect training separation implied underconfidence, and median training/test slopes had Spearman correlation -0.11.Median test slopes were mainly higher with higher true AUC or minimum node size.Source: §4.5.2 · p. 14

4.5.3 Mean squared error (MSE)

  • Median test MSE was 0.010 in scenarios with median training AUC ≥0.99, versus 0.006 in the remaining scenarios.Across all scenarios, median MSE was 0.008.Source: §4.5.3 · p. 16

5 Overall discussion

This discussion examines how random forests learn local probability peaks around training events and how this behavior affects training performance and risk estimates. It considers the implications for probability estimation, including the recommendation against fully grown trees.

5 Overall discussion

  • RFs create local probability peaks around training events—larger regions for clustered events and local peaks for isolated events—especially with deep trees and continuous predictors.Heatmaps also showed that probabilities could change non-smoothly as predictor values changed.Source: §5 · pp. 17–18
  • Because probability peaks are very local, learning through peaks often yields near-perfect training discrimination, while reducing discrimination on new data; for isolated events, that reduction is probably modest because exact-location recurrence is unlikely.Source: §5 · p. 17
  • RF training calibration slopes always exceeded 1, unlike LR’s slope of 1; RF slopes also do not converge to 1 on new data.The authors attribute this difference probably to RF’s pragmatic probability estimation, compared with LR’s principled estimation.Source: §5 · p. 18
  • Simulation results argue against very deep trees for probability estimation; logloss tuning selected min.node.size values of 8 for ovarian cancer, 261 for CRASH3, and 591 for IST.The authors recommend reporting internal and external validation rather than presenting training discrimination alone; calibration slopes require cautious interpretation for RF.Source: §5 · pp. 18–19
  • Bootstrap averaging helps explain RF’s training pattern: 63% of tree-level proportions are near-perfect, driving high discrimination, while the remaining 37% pull estimates away from 0 or 1, yielding slopes > 1.Each training case appears in about 63% of bootstrap samples and is therefore used by about 63% of the forest’s trees.Source: §5 · p. 19
  • The study was limited to included scenarios and logistic DGMs without nonlinear or nonadditive outcome associations; tuning and additional simulation factors remain for future work.The authors note that tuning could improve calibration, while adding simulation factors to the existing 192 scenarios would increase computational cost and complicate interpretation.Source: §5 · p. 20
  • Continuous-variable splits may overfit, increasing training discrimination while decreasing test discrimination; adapted random forests such as cforest use conditional inference trees instead.Source: §5 · p. 21
  • In case studies, tuned adapted random forests produced less optimistic training AUC and similar or slightly better test performance.The authors did not systematically explore tuning or alternative algorithms because their aim was to understand how data and modeling characteristics affected the models.Source: §5 · p. 21
  • Deeply grown random forests can overfit locally, yielding near-perfect training discrimination but reduced discrimination on new data compared with less deeply grown trees.The authors conclude that this runs counter to recommending fully grown trees for probability estimation.Source: §5 · p. 21

Declarations

The declarations state that ethics approval and consent were not applicable, and report funding, data availability, and competing-interest information.

Declarations

  • Ethics approval and consent to participate were both reported as not applicable.Source: Declarations · p. 24
  • Study data and code are openly available in the OSF repository, except for the ovarian cancer dataset, which is not publicly available.Source: Declarations · p. 24

Competing interest

  • The authors reported no conflict of interest and acknowledged FWO grant G097322N, with BVC and DT as supervisors.Source: Competing interest · p. 24
  • Partial findings were presented at the 2023 Young Statisticians Meeting in Leicester and the 2023 ISCB conference in Milan.Source: Acknowledgements · p. 24

Additional File

The additional file includes supplementary dataset-distribution tables, simulation materials, and figures covering probability estimates, performance, calibration, correlations, and mean squared error.

Overview Table for Additional File

A.1 Tables

The supplementary tables include algorithm performance and dataset class distributions, while an accompanying note specifies a fixed-variable CRASH subset for test figures.

A.1 Tables

  • Table S1 compares machine-learning algorithms using PDI for ovarian tumour type and AUC for benign-versus-malignant ovarian malignancy.Source: Table S1 · p. 27
  • Table S2 reports class distributions for the IOTA dataset.Source: Table S2 · p. 28
  • The CRASH test-figure subset fixes propsol=1, papnr=0, Ascites=0, Shadows=0, and loc10=0.Source: §A.1 · p. 28

CLASS ALL TRAIN TEST IN FIGURES S5-S6 (TRAIN)A

The IST dataset footnote specifies the categorical-variable values used for the subset.

CLASS ALL TRAIN TEST IN FIGURES S5-S6 (TRAIN)A

CLASS ALL TRAIN TEST IN FIGURES S10-S11 (TRAIN)A

The supplied material documents data-space probability visualizations for three clinical datasets alongside simulation figures, a main-results table, and details of the simulation algorithm and metrics.

CLASS ALL TRAIN TEST IN FIGURES S10-S11 (TRAIN)A

  • Table S6 is identified as the main simulation-results table and reports interquartile ranges and area under the ROC curve.Source: Table S6 · p. 31

A2 Figures

  • Ovarian-malignancy panels compare random-forest and multinomial-logistic-regression probability estimates for training and test cases.Source: Figure S1 · p. 34, Figure S3 · p. 36
  • CRASH 3 panels show random-forest and logistic-regression probability estimates for three outcomes, with separate training and test cases.Source: Figure S5 · p. 38, Figure S7 · p. 40
  • IST panels compare random-forest and logistic-regression probability estimates for four stroke types, with separate training and test cases.Source: Figure S10 · p. 43, Figure S12 · p. 45

A.3 Simulation Algorithm

  • The simulation algorithm generates correlated multivariate-normal predictors and can binarize values using a threshold.Source: §A.3 · pp. 57–58
  • It combines predictors and coefficients into linear predictors, converts these to probabilities with the logistic function, and draws binary outcomes from a one-trial binomial distribution.Source: §A.3 · p. 58

A.4 Simulation metrics

  • Perfectly separated predicted probabilities are transformed to the closest possible value according to the number of trees and training samples.Source: §A.4 · p. 59
Loading 2402.18612v2…