PaperlayerView exact revision on arXiv

Source-linked AI summary

On the use of Harrell's C for clinical risk prediction via random survival forests

Matthias Schmid, Marvin Wright, Andreas Ziegler

arXiv:1507.03092v2stat.ML

TL;DR

RSF commonly uses log-rank splitting but evaluates prediction accuracy with Harrell’s C, so the split criterion may not optimize the measure of interest. This paper analyzes and tests using Harrell’s C for both splitting and evaluation, finding advantages over log-rank splitting in specified settings and recommending criteria by study scale.

  • Problem

    RSF commonly uses log-rank splitting but evaluates prediction accuracy with Harrell’s C, so the split criterion differs from the measure of interest.

    Source: §1 · p. 3

  • Method

    The paper analyzes Harrell’s C and log-rank splitting and evaluates replacing log-rank splits with C-based node splits in RSF.

    Source: §1 · p. 3, §4 · p. 21

  • Results

    C-based splitting performed better than log-rank splitting in high-signal-to-noise settings and when informative continuous predictors were numerous relative to categorical predictors; differences were little when all predictors were categorical.

    Source: §4 · p. 22

  • Takeaways & Limitations

    The authors recommend Harrell’s C splitting for smaller-scale clinical studies and log-rank splitting for large-scale omics studies.

    Source: §4 · p. 23

    C-based splitting requires estimating IPC weights, adding computational cost and making it infeasible in deeply grown trees with small minimum node size.

    Source: §2.3 · p. 9

Abstract

from arXiv · show

Random survival forests (RSF) are a powerful method for risk prediction of right-censored outcomes in biomedical research. RSF use the log-rank split criterion to form an ensemble of survival trees. The most common approach to evaluate the prediction accuracy of a RSF model is Harrell's concordance index for survival data ('C index'). Conceptually, this strategy implies that the split criterion in RSF is different from the evaluation criterion of interest. This discrepancy can be overcome by using Harrell's C for both node splitting and evaluation. We compare the difference between the two split criteria analytically and in simulation studies with respect to the preference of more unbalanced splits, termed end-cut preference (ECP). Specifically, we show that the log-rank statistic has a stronger ECP compared to the C index. In simulation studies and with the help of two medical data sets we demonstrate that the accuracy of RSF predictions, as measured by Harrell's C, can be improved if the log-rank statistic is replaced by the C index for node splitting. This is especially true in situations where the censoring rate or the fraction of informative continuous predictor variables is high. Conversely, log-rank splitting is preferable in noisy scenarios. Both C-based and log-rank splitting are implemented in the R~package ranger. We recommend Harrell's C as split criterion for use in smaller scale clinical studies and the log-rank split criterion for use in large-scale 'omics' studies.

Highlights

The paper proposes Harrell’s C as an alternative split criterion for random survival forests and compares it with log-rank splitting. The criteria differ in their split choices and end-cut preference, with C-based splitting outperforming log-rank splitting in smaller-scale studies and when censoring is high.

  • Harrell’s C is proposed as a split criterion for random survival forests.Source: Highlights · p. 2
  • Continuous predictors can receive substantially different split points under Harrell’s C and log-rank splitting.Source: Highlights · p. 2
  • The log-rank statistic has a stronger end-cut preference than Harrell’s C.Source: Highlights · p. 2
  • Harrell’s C outperforms log-rank splitting in smaller-scale studies.Source: Highlights · p. 2
  • Harrell’s C outperforms log-rank splitting when the censoring rate is high.Source: Highlights · p. 2

1. Introduction

Random forests use ensembles of trees with randomized predictor selection, and random survival forests apply this approach to right-censored outcomes. Because RSF commonly uses log-rank splitting but is evaluated with Harrell’s C, this work investigates using Harrell’s C for both splitting and evaluation.

  • Random forests average predictions from bootstrap-grown trees while random predictor selection decorrelates trees and reduces ensemble prediction variance.The random selection of predictors distinguishes random forests from earlier bagging methods.Source: §1 · p. 2
  • Random forests have been extended to analyze right-censored event times, alongside methods for variable importance and categorical or continuous outcomes.Source: §1 · p. 2
  • RSF builds survival-tree ensembles using log-rank splits, but its common evaluation criterion is Harrell’s C, leaving split selection misaligned with the performance measure.The passage notes that log-rank splits may not fully optimize Harrell’s C.Source: §1 · p. 3
  • The study investigates whether replacing log-rank splits with Harrell’s C improves RSF prediction performance when C is also used for evaluation.Source: §1 · p. 3
  • The paper analyzes the split criteria theoretically and evaluates C-based splitting in simulations and two cancer-data re-analyses.The authors report systematic improvements in predictive performance from these analyses.Source: §1 · p. 3

2. Methods

The methods describe how RSF grows and evaluates survival-tree ensembles, then develop and apply Harrell’s C as an alternative node-splitting criterion. The analysis assumes independent, identically distributed observations and defines the quantities used to represent outcomes and predictors.

2.1. Random survival forests

  • RSF uses bootstrap samples to grow trees, with ntree and mtry specified before fitting; each node splits on the variable and threshold maximizing the log-rank statistic.Tree growth continues while the number of uncensored observations exceeds the specified terminal-node size.Source: §2.1 · p. 4, §2.2 · p. 5
  • RSF performance is evaluated on independent test data or, when unavailable, out-of-bag data, using terminal-node survival estimates averaged across trees.Source: §2.1 · p. 4, Notation · p. 6

2.2. The C statistic and the log-rank statistic as split criteria

  • The analysis treats Harrell’s C and log-rank splitting as special cases of the Gehan statistic, obtained through different standardization and weighting schemes.Source: §2.2 · p. 4
  • Harrell’s C can replace the log-rank statistic to select the predictor-threshold combination at each node; the section also introduces the statistics and their relationships.Source: §2.2 · p. 5

Notation

  • The data are assumed iid, with possibly right-censored survival times and predictor vectors; tied observed survival times are excluded.Source: Notation · p. 5
  • Predictors are classified as informative when survival-time distributions depend on them, and event counts and at-risk counts are indexed across ordered event times.RSF assigns each observation a score by averaging terminal-node cumulative hazard estimates across trees and event times.Source: Notation · p. 6

Definition of Harrell’s C

Harrell’s C summarizes concordance among comparable survival-time pairs and is adapted for evaluating a candidate split into two child nodes. Its values range from non-informative ranking at 0.5 to perfect association at 1.

  • Harrell’s C is the number of concordant pairs divided by the number of comparable pairs.The event indicator excludes pairs that cannot be compared because the shorter observed time is censored.Source: Definition of Harrell’s C · p. 6
  • Harrell’s C estimates the probability that predicted scores rank two observations consistently with their survival times.A value of 0.5 corresponds to a non-informative prediction rule, while 1 corresponds to perfect association.Source: Definition of Harrell’s C · p. 6
  • Harrell’s C was previously proposed for evaluating RSF predictive performance.Source: Definition of Harrell’s C · p. 6
  • For split evaluation, observations are assigned to child groups and the concordance probability is estimated from pairs crossing those groups.Pairs in the same child node receive a value of 0.5 in the split-specific calculation.Source: Definition of Harrell’s C · pp. 6–7

The log-rank statistic

The log-rank statistic is used to compare groups in survival data and is a popular split criterion in survival trees, including random survival forests.

The log-rank statistic

The Gehan statistic

The Gehan statistic compares survival times across two groups using comparable pairs. It is connected to both the log-rank statistic and Harrell’s C through weighting, standardization, and shared pairwise terms.

The Gehan statistic

  • The Gehan statistic assesses whether survival times in G0 are systematically larger than those in G1.It considers pairs only when the smaller survival time is not censored.Source: The Gehan statistic · p. 7

Relationship between the Gehan statistic and the log-rank statistic

  • The Gehan-Wilcoxon statistic is a weighted version of the log-rank statistic, with greater weight on events at early time points.The Gehan statistic is standardized and squared to obtain the Gehan-Wilcoxon statistic.Source: Relationship between the Gehan… · p. 8

Relationship between the Gehan statistic and Harrell’s C

  • Harrell’s C is linearly related to the Gehan statistic, and both depend on a term involving comparable cross-group pairs.N is the number of possible comparisons between G0 and G1; N0 and N1 count comparable pairs within each group.Source: Relationship between the Gehan… · p. 8

Conclusion about the relationships

Harrell’s C and the log-rank statistic share the Gehan statistic as a basis but transform it differently. The paper describes using Harrell’s C for RSF node splitting and notes computational constraints for inverse-probability weighting.

Conclusion about the relationships

  • Harrell’s C and the log-rank statistic are both based on the Gehan statistic but apply different standardization and weighting schemes.Source: Conclusion about the relationships · p. 8
  • Both criteria include terms that become large for unbalanced groups: the log-rank denominator depends on Y1k · Y0k, while Harrell’s C depends on N0 + N1.The products Y1k · Y0k become small when groups are unbalanced, increasing the statistics.Source: Conclusion about the relationships · p. 9

2.3. Use of Harrell’s C as split criterion in RSF

  • At each RSF node, the proposed procedure evaluates Harrell’s C across candidate variables and split points, switching labels when C < 0.5 and selecting the maximum.Source: §2.3 · p. 9
  • Uno’s C is not proposed for splitting because estimating inverse-probability-of-censoring weights at each node is computationally costly and infeasible in deeply grown trees with small minimum node size.Simulation study 2 showed that conclusions were not altered when Harrell’s C was used instead of Uno’s C.Source: §2.3 · p. 9
  • Both C-based and log-rank splitting are implemented in the R package ranger.Source: §2.3 · p. 9

2.4. Simulation study 1 – threshold selection

Simulation study 1 compared how C and log-rank splitting select thresholds, while simulation study 2 compared their effects on RSF prediction accuracy. The studies varied predictor informativeness, censoring, predictor type, and noise.

2.4. Simulation study 1 – threshold selection

  • For simulation study 1.a), 1,000 observations used a non-informative uniform predictor and 50% censoring to compare threshold selection.Study 1.b) used informative threshold models with true cutpoints 0.25 or 0.75, three censoring rates, and 1,000 replications.Source: §2.4 · p. 10

2.5. Simulation study 2 – predictive performance of RSF

  • Simulation study 2 generated survival times from four informative predictors through a probability tree and Weibull model, then varied censoring and added noise predictors.It evaluated continuous and dichotomized predictors, with sample sizes n = 100 or 300 and 25%, 50%, or 75% censoring.Source: §2.5 · pp. 10–11, Figure 2 · p. 12
  • RSF prediction accuracy was evaluated using Harrell’s C, with Uno’s C also evaluated for continuous predictors, on 500 independent test sets.Models used 500 trees and minimum terminal node size 3.Source: §2.5 · p. 11

2.6. Data analysis 1 – gene expression data to predict the time to distant metastases in breast cancer

The breast cancer analysis assessed RSF with clinical, gene-expression, and added noise variables; the lymphoma analysis examined a high-dimensional gene-expression setting with unknown signal-to-noise ratio.

2.6. Data analysis 1 – gene expression data to predict the time to distant metastases in breast cancer

  • The breast cancer analysis used 196 node-negative patients and 76 gene-expression variables plus five clinical variables to predict time to distant metastases.Survival times were censored at 10 years, and 79.08% were censored.Source: §2.6 · p. 12
  • Breast cancer RSF models were evaluated on held-out samples across three learning sizes, first with clinical variables, then with expression data, and finally with added noise variables.The high-dimensional step added 1,000 to 5,000 non-informative variables with pairwise correlation ρ = 0.5.Source: §2.6 · pp. 12–13

2.7. Data analysis 2 – survival in patients with diffuse large B-cell lymphoma

  • The lymphoma analysis used 7,399 gene-expression values from 240 patients to predict overall survival after chemotherapy, with 42.5% censored survival times.The analysis aimed to compare splitting rules where predictor signal-to-noise ratio was unknown.Source: §2.7 · p. 13
  • Lymphoma RSF models used learning samples of n = 80, 120, or 160 and were evaluated on held-out samples of 80 patients.Each learning size used 100 subsamples.Source: §2.7 · p. 13

3. Results

Across simulations and two medical data analyses, C-based splitting often improved RSF prediction accuracy, particularly with continuous predictors and higher censoring. Its advantage diminished as noise increased, while log-rank splitting could perform better in noisy or dichotomized settings.

3.1. Simulation study 1 – threshold selection

  • C-selected thresholds were more centered and produced more balanced splits than log-rank thresholds for both non-informative and informative continuous predictors.For the non-informative predictor, C thresholds peaked near the interval midpoint while log-rank thresholds peaked near its boundaries.Source: §3.1 · p. 14, Figure 4 · p. 15

3.2. Simulation study 2 – predictive performance of RSF

  • C-based splitting improved RSF performance in most continuous-predictor scenarios, with gains tending to increase at higher censoring; more noise predictors favored log-rank splitting.In simulation study 2.a), performance differences were evaluated using Harrell’s C and Uno’s C, with similar results.Source: §3.2 · pp. 15–16, Figure 5 · p. 17
  • With dichotomized predictors, the split rules produced similar performance in most scenarios; log-rank splitting outperformed C-based splitting in the low-dimensional p = 10 scenario.Most performance-difference boxplots were near zero.Source: §3.2 · p. 16, Figure 6 · p. 18

3.3. Data analysis 1 – gene expression data to predict the time to distant metastases in breast cancer

  • 0.054 (95% CI [0.039, 0.063]) median improvement in Harrell’s C at n = 130 for the combined breast cancer model, about an 8% gain.The combined model included clinical and gene-expression variables; gains were observed across learning sample sizes, while the clinical-only model showed only a small increase.Source: §3.3 · pp. 16–18, Figure 7 · p. 18
  • As added noise variables increased, C-based splitting’s performance decreased relative to log-rank splitting in the high-dimensional breast cancer analysis.Figure 8 summarizes the comparison across added noise-variable counts.Source: §3.3 · p. 18, Figure 8 · p. 19
  • C-based and log-rank splitting selected substantially different variable-importance rankings: only 19 of the 30 top log-rank predictors appeared among the C-based top 30.Estrogen receptor status ranked 10th with C-based splitting and 28th with log-rank splitting; the authors describe Figure 9 as illustrative rather than a systematic investigation.Source: §3.3 · pp. 18–19, Figure 9 · p. 20

3.4. Data analysis 2 – survival in patients with diffuse large B-cell lymphoma

  • 0.028 (95% CI [0.018, 0.041]) median performance gain from C-based splitting at n = 160 in the lymphoma analysis, corresponding to about 5%.C-based splitting showed a gain at every learning sample size; Figure 10 presents the performance differences.Source: §3.4 · pp. 20–21, Figure 10 · p. 21

4. Discussion

The discussion reports that Harrell’s C improves RSF performance in many settings, while log-rank splitting remains preferable in noisy scenarios. It recommends C-based splitting for smaller clinical studies and log-rank splitting for large-scale ‘omics’ studies.

  • Using Harrell’s C for both splitting and evaluation creates a unified analysis strategy.Source: §4 · p. 21
  • Log-rank splitting’s stronger end-cut preference favors RSF performance in noisy scenarios, whereas C-based splitting performs better with high signal-to-noise.Source: §4 · p. 22
  • C-based splitting gains when informative continuous predictors are numerous, but makes little difference when all predictors are categorical.Threshold-selection differences become most apparent with continuous predictors.Source: §4 · p. 22
  • C-based splitting tends to outperform log-rank splitting in data sets with high censoring rates.Source: §4 · p. 22
  • Harrell’s C as the split criterion improves RSF performance across a wide range of settings.Source: §4 · p. 22
  • The authors recommend C-based splitting for smaller clinical studies and log-rank splitting for large-scale ‘omics’ studies.Source: §4 · p. 23

Related papers

Loading 1507.03092v2…