PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Ways to Build and Evaluate Individual Survival Distributions

Humza Haider, Bret Hoehn, Sarah Davis, Russell Greiner

arXiv:1811.11347v1cs.LGstat.ML

TL;DR

Standard survival tools do not provide calibrated, patient-specific probabilities across all future times. The paper develops and evaluates individual survival distribution models, including a whole-distribution calibration measure, finding that mtlr is generally strongest across diverse datasets while survival-curve interpretation remains constrained by curve behavior and prediction-error limitations.

  • Problem

    Risk scores lack survival probabilities, single-time models cover only one time point, and Kaplan-Meier curves provide population averages rather than individual survival distributions.

  • Method

    The paper studies ISD models and evaluates them with Concordance, 1-Calibration, Brier score, L1-loss variants, and D-Calibration across eight datasets.

  • Results

    Across diverse datasets, mtlr was superior or competitive for major performance measures and outperformed or matched other models on calibration; it was D-Calibrated for all datasets and 1-Calibrated across evaluated percentiles.

  • Takeaways & Limitations

    ISD models provide a flexible framework that can support multiple evaluation measures and expose survival probabilities across future times through interpretable curves.

  • Takeaways & Limitations

    The study focuses on fixed feature representations without missing values and right-censored observations, and some predicted curves end above zero or lack a defined median.

Abstract

from arXiv · show

An accurate model of a patient's individual survival distribution can help determine the appropriate treatment for terminal patients. Unfortunately, risk scores (e.g., from Cox Proportional Hazard models) do not provide survival probabilities, single-time probability models (e.g., the Gail model, predicting 5 year probability) only provide for a single time point, and standard Kaplan-Meier survival curves provide only population averages for a large class of patients meaning they are not specific to individual patients. This motivates an alternative class of tools that can learn a model which provides an individual survival distribution which gives survival probabilities across all times - such as extensions to the Cox model, Accelerated Failure Time, an extension to Random Survival Forests, and Multi-Task Logistic Regression. This paper first motivates such "individual survival distribution" (ISD) models, and explains how they differ from standard models. It then discusses ways to evaluate such models - namely Concordance, 1-Calibration, Brier score, and various versions of L1-loss - and then motivates and defines a novel approach "D-Calibration", which determines whether a model's probability estimates are meaningful. We also discuss how these measures differ, and use them to evaluate several ISD prediction tools, over a range of survival datasets.

1 Introduction

The paper motivates individual survival distributions because standard tools provide either relative risk, a probability at one time, or population-average curves rather than patient-specific probabilities across time. It introduces ISD models and evaluation measures for assessing whether their predictions are useful and calibrated.

  • Risk scores provide relative survival measures, single-time models provide one probability, and Kaplan-Meier curves describe population averages rather than individual patients.
  • Individual survival distributions incorporate patient-specific attributes to predict a novel patient’s survival probabilities across multiple time points.
  • Meaningful survival curves are necessary because treatment and prognosis decisions depend on whether predicted probabilities accurately reflect a patient’s actual survival chances.
  • A survival curve maps each time to the predicted probability of surviving at least that long, such as 50% at 11 months or 95% at 2 months.
  • The paper evaluates ISD and related models using Concordance, 1-Calibration, Brier score, L1-loss variants, and novel D-Calibration across eight diverse datasets.

2 Summary of Various Survival Analysis/Prediction Systems

The paper organizes survival systems by whether they return risk or probability, one value or a time range, and population-level or individual-specific predictions. It then presents ISD models that produce patient-specific survival curves and explains their broader functionality and interpretive challenges.

  • The survival-learning setup uses patient features, observed or censored event times, and an indicator distinguishing death from censoring.
  • Survival systems are categorized by result type, temporal scope, and whether predictions are individual-specific or population-level.
  • Risk models such as Cox return a single atemporal score for ranking patients, whereas single-time predictors return a survival probability for one fixed time.
  • 2.5 [P,∞,i]: Individual Survival Distribution, isd (cox-kp, coxen-kp, aft, rsf-km, mtlr): An ISD model produces a patient-specific survival distribution, and the paper considers cox-kp, coxen-kp, aft, rsf-km, and mtlr.
  • D-Calibration evaluates the entire distribution of estimated survival probabilities rather than a probability at only one time.
  • Distributional models can provide probabilities at arbitrary future times, mean or median survival, and visualizable curves, but some curves end above zero or never reach 0.5, making interpretation and evaluation difficult.

3 Measures for Evaluating Survival Analysis/Prediction Models

The paper evaluates survival prediction models with measures targeting discrimination, single-time calibration, distributional accuracy, and calibration across entire individual survival distributions. D-Calibration extends calibration assessment from one prediction time to the full survival curve.

  • Concordance: Concordance measures whether predicted risk rankings agree with observed death-time orderings across comparable patient pairs.It is relevant when the goal is to discriminate between patients, and time-dependent variants evaluate risk scores that vary with time.
  • 1-Calibration: 1-Calibration evaluates single-time survival probabilities by binning predictions and comparing observed event frequencies with expected probabilities using a Hosmer-Lemeshow test.For ten predictions of a 30% event probability, calibration implies approximately three events; the same logic applies across probability bins.
  • 1-Calibration: For Figure 6, the Hosmer-Lemeshow statistics are 5.99 and 321.44, with p-values p = 0.741 and p < 0.001, respectively.The left model passes the test because observed and expected bin values are similar, whereas the right model does not.
  • Comparing calibration measures: D-Calibration and 1-Calibration can disagree: a model may be calibrated at some time points but not others, and the two measures are not equivalent.The paper gives examples of models that are perfectly calibrated by one measure but not the other; an mtlr model’s NACD distribution has χ2 goodness-of-fit p = 0.433.
  • D-Calibration: A model producing survival distributions requires evaluation of the entire curve rather than only a risk score, a single-time probability, or a mean survival time.D-Calibration addresses this distribution-level evaluation need.
  • D-Calibration: D-Calibration bins each patient’s predicted survival probability at their observed death time and applies a goodness-of-fit assessment across those probabilities.Unlike 1-Calibration, which uses a specified time t∗, D-Calibration uses each patient’s full predicted curve and associated death time.

4 Evaluating isd Models

The paper compares six survival-distribution models across eight datasets using discrimination, calibration, and error measures. MTLR generally performs best, but model performance varies with dataset structure, censoring, dimensionality, and model assumptions.

  • Evaluation design: The evaluation compares six models across eight datasets using Concordance, L1-loss, 1-Calibration, Integrated Brier score, and D-Calibration.The models include Kaplan-Meier, cox-kp, coxen-kp, aft, mtlr, and rsf-km.
  • Evaluation design: The eight datasets span 170–2402 patients, 12–7401 features, and 17.23%–86.21% censoring.They were selected to cover varied sample sizes, feature counts, and censoring proportions, while excluding datasets with fewer than 150 instances.
  • Overall results: MTLR performs best on six of eight datasets for Concordance and L1-loss, and seven of eight for Integrated Brier score.This is the reported majority-dataset result for the three measures.
  • Dataset-specific results: On the four Nice datasets, most models perform comparably, while rsf-km performs substantially worse on GBM and GLI across the three reported measures.Aft and cox-kp are nearly as strong as the more complex models, and aft achieves the best L1-loss result on GBM.
  • Dataset-specific results: On high-censor datasets, metric variance is higher on READ because few uncensored patients occur in each test fold, whereas this pattern is absent on larger BRCA.Coxen-kp and mtlr are similar, while rsf-km performs consistently worse across the three measures on both datasets.
  • Dataset-specific results: On high-dimensional datasets, unregularized aft performs poorly across all measures, while coxen-kp, mtlr, and rsf-km perform similarly.Cox-kp failed to run on DBCD and DLBCL, likely because of their large feature counts.
  • 1-Calibration results: MTLR is 1-Calibrated on at least four of eight datasets at the 10th, 25th, 50th, and 90th survival-time percentiles, outperforming the other models.The 90th percentile is especially challenging: mtlr passes on four datasets, while aft, cox-kp, and rsf-km pass on none.
  • D-Calibration results: Both Kaplan-Meier and mtlr pass D-Calibration on every dataset, but only mtlr provides each patient with an individual survival curve.Coxen-kp passes on seven of eight datasets, rsf-km on five, and aft and cox-kp on three each; the histograms show acceptable variability for mtlr and near-perfect alignment for Kaplan-Meier.

5 Discussion

The discussion finds that MTLR generally provides the strongest calibrated individual survival distributions, while model choice depends on dataset dimensionality, censoring, and evaluation metric. It also emphasizes that discrimination and L1-loss alone may not establish useful patient-specific survival estimates.

  • MTLR scored best on Concordance for six of eight datasets, while RSF-KM scored best on the other two.
  • MTLR was the only ISD model D-Calibrated across all datasets, whereas AFT and Cox-KP achieved D-Calibration on only three datasets.The authors suggest AFT's parametric form may limit the shapes of survival curves it can produce.
  • MTLR performed best on Integrated Brier score for seven of eight datasets, but CoxEN-KP, RSF-KM, and MTLR were nearly identical on high-dimensional DBCD.Integrated Brier score combines calibration and discrimination.
  • For high-dimensional datasets, CoxEN-KP, RSF-KM, and MTLR were similarly competitive, whereas RSF-KM performed poorly on low-dimensional datasets and tended to overfit.The authors report that additional meta-parameter tuning did not correct RSF-KM's overfitting.
  • MTLR had the lowest L1-loss on six of eight datasets, but a lowest error does not necessarily make predicted survival times clinically useful.On NACD, MTLR's average L1-loss was 43.97 months, approximately 3.7 years.
  • The discussion recommends simpler models for some Nice datasets, MTLR or CoxEN-KP for high-censor settings with few features, and avoiding RSF-KM when there are fewer than 34 features.For high-dimensional, low-censoring data, MTLR, CoxEN-KP, and RSF-KM were strongest for calibration, while RSF-KM appeared weaker for discrimination.
  • ISD models are valuable because they produce complete patient-specific survival curves, unlike risk scores or single-time probabilities, enabling evaluations beyond patient ranking.The paper notes that single-time models require pre-specifying which time points to predict.

6 Conclusion

The paper surveys individualized survival-distribution models and evaluation criteria, concluding that mtlr generally performs strongly, especially on calibration metrics, while the analysis remains scoped to common right-censored data settings.

  • The analysis assumes fixed features, no missing values, and observations that are either uncensored or right-censored.
  • The paper focuses on individual survival distributions and discusses their advantages for patient-specific survival probabilities and informed decisions.
  • mtlr was typically superior for L1-loss, Integrated Brier score, and Concordance, and outperformed or matched other models on calibration metrics.
  • The paper encourages researchers and practitioners to use ISD models, especially models similar to mtlr, for meaningful survival-analysis tools.

A Extending Survival Curves to 0

The appendix addresses survival curves that stop above zero by linearly extrapolating them to zero, while handling extreme infinite predictions with a Kaplan–Meier-based fallback.

  • Survival curves ending at non-zero probability are incomplete distributions and complicate mean and median survival calculations.
  • The chosen extension draws a line from (0, 1) to the last calculated survival probability and continues it until survival reaches zero.
  • Because the extension cannot cross between curves, it does not change discriminatory criteria.
  • For the Nacd-Col dataset, mtlr curves ended at 68.9 months, while their linear extensions reached 118 months.
  • When unregularized models predict survival probabilities of 1 indefinitely, the method uses a Kaplan–Meier-based value to replace infinite predictions and truncates median survival times accordingly.

B Evaluation Measures Supplementary Information

The appendix provides additional information about the paper’s evaluation measures.

  • The appendix supplies supplementary information about the various evaluation measures.

B.1 Concordance

Concordance evaluates discriminatory ordering using comparable patient pairs, which are defined by observed event ordering despite censoring; ties receive a partial score in the empirical evaluations.

  • Concordance is designed to measure discriminative ability, but censoring makes some patient pairs incomparable.
  • The comparable-pair set is defined as CP(V) = {[i, j] ∈ V × V | t_i < t_j and δ_i = 1}.
  • The Concordance calculation counts how often the predictor orders risk scores correctly for comparable pairs.
  • The empirical evaluations assign tied risk scores or death times a score of 0.5 rather than removing tied pairs.
  • Comparable pairs include cases where an uncensored patient dies before another patient’s censoring time or death time.

B.2 L1-loss, and variants

The paper compares L1-based survival-time losses that handle censoring by discarding censored cases, using hinge bounds, or assigning weighted conditional best guesses. It also motivates Log-L1-loss for relative error and describes 1-Calibration through binned predicted probabilities.

  • L1-loss variants: Uncensored L1-loss removes censored patients, while hinge loss assigns zero loss when censoring occurs before the estimated median survival time.For later censoring, hinge loss uses the positive difference between the censoring time and estimated median survival time.
  • L1-loss variants: The hinge formulation is an optimistic lower bound because it treats unknown death times as the best feasible values under censoring constraints.This can favor models that substantially overestimate survival, especially when many patients are censored.
  • L1-Margin loss: L1-Margin loss assigns censored patients a Best-Guess death time equal to their conditional expected survival time given survival to censoring.The estimate is computed from the survival function and implemented with a Kaplan-Meier estimate from the training data.
  • L1-Margin loss: Best-Guess errors are weighted by censoring time: early censoring receives little weight, whereas late censoring is treated nearly like an observed death.The implementation sets α_k = 1−Ŝ_KM(c_k).
  • Log-L1-loss: Log-L1-loss measures relative rather than absolute error, distinguishing equal absolute errors that have very different proportional meanings.The paper illustrates this with 3 months representing 2.5% error for a 120-month prediction but 300% for a 1-month prediction.
  • 1-Calibration: 1-Calibration compares observed deaths with predicted deaths across ten bins formed by sorting predicted probabilities.Each bin’s expected deaths equal its average predicted probability multiplied by the number of patients, and the Hosmer–Lemeshow test assesses discrepancy.

B.4.2 Integrated Brier score does not involve the Entire Distribution

The Integrated Brier score evaluates prediction error over time but does not use the full probability distribution represented by individual survival curves. The paper contrasts this with D-Calibration, which tests whether probabilities are distributed meaningfully across survival outcomes.

  • Metric distinction: Integrated Brier score does not require survival curves to distribute probability mass meaningfully across all time intervals.A perfect Heaviside curve can achieve IBS(·,·) = 0 even though it assigns no probabilities within intermediate intervals.
  • Metric distinction: D-Calibration penalizes failure to distribute deaths across probability intervals as expected, whereas Integrated Brier score can overlook this issue.Meaningful survival distributions should retain non-zero probabilities beyond a single time point because outcomes are stochastic.
  • Censored Brier score: The censored Brier score uses inverse probability of censoring weights estimated from the censoring Kaplan-Meier curve.The weighting compensates for the increasing likelihood that long-surviving patients are censored.

B.5 D-Calibration

D-Calibration tests whether predicted survival probabilities form the distribution expected under a correct individual survival model. The procedure uses probability bins, incorporates censored patients by spreading their weight across compatible bins, and relies on monotonicity and independent censoring assumptions.

  • Uniformity principle: For uncensored patients, the predicted survival probability at the time of death should be uniformly distributed on [0,1].This follows from the probability integral transform when survival curves are the true curves.
  • Assumptions: D-Calibration assumes right-censored observations provide lower bounds on death times and requires censoring to be independent of death time.The analysis defines observed time as t_i = min{d_i,c_i}.
  • Censoring adjustment: Censored patients are blurred across bins below the survival probability at censoring, reflecting uncertainty about their eventual death time.Early censoring spreads weight broadly, while very late censoring can contribute entirely to one final bin.
  • Limitation: D-Calibration can be artificially supported when many validation patients are censored at time 0, because their weights are spread equally across all bins.This weakness can make the bins appear uniform regardless of how other patients are distributed.
  • Calibration test: Theorem B.2 states that, under a strictly decreasing true survival function, expected bin proportions are equal: E[N_k] = p_{k+1} − p_k.This equal-bin expectation enables a goodness-of-fit test against a uniform distribution.
  • Results: Kaplan-Meier is asymptotically D-Calibrated when survival times are identically distributed, and it passed the D-Calibration test in all eight datasets with p-value 1.000.The paper attributes the high p-values under censoring to smoothing across bins.
  • Metric distinction: D-Calibration and 1-Calibration assess different properties: a model can be perfect on one measure while failing the other.The paper gives explicit examples establishing that neither calibration notion implies the other.

B.6 Other Subtle Points

The paper identifies scope limitations from right-censoring and modeling assumptions, then describes Multi-Task Logistic Regression as a flexible alternative that models survival across multiple time points. MTLR encodes survival as structured binary sequences and smooths the resulting curve.

  • Censoring scope: The evaluation considers only right-censored data, where the censored time is a lower bound on death time.Left- and interval-censoring extensions exist but are outside the datasets analyzed here.
  • Modeling assumptions: The Cox formulation assumes independent features and time-invariant covariate effects, producing survival curves with the same basic shape that do not cross.These are described as strong assumptions of Equation 2.
  • Model comparison: Unlike the described Cox system, MTLR and RSF-KM allow feature importance to vary over time and permit patient survival curves to cross.The paper reports that MTLR is more often D-Calibrated than this predictive Cox system.
  • MTLR representation: MTLR represents survival at multiple time points with binary labels whose legal sequences contain zeros followed by ones.The labels encode whether survival has crossed each time threshold, enforcing the no-zombie structure.
  • MTLR model: MTLR uses time-specific parameter vectors to assign probabilities to legal survival-status sequences.The sequence score sums parameter-feature products over the relevant time range before normalization.
  • Prediction: After training, MTLR evaluates a new patient at the selected time points, uses running sums to form the survival curve, and applies splines for a smooth monotone curve.Regularization, feature selection, censoring treatment, and the number of time points are additional design considerations.

C.3 Extension to Random Survival Forests (rsf-km)

The rsf-km extension derives an individual survival curve by combining Kaplan–Meier curves from the leaf nodes reached by a patient across survival trees. Its curve construction was reverse-engineered, and its Concordance was similar to the original random survival forest.

  • Each terminal node produces a Kaplan–Meier curve from the training instances reaching that node.
  • For a new patient, rsf-km routes the patient through all survival trees and obtains one Kaplan–Meier curve from each reached leaf.
  • rsf-km averages the resulting curves point-wise across time to create the patient’s individual survival curve.The averaging is performed at the relevant death times.
  • The median of the resulting survival curves need not preserve the patient ordering induced by the original forest’s averaged cumulative-hazard risk score.
  • The authors found similar Concordance scores for rsf-km and the original random survival forest, while noting that the curve-generation method was reverse-engineered.

D Detailed Empirical Results

The empirical-results appendices provide tables for Concordance, Integrated Brier score, and multiple L1-loss variants, with performance comparisons organized by dataset and metric. They also document which models achieve the best values under each evaluation measure.

  • The appendix supplies tables corresponding to the paper’s Concordance, Integrated Brier score, and L1-loss figures.
  • Calibration and scoring results: The Integrated Brier score table marks the best model for each dataset by the lowest score and may contain displayed ties caused by rounding.
  • L1-loss results: The Margin-L1-loss table identifies the best-performing model for each dataset using the lowest Margin-L1-loss.
  • L1-loss results: The Uncensored L1-loss table identifies the best-performing model for each dataset using the lowest L1-loss.
  • L1-loss results: The Log-Margin-L1-loss table identifies the best-performing model for each dataset using the lowest Log-Margin-L1-loss.

D.4 1-Calibration

The 1-Calibration analysis evaluates models at five event-time percentiles and summarizes how many of eight datasets each model passes at each time. mtlr is generally the strongest model, while later survival times are more difficult to calibrate.

  • The detailed tables organize results by event-time percentile, and bold values indicate models passing 1-Calibration at p > 0.05.
Loading 1811.11347v1…