PaperlayerView exact revision on arXiv

Source-linked AI summary

Countdown Regression: Sharp and Calibrated Survival Predictions

Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H. Shah, Andrew Ng

arXiv:1806.08324v2cs.LGstat.APstat.ML

TL;DR

Patient-specific time-to-event prediction must contend with censored observations, while useful forecasts need calibrated survival distributions. The paper proposes Survival-CRPS for right- and interval-censored data and reports sharper, calibrated mortality predictions than MLE on two large EHR datasets.

  • Problem

    Censored event times complicate survival prediction, and patient-specific prognosis calls for calibrated survival curves rather than risk ranking alone.

    Source: §1 · p. 1

  • Method

    The paper generalizes CRPS as Survival-CRPS for censored survival outcomes and introduces Survival-AUPRC to evaluate sharpness and calibration.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    Survival-CRPS yielded sharper, calibrated patient-mortality predictions than MLE on two large-scale EHR datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors advocate exploring objectives beyond MLE and evaluating the holistic quality of predicted distributions rather than point estimates.

    Source: §5 · p. 8

    The experiments operationalize mortality outcomes as age at death or age at the last known alive encounter.

    Source: §3 · p. 5

Abstract

from arXiv · show

Probabilistic survival predictions from models trained with Maximum Likelihood Estimation (MLE) can have high, and sometimes unacceptably high variance. The field of meteorology, where the paradigm of maximizing sharpness subject to calibration is popular, has addressed this problem by using scoring rules beyond MLE, such as the Continuous Ranked Probability Score (CRPS). In this paper we present the \emph{Survival-CRPS}, a generalization of the CRPS to the survival prediction setting, with right-censored and interval-censored variants. We evaluate our ideas on the mortality prediction task using two different Electronic Health Record (EHR) data sets (STARR and MIMIC-III) covering millions of patients, with suitable deep neural network architectures: a Recurrent Neural Network (RNN) for STARR and a Fully Connected Network (FCN) for MIMIC-III. We compare results between the two scoring rules while keeping the network architecture and data fixed, and show that models trained with Survival-CRPS result in sharper predictive distributions compared to those trained by MLE, while still maintaining calibration.

1 INTRODUCTION

The introduction argues that MLE can produce overly diffuse survival predictions, while patient-specific forecasts need distributions that are both sharp and calibrated. It presents Survival-CRPS for censored outcomes, Survival-AUPRC for evaluation, and results showing sharper calibrated mortality predictions on two large EHR datasets.

1 INTRODUCTION

  • MLE can yield excessively high-variance distributions in heteroskedastic survival prediction, particularly when censoring is heavy.The introduction contrasts this with point-estimate tasks and risk stratification, where the stated shortcomings may not matter.Source: §1 · pp. 1–2
  • Patient-specific prognosis requires sharp, calibrated survival curves because ranking metrics such as concordance do not answer time-specific questions.The passage contrasts questions about an individual’s risk over different time horizons with comparisons of relative risk across patients.Source: §1 · pp. 1–2
  • CRPS can improve forecast sharpness, but conventional CRPS does not handle the censored observations needed for survival prediction.The introduction describes CRPS as less sensitive to outliers than MLE, though less simple analytically and numerically.Source: §1 · p. 2
  • The paper introduces Survival-CRPS for right- and interval-censored data and Survival-AUPRC to assess sharpness and calibration holistically.It also presents practical recommendations for implementing Survival-CRPS on large-scale data.Source: §1 · p. 2
  • Survival-CRPS produced sharper, calibrated patient-mortality predictions than MLE on two large-scale EHR datasets.Figure 1 illustrates the sharper distributions for right- and interval-censored examples.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors state that their concepts apply to any right- or interval-censored survival prediction problem, not only healthcare.The introduction motivates exploring objectives beyond MLE and evaluating whole predicted distributions rather than point estimates.Source: §1 · p. 2

2 COUNTDOWN REGRESSION

This section formulates Survival-CRPS for censored time-to-event records and describes how to evaluate calibration and sharpness, including with Survival-AUPRC.

2 COUNTDOWN REGRESSION

  • Parametric models map patient features to event-time distributions, with records encoding observed time, censoring status, and an upper event-time bound for interval censoring.The choice of distribution family implies assumptions about the data-generating process.Source: §2 · pp. 2–3

2.1 SURVIVAL-CRPS: PROPER SCORING RULE OBJECTIVE

  • Both Survival-CRPS variants are proper scoring rules for censored outcomes, penalizing mass before censoring and, for interval censoring, after the event bound.Log scoring can be sensitive to heavy-tailed data and yield more conservative, less-sharp predictions.Source: §2.1 · p. 3

2.2 EVALUATION BY SHARPNESS SUBJECT TO CALIBRATION

  • Calibration compares predicted CDFs with observed frequencies at CDF quantiles, while sharpness uses coefficient of variation because variance or entropy alone ignore differences in predicted means.Calibration curves are also assessed by the slopes of their best-fit lines, ideally 1.Source: §2.2 · p. 4

2.3 SURVIVAL-AUPRC: HOLISTIC TIME TO EVENT METRIC

  • Survival-AUPRC, the area under the Survival Precision Recall Curve obtained by exploring precision from 0 to 1, measures how quickly predicted mass concentrates around the true outcome as the precision window expands.For censored observations, the right end of precision intervals is defined by when the event must have occurred for interval-censored cases, or infinity for right-censored cases. The mean score across examples summarizes prediction quality; scores range from 0 for an infinitely dispersed distribution to 1 for a Dirac δ centered at the outcome.Source: §2.3 · p. 5

2.4 IMPLEMENTATION TIPS AND CHOICES

  • The authors recommend Log-Normal for expressivity and numerical stability; Survival-CRPS integrals are approximated using the trapezoid rule.The approximation formulas remain proper scoring rules, and training back-propagates through the approximation.Source: §2.4 · p. 5

3 EXPERIMENTS

The experiments compare four MLE and Survival-CRPS objectives for mortality prediction using STARR and MIMIC-III records with dataset-matched neural networks. Interval-censored Survival-CRPS gives the sharpest distributions, while predictions remain calibrated.

3 EXPERIMENTS

  • The mortality experiments compare right- and interval-censored variants of MLE and Survival-CRPS objectives.For interval censoring, the experiments assume a maximum lifespan of 120 years.Source: §3 · p. 5

3.1 RNN WITH STARR DATA WAREHOUSE

  • STARR’s longitudinal records motivated an RNN that predicts log-normal distribution parameters at each timestep from current and prior data.The dataset covers over 3 million patients and 51 million timesteps; the RNN uses sequences split into train, validation, and test sets at an 8:1:1 ratio.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 FCN WITH MIMIC-III

  • MIMIC-III’s mostly single-admission, short visits motivated a four-layer fully connected network making predictions at discharge.The experiment used 51,015 admissions, of which 70.1% were censored.Source: §3.2 · p. 7

3.3 RESULTS

  • Interval-censored Survival-CRPS yielded the sharpest distributions; MLE assigned over 75% of STARR probability to unreasonable ages beyond 120, and predictions remained well calibrated.The benefit was most pronounced in low-prevalence STARR, but also appeared in moderate-prevalence MIMIC-III.Source: §3.3 · p. 7

4 RELATED WORK

Related work includes survival models with varied prediction forms and assumptions, while the supplied figure and table captions describe calibration plots and sharpness-and-calibration comparisons.

4 RELATED WORK

  • Prior survival approaches include fixed-time classification, discrete-time curves, Gaussian-process models, fixed-shape Weibull models, and models optimized only with MLE.The passage also discusses work on flexible curves with missing covariates, adversarial training, and neural-network calibration.Source: §4 · p. 7
  • Table 1 compares sharpness and calibration for right- and interval-censored MLE and Survival-CRPS variants, with bootstrap confidence intervals.Source: Table 1 · p. 8
  • Figure 5 plots predicted cumulative densities against observed event frequencies at predicted-density quantiles, adjusting which censored observations count at later times.Right-censored cases are excluded beyond censoring; interval-censored cases are reintroduced past 120 years.Source: Figure 5 · p. 8

5 CONCLUSION

The conclusion advocates exploring objectives beyond MLE and evaluating full predictive distributions rather than point estimates. It presents Survival-CRPS and Survival-AUPRC and describes deep, log-normal survival prediction at large scale.

5 CONCLUSION

  • The paper argues for objectives beyond MLE and evaluation metrics that assess full predicted distributions rather than point estimates.Source: §5 · p. 8
  • Survival-CRPS is motivated by CRPS’s ability to yield sharp distributions while maintaining calibration, and the conclusion leaves open whether other scoring rules may work better.Source: §5 · p. 8
  • The paper demonstrates large-scale survival prediction with deep models using a log-normal parameterization, with particular relevance to healthcare.The authors hope the work will be useful to people building and deploying such models.Source: §5 · p. 8

A. INTEGRAL IDENTITIES

The passage defines a log-normal CDF through a Gaussian CDF evaluated at the logarithm of the outcome, then introduces an integral approximation.

A. INTEGRAL IDENTITIES

B. SURVIVAL-CRPS FOR LOG-NORMAL (RIGHT-CENSORED)

This section gives Survival-CRPS formulations for right-censored outcomes and extends them to interval-censored outcomes, including log-normal forms.

B. SURVIVAL-CRPS FOR LOG-NORMAL (RIGHT-CENSORED)

  • For right-censored data, the section generalizes CRPS to a continuous prediction distribution F using the outcome time y and censoring indicator c.It also describes a log-normal form for distributions over the positive reals.Source: B. SURVIVAL-CRPS FOR LOG-NORMAL… · p. 12

C. SURVIVAL-CRPS FOR LOG-NORMAL (INTERVAL-CENSORED)

D. SURVIVAL-AUPRC FOR LOG-NORMAL (INTERVAL-CENSORED)

The section defines Survival-AUPRC for interval-censored, right-censored, and uncensored outcomes, with log-normal-specific forms for each setting.

D. SURVIVAL-AUPRC FOR LOG-NORMAL (INTERVAL-CENSORED)

  • For an interval outcome [L, U], Survival-AUPRC is defined for a general continuous prediction distribution, with a log-normal form using standard-normal φ and Φ and log-transformed endpoints.The displayed expression uses F(U/t) and F(Lt).Source: D. SURVIVAL-AUPRC FOR LOG-NORMAL… · p. 13

E. SURVIVAL-AUPRC FOR LOG-NORMAL (RIGHT-CENSORED)

  • For a right-censored interval outcome [L, ∞), Survival-AUPRC is defined for a general continuous distribution, with a log-normal form based on the standard-normal CDF and log L.Its displayed expression uses 1 − F(Lt).Source: E. SURVIVAL-AUPRC FOR LOG-NORMAL… · p. 14

F. SURVIVAL-AUPRC FOR LOG-NORMAL (UNCENSORED)

  • For a point outcome y, the section defines Survival-AUPRC for a general continuous distribution and gives a log-normal form using the standard-normal CDF and log y.Its displayed expression uses F(y/t) and F(yt).Source: F. SURVIVAL-AUPRC FOR LOG-NORMAL… · p. 14

G. EVALUATION AS BINARY OUTCOME

Figure 6 evaluates discrimination and calibration for the interval-censored Survival-CRPS model at three mortality-prediction horizons.

G. EVALUATION AS BINARY OUTCOME

  • Figure 6 concerns predictions from the interval-censored Survival-CRPS model.Source: Figure 6 · p. 15
  • The figure evaluates discrimination of dichotomous-outcome predictions.Source: Figure 6 · p. 15
  • The evaluation covers 6 months, 1 year, and 5 years.Source: Figure 6 · p. 15

H. INDIVIDUAL PATIENTS IN INTERVAL-CENSORED SURVIVAL-CRPS MODEL

Figure 7 presents individual predicted times to death with 95% intervals; predictions become more confident with repeated EHR interactions, and true times generally fall within the intervals.

H. INDIVIDUAL PATIENTS IN INTERVAL-CENSORED SURVIVAL-CRPS MODEL

  • The figure shows individual patients’ median predicted time to death with 95% intervals.Source: Figure 7 · p. 15
  • Predictions become more confident upon repeated interactions between patients and the EHR.Source: Figure 7 · p. 15
  • True times to death generally lie within the predicted intervals.Source: Figure 7 · p. 15

Related papers

Loading 1806.08324v2…