Source-linked AI summary
Deep Survival Analysis
Rajesh Ranganath, Adler Perotte, Noémie Elhadad, David Blei
TL;DR
EHR data could support actionable survival analysis, but traditional methods are limited by sparse observations and alignment assumptions. The paper introduces deep survival analysis, a hierarchical generative model that jointly represents covariates and survival time, aligns observations by failure time, and handles heterogeneous data. In 313,000 patient records, it better stratifies CHD risk than a clinically validated risk score.
Problem
Traditional survival analysis is difficult to apply to EHR data because observations are sparse and patients require arbitrary synchronization around an initial event.
Method
Deep survival analysis jointly models covariates and survival time in a Bayesian latent-variable framework and aligns observations by failure time.
Results
Deep survival analysis better stratified CHD risk than the clinically validated CHD risk score in 313,000 patient records.
Takeaways & Limitations
The approach handles sparse, heterogeneous EHR observations and supports risk-score development from observational data, including conditions without a known risk score.
Takeaways & Limitations
Traditional synchronization assumptions remain a scope boundary because EHR records begin at varying points in patients’ lifetime and disease progression.
Abstract
from arXiv · showhide
The electronic health record (EHR) provides an unprecedented opportunity to build actionable tools to support physicians at the point of care. In this paper, we investigate survival analysis in the context of EHR data. We introduce deep survival analysis, a hierarchical generative approach to survival analysis. It departs from previous approaches in two primary ways: (1) all observations, including covariates, are modeled jointly conditioned on a rich latent structure; and (2) the observations are aligned by their failure time, rather than by an arbitrary time zero as in traditional survival analysis. Further, it (3) scalably handles heterogeneous (continuous and discrete) data types that occur in the EHR. We validate deep survival analysis model by stratifying patients according to risk of developing coronary heart disease (CHD). Specifically, we study a dataset of 313,000 patients corresponding to 5.5 million months of observations. When compared to the clinically validated Framingham CHD risk score, deep survival analysis is significantly superior in stratifying patients according to their risk.
1. Introduction
Deep survival analysis is proposed to estimate future event times from heterogeneous, sparse EHR data and improve risk stratification for clinical decision support. The approach jointly models covariates and survival time, aligns observations by failure time, and outperforms a validated CHD risk score in the reported study.
- Motivation: EHR-based survival analysis aims to estimate future event times so physicians can take risk-calibrated clinical actions.CHD motivates the task because effective preventive therapies exist, but they are not risk free.
- Motivation: Traditional risk-score development is constrained by missing covariates and arbitrary alignment to an initial event in EHR data.EHR records may begin at different points in patients’ lifetimes and disease progression.
- Contributions: Deep survival analysis jointly models covariates and survival time in a Bayesian framework, simplifying work with missing EHR covariates.Its generative process uses deep exponential families to capture nonlinear latent dependencies between covariates and failure time.
- Contributions: Failure-time alignment replaces artificial time-zero alignment by aligning patients according to observed failure or censoring.This design supports event-centric ordering rather than synchronization around an arbitrary starting event.
- Evaluation: 313,000 patient records were used to assess CHD risk, and deep survival analysis better stratified patients than the clinically validated CHD risk score.The study also included vitals, laboratory measurements, medications, and diagnosis codes after preprocessing.
2. Survival Analysis
Traditional survival analysis models time to an event from a common start, but EHR data challenges this framework through sparsity, arbitrary synchronization requirements, and nonlinear covariate relationships. Failure-aligned analysis instead organizes observations relative to failure while representing censoring as an interval constraint.
- Fundamentals: Survival analysis models time to an event from a common start using observed or censored failure-time observations.Censored observations indicate that failure occurs after a specified time.
- Limitations: Three limitations affect traditional survival methods on EHR data: sparse high-dimensional records, synchronization requirements, and restrictive covariate functions.These limitations complicate applying conventional models to routinely collected EHR observations.
- Limitations: Sparse EHR data makes traditional conditional models difficult to use because they cannot easily handle missing covariates.The records are described as usually high-dimensional and very sparse.
- Limitations: Traditional methods align patients around a synchronization event, although EHR records can begin at varying points in lifetime and disease progression.The paper seeks methods that evaluate risk at any time, not only at synchronization events.
- Failure alignment: Failure-aligned survival analysis measures time to failure from each observation time, decreasing as failure approaches.For censored patients, the relevant failure time is an interval after their last EHR interaction.
3. Deep Survival Analysis
Deep survival analysis is a hierarchical generative model that jointly represents EHR covariates and survival time through latent variables. It aligns records by failure time, accommodates censoring and missingness, and supports heterogeneous clinical data.
- 3.1 Deep Exponential Families: Deep survival analysis jointly models covariates and survival times conditioned on a latent process rather than imputing missing covariates separately.Deep exponential families provide the multilayer latent structure underlying the generative process.
- 3.2 Alignment by Failure: Failure-time alignment measures time backward from the event instead of forward from an artificial common start time.Censored observations are represented as intervals beyond the patient’s last EHR interaction.
- 3.2 Alignment by Failure: Each EHR interaction receives a time-from-event value, allowing interactions to be modeled exchangeably for computational efficiency.Different events define different survival-alignment frames, enabling multiple survival tasks.
- 3.2 Alignment by Failure: The model uses a Weibull distribution for time from event, with positive scale and shape parameters and a censored likelihood based on probability after censoring.The shape parameter controls whether density concentrates near zero, matches the exponential, or centers around its expectation.
- 3.3 Generative Process for Deep Survival Analysis: The latent variable generates observed covariates and failure time, while posterior predictive inference integrates over latent states and remains defined with missing data.Continuous EHR measurements use a Student-t likelihood for robustness to outliers, whereas sparse medication and diagnosis counts are modeled as binary observations.
- 3.3 Generative Process for Deep Survival Analysis: Laboratory values, medications, diagnosis codes, and vitals are modeled as heterogeneous data types conditionally independent given the latent structure.Real-valued measurements and count-valued observations receive separate likelihood treatments.
4. Experimental Setup
The study evaluates deep survival analysis for CHD risk using longitudinal EHR data from a large metropolitan hospital. It compares the model with the validated Framingham risk score using held-out evaluation and predictive likelihood.
- 4.1 Data and Clinical Scenario: 313,000 adult patients from Columbia University Medical Center formed the longitudinal EHR dataset.Patients had at least five months with one or more recorded observations, including inpatient, outpatient, and emergency visits.
- 4.1 Data and Clinical Scenario: Monthly preprocessing aggregated continuous measurements by expected values and encoded discrete medications and diagnoses as binary presence indicators.Observations were included for months in which patients had any recorded data.
- 4.2 Baseline and Model Setup: The Framingham CHD score estimates 10-year risk using gender, age, cholesterol, blood pressure, diabetes, and smoking.Its reported concordance was 0.73 for men and 0.77 for women on curated Framingham Heart Study data, but performance is lower on EHR data.
- 4.2 Baseline and Model Setup: The model used a two-layer rectified-linear network, Gaussian exponential-family components, a Weibull shape of 2, and zero-mean unit-variance Normal priors.All methods ran for 6,000 iterations; the full training run completed in 7.5 hours on a 40-core server.
- 4.2 Baseline and Model Setup: 263,000 patients were used for training, with 25,000 each reserved for validation and testing.Convergence used the validation cohort, while concordance was evaluated on the test cohort.
- 4.3 Evaluation: Concordance compared deep survival analysis with the baseline, while predictive likelihood assessed temporal failure prediction on held-out data.Predictive likelihood was used for internal model validation.
5. Results
The results quantify severe EHR incompleteness and show that deep survival analysis outperforms the baseline CHD risk score, with diagnoses providing the strongest individual data-type predictive likelihood.
- 5.1 Missing observations in EHR data: Only 11.8% of patients had a complete month, while just 1.4% of months were complete for the critical variables examined.The variables were LDL level, HDL cholesterol level, and blood pressure.
- 5.2 Model Performance and Predictive Likelihood: 65.57% concordance was achieved by the baseline CHD risk score on the held-out test set.For the full deep survival model, the best performance was obtained with K=50.
- 5.2 Model Performance and Predictive Likelihood: All deep survival analysis dimensionalities outperformed the baseline risk score in concordance on a held-out set of 25,000 patients.The comparison covered different values of K and the baseline risk score.
- 5.2 Model Performance and Predictive Likelihood: The diagnosis-only model yielded the best predictive likelihood among models using individual data types.The individual-data-type models included vitals, diagnosis codes, laboratory tests, and medications, all with age and gender.
6. Discussion
The paper presents deep survival analysis as a method designed for sparse and heterogeneous EHR observations. It reports more accurate CHD patient stratification than a clinically validated risk score and notes potential for conditions without known risk scores.
- 6. Discussion: Deep survival analysis is built to handle the sparsity and heterogeneity of EHR observations rather than requiring carefully curated research datasets.The model was estimated on data from a large metropolitan hospital in a matter of hours.
- 6. Discussion: Deep survival analysis yielded more accurate patient stratification than a state-of-the-art, clinically validated coronary heart disease risk score.The comparison was made in the context of coronary heart disease.
- 6. Discussion: The approach holds particular promise for developing risk scores from observational data for conditions without a known risk score.