Source-linked AI summary
Deep Recurrent Survival Analysis
Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, Yong Yu
TL;DR
Existing survival-analysis methods often assume event-time distributions or overlook sequential feature patterns. DRSA models conditional event probabilities recurrently and significantly outperforms strong baselines across three real-world tasks.
Problem
Existing methods rely on segmented-data statistics or distributional assumptions, while few model sequential patterns in features for censored time-to-event data.
Method
DRSA recurrently predicts conditional event probabilities and combines them with the probability chain rule to estimate event distributions and survival rates.
Results
DRSA achieves the best C-index scores with significant improvements on all datasets and significant advantages over strong baselines across three real-world tasks.
Takeaways & Limitations
The model provides a flexible survival-analysis approach that captures temporal dependencies without assuming a specific event-time distribution.
Takeaways & Limitations
Extending DRSA to competing-risks prediction remains future work.
Abstract
from arXiv · showhide
Survival analysis is a hotspot in statistical research for modeling time-to-event information with data censorship handling, which has been widely used in many applications such as clinical research, information system and other fields with survivorship bias. Many works have been proposed for survival analysis ranging from traditional statistic methods to machine learning models. However, the existing methodologies either utilize counting-based statistics on the segmented data, or have a pre-assumption on the event probability distribution w.r.t. time. Moreover, few works consider sequential patterns within the feature space. In this paper, we propose a Deep Recurrent Survival Analysis model which combines deep learning for conditional probability prediction at fine-grained level of the data, and survival analysis for tackling the censorship. By capturing the time dependency through modeling the conditional probability of the event for each sample, our method predicts the likelihood of the true event occurrence and estimates the survival rate over time, i.e., the probability of the non-occurrence of the event, for the censored data. Meanwhile, without assuming any specific form of the event probability distribution, our model shows great advantages over the previous works on fitting various sophisticated data distributions. In the experiments on the three real-world tasks from different fields, our model significantly outperforms the state-of-the-art solutions under various metrics.
Introduction
Survival analysis estimates event timing or occurrence probabilities over time despite censored observations, with applications spanning clinical research, information systems, and other fields. DRSA addresses limitations of prior methods by modeling fine-grained conditional event probabilities and capturing sequential feature patterns without assuming a distribution form.
- Motivation: Censoring occurs when events lose tracking during observation, leaving only partial information about the true event time.
- Motivation: Survival analysis estimates time until an event or its occurrence probability at each time point for an observed object.
- Applications: The methodology supports applications including patient survival prediction, customer lifetime estimation, and market modeling.
- Limitations of prior work: Existing deep survival models often use neural networks mainly for feature extraction and rely on assumptions about base probability distributions, creating generalization concerns.
- Proposed approach: DRSA predicts fine-grained survival rates for each individual sample, uses an auto-regressive model to capture sequential feature patterns, and flexibly models survival probabilities without assuming a distribution form.
Related Works
Prior survival-analysis methods include coarse counting-based statistics, distribution-assuming semi-parametric models, and machine-learning approaches that often retain counting-based formulations. Neural methods predict survival or density values directly, but can suffer from sparse gradient signals, motivating recurrent modeling of conditional probabilities and sequential patterns.
- Traditional statistical methods: Traditional methods include non-parametric estimators based solely on counting statistics, which are too coarse-grained for personalized modeling.Examples include the Kaplan-Meier and Nelson-Aalen estimators.
- Traditional statistical methods: Semi-parametric methods such as Cox proportional hazards and Lasso-Cox assume base distribution functions with scaling parameters.
- Machine-learning methods: Machine-learning approaches include survival random forests, Bayesian models, support vector machines, and multi-task learning, while survival forests still rely mainly on counting-based statistics.
- Neural survival methods: Neural methods predict survival rates or density values directly, but density-based prediction can produce sparse gradient signals at neural-network outputs.Recurrent neural networks and LSTMs instead use memory structures to model conditional probabilities and dynamic sequential patterns.
Experiments
The model is evaluated against strong baselines across three real-world tasks, with implementation code published to support reproduction.
- The experiments compare the model with strong baselines across three real-world tasks.
- The study publishes implementation code for reproductive experiments.
Tasks and Datasets
The paper evaluates its models on three real-world survival-analysis tasks using processed datasets. The tasks cover clinical status, music-service visits, and real-time advertising auctions, with task-specific event-time objectives.
- Overview: All compared models are evaluated on three real-world tasks, and the processed full datasets are published.The paper also provides sampled data through its published code.
- CLINIC: CLINIC tracks patient clinic status, aiming to estimate time to death and event probability as baseline physiologic effects wane.The task uses survival analysis to model death-related outcomes over time.
- MUSIC: MUSIC contains roughly 1,000 users’ listening histories from 2004–2009, and predicts elapsed time from one user’s last visit to the next.The event is a user’s visit to the music service Last.fm.
- BIDDING: BIDDING models auction-winning time as bid price using auction-request features, while handling losing censored logs without the true winning price.It supports unbiased winning-probability estimation for individual auctions.
- Preprocessing and splits: CLINIC and MUSIC use training–test ratios of 4:1 and 6:1, respectively; all datasets are one-hot encoded, while BIDDING is preprocessed.True event times are preserved for testing-data performance evaluation.
Evaluation Metrics
The evaluation uses time-dependent C-index to measure event-time ordering and ANLP to assess true event-time likelihood. Statistical significance is tested with Mann–Whitney U for C-index and t-test for ANLP.
- Time-dependent C-index: Time-dependent C-index measures how well models predict the ordering of sample event times at an observing time t.For samples with event times z1 > t and z2 ≤ t, the desired ordering places d1 before d2.
- Average negative log probability: Average negative log probability (ANLP) evaluates forecasting regression by measuring the likelihood of each test sample’s corresponding true event time.ANLP corresponds to the event-time likelihood loss Lz in Eq. (12), using the learned probability function p(z|x).
- Significance testing: Statistical significance is assessed with a Mann–Whitney U test under C-index and a t-test under ANLP.These tests evaluate whether performance improvements over baseline models are statistically significant.
Compared Settings
The study compares DRSA with two traditional statistical methods and five machine-learning methods, including two deep-learning models. The baselines span non-parametric, semi-parametric, parametric, and neural-network approaches.
- Compared methods: The comparison includes two traditional statistical methods and five machine-learning methods, including two deep-learning models.The statistical baselines include KM and Lasso-Cox; the machine-learning baselines include DeepSurv and DeepHit.
- Traditional statistical methods: KM is a non-parametric Kaplan-Meier estimator that counts event probability at each time over the full sample set.It is identified as a statistic-based method.
- Traditional statistical methods: Lasso-Cox is a semi-parametric Cox proportional-hazards model with l1-regularization.It is based on the Cox proportional hazard model.
- Deep-learning baselines: DeepSurv uses a Cox proportional-hazard model with a deep neural network to extract features from sample covariates.DeepSurv is attributed to Katzman et al. (2018).
- Deep-learning baselines: DeepHit predicts the event probability p(z) over the whole time space from input x and is described as state-of-the-art in survival analysis.DeepHit is a deep neural-network model attributed to Lee et al. (2018).
- Proposed model: DRSA is the proposed model, with implementation details provided in the supplemental materials and published code.DRSA is the method evaluated against the comparison models.
Results and Analysis
DRSA achieves the strongest reported performance across the compared methods on event-rate estimation and true event-time prediction. Its sequential hazard-rate modeling, rapid convergence, personalized predictions, and combined loss design explain these results.
- Performance comparison: DRSA achieved the best C-index scores across all compared methods and datasets, with significant improvements reported throughout.Deep learning models generally performed better, attributed to greater feature-extraction capacity.
- Performance comparison: DRSA achieved the best ANLP performance among all methods for true event-time prediction.STM outperformed other conventional machine-learning methods, while DRSA benefited from sequential pattern mining over time.
- Training convergence: DRSA converged quickly, with both ANLP and cross-entropy losses reaching stable values at about the first complete training iteration.Training optimized the ANLP loss L_z and cross-entropy loss L_c.
- Visualization: For an example test sample, DRSA accurately assigned the highest event-time probability to the true event time and produced personalized survival estimates.KM generated the same prediction for every sample, whereas DRSA’s prediction explains its strongest ANLP results.
- Ablation study: Both censored and uncensored partial-likelihood losses contributed to the final prediction, while the full DRSA model optimized all three losses jointly.The three losses are L_z, L_uncensored, and L_censored.
Conclusion
The paper surveys survival-analysis methods and proposes a deep recurrent model for conditional hazard prediction, event-time probability forecasting, and survival-rate estimation. Future work includes extending the model to competing-risks prediction with shared feature embeddings and multi-task learning.
- The paper surveys survival-analysis methods from a modeling perspective and discusses their advantages and disadvantages.
- The proposed deep recurrent neural network flexibly models conditional hazard rates over time.
- The probability chain rule links time-specific hazard predictions to event-time probability forecasting and survival-rate estimation.
- Future work will apply the model to competing-risks prediction using shared feature embeddings and multi-task learning.