Source-linked AI summary
Estimating Missing Data in Temporal Data Streams Using Multi-directional Recurrent Neural Networks
Jinsung Yoon, William R. Zame, Mihaela van der Schaar
TL;DR
Missing measurements are especially challenging in medical time series because streams are sampled at different and irregular times, while common methods discard either temporal or cross-stream information. The paper introduces M-RNN, which jointly interpolates within streams and imputes across streams, and reports large improvements across five real-world medical datasets, with robustness shown in additional experiments.
Problem
Medical missing-data estimation must handle irregular multistream measurements, but common interpolation, imputation, and static approaches ignore important temporal or cross-stream information.
Method
M-RNN is a multidirectional recurrent architecture that jointly trains interpolation and imputation blocks across intra-stream and inter-stream directions, with dropout generating multiple imputations.
Results
The method yields large and statistically significant improvements over state-of-the-art benchmarks across five real-world medical datasets and is more congenial than competing benchmarks.
Takeaways & Limitations
The approach estimates missing measurements using both within-stream and across-stream correlations while preserving relationships between features and labels.
Takeaways & Limitations
The method assumes data is missing at random; data that is not missing at random is left for future work.
Abstract
from arXiv · showhide
Missing data is a ubiquitous problem. It is especially challenging in medical settings because many streams of measurements are collected at different - and often irregular - times. Accurate estimation of those missing measurements is critical for many reasons, including diagnosis, prognosis and treatment. Existing methods address this estimation problem by interpolating within data streams or imputing across data streams (both of which ignore important information) or ignoring the temporal aspect of the data and imposing strong assumptions about the nature of the data-generating process and/or the pattern of missing data (both of which are especially problematic for medical data). We propose a new approach, based on a novel deep learning architecture that we call a Multi-directional Recurrent Neural Network (M-RNN) that interpolates within data streams and imputes across data streams. We demonstrate the power of our approach by applying it to five real-world medical datasets. We show that it provides dramatically improved estimation of missing measurements in comparison to 11 state-of-the-art benchmarks (including Spline and Cubic Interpolations, MICE, MissForest, matrix completion and several RNN methods); typical improvements in Root Mean Square Error are between 35% - 50%. Additional experiments based on the same five datasets demonstrate that the improvements provided by our method are extremely robust.
1 Introduction
Missing measurements are especially difficult and consequential in medical time series because multiple streams are sampled at different, irregular times. The paper proposes M-RNN, which jointly exploits within-stream and across-stream information and improves estimation on real-world medical datasets.
- Medical missing-data estimation is critical for diagnosis, prognosis, treatment, and statistical analysis, especially with irregularly sampled multistream measurements.
- Standard interpolation uses within-stream temporal correlations, whereas imputation uses synchronous across-stream correlations; each ignores the other source of information.
- M-RNN jointly trains interpolation and imputation blocks while processing each stream forward and backward and also modeling inter-stream directions.
- Dropout is used to generate multiple imputed datasets, addressing uncertainty that single imputation does not capture.
- The method is evaluated on five public real-world medical datasets against prior interpolation, imputation, and RNN-based methods.
- Imputed values are more congenial when they preserve the original relationships between features and labels.
2 Related Work
Related methods capture either temporal relationships within streams, synchronous relationships across streams, or neither temporal structure nor both types of correlation. RNN-based approaches incorporate missing data and, in some cases, irregular sampling, but use varied replacement strategies.
- Interpolation captures temporal relationships within each stream but not relationships across streams, while imputation does the reverse.
- Matrix completion treats data as static and therefore ignores its temporal aspect.
- Several RNN methods first replace missing values, update them through hidden-state feedback or EM, and then use reconstructed streams for prediction.
- Methods for irregularly sampled data use measurements, sampling information, and time intervals as RNN inputs, but differ in how missing values are replaced.
3 Problem Formulation
The paper formulates each patient record as time stamps, multistream measurements, and outcomes, allowing measurements to be absent at irregular times. It seeks a function that estimates missing values by minimizing squared error using observed data rather than labels.
- Each patient has a multivariate time series of time stamps, measurements, and labels sampled from an unknown distribution.
- Time stamps are strictly increasing, but intervals between successive observations need not be constant.
- A missing stream at time t is denoted by x^d_t = *, while observed measurements are real values scaled to [0, 1].
- The indicator m^d_t equals 0 when stream d is unmeasured at t and 1 when it is measured.
- δ^d_t records the elapsed time since stream d was previously measured, and Δ collects these elapsed-time vectors across timestamps.
- The estimation function depends on the target stream, timestamp, and entire time-stamp and measurement arrays, but not on labels; empirical loss approximates the unavailable true-distribution loss.
4 Multi-directional Recurrent Neural Networks (M-RNN)
M-RNN combines within-stream interpolation with across-stream imputation in a jointly trained, multi-directional architecture. It estimates missing measurements while controlling parameter growth and supports both single and multiple imputations.
- M-RNN estimates each missing measurement using information from both the same stream over time and other streams at the same time.
- The hierarchical architecture reduces the parameter count from a potential quadratic order in the number of streams to linear order, helping avoid over-fitting.
- The architecture uses an interpolation block followed by an imputation block, trained jointly rather than separately.The interpolation block operates within streams, while the imputation block operates across streams.
- Interpolation Block: The interpolation block uses lagged forward and backward Bi-RNN inputs so the target measurement is excluded from its estimate.
- Imputation Block: The imputation block uses fully connected layers across streams and sets diagonal entries of U to zero to avoid using the missing target value.
- Multiple Imputations: Dropout produces multiple imputations by sampling different neuron masks, whereas setting dropout probability to 0 produces single imputation.
5 Datasets
The study evaluates M-RNN on five public medical datasets spanning intensive care, clinical deterioration, transplantation, and population health. These datasets differ in patient cohorts, stream counts, sampling patterns, and missingness.
- The evaluation uses five public real-world medical datasets: MIMIC-III, clinical deterioration, UNOS-Heart, UNOS-Lung, and UK Biobank.
- MIMIC-III: MIMIC-III includes 23,160 post-2008 Metavision patients and 40 physiological streams comprising 20 vital signs and 20 laboratory tests.
- Clinical Deterioration: The clinical deterioration cohort contains 6,094 hospitalized patients monitored across 38 physiological streams, with vital signs sampled roughly every four hours.
- Clinical Deterioration: Because laboratory tests were sampled about one-sixth as often as vital signs in the deterioration dataset, approximately 5/6 of laboratory-test data is missing by construction.
- UNOS: The UNOS datasets contain 69,205 heart-transplant and 32,986 lung-transplant patients followed yearly from 1985 to 2015 using 34 clinical features.
- UK Biobank: UK Biobank data were collected from 21 assessment centers across England, Wales, and Scotland between 2007 and 2014.
6 Results and Discussions
Across five medical datasets, M-RNN generally improved missing-value estimation over 11 benchmarks, with gains depending on sampling density and available data. Additional experiments show robustness to missingness patterns, while prediction and congeniality benefits were positive but not uniformly significant.
- Imputation accuracy: M-RNN achieved smaller RMSE than all benchmarks across datasets, with statistically significant improvements except versus MissForest on UNOS-Lung.On Deterioration, M-RNN (MI) achieved RMSE 0.0105 versus 0.0215 for Spline interpolation, a 51.2% improvement.
- Cross-dataset performance: M-RNN’s advantage was larger for frequently sampled datasets and smaller for infrequently sampled datasets.The method benefits from exploiting both within-stream and across-stream correlations, but infrequent sampling reduces gains from temporal correlations.
- Uncertainty: Multiple imputation reduced uncertainty compared with single imputation without being presented as improving average imputation performance.On MIMIC-III, the comparison was against Cubic interpolation using box plots of RMSE.
- Combining models: Combining interpolation with iterative imputation produced results similar to the simple imputation component, with the largest RMSE improvement only 0.0018.The iterative imputation stage was described as ignoring most information captured by the interpolation stage.
- Source of gains: The interpolation block contributed most on frequently sampled datasets, whereas the imputation block contributed most on infrequently sampled datasets with many streams.These expectations were supported by the component experiments in Table 4.
- Robustness: M-RNN continued to outperform benchmarks as missingness increased, although its advantage over interpolation benchmarks decreased while its advantage over imputation benchmarks increased.On MIMIC-III, the experiments removed 10%, 20%, 30%, 40%, or 50% of observed data; the original dataset already had 75% missingness.
- Robustness: With too few patient samples, M-RNN degraded badly and could become worse than some benchmarks, whereas fewer measurements per patient also degraded performance.The sample-size experiments used N = 500, 1000, 2000, 4000, 8000, and 16000 patients; the supplied passage does not state the crossover threshold.
- Prediction: M-RNN achieved the best prediction accuracy, but prediction improvements were sometimes smaller and not always statistically significant than imputation improvements.On Deterioration, AUROC was 0.7779 versus 0.7593 for the best benchmark; on UNOS-Heart, it was 0.6855 versus 0.6740 for MissForest.
7 Conclusion
The paper presents M-RNN for reconstructing missing data by exploiting correlations within and across data streams, with large and statistically significant improvements over state-of-the-art benchmarks on real-world medical datasets.
- M-RNN estimates missing data by exploiting correlations both within data streams and across data streams.
- M-RNN is evaluated on a variety of real-world medical datasets.
- M-RNN achieves large and statistically significant improvements over state-of-the-art benchmarks.
Predictions with Alternative Predictive Models
The study compares M-RNN with benchmarks when imputation is followed by several predictive models on the Deterioration dataset. Prediction accuracy varies little across predictive models, and differences from imputation accuracy are not statistically significant.
- Table 7 compares patient state prediction accuracy on the Deterioration dataset across predictive models following imputation.
- Random Forest, Logistic Regression, XGBoost, and RNN predictive models do not yield much different prediction accuracy.
- Prediction accuracy is not perfectly correlated with imputation accuracy, but the differences are not statistically significant.
Effects of Donor Features in UNOS Datasets
The analysis adds donor features to recipient features in the UNOS datasets and finds modest improvements in both missing-value estimation and label prediction.
- Donor features have small effects on imputation and prediction accuracy in the UNOS datasets.
- RMSE for M-RNN (MI) improves from 0.0479 to 0.0451 (5.8%) on UNOS-Heart and from 0.0606 to 0.0579 (4.5%) on UNOS-Lung.
- AUROC for M-RNN (MI) improves from 0.6855 to 0.7153 (9.5%) on UNOS-Heart and from 0.6762 to 0.6883 (3.7%) on UNOS-Lung.
Prediction-oriented M-RNN
The paper trains M-RNN for prediction accuracy using cross-entropy rather than mean square error and evaluates it with AUROC. This yields marginal, statistically nonsignificant improvement while removing preprocessing and imputation requirements.
- Prediction-oriented M-RNN minimizes cross-entropy instead of mean square error and continues to evaluate performance using AUROC.
- Tables 8 and 9 compare missing-value estimation and label prediction with and without donor features in the UNOS datasets.
- Prediction-oriented training improves M-RNN predictions marginally, but the improvement is not statistically significant.
- Prediction-oriented M-RNN creates an end-to-end prediction algorithm that does not require preprocessing or imputation steps.
Implementations
The paper uses off-the-shelf implementations for several benchmark algorithms, drawing on established MATLAB and R packages.
- Spline and Cubic Interpolation use the interp1 package in MATLAB.
- MICE, MissForest, EM, and matrix completion use dedicated R packages.The packages are mice, MissForest, Amelia, and softImpute, respectively.