Source-linked AI summary
Learning to Diagnose with LSTM Recurrent Neural Networks
Zachary C. Lipton, David C. Kale, Charles Elkan, Randall Wetzel
TL;DR
Clinical time series are difficult to mine because they vary in length, are irregularly sampled, and contain missing data. This paper evaluates LSTMs for multilabel classification of 128 diagnoses from 13 PICU measurements, using target replication and related training strategies. LSTM models using raw time series outperform strong baselines, including an MLP using hand-engineered features, while ensembles improve further.
Problem
Clinical time series contain potentially valuable diagnostic information but are difficult to model because of variable lengths, irregular sampling, missing data, and long-term dependencies.
Method
The paper trains LSTM RNNs for multilabel diagnosis classification from 13 clinical measurements, evaluates target replication, auxiliary outputs, and dropout, and compares them with strong baselines.
Results
LSTMs with target replication and dropout outperform an MLP trained on hand-engineered features using only raw time series, while simple LSTM–MLP ensembles outperform both constituent models.
Takeaways & Limitations
LSTM RNNs, especially with target replication, can successfully classify diagnoses from critical-care clinical time series, and combining raw-sequence and engineered-feature models improves performance.
Takeaways & Limitations
The study uses diagnosis labels without timestamps, so it does not yet address early diagnosis by predicting future conditions.
Abstract
from arXiv · showhide
Clinical medical data, especially in the intensive care unit (ICU), consist of multivariate time series of observations. For each patient visit (or episode), sensor data and lab test results are recorded in the patient's Electronic Health Record (EHR). While potentially containing a wealth of insights, the data is difficult to mine effectively, owing to varying length, irregular sampling and missing data. Recurrent Neural Networks (RNNs), particularly those using Long Short-Term Memory (LSTM) hidden units, are powerful and increasingly popular models for learning from sequence data. They effectively model varying length sequences and capture long range dependencies. We present the first study to empirically evaluate the ability of LSTMs to recognize patterns in multivariate time series of clinical measurements. Specifically, we consider multilabel classification of diagnoses, training a model to classify 128 diagnoses given 13 frequently but irregularly sampled clinical measurements. First, we establish the effectiveness of a simple LSTM network for modeling clinical data. Then we demonstrate a straightforward and effective training strategy in which we replicate targets at each sequence step. Trained only on raw time series, our models outperform several strong baselines, including a multilayer perceptron trained on hand-engineered features.
1 INTRODUCTION
Clinical time series offer valuable diagnostic information but are difficult to model because episodes vary in length, observations are irregular and incomplete, and long-term dependencies complicate learning. This paper evaluates LSTMs for multilabel diagnosis classification and introduces target replication as a training strategy.
- PICU clinical time series contain potentially useful information for diagnosis, length-of-stay, future-illness, and mortality prediction.
- Variable episode lengths, irregular sampling, missing values, and long-term dependencies hinder machine learning on clinical time series.
- LSTMs model varying-length sequences and capture long-range dependencies and nonlinear dynamics.
- The study formulates PICU diagnosis recognition as multilabel classification using 13 clinical measurements to classify the 128 most common diagnosis codes.
- Target replication, auxiliary outputs, and dropout are evaluated, and LSTMs with target replication and dropout surpass an MLP trained on hand-engineered features using only raw time series.
2 RELATED WORK
Prior work applied recurrent and feedforward neural networks to medical and temporal clinical data, but not LSTMs for multilabel diagnosis classification in the medical domain. This paper contributes target replication for sequential clinical multilabel classification and reports improved training and generalization.
- Earlier medical applications used neural networks for clinical problems, while prior RNN work covered physiologic signals such as electrocardiograms and glucose measurements.
- Recent research applied deep learning and feedforward networks to psychological conditions, head injuries, Parkinson’s disease, gout, leukemia, and critical illness.
- LSTMs had been used for multilabel classification in music and video action recognition, but published medical applications were absent.
- Temporal clinical machine learning addressed artifact removal, early detection, prediction, clustering, and subtyping using models including latent-factor approaches.
- The paper replicates targets at every time step, using tied recurrent output weights to simplify intermediate optimization objectives.
- Dropout on non-recurrent weights is presented as a regularization approach relevant to recurrent networks trained on modest-scale data.
- The authors identify this work as the first medical LSTM multilabel study and report target replication with faster training and better generalization.
3 DATA DESCRIPTION
The study uses 10,401 anonymized PICU episodes represented by 13 clinical variables and diagnosis labels from an in-house taxonomy. Missing and irregular observations are hourly resampled and imputed, while clinician-assigned labels limit direct comparison with the model.
- The dataset contains 10,401 anonymized PICU episodes, each represented as a multivariate time series of 13 clinical variables.
- Clinical time series are irregularly sampled and include missing values and occasionally missing variables.
- The preprocessing resamples measurements hourly, fills resampling gaps forward and backward, and imputes entirely missing variables with clinically normal values.
- Episodes carry zero or more diagnosis codes from an in-house taxonomy containing 429 labels, with analyses focused on the 128 most common diagnoses.
- Clinician-assigned diagnostic codes make the evaluation a comparison with human experts who have access to substantially more patient information than the LSTM.
4 METHODS
The methods use LSTM recurrent networks for multilabel sequence classification, with target replication and auxiliary outputs designed to improve training and generalization. Regularization combines weight decay with dropout applied only to non-recurrent connections.
- LSTM architecture: The LSTM uses memory cells with forget gates, followed by a sigmoid-activated fully connected layer for multilabel outputs.Log loss is computed at each output.
- LSTM architecture: The simplest architecture processes inputs chronologically and produces predictions only at the final sequence step.Its loss averages log loss across labels.
- Sequential target replication: Target replication produces an output at every sequence step and combines the final loss with the average intermediate-step losses.The model uses the final-step output at prediction time.
- Auxiliary output training: Auxiliary-output training adds extra diagnostic labels as training targets while generating predictions only for the 128 labels of interest at inference.The additional targets are intended to reduce overfitting.
- Regularization: Target replication and auxiliary outputs improve performance and reduce overfitting, while dropout on non-recurrent connections enables larger hidden layers.The experiments use weight decay and dropout as regularization techniques.
5 EXPERIMENTS
The experiments compare LSTMs and baselines for multilabel diagnosis classification, then evaluate dropout, target replication, auxiliary outputs, and complementary ensembles. The best LSTM outperformed the hand-engineered-feature MLP, while combining both models improved performance across metrics.
- Multilabel evaluation methodology: Micro and macro AUC and F1, plus precision at 10, measure multilabel diagnosis performance across all labels.Precision at 10 measures the fraction of true diagnoses among the model’s top 10 predictions; its best possible test score is 0.2281.
- Baseline classifiers: The study evaluates LSTMs, logistic regression, and MLP baselines using raw time series or 143 hand-engineered clinical features.Baselines use fixed windows of the first and last six hours, whereas the strongest MLP uses features summarizing levels, variability, extremes, and trends.
- Results: LSTM-DO-TR, with two 128-cell layers, dropout probability 0.5, and target replication, outperformed the MLP with hand-engineered features.Table 1 summarizes performance across all labels, while Table 2 reports the six diagnoses with the highest LSTM-DO-TR F1 scores.
- Results: Target replication improved performance on all metrics, accelerated learning, reduced overfitting, and enabled earlier correct diagnosis outputs.The linear-gain variant generally performed worse, although it achieved the highest macro F1 among LSTM models.
- Results: Auxiliary outputs improved most metrics and reduced overfitting, but required more training epochs, especially with 301 additional diagnosis labels.Severe class imbalance among extra labels may contribute to slower learning.
- Results: Simple mean or maximum ensembles of the best LSTM and hand-engineered-feature MLP significantly outperformed both constituent models on all metrics.The models differed by diagnosis: the LSTM performed better for intracranial hypertension, whereas the MLP performed better for septic shock.
6 DISCUSSION
The discussion concludes that target replication and other regularization strategies improve clinical time-series diagnosis, while their effects depend on how they are combined. It also links target replication’s advantage over linear gain to the temporal progression of critical illness.
- 6 DISCUSSION: LSTM-DO-TR beat the hand-engineered-feature MLP, and an ensemble combining them improved upon both models.The paper presents this as evidence that LSTMs can classify critical-care diagnoses from clinical time series.
- 6 DISCUSSION: Target replication improved generalization and training speed, whereas uniform intermediate-target weighting outperformed linear gain in this diagnostic task.The authors suggest that linear gain’s emphasis on later evidence may mismatch symptom progression in critical illness.
- 6 DISCUSSION: Because the dataset is modest relative to vision and speech datasets, regularization is critical; target replication, auxiliary outputs, and dropout reduce the generalization gap.The strategies are not fully additive: target replication and dropout worked well together, while dropout and auxiliary outputs appeared to cancel each other’s benefits.
7 CONCLUSION
The paper identifies data representation and task scope as important next steps for clinical LSTM research. Planned extensions address missingness, irregular sampling, richer inputs, interpretability, and broader clinical prediction tasks.
- The authors aim to extend prediction beyond diagnoses to developing conditions, mortality, treatment responses, and future conditions using timestamped diagnoses.
- Current preprocessing discards information about observation timing, missingness, and the variability of frequently measured vital signs.Forward- and back-filling, normal-value imputation, and window-based resampling each remove structure from the clinical time series.
- Future models will distinguish actual measurements from missing or imputed values using indicator variables and architectures that directly handle missingness and irregular sampling.
- Planned extensions incorporate non-sequential inputs such as age, weight, height, treatments, and medications alongside larger clinical datasets.
- The paper also identifies interpretability as an ongoing concern and describes efforts to expose patterns learned by LSTMs to clinical users.
A HOURLY DIAGNOSTIC PREDICTIONS
The models produce diagnosis probabilities at every hourly resampled sequence step, enabling qualitative examination of how predictions develop over time. Target replication leads to earlier correct diagnoses, while combining it with auxiliary outputs produces more confident predictions.
- LSTM-TR reaches correct diagnoses quickly compared with LSTM-Simple, which uses targets only at the final sequence step.
- LSTM-TR,AO generally assigns more confident diagnoses when auxiliary outputs are combined with target replication.
- The four models differ by whether they use final-step targets, target replication, auxiliary outputs, or both techniques.
- The qualitative analysis reports effective predictions for status asthmaticus and acute respiratory distress syndrome, likely using pulmonary-function measurements.
- This step-by-step analysis is preliminary and includes target leakage in a small number of examples caused by back-filling missing values.
B LEARNING CURVES
The learning-curve analysis compares training and validation performance across LSTM variants to assess learning speed and regularization. Target replication accelerates learning modestly, whereas auxiliary outputs slow learning but regularize strongly.
- The evaluation tracks micro AUC and F1 score during training.
- Target replication appears to increase learning speed and provide a small regularizing effect.
- Auxiliary outputs slow learning but impart a strong regularizing effect.
- The plots compare training and validation performance for LSTM-Simple, LSTM-TR, and LSTM-AO.
C PER DIAGNOSIS RESULTS
The per-diagnosis analysis examines classifier performance separately for individual diagnostic codes rather than relying only on aggregate statistics. Results are presented using F1 and AUC measures across diagnoses and model columns.
- Individual diagnostic codes are evaluated with separate F1 and AUC scores.
- The per-diagnosis performance display sorts diagnostic conditions by F1.
- The comparison includes LSTM-DO-TR, an MLP with expert features, and a max ensemble.