PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Missing Data in Clinical Time Series with RNNs

Zachary C. Lipton, David C. Kale, Randall Wetzel

arXiv:1606.04130v5cs.LGcs.IRcs.NEstat.ML

TL;DR

Irregular clinical measurements create missingness patterns in discretized PICU time series, and the paper asks whether those patterns can improve multilabel diagnosis prediction. It models missingness directly alongside imputation, showing gains across model families and predictive value in the tests that clinicians order. The paper also discusses the risk that treatment-pattern signals can shift future data and undermine deployed predictions.

  • Problem

    Irregularly recorded clinical observations produce non-random missingness in discretized sequences, but standard approaches typically handle it through imputation rather than modeling its information.

  • Method

    The paper evaluates imputation and direct missingness modeling for RNN-based multilabel diagnosis prediction, using binary indicators for RNNs and hand-engineered missingness features for linear models.

  • Results

    Missingness indicators improve RNNs, MLPs, and linear models; an LSTM with zero-imputation and indicators achieves a micro AUC of 0.8730.

  • Takeaways & Limitations

    For several diseases, which tests are run can be as predictive as the measurements, making missingness patterns an underused source of predictive information.

  • Takeaways & Limitations

    Treatment-pattern signals may shift future data after deployment, potentially invalidating predictions as models influence treatment protocols.

Abstract

from arXiv · show

We demonstrate a simple strategy to cope with missing data in sequential inputs, addressing the task of multilabel classification of diagnoses given clinical time series. Collected from the pediatric intensive care unit (PICU) at Children's Hospital Los Angeles, our data consists of multivariate time series of observations. The measurements are irregularly spaced, leading to missingness patterns in temporally discretized sequences. While these artifacts are typically handled by imputation, we achieve superior predictive performance by treating the artifacts as features. Unlike linear models, recurrent neural networks can realize this improvement using only simple binary indicators of missingness. For linear models, we show an alternative strategy to capture this signal. Training models on missingness patterns only, we show that for some diseases, what tests are run can be as predictive as the results themselves.

1. Introduction

Clinical time series are irregularly recorded, so discretization creates missingness patterns that may encode caregiver decisions and patient state. The paper models missingness directly, finding benefits for neural and linear models while highlighting risks of treatment-pattern reliance.

  • 1. Introduction: Clinical observations are irregular across patients, variables, and time, producing missing values when represented in fixed-width sequences.These values are typically not missing at random and can reflect caregiver decisions.
  • 1. Introduction: The paper extends RNN-based multilabel diagnosis prediction on PICU data by treating missingness as a feature rather than only imputing values.RNNs use simple binary missingness indicators, while linear models require hand-engineered features to capture the signal.
  • 1. Introduction: Missingness indicators improve RNN, MLP, and linear-model performance, and missingness-only models show that tests run can be as predictive as measurements for several diseases.The methods are presented as applicable to other sequence-prediction problems involving missing values.
  • 1. Introduction: Models that rely on treatment patterns may alter future treatment protocols, shifting the data distribution and invalidating their predictions.The paper discusses this practical and philosophical concern for deployed predictive models.

2. Data

The study uses PICU EHR episodes from Children’s Hospital Los Angeles, converting irregular clinical measurements into hourly sequences for multilabel diagnosis classification. These records contain frequent, variable-specific missingness and multiple diagnosis labels per patient.

  • 2. Data: The dataset contains 10,401 PICU episodes, each covering one patient stay of at least 12 hours with physician-annotated diagnostic codes.The records were extracted from the CHLA EHR system as part of an IRB-approved study.
  • 2. Data: Raw episodes contain irregularly spaced measurements of 13 clinical variables, which are converted into discrete hourly time steps for RNN learning.The hourly representation partitions each episode into fixed intervals.
  • 2. Data: Vital signs are measured about hourly, whereas blood-draw laboratory tests such as glucose are measured about daily, creating variable missingness across sequences.Observation timing also varies across patients and over time, and some variables are absent altogether.
  • 2. Data: Halving the discrete time step would double sequence length and make backpropagation through time more challenging.The authors consider shorter steps warranted for higher-frequency recordings with faster dynamics.
  • 2. Data: The task uses 128 frequent diagnosis labels selected from 429 CHLA diagnosis codes, with labels allowed to overlap.Each selected label has at least 50 positive examples, and patients have an average of 2.24 diagnoses.

3. Recurrent Neural Networks for Multilabel Classification

The diagnosis classifier processes observation sequences with an LSTM RNN and produces multilabel outputs at each sequence step. Training combines losses across labels and time, while inference uses only the final-step output.

  • 3. Recurrent Neural Networks for Multilabel Classification: Given observations x^(1), ..., x^(T), the classifier generates K-dimensional hypotheses y_hat for the true multilabel diagnosis vector y.D is the input dimension, K is the number of labels, t indexes sequence steps, and T is sequence length.
  • 3. Recurrent Neural Networks for Multilabel Classification: The RNN uses LSTM memory cells with forget gates, followed by a fully connected layer and element-wise logistic activations.Binary cross-entropy is applied at each output node.
  • 3. Recurrent Neural Networks for Multilabel Classification: LSTM updates combine inputs from the previous layer and the prior sequence step through input, forget, and output gates.The equations use logistic and tanh activations together with element-wise products.
  • 3. Recurrent Neural Networks for Multilabel Classification: At each sequence step, the loss is averaged across all labels.This provides the per-step multilabel training objective.
  • 3. Recurrent Neural Networks for Multilabel Classification: Target replication supplies static diagnosis targets at every sequence step, providing local error signals for learning across long sequences.The model still uses only the final-step output at inference time.

4. Missing Data

The paper treats missingness as information rather than only an imputation problem, using indicators for RNNs and engineered missingness features for linear models.

  • Imputation strategies: The section evaluates forward-filling, zero imputation, direct missingness indicators, and combinations of imputation with indicators.Imputation and direct modeling are evaluated together because they are not mutually exclusive.
  • Imputation strategies: Zero imputation sets missing measurements to 0, while forward-filling carries forward the previous measurement or uses the training-data median when none exists.Forward-filling assumes the value changed little since its last measurement.
  • Missingness indicators: Binary indicators augment the inputs to mark whether each measurement was imputed, allowing RNNs to learn functions of observations and missingness patterns.The indicator is 1 when a value is imputed and 0 otherwise.
  • Missingness indicators: Linear models use indicators only through fixed additive substitutions, whereas RNN hidden states can represent context-dependent effects of missingness.A linear indicator’s contribution cannot depend on previously observed values or other input evidence.
  • Implicit missingness: RNNs may infer missingness even without indicators by recognizing exact repeats from forward-filling or values set exactly to zero.This creates a possible signal distinguishing filled-in from real values.
  • Hand-engineered features: Engineered features summarize whether variables were measured, measurement frequency, switching frequency, and the relative timing of first and last measurements.The feature set includes indicator means, standard deviations, transitions, and normalized timing features.

5. Experiments

The experiments compare RNNs, MLPs, and logistic regression under alternative imputation and missingness-feature settings using aggregate multilabel metrics. Missingness indicators improve models overall, with the strongest result from an indicator-augmented zero-imputed LSTM.

  • Experimental setup: Experiments compare RNNs with logistic regression and MLP baselines using raw and hand-engineered features.The baselines use fixed representations because they cannot directly process variable-length inputs.
  • Evaluation: Models are evaluated with micro- and macro-averaged AUC and F1 scores, plus Precision at 10 for the top 10 diagnostic suggestions.F1 thresholds are selected using validation performance.
  • Results: The best overall model is an LSTM with zero-imputation and missing data indicators, achieving a micro AUC of 0.8730.It outperforms the strongest MLP baseline and LSTMs without missingness indicators.
  • Results: Adding missing data indicators improves LSTM performance across all metrics for either imputation strategy.All models improve with indicators, but the benefit is smaller for raw-input linear baselines.
  • Results: Hand-engineered missingness features allow logistic regression to capture important information that neural networks can mine automatically.The paper describes these simple features as reasonably effective at capturing that information.
  • Results: LSTMs appear to perform better with zero-filling than with imputed values, unlike the baseline models.The authors suggest LSTMs may implicitly recognize zero-filled missing values.

6. Related Work

The work extends medical missing-data and RNN research by combining missingness indicators with recurrent models for clinical time series. It also addresses the comparatively limited literature on missing data in RNNs.

  • Foundations: The RNN phenotyping methodology builds on Lipton et al. (2016) and a dataset and problem described by Che et al. (2015).The methods use LSTM RNNs trained by backpropagation through time.
  • Missing data and RNNs: Earlier missing-data pattern-recognition literature largely addresses fixed-length feature vectors rather than sequential inputs.The paper situates its contribution within medical informatics and machine learning research on missing values.
  • Missing data and RNNs: The authors report finding no papers that combine missing-data indicators with RNNs, while only a handful address missing data in RNN contexts.Indicator variables themselves were previously proposed by Cohen and Cohen (1975).

7. Discussion

The discussion argues that missingness patterns are an underused predictive signal that RNNs can exploit effectively, while treatment-related signals raise deployment concerns. It also weighs model complexity against feature engineering and identifies clinical prediction and decision support as next steps.

  • Discussion: Missingness indicators improve predictive modeling, and RNNs can mine this signal from sequential data more effectively than linear models.The paper also introduces missingness-pattern features that improve linear models.
  • 7.1 The Perils and Inevitability of Modeling Treatment Patterns: Treatment patterns can be predictive, but deployment may shift future data distributions and invalidate offline predictions.The paper illustrates this concern with treatment-driven signals in clinical risk prediction.
  • 7.1 The Perils and Inevitability of Modeling Treatment Patterns: Learning treatment signal may be difficult to avoid because imputation and physiologic measurements can both retain information about care.The paper notes that medications and procedures may be reflected in observed measurements.
  • 7.2 Complex Models or Complex Features?: RNNs achieved state of the art clinical time-series classification with simple features and outperformed linear models.The experiments found that extensive feature engineering narrowed, but did not eliminate, the performance gap.
  • Future Work: The paper proposes validating the methodology on sepsis, mortality, and length-of-stay prediction and extending it toward clinical decision support.It frames policy imitation as a possible route to real-time decision support.

Appendix A. Per Diagnosis Classification Performance

Appendix A reports per-diagnosis classification results for LSTM models using imputed measurements, imputation plus missing indicators, and missing indicators alone. The table is organized by diagnostic-code performance, with rows sorted by F1 score for the imputation-plus-indicators model.

  • Appendix A. Per Diagnosis Classification Performance: Per-diagnosis results compare measurements, measurements plus missing indicators, and indicators-only LSTM models.The reported metrics are AUC and F1 for individual diagnostic codes.
  • Appendix A. Per Diagnosis Classification Performance: Rows are sorted in descending order by the F1 score of the imputation-plus-indicators model.This ordering provides the table’s per-diagnosis ranking criterion.
  • Appendix A. Per Diagnosis Classification Performance: F1 scores use thresholds selected to optimize per-disease validation F1 and are therefore threshold-sensitive.The appendix notes that some thresholds were based on small validation sets.

Appendix B. Missing

Appendix B summarizes sampling and missingness for the paper’s 13 variables. It reports average sampling frequency, the fraction of episodes missing a variable entirely, and missingness in discretized sequences.

  • Appendix B. Missing: The first statistic is the average number of measurements per hour among episodes with at least one measurement.Episodes in which a variable is missing entirely are excluded from this calculation.
  • Appendix B. Missing: The second statistic is the fraction of episodes in which a variable has zero measurements.This captures complete absence of a variable within an episode.
  • Appendix B. Missing: The third statistic is the missing rate in the resulting discretized sequences.This measures missingness after observations are represented in discrete time steps.
  • Appendix B. Missing: The appendix reports sampling rates and missingness statistics for all 13 features.The statistics characterize how often variables are measured and absent.
Loading 1606.04130v5…