PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Neural Networks for Multivariate Time Series with Missing Values

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, Yan Liu

arXiv:1606.01865v2cs.LGcs.NEstat.ML

TL;DR

Missing values and their patterns can be informative for time-series prediction, yet existing methods have limited ability to exploit them. The paper introduces GRU-D, which integrates masking and time intervals into GRU-based models, and reports improved performance across clinical and synthetic classification datasets.

  • Problem

    Missing values and missing patterns can correlate with target labels, but limited prior work exploits this information for imputation and prediction.

  • Method

    GRU-D incorporates masking and time intervals directly into a GRU architecture, applying them to inputs and network states with trainable decay mechanisms.

  • Results

    GRU-D outperforms strong GRU-with-imputation models and other baselines, achieving the best AUC on mortality tasks across MIMIC-III and PhysioNet.

  • Takeaways & Limitations

    Modeling missingness jointly with recurrent prediction can utilize informative patterns while capturing long-term temporal dependencies in multivariate time series.

Abstract

from arXiv · show

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provides useful insights for better understanding and utilization of missing values in time series analysis.

1 INTRODUCTION

Multivariate time series often contain informative missingness, but existing approaches do not systematically integrate missing patterns into recurrent models for classification. GRU-D addresses this gap by incorporating masking and time intervals into GRU-based prediction.

  • Missing observations arise routinely in applications including health care, geoscience, astronomy, and biology.
  • Missing values and their patterns can provide information about supervised-learning target labels.
  • Existing methods include omitting missing data, interpolation, spectral analysis, kernel methods, multiple imputation, and EM algorithms.
  • These methods often separate imputation from prediction and fail to exploit missing patterns effectively.
  • Prior recurrent approaches concatenated missing entries or timestamps with inputs, but did not systematically model missing patterns for time-series classification.
  • GRU-D incorporates masking and time intervals into GRU inputs and hidden states, jointly training the components while modeling temporal dependencies.

2 RNN MODELS FOR TIME SERIES WITH MISSING VARIABLES

This section defines RNN-based approaches for time-series classification with missing variables and introduces GRU-D, which models missingness through masking and trainable time-dependent decays.

  • Data representation: Missing observations are represented with masking vectors that indicate which variables are missing at each time step.The time series also records observation timestamps and time intervals between measurements.
  • GRU-RNN for classification: Standard GRU models process sequential observations with reset and update gates, sharing parameters across time steps and supporting variable-length sequences.A final sigmoid or soft-max layer produces the classification output from the last GRU state.
  • Existing approaches: Mean imputation, forward imputation, and concatenating missingness indicators each address missing values but do not fully exploit missingness patterns.Imputation cannot distinguish imputed from truly observed values, while simple concatenation does not exploit their temporal structure.
  • GRU-D: The decay formulation keeps each decay rate monotonically decreasing between 0 and 1 through an exponentiated negative rectifier.The decay parameters are learned jointly with the other GRU parameters.
  • GRU-D: GRU-D uses input and hidden-state decays to incorporate how long variables have been missing while jointly training the model parameters.Input values decay toward an empirical mean, hidden states decay before the next update, and masking vectors enter the GRU equations directly.
  • Model interpretation: PhysioNet mortality analysis shows that GRU-D learns variable-specific input and hidden decays related to missing rates and clinically relevant measurements.Variables with smaller missing rates have more spread-out hidden-decay parameters, indicating greater variation in how missingness affects hidden states.

3 EXPERIMENTS

The experiments evaluate GRU-D against non-RNN and RNN baselines on synthetic and real-world clinical classification tasks, including informative missingness, early prediction, and varying training sizes. GRU-D achieves the best reported performance across the evaluated real-data settings and improves early prediction efficiency.

  • Experimental design: Experiments compare GRU-D with non-RNN and RNN baselines on one synthetic and two real-world health-care datasets.The evaluation covers classification, early prediction, different training sizes, and informative missingness.
  • Informative missingness: The synthetic Gesture experiments vary the correlation between missing rates and labels while keeping the missing rate fixed.Higher correlation represents more informative missingness, testing whether models distinguish useful missing patterns.
  • Real-data prediction: GRU-D achieves the best AUC score on mortality prediction for both MIMIC-III and PhysioNet.The passage reports that all models except random forest improve when missingness indicators are included with inputs.
  • Real-data prediction: GRU-D performs best in average AUC across multi-task predictions and in most individual tasks on PhysioNet and MIMIC-III.The evaluated multi-task settings include four PhysioNet tasks and 20 MIMIC-III ICD-9 code tasks.
  • Early prediction: GRU-D reaches the same AUC as RF-simple at 36 hours, whereas RF-simple requires 48 hours on the MIMIC-III mortality task.GRU-D also has at least 2.5% higher AUC than GRU-simple after 30 hours.
  • Scalability: With larger MIMIC-III training sets, GRU-D achieves the best results on the larger datasets while non-RNN improvements remain limited.All models improve with more training samples, but the reported performance gap between RNN and non-RNN baselines grows.

4 SUMMARY

The paper proposes a GRU-based model that incorporates informative missingness directly into the recurrent architecture. Experiments on synthetic and health-care datasets show promising empirical performance and motivate future work on missing-not-at-random data.

  • 4 SUMMARY: GRU-D incorporates masking and time intervals directly inside a GRU architecture to handle informative missing values.The model jointly uses these missingness representations within the recurrent model.
  • 4 SUMMARY: Experiments on synthetic and real-world health-care datasets show promising results and provide insights into missing values in multivariate time series.Future work will study deep learning for missing-not-at-random data and theoretical analysis of missing-value solutions.

A.1 INVESTIGATION OF RELATION BETWEEN MISSINGNESS AND LABELS

The paper measures each variable’s missing rate and tests its correlation with prediction labels. On MIMIC-III, missing rates with low values are often strongly correlated with labels, supporting the usefulness of missingness patterns.

  • Missing rates are computed for each variable and correlated with prediction labels across time series.The prediction tasks include mortality and ICD-9 diagnosis categories.
  • Low missing-rate values are often highly positively or negatively correlated with labels on MIMIC-III.
  • Distinct correlations between missingness and labels demonstrate the usefulness of missingness patterns for prediction.

A.2 GRU-D MODEL VARIATIONS

Figure 8 graphically illustrates variations of the proposed GRU models.

  • Figure 8 presents graphical illustrations of proposed GRU model variations.
  • The figure compares architectural variations of the proposed models.
  • The figure provides a visual overview of the proposed GRU model family.

A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS

The paper varies where trainable decay is applied within GRU-D. Simplified variants decay only inputs or hidden states, while GRU-DM independently decays masking variables.

  • A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS: GRU-D applies trainable decays to both input and hidden-state transitions to capture temporal missing patterns.
  • A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS: GRU-DI decays only the input, targeting the direct impact of missing values.
  • A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS: GRU-DS decays only the hidden state, targeting the indirect impact of missingness.
  • A.2.1 GRU MODEL WITH DIFFERENT TRAINABLE DECAYS: GRU-DM replaces the masking input and decays each variable independently using a diagonal masking-decay matrix.

A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL

GRU-IMP treats missing values as latent variables and trains a classifier using a goal-oriented imputation objective. It optimizes a lower bound through sequential Monte Carlo approximation, using predicted means at test time.

  • A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL: GRU-IMP treats missing values as latent variables in a probabilistic graphical model for goal-oriented imputation.
  • A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL: Training targets the marginalized log-conditional probability of the correct label given observed values.
  • A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL: Because the exact objective and its lower bound are intractable, the method approximates the lower bound by sampling missing variables sequentially.
  • A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL: The missing-variable distribution at each time step is conditioned only on previous observations.
  • A.2.2 GRU-IMP: GOAL-ORIENTED IMPUTATION MODEL: At test time, GRU-IMP fills missing variables with their predicted means rather than Monte Carlo samples.

A.2.3 COMPARISONS OF RELATED RNN MODELS

Prior RNN work modeled irregular timestamps or missing values in EHR data, but did not explicitly model missing patterns. The paper distinguishes its approach from these models through explicit masking and interval representations.

  • Earlier EHR RNNs predicted diseases or diagnoses using irregular timestamps or missing-value data.
  • These models did not explicitly capture or model missing patterns within their RNNs.
  • The cited timestamp inputs were associated with GRU-simple, which uses intervals without masking.

A.3 SUPPLEMENTARY EXPERIMENT DETAILS

The supplementary experiments describe dataset and model-size reporting, multi-task prediction design, and comparative evaluations across MIMIC-III and PhysioNet. GRU-D achieved the best average AUC on both datasets and performed best among the tested GRU variations.

  • Supplementary experiment details: Dataset statistics report sample counts, input-variable counts, time-step lengths, and mean missing rates for three datasets.
  • Supplementary experiment details: Models were sized to have comparable parameter counts, iterations, and training times for fair GRU-RNN comparisons.
  • Supplementary experiment details: Multi-task models replace softmax with sigmoid outputs and apply a comorbidity-based prior regularizer to the prediction layer.
  • Supplementary experiment details: The MIMIC-III evaluation plots AUC by ICD-9 diagnosis category, while the PhysioNet plots show AUC for four prediction tasks.
  • Supplementary experiment details: GRU-D achieved the best average AUC on both datasets and won 11 of 20 ICD-9 prediction tasks.
  • Supplementary experiment details: GRU-D performed best among the tested trainable-decay, interval-only, and masking-only GRU variations.
Loading 1606.01865v2…