PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Series Prediction using Deep Learning Methods in Healthcare

Mohammad Amin Morid, Olivia R. Liu Sheng, Joseph Dunbar

arXiv:2108.13461v3cs.LG

TL;DR

Traditional machine learning struggles with high-dimensional healthcare data and with representing temporal dependencies in patient records. This paper systematically reviews deep learning studies using structured patient time series, classifying their contributions across ten research streams and identifying research gaps. The review highlights technical advances while noting open questions around learning strategies and scalability.

  • Problem

    Traditional ML requires labor-intensive feature selection and feature engineering that may not adequately capture temporal patterns and dependencies in patient data.

  • Method

    The authors systematically reviewed deep learning research using structured patient time series for healthcare prediction and organized findings into ten research streams.

  • Results

    The review identifies ten research streams spanning models, missing values, irregularity, representations, static data, attention, interpretation, ontologies, learning strategies, and scalability.

  • Takeaways & Limitations

    The review summarizes research insights, identifies critical gaps, and suggests future opportunities for deep learning with patient time series data.

  • Takeaways & Limitations

    Learning strategies for integrating medical knowledge into deep learning networks remain a wide-open research gap, and scalability also requires further study.

Abstract

from arXiv · show

Traditional machine learning methods face two main challenges in dealing with healthcare predictive analytics tasks. First, the high-dimensional nature of healthcare data needs labor-intensive and time-consuming processes to select an appropriate set of features for each new task. Second, these methods depend on feature engineering to capture the sequential nature of patient data, which may not adequately leverage the temporal patterns of the medical events and their dependencies. Recent deep learning methods have shown promising performance for various healthcare prediction tasks by addressing the high-dimensional and temporal challenges of medical data. These methods can learn useful representations of key factors (e.g., medical concepts or patients) and their interactions from high-dimensional raw or minimally-processed healthcare data. In this paper we systematically reviewed studies focused on advancing and using deep neural networks to leverage patients structured time series data for healthcare prediction tasks. To identify relevant studies, MEDLINE, IEEE, Scopus and ACM digital library were searched for studies published up to February 7th 2021. We found that researchers have contributed to deep time series prediction literature in ten research streams: deep learning models, missing value handling, irregularity handling, patient representation, static data inclusion, attention mechanisms, interpretation, incorporating medical ontologies, learning strategies, and scalability. This study summarizes research insights from these literature streams, identifies several critical research gaps, and suggests future research opportunities for deep learning in patient time series data.

1 INTRODUCTION

Healthcare predictive analytics increasingly uses electronic health and claims data, but traditional machine learning struggles with high dimensionality and sequential patient information. Deep learning is presented as a response to these challenges.

  • Healthcare data recorded in EHR and administrative claims systems has expanded to millions of individuals over many years.
  • ML has been applied to tasks including early triage, physiologic decompensation, high-cost patient identification, and complex disease characterization.
  • Traditional ML depends on feature engineering to represent sequential patient data and may miss temporal patterns and dependencies among medical events.
  • Traditional ML requires labor-intensive feature selection from thousands of healthcare variables for each prediction task.Medical experts may also need to extract, preprocess, and clean data from different sources.

2 METHOD

The review systematically organizes research on deep learning for structured patient time series prediction. It extracts common study characteristics, classifies technical contributions into ten categories, and compares their strengths, weaknesses, and research opportunities.

  • The review aims to organize findings and identify future opportunities in deep learning for structured healthcare time series prediction.
  • The authors formulated questions about data representation, incomplete and irregular time series, model effectiveness, medical resources, interpretability, and scalability.
  • Each study was characterized using ten attributes covering tasks, databases, inputs, preprocessing, representations, architectures, temporality, performance, benchmarks, and interpretation.
  • Technical contributions were classified into ten categories, including patient representation, missing values, temporal irregularity, attention, ontologies, learning strategy, interpretation, and scalability.
  • The review compares the strengths and weaknesses of identified techniques and highlights associated research gaps and future opportunities.
  • Eligible studies were English-language original peer-reviewed research using deep learning with structured patient time series from EHR or claims data.

3 RESULTS

The review screened a large literature and identified 77 included studies, with most published after 2018. The studies span ten contribution categories and concentrate on several recurring prediction tasks and data sources.

  • 1,524 studies were initially identified; after duplicate removal, screening, and full-text exclusions, 77 studies were included.The exclusions covered inappropriate data types, missing multivariate temporal data, absent deep learning, unstructured data, and incomplete study characteristics.
  • 77% of included studies were published after 2018, indicating a recent surge of interest in deep learning for healthcare prediction.
  • Mortality, heart failure, readmission, and next-visit diagnosis were the most studied prediction tasks.
  • MIMIC was the most popular data source among the reviewed studies.
  • The literature's contributions fall into ten categories, including patient representation, missing values, deep learning models, temporal irregularity, attention, ontologies, and scalability.

3.1 Patient representation

Patient representations mainly use sequences or longitudinal matrices, with the preferred form depending on input type and task. Reviewed studies also examine time granularity, temporal features, embeddings, and coding strategies.

  • Patient time series representations are broadly divided into sequence and matrix representations.
  • 57% of researchers used sequence representation, making it slightly more prevalent than matrix representation.
  • Sequence representations are more common for numeric inputs, whereas matrix representations are more common for categorical diagnosis and procedure codes.
  • One study found matrix representation superior to sequence representation for COPD readmission prediction using a large administrative claims database.
  • Studies using matrix representations reported improved performance with coarse weekly or monthly time granularities rather than finer intervals.
  • Temporal summary measures improved interpretability of extracted features without degrading prediction performance.
  • Pre-trained embedding layers outperformed trainable layers by 2% in recall for next-visit diagnosis and by 1.5% in AUC for heart failure compared with medical group coding.

3.2 Missing value handling

Missing-value handling in patient time-series prediction commonly relies on simple imputation or expert knowledge, while modeling missingness patterns can improve predictive performance. The reviewed literature also reports several recurrent architectures and training strategies used across healthcare tasks.

  • Missing value handling: Simple imputation with zero, median, or forward-backward methods, along with expert domain knowledge, has been the most common missing-value strategy.
  • Missing value handling: Masking vectors have been used as separate inputs to represent value availability across discharge diagnosis, readmission, acute kidney injury, mortality, and length-of-stay prediction.
  • Missing value handling: Approximately 2% AUC improvement was reported when missing patterns outperformed the masking vector technique across mortality, length-of-stay, surgery recovery, and cardiac-condition prediction.
  • Deep learning models: GRU and LSTM models were the most frequently used architectures, appearing in 84% of reviewed studies.
  • Deep learning models: Bidirectional GRU and LSTM variants consistently outperformed unidirectional counterparts across multiple healthcare prediction tasks.
  • Deep learning models: Target replication showed that predicting at multiple time points can be more effective than supervising only the final time-stamped entry.

3.4 Addressing temporal irregularity

Patient time-series data exhibit both visit and feature irregularity, and reviewed methods explicitly model time intervals or decay patterns. Accounting for both irregularity types or emphasizing recent visits produced reported performance gains in specific prediction tasks.

  • Temporal irregularity: Visit irregularity concerns variable intervals between a patient’s visits, whereas feature irregularity concerns measurements recorded at different times and frequencies within one visit.
  • Visit irregularity: Models using time intervals between visits as separate inputs were reported effective for next-visit diagnosis, heart failure, vascular disease, hospital mortality, and readmission prediction.
  • Visit irregularity: LSTM modifications that assigned higher weights to recent visits outperformed traditional LSTM architectures by 3% in AUC for a highly frequent benchmarking task.
  • Feature irregularity: Capturing feature and visit irregularity decreased MSE by up to 5% compared with models capturing visit irregularity alone.

3.5 Attention mechanism

Attention mechanisms assign unequal weights to visits or medical events and are commonly used to improve model interpretability. Where studies reported differential effects, location-based, visit-level, and medical-code attention yielded task-specific performance gains.

  • Attention mechanism: Attention mechanisms assign and optimize distinct weights for visits or medical events according to their relevance to the prediction task.
  • Reporting pattern: Most attention studies did not report differential prediction-performance improvements attributable to attention.
  • Performance effects: Location-based attention improved next-visit diagnosis by 4% in AUC, readmission by 2.4% in F1-Score, and mortality by 13% in F1-Score.
  • Performance effects: Visit-level attention and medical-code attention each produced an approximate 4% increase in F2-Score for hospital readmission.
  • Attention architectures: A crossover attention model used distinct bidirectional GRUs and attention weights for diagnosis and medication codes.
  • Interpretability: Most studies used attention to improve interpretability by highlighting important visits or medical codes at patient or population level.

3.6 Incorporation of medical ontologies

Reviewed studies incorporated medical ontologies and knowledge graphs into embeddings, attention, and recurrent models to represent relationships among medical concepts. These approaches produced reported gains for heart-failure prediction and supported richer concept representations.

  • Knowledge graphs: Knowledge-graph relationships between diseases were associated with an approximate 2% AUC margin for heart-failure prediction.
  • Ontology construction: Medical ontology trees establish hierarchies of medical concepts through successive parent-child relationships for use in deep-learning models.
  • Ontology representations: Ontology-based embeddings represented abstract medical concepts for heart-failure prediction, while later work allowed multiple parents and ordered ancestor sets.
  • Ontology-enhanced attention: Ontology trees used to calculate GRU attention weights achieved a 3% accuracy increase over an earlier ontology-based approach for the same prediction task.
  • Medical concept embeddings: SkipGram embeddings augmented with medical n-gram tokens captured ancestral information for diseases or drugs in next-visit diagnosis prediction.

3.7 Static data inclusion

Static data inclusion methods combine patient-level information with temporal inputs because demographics and prior history can support accurate healthcare predictions.

  • Static data integration: Appending static patient data to a final fully connected layer is the most common integration approach.This method has been applied to readmission, length-of-stay, and mortality prediction tasks.
  • Static data integration: Independent feedforward networks can process static features before combining their outputs with temporal representations.One study fed 342 static features into a separate network before combining them with temporal data in a GRU layer.
  • Static data integration: Static information has been incorporated by modifying LSTM processes or by repeating it at every temporal input point.Repeated static inputs included demographic data, vascular risk factors, and nursing-level assessments for heart failure prediction.
  • Research gap: The review found no study comparing static-data inclusion methods against solid benchmarks.This leaves the relative performance of the approaches unresolved.

3.8 Learning strategy

The reviewed studies used cost-sensitive, multi-task, and transfer learning to extend supervised prediction, alongside several interpretation strategies for understanding model outputs.

  • Learning strategies: Multi-task learning jointly predicted mortality, length-of-stay, and phenotyping using LSTM, Bi-LSTM, and GRU architectures.A seminal study reported a significant contribution over state-of-the-art approaches.
  • Learning strategies: Domain adaptation outperformed task adaptation when data were small, whereas task adaptation was superior otherwise.For medium-sized data, fine-tuning outperformed learning from scratch with feature extraction.
  • Interpretation: Explainable attention commonly highlighted visits and medical codes influencing predictions through learned attention weights.Other interpretation methods included pooling, convolutional layers, DeepLIFT, and Shapley values.

3.10 Scalability

Scalability studies examined deep time-series models across datasets, feature dimensions, and patient populations, often finding performance parity with traditional machine learning in some settings.

  • Dataset scalability: RETAIN achieved a similar AUC across ten hospital EHR datasets, while higher dimensionality improved prediction performance.A separate study evaluated RETAIN on approximately four million UK National Health Service patients and reported the same observation.
  • Dimensional scalability: For lower-dimensional data, traditional ML performance was comparable to DL, while DL performed better for high-dimensional data.This comparison used mortality, length-of-stay, and phenotype prediction with up to 136 features.
  • Dataset scalability: Strong traditional supervised ML methods achieved comparable performance to deep learning on approximately 103 million medical claims and 17 million pharmacy claims.The evaluation included 111,000 patients.

4 DISCUSSION

The discussion synthesizes evidence across representation, missingness, architectures, irregularity, learning, and interpretation, while identifying areas needing broader comparisons and further study.

  • Patient representation: Pre-trained embedding layers outperformed naïve embeddings and medical-code groupings for EHR data, whereas no significant difference was observed for claims data.The review recommends sequence representations with pre-trained embeddings for EHR data and matrix representations for claims data.
  • Missing value handling: Masking vectors remain the suggested missing-value strategy because they have been evaluated across more healthcare applications and missingness levels.Learning missing values was more effective when missingness was highly correlated with the mortality target.
  • Deep learning models: Bidirectional recurrent networks generally performed strongly, while GRU offered lower complexity with performance comparable to LSTM.GRU and LSTM sometimes differed by around 1% in AUC, but other studies found no significant difference.
  • Temporal irregularity: Time-interval approaches were the most commonly published and tested strategy for handling visit irregularity.The review calls for additional benchmarking of memory-mechanism adjustments for visit or feature irregularity.
  • Learning strategy: Transfer learning was the least studied strategy, with one citation showing effectiveness for task and domain adaptation.The review identifies transfer learning as a wide-open research opportunity in deep time-series prediction.
  • Interpretation: Feature-importance reporting is recommended because it is less complex and can provide useful clinical implications with little added effort.Population-level interpretation remained limited, with only one study extracting disease-associated CNN motifs.

5 CONCLUSION

The review organizes deep learning research for structured patient time series into ten contribution categories, spanning representation, modeling, data handling, interpretation, learning strategies, and scalability. It also identifies unresolved benchmarking, evaluation, architectural, and review-scope gaps that motivate future work.

  • The reviewed literature is classified into ten categories covering patient representation, missing values, deep learning models, temporal irregularity, attention, ontologies, static data, learning strategies, interpretation, and scalability.
  • Patient representation: Sequence and matrix representations are the two common approaches for representing patient data, while medical codes can be embedded from scratch, pretrained, or grouped clinically.
  • Missing value handling: Missingness can encode caregiver decisions, making masking vectors and learned missingness patterns important approaches for handling patient time-series data.
  • Research gaps: Future research should benchmark temporal-irregularity methods across hospital settings, examine advanced architectures, and compare deep learning with traditional ensembles where their performance may be comparable.
  • Attention mechanisms: Attention commonly highlights important visits or medical codes for interpretability, but studies rarely evaluate its differential effect on prediction performance or separate visit-level from code-level contributions.

Database

The database section organizes the review’s extracted study information, abbreviations, architectures, patient datasets, prediction tasks, input representations, and performance results. It also records temporal processing, missing-value strategies, interpretability methods, and benchmarking models across reviewed studies.

  • The supplementary material includes an abbreviation list and a table of reviewed studies’ extracted features aligned with Table 1.
  • Reviewed studies use recurrent, convolutional, attention-based, graph-based, and embedding-enhanced architectures for healthcare prediction.
  • The reviewed datasets include EHR and claims data from thousands to hundreds of thousands of patients, with tasks including mortality, diagnosis, readmission, and adverse-event prediction.
  • Patient time series are represented using temporal matrices, multi-hot vectors, bag-of-words features, graphs, and time-since-last-visit information.
  • Studies report varied performance measures, including AUC, Accuracy@k, Precision@k, Recall@k, and F-1, often alongside logistic regression, SVM, random forest, or RNN benchmarks.
  • Missing-value strategies in the reviewed studies include last-value or manual imputation, mean and latest-value imputation, zero-value imputation, and forward imputation.
Loading 2108.13461v3…