Source-linked AI summary
Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, Jimeng Sun
TL;DR
Doctor AI addresses the challenge of predicting heterogeneous future clinical events from longitudinal EHR data without relying on separate specialized models. It uses an RNN with multilabel outputs for diagnoses and medications, achieving up to 79% recall@30 and supporting transfer across institutions, while time-interval prediction remains difficult and incorrect predictions can harm patients.
Problem
Existing predictive work often targets limited outcomes, while clinical practice requires predictions across heterogeneous scenarios and future diagnoses, medications, and visit timing.
Method
Doctor AI uses an RNN to learn patient representations from timestamped longitudinal EHR codes, with multilabel predictions for diagnoses and medications and Skip-gram-based initialization.
Results
79.58% recall@30 was achieved on real-world EHR data, with the model significantly outperforming many baselines and adapting across hospitals.
Takeaways & Limitations
Doctor AI provides clinically meaningful diagnostic predictions and shows potential for differential-diagnosis assistance and transfer to institutions with smaller datasets.
Takeaways & Limitations
Accurate prediction of visit-time intervals remains challenging without sensitive personal features, and incorrect predictions may harm patients.
Abstract
from arXiv · showhide
Leveraging large historical data in electronic health record (EHR), we developed Doctor AI, a generic predictive model that covers observed medical conditions and medication uses. Doctor AI is a temporal model using recurrent neural networks (RNN) and was developed and applied to longitudinal time stamped EHR data from 260K patients over 8 years. Encounter records (e.g. diagnosis codes, medication codes or procedure codes) were input to RNN to predict (all) the diagnosis and medication categories for a subsequent visit. Doctor AI assesses the history of patients to make multilabel predictions (one label for each diagnosis or medication category). Based on separate blind test set evaluation, Doctor AI can perform differential diagnosis with up to 79% recall@30, significantly higher than several baselines. Moreover, we demonstrate great generalizability of Doctor AI by adapting the resulting models from one institution to another without losing substantial accuracy.
1. Introduction
Doctor AI addresses the need for flexible prediction across heterogeneous clinical scenarios by using an RNN to predict diagnoses, medications, and visit timing from longitudinal EHR data. It also evaluates Skip-gram initialization and cross-institution transfer learning.
- Motivation: Existing predictive models usually target limited outcomes, whereas day-to-day clinical practice involves heterogeneous scenarios requiring flexible prediction.Developing specialized models one by one is impractical.
- Approach: Doctor AI uses a recurrent neural network to represent longitudinal patient history and predict diagnoses, medication orders, and the time to the next visit.The model performs multilabel prediction over disease and medication categories.
- Results: Above 64% recall@10 and 79% recall@30 were achieved for diagnosis prediction, indicating potential for differential diagnosis assistance.These results were reported for the trained RNN.
- Results: Skip-gram initialization improved RNN performance in both accuracy and speed.The initialization scheme uses learned medical-code embeddings.
- Results: Cross-institution transfer learning retained substantial predictive accuracy when models were adapted between medical institutions.This suggests a possible approach for health systems with smaller patient datasets.
2. Related Work
Prior work models temporal clinical data with discretized or continuous-time approaches, disease-specific or general-purpose progression models, and emerging neural methods. The paper motivates Doctor AI by emphasizing scalability, domain-knowledge requirements, and the sequential structure of EHR data.
- Temporal sequence models: Multilabel event sequences can be modeled with discretized time-series methods or continuous-time point-process approaches.Continuous-time approaches include Markov chains, Bayesian networks, Cox processes, and Hawkes processes.
- Temporal sequence models: Continuous-time methods trade computational efficiency against the ability to represent complex nonlinear temporal dynamics.The passage contrasts intensity-based methods with continuous-time Markov chains and notes limitations of Hawkes processes.
- Disease progression modeling: Disease progression research includes specific-purpose models focused on individual diseases and general-purpose models covering broader disease ranges.General-purpose approaches include Hawkes-process, discretized-time, and Markov-jump-process models.
- Limitations of prior work: Previous clinical progression datasets were typically much smaller, creating scalability challenges for broader modeling approaches.The largest cited dataset contained 13,180 patients and 8,722 codes, smaller than the dataset used in this work.
- Deep learning for EHR: Early deep-learning studies applied neural networks to EHR, but many did not fully address its sequential nature.RNN-based sequence prediction was identified as especially related work.
3. Cohort
The cohort comprises de-identified, timestamped primary-care EHR records from Sutter Health, covering diagnoses, medications, procedures, and problem-list data. After excluding patients with fewer than two visits, the dataset contains 263,706 patients averaging 54.61 visits each.
- Population and source of data: The source population was primary-care patients from Sutter Health Palo Alto Medical Foundation, whose EHR used Epic Systems for more than a decade.The data came from a density-sampled case-control study for heart failure.
- Population and source of data: The dataset contains de-identified encounter orders, medication orders, problem-list records, and procedure orders.Inputs include ICD-9, medication, and procedure codes.
- Data processing: 263,706 patients remained after excluding those with fewer than two visits, averaging 54.61 visits per person.All codes were timestamped at patients’ visit times, with multiple codes in one visit sharing a timestamp.
4. Methods
Doctor AI models longitudinal EHR events with a GRU-based recurrent architecture to jointly predict codes at the next visit and the time until that visit. High-dimensional multilabel inputs are embedded, processed into patient-status representations, and mapped to code and duration outputs.
- Each patient event is represented as a timestamped multi-hot vector of medical codes, with the number of events varying across patients.
- Doctor AI uses GRU recurrent units to learn a real-valued patient-status representation at each timestamp.The implementation can stack multiple recurrent layers.
- The model concatenates each visit’s multilabel code vector with the duration since the previous event before processing it.
- A Softmax output predicts diagnosis and medication codes for the next visit, while a ReLU output predicts the time until that visit.
- The joint objective combines cross-entropy loss for code prediction with squared loss for visit-duration prediction.
- Almost 40,000-dimensional inputs are projected into a lower-dimensional embedding space before entering the GRU.
5. Results
Doctor AI uses RNNs to forecast diagnoses, medications, and visit timing from longitudinal EHR data, outperforming baseline algorithms across prediction settings. Results also support Skip-gram initialization, knowledge transfer across institutions, and improved performance with longer patient histories, while visit-time prediction remains difficult.
- Prediction performance: Doctor AI outperformed baseline algorithms across diagnosis-only, medication-only, and joint prediction of diagnoses, medications, and next-visit duration.Recall was lower for the joint task because its hypothesis space was larger.
- Prediction performance: RNNs learned patient representations by accumulating relevant information from medical histories and current codes, outperforming hand-picked frequency features.The learned representations provide a proposed explanation for the superior RNN performance.
- Model variants: Skip-gram-initialized RNNs consistently outperformed directly initialized RNNs except for medication Recall@30, where differences were insignificant.Multiple layers improved representation learning overall, although one layer might suffice for medication dynamics.
- Visit-time prediction: Although Doctor AI significantly improved next-visit duration prediction over baselines, accurate time-interval prediction remained challenging.The authors attribute the difficulty to personal factors rarely included in EHR data, including financial status, residence, transportation, and lifestyle.
- History length: Longer patient histories were evaluated for their relationship with diagnosis-prediction performance using 5,800 patients with more than 100 visits.The experiment measured mean and standard error of recall at visits occurring at different points in patients’ histories.
6. Conclusion
Doctor AI learns patient representations from longitudinal EHR records to predict future events, achieving strong diagnostic performance and cross-hospital adaptability. The authors note that incorrect predictions can harm patients and that exceeding average physician performance remains future work.
- 79.58% recall@30 significantly outperformed many baselines on large real-world EHR datasets.
- Doctor AI learned from one hospital could be adapted to another hospital.
- Medical experts found that Doctor AI provided clinically meaningful diagnostic results and mimicked physicians’ predictive power.
- Incorrect predictions can degrade patient health, and future work aims to exceed average physician performance.
Appendix A. Description of Gated Recurrent Units
The GRU updates its hidden state through reset and update gates rather than allowing the previous state and current input to directly determine it. These gates regulate past-information retention and input influence.
- The input, update gate, reset gate, intermediate memory unit, and hidden layer are defined at each timestep.
- The reset gate controls whether the intermediate memory unit disregards past hidden-state values.
- The update gate controls how much information from the previous hidden state propagates to the current hidden state.
- This gating is useful because information relevant to predicting future diagnoses, medications, or visit timing is difficult to identify.
Appendix B. Learning the Skip-gram vectors from the EHR
The study learns shared low-dimensional representations of diagnosis, medication, and procedure codes from their temporal contexts in EHR data using Skip-gram.
- Skip-gram learns real-valued multidimensional vectors that capture latent representations of medical codes.
- Diagnosis, medication, and procedure codes are arranged temporally and embedded in a shared lower-dimensional space.
- A context window of 5 codes to each side is used during Skip-gram training.
Appendix C. Details of the training procedure of multilayer perceptron
The multilayer perceptron uses a 2,000-unit hidden layer, L2 regularization, and task-specific activation functions for categorical and time-interval prediction.
- The multilayer perceptron has a hidden layer with width 2,000 and applies L2 regularization to all weight matrices.
- The first and output layers use tanh and softmax activations, respectively.
- Rectified linear units are used for predicting time intervals.
Appendix D. Case study
Doctor AI’s case studies compare predicted diagnoses with true future diagnoses and illustrate how its predictions can resemble clinical reasoning. The examples also examine behavior for frequent and infrequent disease codes.
- Doctor AI’s predicted diagnoses contained most, if not all, true diseases across five patient visits.In one case, the top three predicted diseases matched the true diseases.
- The model produced clinically plausible predictions even when some guessed codes were incorrect.Examples included cataracts and refractive disorders for visual disturbances, and allergic rhinitis based on prior history.
- Table 4 compares Doctor AI’s diagnoses for frequent and infrequent disease codes after 200 time steps.