Source-linked AI summary
Scalable and accurate deep learning for electronic health records
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj, Peter J. Liu, Xiaobing Liu, Mimi Sun, Patrik Sundberg, Hector Yee, Kun Zhang, Gavin E. Duggan, Gerardo Flores, Michaela Hardt, Jamie Irvine, Quoc Le, Kurt Litsch, Jake Marcus, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H. Shah, Atul J. Butte, Michael Howell, Claire Cui, Greg Corrado, Jeff Dean
TL;DR
Traditional EHR prediction requires labor-intensive curation that discards much available information. The paper represents entire raw records in FHIR and applies deep learning across two hospitals, achieving better performance than traditional models across multiple clinical tasks while providing chart-based attribution.
Problem
Traditional EHR predictive modeling requires task-specific curated variables and labor-intensive preprocessing, limiting use of the record’s available information.
Method
The study represents entire raw EHR records, including free-text notes, as sequential FHIR-based inputs for deep-learning prediction without manual feature harmonization.
Results
Deep learning outperformed traditional models across mortality, readmission, length-of-stay, and diagnosis prediction tasks, including mortality AUROC 0.92–0.94 versus 0.91 in prior EHR models.
Takeaways & Limitations
The approach supports scalable predictions across multiple hospitals and clinical outcomes, with attribution methods that highlight evidence in individual patient charts.
Takeaways & Limitations
The study is retrospective, and prospective trials are needed to determine whether accurate predictions improve care.
Abstract
from arXiv · showhide
Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two U.S. academic medical centers with 216,221 adult patients hospitalized for at least 24 hours. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting in-hospital mortality (AUROC across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed state-of-the-art traditional predictive models in all cases. We also present a case-study of a neural-network attribution system, which illustrates how clinicians can gain some transparency into the predictions. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios, complete with explanations that directly highlight evidence in the patient's chart.
INTRODUCTION
Routine EHR data are vast and rich, yet traditional predictive modeling uses limited variables and requires labor-intensive, task-specific preparation. The paper proposes deep learning over the entire EHR, including free-text notes, to learn predictors and their interactions directly.
- 80% of analytic-model effort is widely held to involve preprocessing, merging, customization, and cleaning rather than analysis.
- Thousands of potential EHR predictors, especially free-text notes, make comprehensive modeling difficult.
- Using only limited variables can produce imprecise predictions, including false alarms that contribute to alert fatigue and false negatives that miss clinically important events.
- Deep learning can handle large, messy, multimodal data and learn representations of important factors and interactions without investigators specifying them in advance.
- The proposed approach uses the entire EHR, including notes, to learn harmonization and medical-event prediction jointly rather than feeding curated variables into a statistical model.
Related work
Prior EHR prediction studies generally use few variables, traditional models, single-center data, and labor-intensive harmonization. This paper instead presents a generic raw-EHR-to-FHIR pipeline and evaluates deep learning across hospitals and prediction problems.
- A systematic review found a median of 27 variables, traditional generalized linear models, and single-center data in EHR predictive models.
- Multi-site EHR modeling commonly requires task-specific variable selection plus labor-intensive extraction and normalization because standards and semantic interoperability are limited.
- OMOP-style standardization supports consistent cross-site model development but accommodates only part of the original data.
- FHIR provides a consistent, hierarchical, extensible container format that simplifies data interchange, but does not ensure semantic consistency.
- The contribution combines a raw-EHR-to-FHIR pipeline without manual feature harmonization with deep-learning evaluation across two general hospitals and diverse prediction settings.
RESULTS
The study evaluated 216,221 hospitalizations using large sequential EHR representations from two hospitals. Deep learning showed strong discrimination for mortality, readmission, and length-of-stay prediction, outperforming traditional baselines.
- 216,221 hospitalizations involving 114,003 patients were included, with in-hospital mortality in 2.3% and unplanned 30-day readmission in 12.9%.
- 46,864,534,945 EHR tokens were available for discharge-time predictions across both datasets.
- Mortality: AUROC for inpatient mortality at 24 h was 0.95 for Hospital A and 0.93 for Hospital B, exceeding the aEWS baseline values of 0.85 and 0.86.
- Mortality: The mortality work-up-to-detection ratio at 24 h was 7.4 vs 14.3 in Hospital A and 8.0 vs 15.4 in Hospital B for the deep learning model versus aEWS.
- Readmissions: AUROC for unexpected 30-day readmission at discharge was 0.77 for Hospital A and 0.76 for Hospital B, above modified-HOSPITAL values of 0.70 and 0.68.
- Long length of stay: AUROC for long length of stay at 24 h was 0.86 for Hospital A and 0.85 for Hospital B, above modified-Liu values of 0.76 and 0.74.
Inferring discharge diagnoses
The model predicted patients’ full discharge-diagnosis codes at admission, after 24 h, and before discharge coding. Frequency-weighted accuracy increased during hospitalization and reached 0.90 at discharge for both hospitals.
- The model classified all discharge diagnosis codes at admission, after 24 h, and immediately before discharge diagnoses were coded.
- Frequency-weighted AUROC was 0.87 for Hospital A and 0.86 for Hospital B at admission.
- Frequency-weighted AUROC increased to 0.88–0.89 at 24 h and 0.90 for both hospitals at discharge.
- The case study showed attribution to notes, antibiotics, nursing documentation, and concepts such as “pleurx,” rather than only conventional laboratory and vital-sign predictors.
DISCUSSION
The study shows that deep learning applied to entire raw EHRs can produce accurate, scalable predictions across diverse clinical tasks and healthcare settings without hand-selecting predictors. Its attribution case study offers chart-based evidence for predictions, while important limitations remain around cross-site transfer, computational demands, and prospective clinical utility.
- DISCUSSION: Deep learning using entire EHRs outperformed traditional clinical predictive models across mortality, readmission, length-of-stay, and diagnosis tasks.The approach used one data structure across multiple outcomes and prediction settings.
- DISCUSSION: More than 46 billion EHR data points were considered for discharge predictions, enabling more accurate predictions earlier in hospitalization than traditional models.The models incorporated the full patient chart, including free-text notes.
- DISCUSSION: 0.92–0.94 vs 0.91 AUROC was reported for mortality, 0.75–0.76 vs 0.69 for unexpected readmission, and 0.85–0.86 vs 0.77 for increased length of stay.The comparisons were made against published EHR models, while study-specific baselines were also implemented.
- DISCUSSION: Models trained without data harmonization may have limited transfer across sites, and development remains computationally intensive and dependent on specialized expertise.Running predictions on a new patient takes only a few milliseconds, but the current study did not isolate the incremental contribution of each data type.
- DISCUSSION: Attribution visualizations highlighted medications, nursing documentation, and clinical-note evidence that contributed to an individual mortality prediction.The authors present this as a proof-of-concept and note that further research is needed on applicability, cognitive impact, and clinical utility.
Datasets
The study used de-identified EHR data from two academic medical centers and represented entire patient records as temporally ordered FHIR-based token sequences.
- Datasets: EHR data came from UCSF and the University of Chicago Medicine, covering demographics, orders, diagnoses, procedures, medications, laboratory values, vital signs, and flowsheets.The UCM dataset additionally contained de-identified free-text medical notes.
- Datasets: All datasets were de-identified and stored in encrypted, access-controlled, audited sandboxes.Institutional review boards approved the study with waiver of informed consent or exemption.
- Datasets: A single temporal data structure represented the entire EHR for all predictions instead of requiring custom hand-created datasets for each outcome.Events were organized by patient and time using FHIR resources.
- Datasets: FHIR resources supplied event attributes that were split into discrete tokens, while clinical notes were tokenized word by word.The resulting sequence contained time-ordered tokens from the beginning of a patient’s record to prediction.
Outcomes
The study evaluated one shared EHR representation across outcomes spanning mortality, readmission, hospital utilization, and diagnosis prediction, using information available before each prediction.
- Outcomes: The outcomes covered death, readmissions, length of stay, and patients’ diagnoses across divergent clinical and operational domains.These represented clinical outcome, care quality, resource utilization, and understanding of patient problems, respectively.
- Outcomes: Inpatient mortality was defined as a discharge disposition of “expired.”The model predicted impending inpatient death.
- Outcomes: 30-day unplanned readmission was defined as admission within 30 days after discharge from an eligible index hospitalization.Only readmissions to the same institution were included, and billing diagnoses and procedures from the index hospitalization were excluded from prediction.
- Outcomes: Long length of stay was defined as hospitalization lasting at least 7 days.Seven days was approximately the 75th percentile of hospital stays for most services across the datasets.
- Outcomes: Diagnosis prediction targeted the entire set of primary and secondary ICD-9 billing diagnoses from 14,025 possible codes.This outcome was evaluated alongside mortality, readmission, and length-of-stay prediction.
Prediction timing
Predictions were generated at defined points during hospitalization using the same algorithm and separate hospital datasets, with architectures suited to records of varying length and density.
- Prediction timing: Mortality predictions were made every 12 hours from 24 hours before admission through 24 hours after admission.The primary mortality prediction used the 24-hours-after-admission time point.
- Prediction timing: Readmission and diagnosis predictions were made at admission, 24 hours after admission, and discharge.These timings were part of the retrospective study design.
- Prediction timing: The cohort included adults hospitalized for at least 24 hours so prediction time points used identical cohorts.Admissions for patients aged 18 years or older were included.
- Prediction timing: Patients typically excluded from readmission studies, including those discharged against medical advice, were retained to simulate real-time prediction.Such exclusions would not be known when making earlier hospitalization predictions.
- Prediction timing: Hospitalizations without ICD-9 diagnoses were excluded only from diagnosis prediction, while remaining predictions included them.These encounters generally followed the transition to ICD-10 after October 2015.
- Prediction timing: The same modeling algorithm was applied to both hospitals, which were treated as separate datasets and reported separately.Three architectures were developed: LSTM, attention-based TANN, and boosted time-based decision stumps.
Comparison to previously published algorithms
The study compared deep learning models with published baseline algorithms and evaluated discrimination, calibration, alert burden, and multi-diagnosis accuracy using held-out test data.
- Comparison to previously published algorithms: Published baseline algorithms included aEWS for mortality, mHOSPITAL for readmission, and mLiu for long length of stay.These logistic models used variables drawn from established scores and prior predictive modeling work.
- Comparison to previously published algorithms: Attribution mechanisms highlighted patient data elements that influenced each neural-network prediction.This addressed the common criticism that neural networks provide little insight into prediction factors.
- Comparison to previously published algorithms: Patients were split into development (80%), validation (10%), and test (10%) sets, with test accuracy reported after final evaluation.The test set remained hidden until final evaluation to prevent overfitting.
- Comparison to previously published algorithms: Model discrimination was assessed with AUROC, calibration with predicted-versus-empirical probability curves, and alert burden with the work-up to detection ratio at 80% sensitivity.The work-up to detection ratio is also called the number needed to evaluate.
- Comparison to previously published algorithms: Full diagnosis prediction used macro-weighted AUROC and micro-weighted F1 across diagnosis classes.The diagnosis task involved between 1 and 228 codes per hospitalization.
- Comparison to previously published algorithms: The micro-weighted F1 score used one validation-selected threshold for all diagnosis classes, without confidence intervals because of computational complexity.The F1 score is the harmonic mean of positive-predictive-value and sensitivity.
Code availability
The FHIR format is publicly available, while the transformation pipeline and models rely on Google’s non-shareable internal computation platforms. Supplementary information accompanies the paper.
- The FHIR format used in this work is available through Google’s GitHub repository.
- The transformation of FHIR data into TensorFlow training examples and the models depend on Google’s internal distributed computation platforms.
- Those internal platforms cannot reasonably be shared, so the authors emphasize detailed construction and design descriptions in the Methods and Supplementary Materials.
- Supplementary information accompanies the paper on the npj Digital Medicine website.
- The authors declare no competing interests.
- Springer Nature remains neutral regarding jurisdictional claims in published maps and institutional affiliations.