Source-linked AI summary
Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data
Ethan Steinberg, Ken Jung, Jason A. Fries, Conor K. Corbin, Stephen R. Pfohl, Nigam H. Shah
TL;DR
Clinical prediction from EHR data is hindered by complex records and limited labeled data. This work develops clinical language model based representations and finds improved performance across five prediction tasks, especially with small training sets.
Problem
EHR records are variable-length, high-dimensional, sparse, and temporally complex, making fixed-length representations for clinical prediction time-consuming and task-dependent.
Method
The authors develop a clinical language model and derive clinical language model based representations, evaluating them against count-based, other learned, and end-to-end neural representations.
Results
Across five prediction tasks, language model based representations outperformed alternative representations and end-to-end neural networks, with a 19% average AUROC improvement at the smallest sample sizes.
Takeaways & Limitations
Language model based representations are a useful technique for developing clinical prediction models from EHR data, particularly when labeled training samples are scarce.
Takeaways & Limitations
Findings are limited to five clinical outcomes, and the study does not assess generalization of representations learned at one institution to other sites.
Abstract
from arXiv · showhide
Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. This process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient representation schemes inspired from techniques in natural language processing can increase the accuracy of clinical prediction models by transferring information learned from the entire patient population to the task of training a specific model, where only a subset of the population is relevant. Such patient representation schemes enable a 3.5% mean improvement in AUROC on five prediction tasks compared to standard baselines, with the average improvement rising to 19% when only a small number of patient records are available for training the clinical prediction model.
1 Introduction
EHR data is structurally complex and often limited for specific clinical outcomes, making manual representations and end-to-end models insufficiently reliable. The paper proposes reusable patient representations learned from large EHR populations using clinical language modeling, then evaluates them across prediction tasks and data regimes.
- Motivation: EHRs are variable-length, high-dimensional, sparse, and temporally and hierarchically structured, complicating fixed-length patient representation.Patient records contain irregular visits with diagnosis, procedure, medication, laboratory, text, and image data.
- Motivation: Manual feature transformations are time-consuming and task-dependent, leaving much of EHR temporal and hierarchical structure underutilized.
- Representation learning: Transfer learning uses information learned on a large task or population to improve performance on a different task with fewer labeled records.Representation learning performs this transfer by constructing fixed-length representations for downstream tasks.
- Proposed approach: The paper proposes clinical language model based representations (CLMBR), derived from an improved generative sequence model for EHR data.The study compares CLMBR with published representation-learning methods, count representations, and end-to-end deep neural networks across five prediction tasks.
- Evaluation: The evaluation examines whether representation-learning gains vary with labeled training-set size and supervised learning algorithm, while comparing CLMBR with prior clinical language modeling.
2 Materials and Methods
The study compares four EHR representation categories across five clinical outcomes, using logistic regression, gradient boosted trees, and an end-to-end GRU across varying training-set sizes. It evaluates count-based, Word2Vec, LSI, and CLMBR representations using de-identified longitudinal EHR data and time-based splits.
- Four representation categories—Counts, Word2Vec, LSI, and CLMBR—were evaluated with logistic regression and gradient boosted trees for five clinical outcomes.An end-to-end GRU using raw EHR sequences served as an additional baseline.
- 2.1 Data: The dataset contained 3.4 million de-identified patient records from Stanford Hospital and Lucile Packard Children’s Hospital spanning 1990–2018.Records were represented as time-ordered sequences of daily diagnosis, procedure, medication, and laboratory-order codes.
- 2.1 Data: The study excluded quantitative laboratory results, vital signs, clinical notes, images, and explicit linkages between codes, retaining 21,664 filtered codes.
- 2.2 Experimental Setup: Models were evaluated on five outcomes using discrimination accuracy across multiple training-set sizes.
- 2.2 Experimental Setup: Time-based splitting used data through December 2015 for training, January–July 2016 for tuning, and August 2016–August 2017 for held-out testing.The design addressed potential temporal non-stationarity in EHR data.
- 2.3 Representations: Count-based baselines varied ontology expansion and time binning, while Word2Vec and LSI representations were evaluated with and without ontology expansion.LSI used patient records as documents and codes as words, with representation sizes of 400 and 800.
- 2.3.4 Clinical Language Model Based Representations - CLMBR: CLMBR modeled next-day multi-label code prediction from prior patient-record days using a GRU and extracted lower-dimensional representations through a linear layer.The approach used binary relevance for multi-label prediction and hierarchical decomposition based on UMLS ontologies for code probabilities.
- 2.3.4 Clinical Language Model Based Representations - CLMBR: The language-model factorization assumes codes are conditionally independent given the preceding record history, an assumption likely violated by correlated codes within encounters.The authors nevertheless report that the approach worked well in practice.
3 Results
The experiments evaluated representation quality across training-set sizes, prediction-model types, and language-modeling objectives. CLMBR performed best overall, with larger benefits in smaller-data settings and logistic regression particularly effective with CLMBR representations.
- Effect of Training Set Size: CLMBR representations performed best as training-set size decreased, while word2vec and LSI provided smaller benefits over count-based representations.AUROC decreased for all representation types as datasets shrank; word2vec outperformed LSI among the other learned representations.
- Effect of Training Set Size: CLMBR increased AUROC across all outcomes and training-set sizes, with larger gains at smaller sample sizes.The benefit diminished as training-set size increased.
- Prediction Model Type: With count-based representations, gradient boosted trees outperformed L2-regularized logistic regression by 4% to 20.7% across outcomes.The reported gaps ranged from 4% for 30-day readmission to 20.7% for inpatient mortality.
- Prediction Model Type: With CLMBR representations, logistic regression outperformed gradient boosted trees, which offered no improvement at large sample sizes and often hurt performance at smaller sizes.This reverses the model-type pattern observed with count-based representations.
- Language Modeling Objective: The study also evaluated representations derived from CLMBR and DoctorAI language-modeling objectives.Table 4 reports AUROCs and differences between the two language-model-based representation approaches.
4 Discussion
Language-model-based representations outperformed alternative representations across training-set sizes, with the largest gains in small samples. The discussion also identifies computational, generalization, and future-data limitations.
- Findings: Language-model-based representations, including DoctorAI and CLMBR, outperformed alternative representations across training-set sizes and clinical prediction outcomes.They also outperformed end-to-end neural networks in the large-sample regime.
- Findings: The average AUROC improvement from language-model-based representations was 19% at small sample sizes and remained present with over 200,000 samples.With enough data, count-based representations were only 3.5% worse than CLMBR.
- Limitations: Language-model-based representations require significant upfront computation to train and tune, although the cost can be amortized across outcomes at an institution.The paper notes that later language-model work has reduced training costs.
- Limitations: The findings are limited to five clinical outcomes, and cross-institutional generalization of CLMBR representations was not explored.The authors also note that increasing EHR data availability could reduce current gains and potentially restore an end-to-end neural-network advantage.
5 Conclusion
The paper developed and evaluated language-model-based EHR representations across clinical outcomes and dataset sizes. These representations performed especially well with small samples, and logistic regression worked particularly well with them.
- Conclusion: Language-model-based patient representations outperformed three other representation schemes and end-to-end neural networks across varied clinical outcomes and dataset sizes.The conclusion describes this as an evaluation across a variety of clinical prediction settings.
- Conclusion: 19% was the average AUROC improvement at the smallest sample sizes.The conclusion identifies small-sample performance as the strongest improvement.
- Conclusion: Logistic regression worked particularly well with language-model-based representations, potentially enabling faster and cheaper clinical prediction-model development.The conclusion presents this as a practical implication of the representation approach.
- Conclusion: The results suggest that language-model-based representations are useful for developing better clinical prediction models from EHR data.This is the paper’s stated concluding implication.
Author Information
The paper lists Stanford University, Stanford, CA, USA, as the authors’ affiliation and describes contributions spanning experiments, code, figures, discussion, and manuscript revision.
- Affiliation: The listed affiliation is Stanford University in Stanford, California, USA.
- Contributions: Ethan Steinberg conducted the primary experiments and created the initial manuscript drafts.
- Contributions: The authors contributed code, discussion, figure design, and manuscript revision across the project.
A Language Model Hyperparameter Grid
This section presents the language model hyperparameters used in the study.
- Table 5 lists the language model hyperparameters.
B Best Language Model Hyperparameters
This section presents the best hyperparameter configuration for the language model.
- Table 6 lists the best language model hyperparameters.
C End-to-end GRU Model Hyperparameter Grid
This section presents the hyperparameters for the end-to-end GRU model.
- Table 7 lists the end-to-end GRU model hyperparameters.
D Best End-to-end GRU Model Hyperparameters
This section presents best hyperparameter tables for GRU models across five prediction tasks.
- Table 6 lists inpatient mortality GRU best hyperparameters.
- Table 9 lists long admission GRU best hyperparameters.
- Table 10 lists ICU transfer GRU best hyperparameters.
- Table 11 lists 30-day readmission GRU best hyperparameters.
- Table 12 lists abnormal HbA1c GRU best hyperparameters.
E Best Prediction Model/Representation Hyperparameters On All Data
This section lists the best prediction-model or representation hyperparameters for five clinical prediction tasks. The tasks span inpatient mortality, long admission, ICU transfer, 30-day readmission, and abnormal HbA1c.
- The inpatient mortality task has a dedicated table of best hyperparameters.
- The long admission task has a dedicated table of best hyperparameters.
- The ICU transfer task has a dedicated table of best hyperparameters.
- The 30-day readmission task has a dedicated table of best hyperparameters.
- The abnormal HbA1c task has a dedicated table of best hyperparameters.