Source-linked AI summary
BEHRT: Transformer for Electronic Health Records
Yikuan Li, Shishir Rao, Jose Roberto Ayala Solares, Abdelaali Hassaine, Dexter Canoy, Yajie Zhu, Kazem Rahimi, Gholamreza Salimi-Khorshidi
TL;DR
BEHRT addresses the challenge of improving future-diagnosis prediction in EHR while accounting for EHR-specific challenges. It adapts Transformer and BERT-inspired sequence modeling to EHR, and outperformed leading deep EHR models by approximately 8% absolute improvement across tasks.
Problem
The paper addresses the need for improved prediction of future diagnoses while accounting for EHR-specific challenges.
Method
BEHRT adapts Transformer and BERT-inspired sequence modeling to EHR by representing visits as sentences and patient histories as documents for self-attention, positional encoding, and masked language modeling.
Results
∼8% absolute improvement over the best deep EHR models in multi-label diagnosis prediction across a range of diseases.
Takeaways & Limitations
BEHRT supports flexible EHR representations and attention-based visualization of contextual disease trajectories for multimorbid patients.
Takeaways & Limitations
Caliber consolidates specific codes into 301 disease codes and fails to map many other codes, limiting the comprehensiveness of the CPRD dataset representation.
Abstract
from arXiv · showhide
Today, despite decades of developments in medicine and the growing interest in precision healthcare, vast majority of diagnoses happen once patients begin to show noticeable signs of illness. Early indication and detection of diseases, however, can provide patients and carers with the chance of early intervention, better disease management, and efficient allocation of healthcare resources. The latest developments in machine learning (more specifically, deep learning) provides a great opportunity to address this unmet need. In this study, we introduce BEHRT: A deep neural sequence transduction model for EHR (electronic health records), capable of multitask prediction and disease trajectory mapping. When trained and evaluated on the data from nearly 1.6 million individuals, BEHRT shows a striking absolute improvement of 8.0-10.8%, in terms of Average Precision Score, compared to the existing state-of-the-art deep EHR models (in terms of average precision, when predicting for the onset of 301 conditions). In addition to its superior prediction power, BEHRT provides a personalised view of disease trajectories through its attention mechanism; its flexible architecture enables it to incorporate multiple heterogeneous concepts (e.g., diagnosis, medication, measurements, and more) to improve the accuracy of its predictions; and its (pre-)training results in disease and patient representations that can help us get a step closer to interpretable predictions.
1 Introduction
Precision healthcare seeks personalised prediction, prevention, and intervention, while EHR deep learning has progressed toward models that learn from complex longitudinal records. BEHRT builds on Transformer architecture and prior EHR sequence models to improve future-diagnosis prediction.
- Precision healthcare aims to improve care through precise and personalised prediction, prevention, and intervention.
- Large EHR systems link diagnoses, interventions, laboratory tests, and other concepts across visits and years.
- Traditional EHR models depend on expert-designed features, whereas deep learning can learn representations from raw or minimally processed data.
- Early deep EHR models often overlooked irregular visit intervals and event order, motivating sequence models that represent medical histories over time.
- BEHRT builds on Transformer architecture and prior EHR deep-learning research to improve the accuracy of future-diagnosis prediction.
2 Materials and Methods
The study prepares linked CPRD records as temporally ordered diagnosis sequences and introduces BEHRT, a Transformer-based model for multitask future-diagnosis prediction. Its architecture combines contextual representations with age, position, and visit-segment information, while evaluation uses multiple prediction horizons and patient-averaged APS and AUROC.
- Data source and preprocessing: CPRD data were filtered for HES linkage eligibility, CPRD quality standards, and at least five visits, leaving 1.6 million patients from an initial 8 million.The retained cohort was used to train and evaluate BEHRT.
- Data source and preprocessing: Diagnoses from ICD-10 and Read Codes were mapped to an expert-checked Caliber dictionary, producing 301 diagnosis codes.Procedure codes were excluded before mapping.
- Data source and preprocessing: Each patient’s record was represented as a temporally ordered sequence of diagnosis-containing visits, with CLS marking history start and SEP separating visits.Although visits may contain diagnoses, medications, and measurements, this study modeled diagnoses only.
- BEHRT architecture: BEHRT adapts BERT’s Transformer design to EHR by treating diagnoses as words, visits as sentences, and medical histories as documents for self-attention and masked language modeling.The architecture pre-trains bidirectional contextual representations for downstream disease prediction and phenomapping.
- BEHRT architecture: BEHRT combines disease, position, age, and segment embeddings to represent diagnoses, timing, and care-delivery patterns, while allowing additional heterogeneous concepts through added embeddings.The model uses feedforward networks to examine sequential order rather than traditional RNN or CNN architectures.
3 Results
BEHRT’s results show that its pretrained representations capture clinically meaningful disease relationships and long-range dependencies, while supporting downstream multi-disease prediction. The model’s evaluation spans embedding quality, attention behavior, and prediction performance across tasks and diseases.
- Pre-training: 0.6597 precision was achieved on the MLM pre-training task with a six-layer, 12-head architecture, hidden size 288, and intermediate size 512.Bayesian optimisation selected the reported architecture and hyperparameters.
- Disease embeddings: Diseases mapped into 288-dimensional vectors formed groups reflecting known co-occurrence and clinical relationships in the t-SNE projection.The embedding evaluation combined visual comparison with medical knowledge and downstream prediction.
- Disease embeddings: Gender-specific diseases formed naturally separated groups, suggesting the embeddings inferred contextual factors such as gender without explicit gender input.Examples include female-specific diseases being distant from male-specific diseases.
- Disease embeddings: Disease clusters generally aligned with Caliber chapters while also revealing clinically correlated relationships that crossed chapter boundaries.The map represents disease correspondence learned from 1.6 million individuals’ EHRs.
- Attention analysis: Self-attention identified long-range relationships, including a strong connection between Rheumatoid Arthritis and later Enthesopathies and synovial disorders.This illustrates attention extending beyond temporally adjacent events.
- Prediction performance: BEHRT was evaluated as a pretrained feature extractor for multi-disease prediction, with additional disease-level analysis reporting relatively high precision and recall for several conditions.The disease-level analysis included Epilepsy, Primary Malignancy Prostate, Polymyalgia Rheumatica, Hypo or hyperthyroidism, and Depression.
4 Conclusions and Future Works
BEHRT combines pretrained EHR representations, flexible multimodal embeddings, and attention-based disease relationships to support prediction and disease-trajectory analysis. It outperformed leading deep EHR models across near-future diagnosis tasks, while remaining limited by code-mapping coverage and noise.
- BEHRT can be pretrained on a large dataset and fine-tuned with small additional training for downstream tasks.
- BEHRT’s attention mechanism can reveal long-range disease associations beyond temporal adjacency.
- BEHRT outperformed leading deep EHR models by ∼8% absolute across multi-label near-future diagnosis tasks.
- Its four embeddings capture diseases, age, segment, and position, while future extensions could add medication, tests, and interventions with minimal architectural changes.
- Disease embeddings capture trajectory-based disease relationships, while attention correspondences illustrate contextual influences among diseases in multimorbid patients.
- The study plans improved models, additional medical features, more stable code mappings, and disease-specific prediction analyses.
A Hyperparameter Tuning
The study used Bayesian optimization to tune the MLM pretraining and searched hyperparameters for Deepr and RETAIN to support model comparisons.
- Bayesian optimization was used to select MLM pretraining hyperparameters.
- The tuning process also searched hyperparameters for Deepr and RETAIN to ensure comparable model evaluation.
B Disease-wise Model Performance
Disease-wise BEHRT performance was evaluated with APS and AUROC for the next-6-month prediction task, reporting results across disease codes and chapters.
- The analysis includes disease codes with an occurrence ratio of at least 0.01 and describes each code’s chapter alongside APS and AUROC.
- Reported disease examples include glaucoma with APS 0.250573 and AUROC 0.885547, and hyperplasia of prostate with APS 0.249208 and AUROC 0.894444.
C Comparison of APS/AUROC across Models
Across the shared next-6-month task and dataset, BEHRT’s disease-level predictions generally occupy the graph’s upper-right region relative to Deepr and RETAIN, indicating higher APS and AUROC for most diseases.
- BEHRT’s predictions remain mostly in the graph’s upper-right quadrant, denoting higher APS and AUROC than the other two models.
- BEHRT, Deepr, and RETAIN were compared on the same dataset and next-6-month task for each disease.