Source-linked AI summary
AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease
Chengsheng Mao, Jie Xu, Luke Rasmussen, Yikuan Li, Prakash Adekkanattu, Jennifer Pacheco, Borna Bonakdarpour, Robert Vassar, Guoqian Jiang, Fei Wang, Jyotishman Pathak, Yuan Luo
TL;DR
Predicting which patients with MCI will progress to AD is a research area with limited EHR-based evidence, while clinical notes contain rich information relevant to risk modeling. The paper develops AD-BERT, pretrained on AD-related clinical notes, and reports that it achieved the highest performance among commonly used NLP models.
Problem
Research analyzing EHRs for MCI-to-AD progression prediction remains limited, although clinical notes contain information such as family history, laboratory measurements, treatments, and self-report scores.
Source: Introduction · p. 3Method
The study develops AD-BERT, pretrained on AD-related clinical notes, and combines patient note-section embeddings by MaxPooling for MCI-to-AD risk prediction.
Source: Framework · p. 6, Discussion · p. 10Results
AD-BERT achieved the highest performance among commonly used NLP models for MCI-to-AD prediction.
Source: Discussion · p. 10Takeaways & Limitations
The authors conclude that BERT models provide a solution for analyzing clinical notes to predict MCI-to-AD progression.
Source: Conclusion · p. 12The study considers only MCI and AD, a small fraction of the complex, long-term AD progression process.
Source: Discussion · p. 11
Abstract
from arXiv · showhide
Objective: We develop a deep learning framework based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model using unstructured clinical notes from electronic health records (EHRs) to predict the risk of disease progression from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD). Materials and Methods: We identified 3657 patients diagnosed with MCI together with their progress notes from Northwestern Medicine Enterprise Data Warehouse (NMEDW) between 2000-2020. The progress notes no later than the first MCI diagnosis were used for the prediction. We first preprocessed the notes by deidentification, cleaning and splitting, and then pretrained a BERT model for AD (AD-BERT) based on the publicly available Bio+Clinical BERT on the preprocessed notes. The embeddings of all the sections of a patient's notes processed by AD-BERT were combined by MaxPooling to compute the probability of MCI-to-AD progression. For replication, we conducted a similar set of experiments on 2563 MCI patients identified at Weill Cornell Medicine (WCM) during the same timeframe. Results: Compared with the 7 baseline models, the AD-BERT model achieved the best performance on both datasets, with Area Under receiver operating characteristic Curve (AUC) of 0.8170 and F1 score of 0.4178 on NMEDW dataset and AUC of 0.8830 and F1 score of 0.6836 on WCM dataset. Conclusion: We developed a deep learning framework using BERT models which provide an effective solution for prediction of MCI-to-AD progression using clinical note analysis.
Introduction
MCI can progress to AD, but prediction research using EHRs remains limited despite the rich information in clinical notes. The study develops an EHR-note BERT framework for this task and addresses class imbalance with stratified batches.
Introduction
- AD is described as a prevalent progressive neurological disorder and a major cause of death in the United States.The introduction reports that nearly 75% of people with AD remain undiagnosed globally, attributing this to stigma and lack of awareness.Source: Introduction · p. 3
- Risk prediction matters because MCI may or may not progress to AD dementia, and clinical notes contain information relevant to modeling that risk.Prior approaches have used biomarkers, imaging, genetics, and clinical measures; EHR notes can include family history, laboratory measurements, treatments, and self-report scores.Source: Introduction · p. 3
- AD-related clinical notes may use different terminology than general or other-disease text, motivating BERT models specific to MCI-to-AD prediction.The paper gives “memory loss” as a phrase appearing more frequently in MCI- or AD-related notes than in general text.Source: Introduction · p. 3
- EHR-based research on MCI-to-AD progression remains limited, and the paper develops a BERT framework validated on an independent dataset.The authors also identify class imbalance between patients who progress and those who do not.Source: Introduction · p. 3
- Stratified batch sampling ensures equal proportions of case and control samples in each batch to address class imbalance.The paper notes that an imbalanced dataset can produce batches containing only controls, causing training to fail.Source: Introduction · p. 4
Related Work
Prior NLP work has applied conventional machine learning, UMLS-enhanced methods, deep learning, and transformer models to clinical text. The paper identifies MCI-to-AD prediction from clinical records as an unaddressed application of transformer-based models.
Related Work
- Prior AD-related clinical-note studies predicted cognitively unimpaired-to-MCI progression or detected cognitive decline, but the passage reports no prior MCI-to-AD progression study using clinical narratives.The cited examples report an LSTM AUC of 0.75 and F1 Score of 0.46 for cognitively unimpaired-to-MCI prediction, and a recurrent neural network with AUC and AUPRC over 0.9 for cognitive-decline detection.Source: Related Work · p. 4
- Clinical-note NLP research includes word-feature and conventional machine-learning methods, UMLS-enhanced methods, deep learning, and work on interoperability.The passage also describes prior EHR research on cognitive decline and cognitively unimpaired-to-MCI progression.Source: Related Work · p. 4
- Transformer variants pretrained on biomedical and clinical corpora have been applied to phenotyping and clinical prediction, but the passage reports no transformer study of AD-related clinical records.Examples include predicting COVID-19 complications from pre-existing conditions and AKI prediction using a language model initialized with BERT weights.Source: Related Work · p. 4
Study cohort
The study assembled MCI cohorts from NMEDW and WCM using diagnosis codes, defining cases by subsequent AD diagnosis and controls as patients not yet diagnosed with AD. It used progress notes before the first MCI diagnosis for risk prediction.
Study cohort
- The cohorts comprised 396 cases and 3,261 controls at NMEDW, and 548 cases and 2,015 controls at WCM, identified between 2000 and 2020.MCI was identified using ICD-9 331.83 and ICD-10 G31.84; AD cases were identified using ICD-9 331.0 and ICD-10 G30.*.Source: Study cohort · pp. 4–5
- Progress notes before the first MCI diagnosis were collected for prediction; patients with only one encounter were excluded because their records might be incomplete.Figure 1 depicts the inclusion and exclusion criteria for both cohorts.Source: Study cohort · p. 5, Figure 1 · p. 5
Preprocessing
The study deidentified and cleaned EHR progress notes, split them into sections, and represented each patient by pooling section embeddings.
Preprocessing
- Philter was used to remove protected health information, including names, addresses, and phone numbers, from clinical notes.The notes were also cleaned by removing non-ASCII characters and replacing multiple contiguous spaces with one blank space.Source: Preprocessing · p. 5
- Each note was split at newline characters, and each section was processed independently as an input for section representation.The patient embedding was generated by MaxPooling the section representations.Source: Preprocessing · p. 5
- All section embeddings for a patient were combined by MaxPooling to form a patient-level vector representation.The preprocessing description ties this representation to the sections of the patient's notes.Source: Preprocessing · p. 5
Framework
AD-BERT is pretrained on AD-related clinical notes and then fine-tuned to predict MCI-to-AD progression. It combines a patient’s section embeddings with MaxPooling before producing a progression probability.
Framework
- AD-BERT is pretrained from BC-BERT on about 37,000 NMEDW clinical progress notes using masked language modeling and next sentence prediction.The notes average 6,250 characters.Source: Framework · p. 5
- For prediction, AD-BERT independently embeds each note section, and MaxPooling combines a patient’s section embeddings into one representation.Source: Framework · pp. 5–6
- A fully connected linear layer with sigmoid activation converts the patient representation into a probability of MCI-to-AD progression.Source: Framework · p. 6
Configuration
The study uses stratified training and validation splits, plus stratified batches and weighted loss to address class imbalance during model training.
Configuration
- Each cohort was stratified into 80% training and 20% testing, with one-fifth of the training data reserved for validation and model selection.Source: Configuration · p. 6
- To address class imbalance and small-batch sampling bias, training used a stratified batch sampler and weighted cross-entropy loss, with batch size 4 and maximum sequence length 32.WCM served as an independent data source for validation.Source: Configuration · p. 6
Results
AD-BERT achieved the best performance across prediction settings on NMEDW and all but one setting on WCM, outperforming the compared baseline models. The study also examined cohort characteristics, time-window definitions, and model attention.
Results
- The cohorts included 3,657 NMEDW and 2,563 WCM patients; average conversion time was 723 days and 742 days, respectively.Conversion time runs from first MCI diagnosis to first AD diagnosis.Source: Results · p. 6
- Prediction settings included no-restrict and 6-month, 1-year, and 2-year windows; window-specific controls had to have a last encounter after the window.Patients with a last encounter inside the window and no detected AD diagnosis were excluded; Figure 3 illustrates the sample definitions.Source: Results · pp. 6–7, Figure 3 · p. 8
- AUC was 0.8830 and F1 score was 0.6836 for no-restrict WCM prediction; AD-BERT performed best in all settings except 1-year prediction, with increases of 5.5% in F1 and 0.6% in AUC over BioBERT.Source: Results · p. 8
- AUC was 0.8170 and F1 score was 0.4178 for no-restrict NMEDW prediction; AD-BERT performed best across all settings, with increases of 2% in F1 and 0.5% in AUC over BioBERT.The comparison includes the performance table and prose results.Source: Results · p. 8, Table 4 · p. 9
- Deep learning models outperformed bag-of-words, and AD-BERT pretrained on NMEDW also performed well on WCM.Source: Results · p. 9
- In an attention visualization, AD-BERT assigned greater attention to “memory,” “MCI,” and “difficulty recalling dates” than to other terms.Source: Results · p. 9, Figure 4 · p. 10
Discussion
AD-BERT, pretrained on AD-related clinical notes, achieved the highest performance among commonly used NLP models for MCI-to-AD prediction. The discussion highlights clinical notes’ routine availability and reports constraints involving portability, missing data, and the study’s focus on two disease stages and notes alone.
Discussion
- AD-BERT achieved the highest performance among commonly used NLP models, and its NMEDW-pretrained version also performed well on WCM.The authors say clinical notes are more routinely available for larger patient samples than biomarker data.Source: Discussion · pp. 10–11
- The 2-year prediction model had the best F1 score, while the 6-month model had the worst.The authors expected longer-window prediction to be easier than shorter-window prediction.Source: Discussion · pp. 10–11
- The stratified batch sampler addresses imbalance; without it, all BERT-based models predicted every sample as control, yielding 0 recall and NA precision.Source: Discussion · p. 11
- The study trained AD-BERT on NMEDW and validated it on WCM, but this does not guarantee comparable performance in other healthcare systems.The authors cite variation in clinical documentation and workflows and call for validation across multiple systems.Source: Discussion · p. 11
- Missing ICD codes could lead to AD encounters being excluded and case samples being assigned to the control group.The study also excluded patients without progress notes at or before MCI diagnosis; these patients were about 0.2% in NMEDW.Source: Discussion · p. 11
- The study considered only clinical notes and only the MCI-to-AD stages of the broader AD progression process.The authors propose future work combining notes with biomarkers and other structured data, and studying additional progression stages.Source: Discussion · p. 11
Conclusion
The framework uses BERT to predict MCI-to-AD risk from EHR clinical notes, with independent-dataset validation and a stratified batch sampler for class imbalance.
Conclusion
- The BERT-based framework predicts MCI-to-AD risk from EHR clinical notes and was validated on an independent dataset from another institution.The study presents it as a potential solution for clinical-note analysis of MCI-to-AD progression.Source: Conclusion · p. 12
- Unstructured clinical notes contain potentially useful progression information, but extracting predictive signals from them is challenging.Source: Conclusion · p. 12
- A stratified batch sampler was designed to address class imbalance between MCI patients who progress to AD and those who do not.Source: Conclusion · p. 12