Source-linked AI summary
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Kexin Huang, Jaan Altosaar, Rajesh Ranganath
TL;DR
Clinical notes contain clinically valuable information that is difficult to use because of their unstructured, sparse, and high-dimensional nature. The paper develops ClinicalBERT, a bidirectional-transformer representation specialized to clinical notes, and evaluates it on semantic similarity and 30-day readmission prediction. ClinicalBERT correlates more accurately with physician judgments and outperforms competitive methods on readmission prediction, while its attention weights can be visualized and its parameters are released.
Problem
Clinical notes provide richer patient information than structured EHR features but are difficult to model because they are unstructured, high-dimensional, sparse, and long.
Method
ClinicalBERT applies BERT to clinical corpora, learning token representations from medical notes for downstream clinical tasks including readmission prediction.
Results
ClinicalBERT more accurately captures physician-assessed clinical word similarity and outperforms competitive deep language models on 30-day hospital readmission prediction.
Takeaways & Limitations
ClinicalBERT representations support clinical prediction and interpretable inspection of note elements relevant to readmission risk, with models and parameters released for reuse.
Abstract
from arXiv · showhide
Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.
1 Introduction
Clinical notes offer richer patient information than structured EHR features but are difficult to model because they are unstructured, high-dimensional, sparse, and long. ClinicalBERT addresses this challenge for clinical prediction, including dynamically updated 30-day readmission risk.
- Clinical notes are difficult for machine-learning models because they are unstructured, high-dimensional, sparse, and contain long-range dependencies.
- Clinical notes describe symptoms, diagnoses, radiology results, daily activities, and patient history beyond structured features.
- ClinicalBERT processes notes during admission and dynamically assigns a risk score for 30-day hospital readmission.
- ClinicalBERT improves readmission prediction over methods centered on discharge summaries and can be applied at any timepoint after admission.
- ClinicalBERT captures relationships between medical concepts, supports interpretable attention visualizations, and can be adapted to other clinical tasks.
2 Methods
ClinicalBERT learns clinical-text representations with transformer-based self-attention and clinical-note pre-training, then adapts them to 30-day readmission prediction.
- Representation learning: ClinicalBERT applies BERT to clinical corpora to learn representations for downstream clinical tasks.The model is demonstrated on hospital readmission prediction.
- Clinical text embedding: Clinical-note tokens combine token, segment, and position embeddings, with a [CLS] token inserted for classification tasks.The [CLS] representation is later used for readmission prediction.
- Transformer architecture: BERT uses transformer self-attention to capture long-range interactions between elements of an input sequence.Queries, keys, and values are derived from input-token embeddings using learned weights.
- Readmission prediction: For fine-tuning, ClinicalBERT maps the [CLS] representation through a linear layer and sigmoid to estimate readmission probability within 30 days.Model parameters are fine-tuned by maximizing the binary classifier’s log-likelihood.
- Pre-training: ClinicalBERT is pre-trained on clinical notes using masked language modeling and next sentence prediction.The objective combines log-likelihoods for masked-token predictions and sentence-consecutiveness classification.
3.1 Data
The study uses MIMIC-III, an intensive-care-unit electronic health-record dataset containing admissions, patients, and de-identified clinical notes from 2001–2012.
- Dataset: MIMIC-III contains 58,976 unique hospital admissions from 38,597 intensive-care-unit patients at Beth Israel Deaconess Medical Center between 2001 and 2012.
- Dataset: The dataset includes 2,083,180 de-identified notes associated with the admissions.
3.2 Empirical Study I: Language Modeling and Clinical Word Similarity
ClinicalBERT is evaluated as a representation model for clinical notes through language modeling and physician-rated clinical concept similarity. It improves over BERT on clinical language modeling and more accurately matches physician judgments of medical-term similarity than comparison models.
- Language Modeling: ClinicalBERT improves over BERT on masked language modeling and next sentence prediction using MIMIC-III clinical notes.Table 1 reports five-fold average accuracy for both tasks.
- Clinical Word Similarity: ClinicalBERT represents medical terms by averaging subword hidden states from the sum of its last four encoder layers.The representation is computed after feeding ClinicalBERT token sequences corresponding to each medical term.
- Clinical Word Similarity: Model quality is measured by Pearson correlation between cosine similarity of concept embeddings and physician ratings of medical-concept similarity.Higher correlation indicates closer agreement with human-rated similarity.
- Clinical Word Similarity: ClinicalBERT more accurately correlates with physician judgments than Word2Vec and FastText trained on MIMIC-III clinical notes.Word2Vec and FastText are trained on the full MIMIC-III data for comparison, while ClinicalBERT is also pretrained on the full dataset.
3.3 Empirical Study II: 30-Day Hospital Readmission Prediction
ClinicalBERT models 30-day readmission from clinical notes and combines subsequence predictions to handle long patient records. It is evaluated against language-model and simpler baselines, with attention used to inspect predictive terms.
- Models: ClinicalBERT predicts hospital readmission from clinical notes and is compared with bag-of-words, bi-LSTM, and BERT baselines.
- Data and task: The cohort labels admissions as readmitted when patients return within 30 days, while excluding in-hospital deaths.
- Scalable Readmission Prediction: ClinicalBERT concatenates and splits patients’ notes into fixed-length subsequences, then aggregates their readmission predictions.
- Scalable Readmission Prediction: 3–8%: Equation (4) outperforms using the mean prediction for each subsequence alone.The formula trades off maximum and mean subsequence probabilities to reduce effects from irrelevant or noisy subsequences.
- Evaluation: ClinicalBERT outperforms competitors on discharge-summary readmission prediction and on early-admission clinical-note prediction.The evaluation reports 5-fold cross-validation for discharge summaries and five independent runs for early-admission notes.
- Interpretability: Attention weights expose terms and interactions associated with readmission, supporting inspection of model predictions.The model uses attention mechanisms to visualize which clinical-note terms are predictive.
4 Guidelines on using ClinicalBERT in Practice
ClinicalBERT is pretrained on ICU notes from one Boston hospital, so practical use should adapt it to the practitioner’s institutional EHR data.
- ClinicalBERT is pretrained on MIMIC-III patients from ICUs in one Boston hospital.
- Because notes vary by institution and clinical setting, practitioners are recommended to retrain ClinicalBERT on their private EHR dataset.The adapted model can then support downstream tasks such as mortality or length-of-stay prediction.
5 Discussion
ClinicalBERT learns clinical-text representations that capture physician-assessed semantic relationships and improve 30-day readmission prediction. The discussion identifies long-note modeling and dataset scale as boundaries for future use.
- ClinicalBERT captures physician-assessed semantic relationships in clinical text and outperforms a deep language model on 30-day readmission prediction.
- ClinicalBERT yields a large relative increase in recall at a fixed rate of false alarms.
- The max and sum operations used for long notes may not capture correlations within those notes.
- MIMIC-III is small compared with hospitals’ larger internal collections, motivating retraining on institutional notes rather than relying only on pretrained MIMIC-III embeddings.
A Hyperparameters and training details
ClinicalBERT is initialized from BERT Base, trained with 768-dimensional representations and 512-token sequences, and fine-tuned with a neural classifier for readmission prediction.
- ClinicalBERT is initialized with released BERT Base parameters and uses 768-dimensional representations.
- The maximum supported input sequence length is 512 tokens, with pretraining beginning on shorter sequences.
- Fine-tuning uses three epochs, batch size 56, learning rate 2x10^-5, and early stopping based on validation loss.
- The binary classifier has layer shapes 768 x 2048, 2048 x 768, and 768 x 1.
- The Bi-LSTM baseline uses Word2Vec embeddings, 200 output units, dropout 0.1, global max pooling, and fully connected layers.
- Experiments use two Intel Xeon E5-2670v2 CPUs, 128GB RAM, and two NVIDIA Tesla P40 GPUs.
B Preprocessing Notes for Pretraining ClinicalBERT
ClinicalBERT preprocessing lowercases text, removes line breaks and selected special characters, and segments notes into sentences for next sentence prediction pretraining.
- ClinicalBERT preprocessing lowercases words and removes line breaks, carriage returns, de-identified brackets, and selected special characters.
- SpaCy sentence segmentation is used to divide each clinical note into sentences.
- Next sentence prediction pretraining requires two sentences at every iteration.