Source-linked AI summary
Publicly Available Clinical BERT Embeddings
Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, Matthew B. A. McDermott
TL;DR
Clinical text differs from general and biomedical text, yet publicly available pre-trained clinical BERT models were lacking. The paper pre-trains and releases clinical and discharge-summary BERT variants, finding gains on non-de-identification tasks but not on two de-identification tasks. These models provide a public clinical NLP resource, although their training requires substantial computation and their scope is limited by the source data and model design.
Problem
Clinical narratives differ linguistically from general and biomedical text, while publicly available pre-trained BERT models for clinical text were lacking.
Method
The authors pre-train and publicly release Clinical BERT and Discharge Summary BERT variants, including models initialized from BERT-Base or BioBERT.
Results
Clinical embeddings improve performance on three non-de-identification clinical tasks, but clinical BERT offers no improvements on two de-identification tasks.
Takeaways & Limitations
The released embeddings provide a community resource for clinical NLP without requiring researchers to reproduce MIMIC-scale pre-training.
Takeaways & Limitations
Training requires roughly 17–18 days on one GPU, and MIMIC covers intensive-care notes from a single institution.
Abstract
from arXiv · showhide
Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.
1 Introduction
Clinical narratives differ linguistically from general and biomedical text, motivating specialized clinical BERT models. This work publicly releases such models and evaluates their performance across clinical NLP tasks.
- Clinical narratives differ in linguistic characteristics from general text and non-clinical biomedical text, motivating specialized clinical BERT models.
- The work builds and publicly releases a clinical embedding model.
- The released models include BERT-Base and BioBERT-finetuned variants trained on all clinical notes or discharge summaries.
- Clinical-specific contextual embeddings improve over general-domain and BioBERT results across two clinical NER tasks and one medical natural language inference task.
- General BERT and BioBERT outperform clinical BERT on two de-identification tasks, which the authors relate to differences in de-identification context.
2 Related Work
Contextual embeddings address limitations of fixed word vectors and have been extended from general and biomedical text toward clinical corpora. Prior clinical models showed task gains, but publicly released clinical BERT resources remained limited.
- Contextual Embeddings in General: Traditional word vectors assign a single representation to all meanings of a word, whereas ELMo and BERT provide context-sensitive representations.
- Contextual Embeddings in General: BERT is examined instead of ELMo or non-contextual embeddings because it has generally outperformed them across varied tasks, including clinical tasks.
- Biomedical and Clinical Models: BioBERT pre-trains BERT on biomedical research articles and reports improved performance on several biomedical NLP tasks.
- Biomedical and Clinical Models: Clinical-domain contextual models had demonstrated improvements on clinical tasks, including de-identification and named entity recognition.
- Biomedical and Clinical Models: A clinical-note BERT achieved state-of-the-art results on four corpora, but did not release its embeddings publicly or examine clinical fine-tuning of BioBERT.
3 Methods
The authors pre-train clinical BERT variants on MIMIC notes, fine-tune them for several clinical tasks, and release the resulting embeddings. The approach uses standard BERT training and relatively simple downstream classifiers, while requiring substantial computation.
- Data: The models use approximately 2 million notes from the MIMIC-III v1.4 database for clinical-text pre-training.
- Data: Clinical BERT uses all note types, while Discharge Summary BERT uses only discharge summaries to tailor the corpus to downstream tasks.
- Data: The authors retain notes overlapping task corpora because those corpora are much smaller than the full MIMIC corpus.
- BERT Training: Clinical BERT is initialized from BERT-Base and Clinical BioBERT from BioBERT, with fine-tuned embeddings passed through a single linear classification layer.
- BERT Training: The downstream setup applies token-level classification to NER and de-identification, and sentence-level classification to MedNLI.
- Computational Cost: Training and preprocessing require roughly 17–18 days on one GeForce GTX TITAN X GPU, potentially exceeding some laboratories’ resources.
- Tasks: The evaluation covers MedNLI and four i2b2 tasks, including concept extraction, entity extraction, and de-identification.
- Tasks: The de-identification datasets use synthetic, realistic PHI masks, unlike MIMIC’s sentinel PHI markers.
4 Results & Discussions
Clinical BERT improves performance on several non-de-identification clinical NLP tasks, while de-identification performance remains limited by distribution differences between MIMIC and task data. Qualitative comparisons also suggest stronger clinical-domain cohesion, but the models have important architectural and institutional limitations.
- Clinical NLP Tasks: 82.7% accuracy on MedNLI exceeded the prior 73.5% state of the art.Clinical BERT achieved the reported new state of the art using the InferSent result as the prior comparison.
- Clinical NLP Tasks: Clinical BERT improved over general BERT or BioBERT on MedNLI, i2b2 2010, and i2b2 2012, but not on either de-identification task.The reported gains cover three of five evaluated tasks; i2b2 2006 and i2b2 2014 were the exceptions.
- De-identification: De-identification data differ from MIMIC because synthetic PHI masks replace MIMIC’s sentinel markers, altering sentence structures encountered at task time.The authors argue this distribution drift is especially damaging for contextual models trained on recurring sentinel-token patterns.
- Limitations & Future Work: Discharge Summary BERT improved over Clinical BERT on one task, suggesting that note-type-specific corpora can provide selective benefits.The authors also identify the lack of advanced architectures atop the embeddings and MIMIC’s single-institution ICU source as limitations.
- Qualitative Embedding Comparisons: Clinical BERT’s nearest neighbors showed greater cohesion around medical and clinical-operations terms than BioBERT’s.The paper gives “Discharge” as an example: Clinical BERT’s associated words were all meaningful in a clinical-operations context.
5 Conclusion
The paper pretrains and publicly releases clinically oriented BERT models, including models trained on clinical text and models fine-tuned atop BioBERT. It reports robust superiority over general-domain or BioBERT embeddings for non-de-identification tasks and selective benefits from note-type-specific corpora.
- The authors pretrain and publicly release clinically oriented BERT models trained on clinical text or fine-tuned atop BioBERT.
- Clinical embeddings were superior to general-domain or BioBERT embeddings for non-de-identification tasks.
- Note-type-specific corpora produced further selective performance benefits.
A MIMIC Notes
MIMIC notes are distributed across 15 note types, and many are semi-structured with section headers separating free-text paragraphs.
- 15 note types make up the MIMIC notes collection.
- Many MIMIC note types are semi-structured, using section headers to separate free-text paragraphs.
B BERT Training Details
The pre-training experiments use the TensorFlow implementation of BERT.
- The pre-training experiments use the TensorFlow implementation of BERT.
B.1 Pre-training
The models were pretrained for 150,000 steps using a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5.
- 150,000 steps were used for pre-training with batch size 32, maximum sequence length 128, and learning rate 5 · 10−5.Models pretrained for 300,000 steps showed no significant differences in downstream task performance.
B.2 Fine-tuning
Downstream tasks were fine-tuned across learning rates, batch sizes, and epoch counts, with maximum sequence length fixed at 150.
- Learning rates of 2 · 10−5, 3 · 10−5, and 5 · 10−5, batch sizes of 16 and 32, and epochs of 3 and 4 were explored.NER tasks also included 2 epochs; i2b2 2014 used only 2 epochs due to time constraints.
- 150 was the maximum sequence length used across all downstream tasks.
C IOB Format
IOB format encodes span-based named entity recognition labels with Beginning, Inside, and shared Outside subclasses, adding granularity over span positions.
- IOB reclassifies each entity class into Beginning, Inside, and Outside subclasses for span-based NER.
- Beginning marks the first word of a span, while Inside marks words within that span.
- The Outside label marks words outside any span and replaces the shared “no class” label for extraneous words.
- The example labels “Null Null Null Problem Problem” become “O O O B-Problem I-Problem” in IOB format.