Source-linked AI summary
GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records
Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, Christopher A Harle, Gloria Lipori, Duane A Mitchell, William R Hogan, Elizabeth A Shenkman, Jiang Bian, Yonghui Wu
TL;DR
Large clinical language models could help process unstructured EHRs, but evidence for billion-parameter models in this domain has been limited. This study develops and evaluates GatorTron, finding that it outperforms existing biomedical and clinical transformers across five clinical NLP tasks.
Problem
Evidence is limited on how billion-parameter clinical language models can support medical AI systems using unstructured EHRs.
Method
The study trains GatorTron from scratch on over 90 billion words and evaluates parameter scaling across five clinical NLP tasks.
Results
GatorTron outperformed existing biomedical and clinical transformers on all five evaluated clinical NLP tasks.
Takeaways & Limitations
Scaling clinical transformers from 110 million to 8.9 billion parameters produced a large clinical model with stronger performance across the evaluated tasks.
Takeaways & Limitations
Even large models such as GatorTron still have difficulty identifying key information from longer clinical narratives.
Abstract
from arXiv · showhide
There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model - GatorTron - using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve 5 clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.
Introduction
Clinical narratives contain rich patient information but are difficult for computational models to use, motivating NLP for medical AI systems [5,6,7]. This study introduces GatorTron, trained from scratch on >90 billion words, and evaluates parameter and data scaling across five clinical NLP tasks.
- Introduction: GatorTron was trained from scratch on >90 billion words, including de-identified UF Health clinical notes, PubMed articles, and Wikipedia.The corpus combines clinical and nonclinical text sources.
- Introduction: The parameter-scaling experiment compared GatorTron models with 345 million, 3.9 billion, and 8.9 billion parameters.These were designated base, medium, and large models, respectively.
- Introduction: The data-scaling experiment compared GatorTron-base trained on the full corpus with a version trained on a random sample containing one-quarter of the corpus.This comparison tested whether increasing training-data size benefits downstream clinical NLP tasks.
- Introduction: The evaluation covered clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference, and medical question answering.These tasks span information extraction, similarity, inference, and question answering.
- Introduction: 5 clinical NLP tasks were outperformed by GatorTron relative to previous biomedical and clinical transformer models, while scaling clinical transformers from 110 million to 8.9 billion parameters.The study presents this scaling as evidence for the benefit of larger clinical transformer models.
Results
Scaling training data from one-quarter to the full corpus improved four tasks, while increasing model size from 345 million to 8.9 billion parameters improved all five clinical NLP tasks. GatorTron-large generally achieved the strongest results, with monotonic scaling gains for natural language inference and medical question answering but not semantic textual similarity.
- Natural language inference: GatorTron-large achieved 0.9020 NLI accuracy, outperforming BioBERT and ClinicalBERT by 9.6% and 7.5%, respectively, with monotonic improvement across model sizes.All GatorTron models outperformed the existing biomedical and clinical transformers.
- Assess semantic textual similarity: GatorTron-medium achieved the best semantic textual similarity Pearson correlation of 0.8903, while GatorTron-large scored 0.8896 and outperformed GatorTron-base.All GatorTron models outperformed existing biomedical and clinical transformer models, but performance did not improve consistently with model size.
- Scale up the size of training data and the number of parameters: Increasing parameters from 345 million to 8.9 billion improved all five clinical NLP tasks, while full-corpus training improved four tasks except medication-related-question F1.These results suggest GatorTron scales for canonical clinical downstream tasks and that performance had not reached its limit.
Discussion
GatorTron scaled clinical transformers to 8.9 billion parameters using a corpus exceeding 90 billion words and outperformed existing biomedical and clinical transformers across five clinical NLP tasks. Larger models especially improved complex tasks, while remaining limited in handling key information in long paragraphs.
- GatorTron outperformed existing biomedical and clinical transformers on all 5 evaluated clinical NLP tasks across 6 benchmark datasets.The models were evaluated at phrase, sentence, and document levels.
- Scaling from 345 million to 8.9 billion parameters produced monotonic improvements on 4 of 5 tasks, excluding semantic textual similarity.GatorTron also outperformed BioMegatron36, which had a similar model size but was trained on a much smaller corpus.
- Larger models yielded remarkable gains on natural language inference and medical question answering but only moderate gains on clinical concept and medical relation extraction.This pattern indicates that model scaling is more helpful for complex NLP tasks than for easier tasks.
- Larger GatorTron models converged faster than the smaller model during masked-language-model pretraining.Training and validation losses were monitored using a 5% clinical-text subset to determine stopping time.
- GatorTron can support computable-phenotype cohort identification, clinical decision support, pharmacovigilance, clinical-text deduplication, medical-knowledge mining, and patient-facing medical AI.Despite these applications, the models still have difficulty identifying key information in longer paragraphs for NLI and MQA.
- GatorTron improved clinical information extraction by modeling longer phrases and semantic categories more accurately than ClinicalBERT.It identified the complete phrase “a mildly dilated ascending aorta” and classified “kidney protective effects” as “TREATMENT,” unlike ClinicalBERT.
Data Source
The study primarily used clinical narratives from the UF Health Integrated Data Repository, collected under institutional review board approval. The corpus combined UF Health notes with MIMIC-III, PubMed, and Wikipedia to exceed 90 billion words.
- Data Source: UF Health clinical narratives formed the study’s primary data source, and the study was approved by the UF Institutional Review Board (IRB202100049).The data came from the UF Health IDR, a research data warehouse.
- Data Source: >90 billion words were assembled by merging the UF Health clinical corpus with MIMIC-III, PubMed, and Wikipedia corpora.The added corpora contributed 0.5 billion, 6 billion, and 2.5 billion words, respectively.
Preprocessing and de-identification of text
The study applied minimal preprocessing to clinical text, including cleaning, normalization, tokenization, and sentence boundary detection. UF Health text underwent additional de-identification to remove protected health information.
- Minimal preprocessing removed empty or duplicate notes, standardized UTF-8 encoding, eliminated illegal strings, normalized special characters, tokenized text, and detected sentence boundaries.
- UF Health clinical text received additional de-identification to remove protected health information.
- The de-identification system followed HIPAA safe-harbor rules for 18 protected-health-information categories and replaced identified values with dummy strings such as [**NAME**].
Study design
The study pretrained GatorTron on >90 billion words with masked language modeling, fine-tuned it on five clinical NLP tasks, and evaluated scaling across three model sizes against established biomedical and clinical transformers.
- Study design: GatorTron was pretrained on >90 billion words with masked language modeling and then supervised-fine-tuned for five clinical NLP tasks.The study adopted a BERT architecture implemented in Megatron-LM.
- Study design: The study compared GatorTron-base, GatorTron-medium, and GatorTron-large, containing 345 million, 3.9 billion, and 8.9 billion parameters, respectively.The large model used model parallelism across four GPUs, whereas the base and medium models were loaded into one GPU for distributed training.
- Study design: The three GatorTron models were evaluated against ClinicalBERT, BioBERT, and BioMegatron across clinical concept extraction, relation extraction, semantic textual similarity, natural language inference, and medical question answering.ClinicalBERT had 110 million parameters, BioBERT 345 million, and BioMegatron 1.2 billion; evaluation used six public benchmark datasets.
Training environment · GatorTron Model Configuration · Train GatorTron models from scratch
GatorTron was trained from scratch on a large distributed GPU infrastructure, using three transformer configurations spanning 110 million to 8.9 billion parameters. Pretraining used a >90-billion-word corpus, BERT-style self-supervision, and model parallelism for the largest model.
- Training environment: GatorTron training used 992 NVIDIA DGX A100 GPUs across 124 superPOD nodes, with data-level and model-level parallelism implemented by Megatron-LM.Training progress was monitored using training and validation loss, and stopped when the loss plot became flat.
- GatorTron Model Configuration: The medium model scaled to 3.9 billion parameters, approximately 10 times the base setting, while the large model scaled to 8.9 billion parameters.The large model was similar in size to BioMegatron, which had 8.3 billion parameters.
- GatorTron Model Configuration: The study developed three GatorTron configurations, selecting layers, hidden sizes, and attention heads using depth-to-width allocation guidelines and prior BioMegatron experience.Table 3 provides the technical details of the three settings.
- Train GatorTron models from scratch: The models pretrained a vocabulary from scratch with byte-pair encoding on a corpus containing >90 billion words and retained a BERT-style architecture.This established the tokenization and architecture for training GatorTron from scratch.
- Train GatorTron models from scratch: Pretraining used masked language modeling and sentence-order prediction, randomly masking 15% of input tokens and predicting the order of two consecutive text segments.The sentence-order task presented consecutive sentences in random orders and classified whether their order was correct.
- Train GatorTron models from scratch: Because the 8.9-billion-parameter large model could not fit on one GPU, it was sliced into four pieces for distributed model-parallel training, unlike the base and medium models.The default BERT loss function was used, and Fig. 4 illustrates the large-model distributed training setup.
Existing transformer models for comparison · Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark
The study compares GatorTron with established biomedical and clinical transformer models, then fine-tunes it across five clinical NLP tasks using expert-annotated benchmark datasets. Task-specific formulations, output layers, losses, datasets, and evaluation metrics are defined for concept extraction, relation extraction, semantic similarity, inference, and question answering.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: GatorTron models were fine-tuned for five clinical NLP tasks using expert annotations from six public benchmark datasets, with task-specific output layers and cross-entropy or mean square error losses.Classification tasks used cross-entropy loss, whereas regression tasks used mean square error loss.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: Clinical concept extraction was formulated as BIO sequence labeling with classification layers, using i2b2 and n2c2 challenge datasets and precision, recall, and F1-score for evaluation.Unified classifiers handled datasets without overlapping concepts, while separate models recognized each concept category when concepts overlapped.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: Medical relation extraction was treated as classification over heuristically generated concept pairs, marked with entity boundaries and evaluated on medication–adverse-event relations using precision, recall, and F1-score.The 2018 n2c2 dataset was used for this evaluation.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: Semantic textual similarity was formulated as regression over sentence-level representations from the 2019 n2c2/OHNLP challenge dataset and evaluated with Pearson correlation.A linear regression layer produced similarity scores, with mean square error as the loss.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: Natural language inference was formulated as three-way classification of entailment, contradiction, and neutral on MedNLI, with accuracy used for comparison.The premise and hypothesis were concatenated with a [SEP] token before classification.
- Fine-tune GatorTron for 5 clinical NLP tasks, evaluation matrices, and benchmark: Medical question answering used machine reading comprehension on the emrQA dataset, predicting answer spans from question–context sequences and evaluating F1-score and exact match.Inputs used two linear layers for answer start and end positions; long contexts were processed with sliding windows because the model supported a maximum length of 512 tokens.
Data availability
Benchmark datasets supporting the study’s findings are available from the official websites of natural language processing challenges, subject to Data Use Agreements.
- Benchmark datasets supporting the study’s findings are available from official natural language processing challenge websites with Data Use Agreements.
2. MedNLI dataset: https://physionet.org/content/mednli/1.0.0/ · 4. MIMIC III dataset: https://physionet.org/content/mimiciii/1.4/ · Legends of Figures
The supplied section materials note privacy restrictions on UF Health IDR clinical notes and public availability of GatorTron models trained on more than 90 billion words. Figure-related materials provide demographic, correspondence, address, and statistical-information metadata.
- 4. MIMIC III dataset: https://physionet.org/content/mimiciii/1.4/: UF Health IDR clinical notes are not publicly available because they contain patient privacy information.
- 4. MIMIC III dataset: https://physionet.org/content/mimiciii/1.4/: GatorTron models pretrained on more than 90 billion words are publicly available.
- Legends of Figures: Figure materials state that ages were calculated as of September 2022.
- Legends of Figures: The materials include a Materials & Correspondence section.
- Legends of Figures: The listed institutional location is the Clinical and Translational Research Building.
- Legends of Figures: The listed address is 2004 Mowry Road, PO Box 100177, Gainesville, FL, USA, 32610.
- Legends of Figures: The figure materials report no statistical information.