PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Still Need Clinical Language Models?

Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, Emily Alsentzer

arXiv:2302.08091v1cs.CL

TL;DR

The paper asks whether specialized clinical models remain necessary when general-domain LLMs encode medical knowledge. It evaluates 12 models across three EHR tasks and finds that smaller clinical models outperform in-context approaches while matching or surpassing much larger general-domain models.

  • Problem

    It remains unclear whether general-web-trained LLMs are the right tools for specialized, safety-critical clinical text, despite evidence that large LLMs encode medical knowledge.

  • Method

    The authors evaluate 12 language models on three EHR parsing and reasoning tasks, including T5-Base and T5-Large models pretrained from scratch on MIMIC-III and MIMIC-IV clinical notes.

  • Results

    Smaller specialized clinical models substantially outperform all in-context learning approaches and can match or outperform much larger general-text models.

  • Takeaways & Limitations

    In-context learning with extremely large LLMs is not a sufficient replacement for finetuned specialized clinical models, supporting domain-specific model development.

  • Takeaways & Limitations

    The study does not test reasoning over long clinical text, clinical-text generation, model-compression methods, or parameter-efficient training, and compares some different architectures.

Abstract

from arXiv · show

Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.

1 Introduction

The paper asks whether specialized clinical language models remain useful despite strong general-domain LLMs, and evaluates alternative strategies for clinical tasks involving EHR notes.

  • Motivation: Recent LLMs encode substantial clinical knowledge, raising questions about the continued role of smaller models specialized for clinical text.Examples include GPT-3 competing with or outperforming smaller models on several clinical tasks and ChatGPT passing the USMLE.
  • Approaches: Healthcare systems can pretrain specialized models from scratch or from general checkpoints, finetune general-purpose models, or use prompted LLMs without additional finetuning.These four approaches correspond to the rows considered in Figure 1.
  • Study design: The study evaluates 12 language models on three clinical tasks that test parsing and reasoning over EHR notes.The experiments also train T5-Base and T5-Large from scratch on MIMIC-III and MIMIC-IV clinical notes.
  • Findings: 345M-parameter specialized clinical models substantially outperform all in-context learning approaches, including when finetuned on limited annotated data.The reported comparison covers the three clinical tasks and the evaluated in-context learning approaches.
  • Findings: Pretraining on clinical tokens enables smaller, more parameter-efficient models that match or outperform much larger models trained on general text.The paper releases its code and models under the PhysioNet Credentialed Health Data license and data use agreement.

2 Background & Related Work

Clinical notes differ substantially from general text, motivating domain adaptation, but specialized pretraining has financial and privacy costs while in-context learning introduces safety and data-governance concerns.

  • Clinical-text characteristics: Clinical notes contain grammatical errors, undefined abbreviations, and domain-specific terminology that distinguish them from general-domain text.These properties make clinical notes a distinct input domain for language models.
  • Model definitions: Specialized clinical models are pretrained on clinical notes, whereas general-purpose models are trained mostly on open-domain web text.Specialized models may be trained from scratch or further pretrained from biomedical or general-domain checkpoints through DAPT.
  • Domain adaptation: Further pretraining on clinical text may help general models adapt to the domain shift caused by semi-structured and abbreviated clinical notes.The text contrasts clinical notes with the more grammatical biomedical and general text used for other models.
  • Costs and privacy: Clinical pretraining incurs high upfront costs and may leave models retaining sensitive patient information from hospital records.The paper notes that unknown leakage risk has prevented release of some multi-billion-parameter clinical models.
  • In-context learning: In-context learning uses task examples in prompts without modifying weights, making it attractive when labeled clinical data are limited.However, large ICL models may raise security and privacy concerns and can generate realistic but factually incorrect content in medical settings.
  • Open question: Prior clinical performance from GPT-3 and ChatGPT does not establish whether ICL transfers to parsing semi-structured, abbreviation-laden clinical notes.The paper identifies this as an unresolved question for the tasks studied here.

3 Experimental Setup

The experiments compare clinical and general language models across three EHR-focused tasks, including newly trained Clinical-T5 variants and larger general-domain or in-context baselines.

  • Evaluation: The study evaluates 12 language models on three MIMIC-derived clinical tasks selected to test parsing and reasoning over clinical notes.The setup includes models spanning specialized, general-domain, and very large in-context learning systems.
  • Tasks: MedNLI measures accuracy for deciding whether a doctor-written hypothesis is entailed, neutral, or contradicted by a clinical-note premise.The premise is taken directly from a clinical note.
  • Tasks: RadQA is extractive question answering over radiology reports, evaluated with token-level F1 and exact string match.Questions concern the clinical reason for imaging, and answers are extracted from the report when available.
  • Tasks: CLIP is multi-label classification of follow-up information in discharge-summary sentences, evaluated with micro and macro F1-Score.Labels include patient-specific, appointment, medication, lab, procedure, imaging, and other appointment-related information.
  • Specialized models: The study includes BioClinRoBERTa and GatorTron, both 345M-parameter BERT-Large encoder models trained on clinical or mixed biomedical and clinical data.BioClinRoBERTa uses PubMed and MIMIC-III, while GatorTron also includes Wikipedia and University of Florida Health notes.
  • Clinical-T5 models: The authors train Clinical-T5 variants on approximately 1.2B MIMIC words, including checkpoint-initialized Base and from-scratch Base and Large models.Clinical-T5-Base-Ckpt uses T5-Base; Clinical-T5-Base and Clinical-T5-Large are initialized from scratch.
  • Baselines: General-domain comparisons include RoBERTa, T5-Base, T5-Large, PubMedGPT, and T5-XL, while ICL comparisons use GPT-3 and T5-Flan-XXL.The matched-architecture comparisons contrast general-domain and clinical pretraining setups.

4 Clinical Models Are Parameter Efficient

Clinical pretraining makes smaller models competitive with or better than substantially larger general-domain models across clinical tasks. Although training from scratch has a high initial cost, domain-specific models can reduce downstream computational costs and reach break-even with sufficient inference use.

  • Clinical-T5-Base-Ckpt and Clinical-T5-Base outperform T5-Base, while Clinical-T5-Large outperforms T5-Large.Training from scratch produces the largest gains, including +3% on RadQA and +5% on CLIP over Clinical-T5-Base-Ckpt.
  • Clinical-T5-Base slightly outperforms T5-Large, which is 3.5× larger, across all three clinical tasks.
  • Clinical-T5-Large slightly outperforms or performs similarly to T5-XL, despite T5-XL being 3.5× larger.This indicates comparable or better performance with significantly fewer computational resources.
  • 4.1 When Is Pretraining From Scratch More Efficient?: Specialized-model cost includes pretraining, finetuning, and inference, whereas the comparison general model incurs finetuning and inference costs.The cost model uses parameter count, token counts, and tokenizer-specific token weights under full-model finetuning.
  • 4.1 When Is Pretraining From Scratch More Efficient?: Clinical tokenization requires roughly 1 clinical token for every 1.12 general tokens, while the evaluated vocabularies overlap by roughly 65%.
  • 4.1 When Is Pretraining From Scratch More Efficient?: Approximately 40B inference tokens are needed to recover the cost of pretraining Clinical-T5-Large from scratch when compared with T5-XL, ignoring finetuning costs.The estimate corresponds to roughly 2.5 years for a single task run over each note once at the cited health-system scale.

5 In-Domain Tokens Are More Valuable

Clinical pretraining data improves performance and efficiency beyond what model size or total pretraining compute alone provide. Across comparisons, specialized clinical models outperform general-domain alternatives, including much larger models, while requiring fewer FLOPs at comparable performance.

  • Model size and training data: BioClinRoBERTa and GatorTron achieve the highest performance on all tasks despite being less than 12% the size of T5-XL.RoBERTa uses more total FLOPs than these clinical models but does not outperform them, suggesting training-data composition matters more than total FLOPs alone.
  • Model size and training data: GatorTron scores 10 points worse than PubMedGPT on the USMLE, yet smaller clinical models outperform PubMedGPT on the evaluated clinical-note tasks.The comparison suggests conventional medical knowledge and the ability to parse clinical notes are distinct capabilities.
  • Compute efficiency: Across comparable FLOPs or performance, clinical models outperform general models, and T5 comparisons isolate the contribution of clinical tokens.The T5 models share architecture and training process while varying in model size and pretraining domain.
  • Compute efficiency: Clinical-T5-Large matches T5-XL on MedNLI while requiring 3.5 times fewer FLOPs.The broader comparisons indicate that clinical models are more computationally efficient at a fixed performance level.

6 In-Context Learning Underperforms Task Specific Models

The study compares in-context learning with finetuned models under limited-data conditions across three clinical tasks. Finetuned models generally outperform in-context approaches, while the advantage of specialized clinical pretraining varies by dataset and data availability.

  • Overall comparison: Except for RadQA at 1% data, GPT-3 and T5-Flan-XXL perform worse than GatorTron at every ablation point.The ablation uses 1%, 5%, 10%, 25%, and 100% of available training data and reports means over three random seeds.
  • Overall comparison: Models finetuned on all available data significantly outperform every in-context learning approach across all tasks.The comparison includes GPT-3 and Flan-T5-XXL in-context learning against finetuned clinical and non-clinical models.
  • Dataset-dependent few-shot performance: On MedNLI, BioClinRoBERTa and GatorTron outperform GPT-3 in every resource-restricted setting.On RadQA, GPT-3 and Flan-T5-XXL outperform smaller specialized models only when those models use 1% of training data, equal to 49 question-answer pairs.
  • Dataset-dependent few-shot performance: All models outperform GPT-3 and Flan-T5-XXL on CLIP even when trained with only 5 discharge summaries.The passage attributes this result to aggressive sentence segmentation and nonspecific task labels in CLIP.
  • Dataset-dependent few-shot performance: On two of three datasets, the 11B Flan-T5-XXL model outperforms the 175B GPT-3 model.Flan-T5-XXL can be run locally with in-context learning on a single GPU, particularly using libraries such as DeepSpeed.
  • Value of clinical pretraining: Clinical models show their largest performance gap over non-clinical models at 5% and 10% finetuning data, with the gap diminishing as data increases.This pattern is especially clear on RadQA and CLIP.

7 Limitations & Future Work

The evaluation omits several clinically important settings, including long-text reasoning and clinical-text generation, and leaves some architectural and training comparisons unresolved. The study also highlights a segmentation issue that can make in-context learning difficult.

  • Scope and comparison limits: The study does not test reasoning over long clinical text, clinical-text generation, model compression, parameter-efficient training, instruction-tuned models, or ChatGPT.It also cannot isolate pretraining-data effects from architecture because some comparisons span different architectures.
  • Data-processing limitation: Aggressive sentence segmentation can label headers such as “Discharge Instructions:” with the following sentence’s label.This creates challenging in-context-learning examples, although extensive heuristics might alleviate the issue.

8 Conclusion

The paper evaluates whether smaller clinical language models remain useful against domain-agnostic LLMs by testing 12 models across three EHR tasks. Its results suggest that specialized clinical models are more parameter-efficient than larger general-domain models.

  • Study design: 12 models spanning 220M to 175B parameters are evaluated on 3 clinical tasks that test parsing and reasoning over electronic health records.The experiments focus on whether smaller specialized clinical models remain useful despite impressive domain-agnostic LLMs.
  • Conclusion: Smaller models tailored for clinical text are more parameter efficient than larger domain-agnostic models.

A MIMIC Preprocessing and Model Training

The appendix describes preprocessing MIMIC notes, testing tokenization choices, and training specialized T5 models from scratch or with continued pretraining. It also outlines finetuning procedures and task-specific compute settings.

  • MIMIC preprocessing: MIMIC-III and MIMIC-IV notes are combined while avoiding repeated patient records, and duplicate MIMIC-III notes are removed using charttime, storetime, and cgid.After preprocessing, MIMIC-III contains 430M words.
  • Tokenization: The selected tokenizer adds special DEID tokens and replaces existing de-identification tags before pretraining.Three tokenization methods were compared, including unmodified tags and replacement with realistic PHI.
  • Clinical-T5 training: Clinical-T5-Base and Clinical-T5-Large are pretrained from scratch on MIMIC notes for approximately 40B and 38B tokens, respectively.Clinical-T5-Base uses 310K steps, while Clinical-T5-Large uses 780K steps.
  • Clinical-T5 training: Clinical-T5-Base-Ckpt is initialized from T5-Base and trained for an additional 100K steps on MIMIC notes.The reported setup uses 8 A6000 GPUs with batch-size 32 per GPU and 40K warm-up steps.
  • Finetuning setup: Finetuning uses task-specific learning rates, optimizers, convergence criteria, batch sizes, and hardware across the evaluated models.The appendix reports different learning-rate procedures for T5, PubMedGPT, ClinicalBERT, GatorTron, ClinicalLongformer, RoBERTa, and BioClinRoBERTa.

B.3 Task-Specific Details

The appendix specifies how T5 and encoder-only models generate outputs for MedNLI, RadQA, and CLIP, then compares expanded model results and continued-pretraining steps. Clinical models generally retain an efficiency advantage over larger or biomedical alternatives.

  • General task setup: Encoder-only models and PubMedGPT use task-specific linear layers, while T5 models generate labels or extracted text with beam search.
  • MedNLI and RadQA: For MedNLI, T5 models use beam-search width 3; for RadQA, width 1 avoids inconsistent gains in exact match at the expense of F1.
  • CLIP: For CLIP, T5 models generate alphabetically ordered comma-delimited label lists using a context window of 256.CLIP is evaluated with micro and macro F1-Score across up to seven labels per sentence.
  • Model comparisons: ClinicalBERT performs similarly to T5-Base, while ClinicalLongformer performs similarly to T5-Large; SciFive models outperform their T5 counterparts but not Clinical-T5 models.The authors suggest the difference may reflect scratch training on clinical tokens versus DAPT on biomedical tokens.
  • Continued pretraining: 20K continued-pretraining steps provide a reasonable boost over T5-Base, while 20K to 80K steps add no apparent gains and 100K steps improve performance versus 80K.The appendix attributes the late improvement possibly to the learning-rate scheduler.

C.2 RadQA

On RadQA, ClinicalBERT performs poorly, while ClinicalLongformer is comparable to Clinical-T5-Base-Ckpt; biomedical models outperform similarly sized general models but not their clinical counterparts.

  • ClinicalBERT performs extremely poorly on RadQA, whereas ClinicalLongformer performs similarly to Clinical-T5-Base-Ckpt.
  • The RadQA results are reported in Table 9 with mean performance and standard deviation for models trained with at least three random seeds.
  • MedNLI, SciFive and SciFive-Large outperform T5-Base and T5-Large, respectively, but neither surpasses its clinical equivalent.

D Additional Details about In Context Learning Experiments

The in-context learning experiments use manually selected prompts and a small number of examples, but performance is highly sensitive to prompt wording and output handling.

  • Prompts were selected from approximately 5–10 candidates per task using one to three randomly sampled in-context examples.Selection used validation performance on a random sample of 200 examples.
  • For CLIP, generating all labels at once performed similarly to generating each label independently while reducing inference-time costs.The all-at-once strategy was therefore used for GPT-3.
  • Flan-T5-XXL’s MedNLI validation accuracy fell from 83.5% to 62% under a different prompt.The result illustrates strong dependence on prompt wording.
  • GPT-3 and Flan-T5-XXL outputs required post-processing to map generated text into task labels.RadQA answers that could not be directly extracted from the report were still scored with F1; enforcing extractive outputs would reduce F1 to approximately 40 for both models.
  • GPT-Neo-X was removed from the remaining experiments after scoring below 40% on MedNLI in initial experiments.
Loading 2302.08091v1…