Source-linked AI summary
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
TL;DR
Biomedical NLP has abundant unlabeled text, yet common practice adapts general-domain language models whose vocabulary and pretraining data may not fit the target domain. This paper builds the BLURB benchmark and systematically evaluates biomedical pretraining and fine-tuning choices, finding that training from scratch on in-domain data outperforms continual pretraining and supports state-of-the-art results across diverse tasks.
Problem
The paper asks whether general-domain initialization and mixed-domain pretraining remain beneficial for biomedical NLP when abundant biomedical text is available.
Method
The authors compile BLURB from public biomedical datasets and compare domain-specific pretraining and task-specific modeling choices across biomedical NLP tasks.
Results
Domain-specific pretraining from scratch substantially outperforms continual pretraining and yields state-of-the-art results across a wide range of biomedical NLP applications.
Takeaways & Limitations
Biomedical language models can use in-domain pretraining as a strong foundation, while some complex BERT-era practices, such as BIO tagging, are unnecessary for NER.
Takeaways & Limitations
The study's strategy assumes sufficient in-domain text; mixed-domain pretraining may make more sense when the target domain has little text of its own.
Abstract
from arXiv · showhide
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.
1 INTRODUCTION
Biomedical NLP pretraining traditionally extends general-domain language models, but this paper tests whether abundant in-domain text makes domain-specific pretraining from scratch preferable. It introduces a comprehensive benchmark and reports broad performance gains, while reassessing common fine-tuning practices.
- Existing NLP pretraining generally uses general-domain corpora such as Wikipedia, BookCorpus, newswire, and Web text.
- The paper questions the assumption that biomedical pretraining should initialize from a general-domain language model.It frames continual pretraining as transfer from general text to specialized biomedical text.
- Domain-specific pretraining from scratch substantially outperforms continual pretraining of generic language models on downstream biomedical applications.
- BLURB is a comprehensive biomedical NLP benchmark compiled from publicly available datasets to enable systematic comparisons.
- BERT-based experiments find that simple IO tagging can match or outperform BIO and more complex NER tagging schemes.
- The authors release pretrained and task-specific models and create a public benchmark leaderboard to accelerate biomedical NLP research.
2.1 Language Model Pretraining
The paper reviews transformer-based language-model pretraining, including domain-adapted tokenization, contextual encoding, and self-supervised objectives. BERT masks input tokens for prediction and can augment this objective with whole-word masking or adversarial perturbations.
- WordPiece derives a subword vocabulary by combining existing units, with vocabulary size and casing selected for the pretraining corpus.
- BERT-style models use transformer architectures with multi-head self-attention to model long-range textual dependencies.
- Self-Supervision: Masked Language Modeling randomly replaces input tokens and trains the model to predict the original tokens using cross-entropy loss.
- Self-Supervision: BERT and RoBERTa select 15% of input tokens for masking, with 80% replaced by [MASK], 10% unchanged, and 10% randomly replaced.
- Self-Supervision: Whole-word masking masks every subword when one subword of a word is selected, encouraging contextual semantic modeling.
- Self-Supervision: Adversarial pretraining perturbs input embeddings to maximize adversarial loss while optimizing the standard MLM objective.
2.2 Biomedical Language Model Pretraining
Biomedical language-model pretraining compares continual adaptation of general-domain models with training from scratch on biomedical text. Because biomedicine has abundant in-domain data, the paper argues that an in-domain vocabulary and corpus provide the stronger foundation.
- Mixed-Domain Pretraining: Continual pretraining initializes a general-domain BERT model and continues MLM and NSP training on biomedical text.
- Mixed-Domain Pretraining: Continual pretraining retains the original BERT vocabulary, which is not representative of biomedical terminology.
- Mixed-Domain Pretraining: SciBERT generates its vocabulary and trains from scratch but mixes biomedical and computer-science text, leaving it out-domain for biomedical applications.
- Mixed-Domain Pretraining: Biomedicine has over thirty million PubMed abstracts and adds over a million annually, weakening the rationale for relying on related out-domain text.
- Domain-Specific Pretraining from Scratch: Domain-specific pretraining from scratch uses an in-domain vocabulary and purely biomedical data, avoiding limitations of mixed-domain initialization.
- Domain-Specific Pretraining from Scratch: Domain-specific pretraining with an in-domain vocabulary shows clear advantages over continual and mixed-domain pretraining.
2.3 BLURB: A Comprehensive Benchmark for Biomedical NLP
BLURB is a comprehensive biomedical NLP benchmark designed to enable head-to-head evaluation across PubMed-based applications. It covers diverse task types and reports macro-averaged performance across those types.
- BLURB addresses prior benchmark limitations by broadening biomedical task coverage and separating PubMed-based applications from clinical applications.
- BLURB focuses on PubMed-based biomedical applications while leaving clinical and other verticals for future work.
- The datasets are drawn from publicly available resources and prioritize datasets used in recent biomedical language-modeling work.
- The benchmark includes NER, PICO extraction, relation extraction, sentence similarity, document classification, and question answering.
- BLURB computes each task-type average first, then reports the macro average across task types rather than averaging all datasets directly.
2.4 Task-Specific Fine-Tuning
The paper uses a shared architecture and fine-tuning procedure to compare pretrained language models across biomedical tasks. Task-specific modeling choices vary by prediction target, including token tagging, classification, regression, and relation representation.
- A General Architecture for Fine-Tuning Neural Language Models: Pretrained language models produce contextual token representations, which a task-specific prediction model converts into application outputs.
- A General Architecture for Fine-Tuning Neural Language Models: The fine-tuning pipeline transforms inputs, tokenizes them with the model vocabulary, computes contextual representations, and applies a prediction module.
- A General Architecture for Fine-Tuning Neural Language Models: The study applies the same fine-tuning procedure to all BERT models and tasks, jointly updating task-specific and underlying language-model parameters.
- Named Entity Recognition (NER): NER is formulated as sequential token labeling, with BIO as the standard tagging scheme and linear, LSTM, or CRF classification layers as alternatives.
- PICO: PICO extraction is treated as sequential tagging, allowing a token to belong to multiple elements such as participants and interventions.
- Relation Extraction: Relation extraction represents entity pairs using markers or dummy tokens and predicts relations from [CLS] or pooled mention representations.
2.5 Experimental Settings
The experiments pretrain PubMedBERT from scratch on a large filtered PubMed corpus and compare it with publicly released BERT-family models. Fine-tuning uses standardized optimization settings, while acknowledging computational and variance constraints.
- PubMedBERT is pretrained from scratch on 14 million PubMed abstracts containing 3.2 billion words and 21 GB of text.
- The pretraining procedure uses Adam, a slanted triangular learning-rate schedule, 62,500 steps, and batch size 8,192.
- PubMedBERT uses uncased modeling and whole-word masking at a 15% masking rate after cased and uncased versions showed similar preliminary performance.
- Comparisons include BERT, RoBERTa, BioBERT, SciBERT, ClinicalBERT, and BlueBERT, with several continual-pretraining models retaining BERT’s vocabulary.
- The primary comparison uses BERT-BASE, while deeper BERT-LARGE exploration is left for future work.
- Task-specific results may vary across random seeds, so the study averages ten runs for BIOSSES, BioASQ, and PubMedQA.
3 RESULTS
The evaluation compares domain-specific and mixed-domain pretraining across BLURB, then examines pretraining and fine-tuning choices. PubMedBERT generally performs best, while in-domain vocabulary and simpler BERT fine-tuning practices are supported by the ablations.
- 3.1 Domain-Specific Pretraining vs Mixed-Domain Pretraining: The study compares pretrained BERT models on BLURB using the same task-specific fine-tuning process across six biomedical NLP tasks.BLURB aggregates average test results for NER, PICO, relation extraction, sentence similarity, document classification, and question answering.
- 3.1 Domain-Specific Pretraining vs Mixed-Domain Pretraining: PubMedBERT consistently outperforms other BERT models on most biomedical NLP tasks, often by a significant margin.Its gains are largest against models trained with out-domain text; PubMedBERT also generally improves over BioBERT despite both using PubMed text.
- 3.2 Ablation Study on Pretraining Techniques: In-domain PubMed vocabulary performs significantly better than the original Wikipedia-and-BookCorpus vocabulary, while whole-word masking consistently improves results.The in-domain vocabulary also shortens downstream inputs, which the paper says makes learning easier.
- 3.2 Ablation Study on Pretraining Techniques: General-domain pretraining provides no advantage here, even with in-domain vocabulary; PubMed pretraining from scratch reaches similar performance with half the compute and gains significantly with equal compute.Continual pretraining with in-domain vocabulary slightly degrades performance, whereas PubMedBERT improves when given the same compute.
- 3.2 Ablation Study on Pretraining Techniques: Adding PubMed Central full text slightly degrades performance initially, but 100K steps—60% longer training—slightly surpasses abstract-only PubMedBERT overall.The improvement is mixed across tasks, with some gaining and others losing; the paper attributes this partly to noisier, less task-matched full text and insufficient additional training.
- 3.3 Ablation Study on Fine-Tuning Methods: For BERT-based NER, complex tagging schemes offer only minuscule differences, while a Bi-LSTM provides no substantial gain over a linear layer.The findings suggest that self-attention reduces the need for sequential modeling and position-sensitive tagging.
4 DISCUSSION
The paper argues that transfer across domains is not uniformly beneficial when abundant in-domain biomedical text is available. It also introduces BLURB to support comprehensive evaluation across biomedical NLP tasks.
- Mixed-domain pretraining may not help when sufficient in-domain text is available, as shown by comparisons of clinical BERTs with PubMedBERT.The results suggest that even related clinical notes may add little when PubMed provides abundant biomedical text.
- The study concludes that different types of transfer learning should be distinguished and evaluated separately across situations.
- BLURB provides a biomedical NLP leaderboard with thirteen datasets spanning six tasks.
- Biomedical NLP research covers tasks from named entity recognition to relation extraction, question answering, and evidence-based medical information extraction.
5 CONCLUSION
The paper reports that domain-specific pretraining from scratch outperforms continual pretraining from general-domain models across biomedical NLP applications. It supports this study with BLURB and released models, while identifying broader benchmark coverage and domain extensions as future directions.
- BLURB is a comprehensive biomedical NLP benchmark covering diverse tasks including named entity recognition, relation extraction, document classification, and question answering.
- The authors release biomedical BERT models and establish a BLURB-based leaderboard to accelerate biomedical NLP research.
- Future work includes exploring domain-specific pretraining strategies, adding biomedical NLP tasks, and extending BLURB to clinical and other high-value domains.