Source-linked AI summary
Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing
Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon
TL;DR
Biomedical NLP must process rapidly growing text despite scarce labeled data, making large pretrained models difficult to fine-tune reliably. The paper systematically evaluates pretraining settings and stabilization techniques across BLURB, finding setting-dependent strategies that improve stability and establish state-of-the-art results. It also identifies adversarial training and prompt-based learning in biomedical NLP as future work.
Problem
Biomedical NLP fine-tuning is difficult because rapidly growing biomedical text requires curation while task-specific labeled datasets are scarce.
Method
The paper systematically studies fine-tuning stability across BLURB’s six tasks and thirteen datasets, varying pretraining settings, model sizes, and stabilization techniques.
Results
Fine-tuning instability is prevalent in low-resource biomedical tasks and worsens with alternative pretraining settings and larger models, while layer-specific adaptations improve stability and benchmark performance.
Takeaways & Limitations
Freezing lower layers, layerwise decay, or top-layer reinitialization should be selected according to model configuration and end task, while domain-specific pretraining produces more robust models.
Takeaways & Limitations
The study leaves adversarial training and prompt-based learning in biomedical NLP for future work.
Abstract
from arXiv · showhide
Motivation: A perennial challenge for biomedical researchers and clinical practitioners is to stay abreast with the rapid growth of publications and medical notes. Natural language processing (NLP) has emerged as a promising direction for taming information overload. In particular, large neural language models facilitate transfer learning by pretraining on unlabeled text, as exemplified by the successes of BERT models in various NLP applications. However, fine-tuning such models for an end task remains challenging, especially with small labeled datasets, which are common in biomedical NLP. Results: We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that finetuning performance may be sensitive to pretraining settings, especially in low-resource domains. Large models have potential to attain better performance, but increasing model size also exacerbates finetuning instability. We thus conduct a comprehensive exploration of techniques for addressing fine-tuning instability. We show that these techniques can substantially improve fine-tuning performance for lowresource biomedical NLP applications. Specifically, freezing lower layers is helpful for standard BERT-BASE models, while layerwise decay is more effective for BERT-LARGE and ELECTRA models. For low-resource text similarity tasks such as BIOSSES, reinitializing the top layer is the optimal strategy. Overall, domainspecific vocabulary and pretraining facilitate more robust models for fine-tuning. Based on these findings, we establish new state of the art on a wide range of biomedical NLP applications. Availability and implementation: To facilitate progress in biomedical NLP, we release our state-of-the-art pretrained and fine-tuned models: https://aka.ms/BLURB.
1 Introduction
Biomedical text is expanding faster than manual curation can scale, while scarce labeled data makes biomedical NLP fine-tuning especially difficult. This paper systematically studies how pretraining choices and stabilization methods affect fine-tuning across biomedical applications.
- Motivation: Biomedical publications and clinical notes are growing rapidly, making manual curation difficult to scale.PubMed adds thousands of papers daily, while clinical records provide steadily expanding text resources.
- Motivation: NLP and self-supervised pretraining offer a route to automatically extract candidate findings and transfer knowledge from unlabeled text.Large neural language models have succeeded across NLP applications, but end-task fine-tuning remains challenging.
- Research gap: Biomedical NLP has unusually scarce task-specific data because annotation requires domain expertise; BIOSSES contains 100 examples versus 8,628 in general-domain STS.This data scarcity makes low-resource biomedical fine-tuning a central challenge.
- Study scope: The study evaluates fine-tuning stability using BLURB, a comprehensive biomedical benchmark spanning 6 tasks and 13 datasets.The benchmark provides a common basis for systematic comparison across biomedical applications.
- Findings: Pretraining settings strongly affect low-resource fine-tuning: segment IDs matter for some semantic tasks, while larger models and ELECTRA can worsen instability.Skipping next-sentence prediction has negligible effect, but BERT-LARGE and ELECTRA often hurt downstream performance in this setting.
- Findings: Layer-specific stabilization methods have setting-dependent benefits, and domain-specific vocabulary and pretraining support robust models that achieve state-of-the-art biomedical NLP performance.Freezing lower layers helps BERT-BASE, layerwise decay helps BERT-LARGE and ELECTRA, and top-layer reinitialization is optimal for low-resource text similarity.
2 Methods
The paper reviews transformer language-model components and pretraining objectives, then studies fine-tuning stability and layer-specific adaptations across biomedical tasks. Its methodology emphasizes comparisons across model sizes, pretraining settings, and data regimes.
- Neural language models: The study uses uncased WordPiece tokenization, whose vocabulary is built by selecting subword combinations through a unigram-language-model likelihood criterion.The vocabulary is initialized from characters and delimiters and expanded until a prespecified size is reached.
- Neural language models: BERT uses transformer layers with token, position, and segment embeddings to produce contextual representations for all input tokens.BERT-BASE has 12 layers and 110 million parameters, whereas BERT-LARGE has 24 layers and 340 million parameters.
- Pretraining objectives: BERT pretraining combines masked language modeling, which predicts replaced tokens, with next-sentence prediction, which classifies sentence-order relationships.The paper ablates NSP and segment IDs to assess their effects on downstream fine-tuning.
- Pretraining objectives: ELECTRA pretrains a discriminator using tokens proposed by an MLM-based generator, then discards the generator before downstream fine-tuning.The discriminator classifies whether each token is original, rather than predicting only masked positions.
- Experimental motivation: Biomedical datasets are often much smaller than general-domain counterparts, motivating systematic evaluation of advanced stabilization techniques beyond simple optimization adjustments.The paper contrasts BIOSSES with STS and notes that BLURB question-answering datasets contain only a few hundred instances.
- Fine-tuning stabilization: The study compares freezing lower layers, layerwise learning-rate decay, and top-layer reinitialization as layer-specific stabilization methods.These methods are motivated as ways to mitigate optimization difficulties associated with deep networks.
3 Results
The study evaluates how pretraining settings, optimization, model size, and layer-specific adaptation affect biomedical NLP fine-tuning stability. Stabilization techniques and domain-specific pretraining improve performance, culminating in a new BLURB state of the art.
- Study setup: BLURB spans six tasks and thirteen datasets, including low-resource sentence similarity and question-answering applications.The study uses BLURB to assess fine-tuning stability across biomedical NLP settings.
- Alternative pretraining settings: NSP usually has little effect, but single-sequence pretraining substantially reduces BIOSSES performance while improving PubMedQA performance.Separate segment IDs remain better for BIOSSES and BioASQ, whereas single-sequence choices can benefit PubMedQA.
- Alternative pretraining settings: ELECTRA performs worse on most biomedical tasks, suffers a catastrophic text-similarity drop, and is generally harder to fine-tune than MLM-based BERT.The paper reports no significant biomedical advantage for ELECTRA without stabilization techniques.
- Optimization adjustments: Optimization adjustments prevent some catastrophic drops but do not always stabilize fine-tuning, particularly under alternative pretraining settings.The adjustments include longer training and ADAM bias correction; BIOSSES remains sensitive when either is omitted.
- Layer-specific adaptation: Freezing lower layers helps standard BERT models, layerwise decay benefits ELECTRA models, and top-layer reinitialization is optimal for sentence similarity.The effects vary across tasks and pretraining settings, with layer-specific methods substantially reducing instability.
4 Discussion
The paper contrasts prior general-domain stability findings with biomedical NLP, where instability affects even BASE models and is intensified by pretraining choices and model scale. It evaluates layer-specific adaptations and identifies task- and setting-dependent stabilization strategies.
- Biomedical fine-tuning instability can affect BASE models, unlike prior findings that emphasized LARGE models and simple optimization adjustments.The paper finds longer training and ADAM debiasing helpful but insufficient for biomedical stabilization.
- Layer-specific adaptation substantially improves stabilization, with the best strategy depending on the task and pretraining setting.The study focuses on methods that are generalizable and easily implemented.
- Multi-task learning can mitigate low-resource biomedical challenges but generally requires multiple related datasets, such as for named-entity recognition.The paper notes that NER tasks are relatively easy with domain-specific pretrained models.
- The study leaves adversarial training and prompt-based learning in biomedical NLP for future work.Both methods are identified as potentially useful for instability or low-resource settings.
5 Conclusion
The paper presents a comprehensive study of fine-tuning large neural language models for biomedical NLP. It finds instability is prevalent in low-resource tasks, worsened by alternative pretraining settings and LARGE models, and uses stabilization findings to establish new BLURB state of the art.
- Fine-tuning instability is prevalent for low-resource biomedical tasks and is further exacerbated by alternative pretraining settings and LARGE models.
- Systematic evaluation identifies best practices for stabilization and establishes new state of the art on the BLURB benchmark.