Source-linked AI summary
Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
Yifan Peng, Shankai Yan, Zhiyong Lu
TL;DR
Biomedical language representation research lacked a public benchmark supporting fair comparison across biomedical and clinical text. The paper introduces BLUE, evaluates BERT and ELMo baselines across five tasks and ten datasets, and finds that BERT pretrained on PubMed abstracts and MIMIC-III performs best overall.
Problem
Biomedical language representation research lacked publicly available benchmarking for comparing models across biomedical and clinical text.
Method
The paper introduces BLUE, a benchmark with five tasks and ten corpora, and evaluates BERT and ELMo baselines using released datasets, code, and pretrained models.
Results
BERT-Base pretrained on PubMed abstracts and MIMIC-III achieves the best results across five tasks and is superior to other models in clinical and biomedical domains.
Takeaways & Limitations
BLUE provides resources for evaluating and analyzing biomedical language representation models and identifying future directions for their development.
Takeaways & Limitations
Performance on the very small BIOSSES test set was unstable, so the study selected the best development-set model after several runs.
Abstract
from arXiv · showhide
Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.
1 Introduction
The paper introduces BLUE to address the lack of public biomedical benchmarking and support evaluation of language representations across diverse biomedical and clinical tasks. It also establishes BERT and ELMo baselines, finding the strongest performance from BERT pretrained on PubMed abstracts and MIMIC-III clinical notes.
- The benchmark addresses the absence of publicly available benchmarking for language representations in biomedicine.
- BLUE provides five biomedical text-mining tasks spanning ten corpora across biomedical literature and clinical notes.The datasets vary in size and difficulty and represent common biomedicine text-mining challenges.
- BERT and ELMo baselines are evaluated to better understand the challenges posed by BLUE.Both are described as state-of-the-art language representation models in general-purpose NLP.
- BERT pretrained on PubMed abstracts and MIMIC-III clinical notes achieves the best results and is significantly superior to other models in the clinical domain.
- The paper releases datasets, data-construction and evaluation code, pretrained BERT models, and baseline results.
2 Related work
Related work develops biomedical language representations through transfer learning, but existing evaluations use inconsistent datasets and sizes. BLUE responds with a broader and more reproducible benchmark spanning biomedical and clinical text genres and additional task types.
- Transfer learning pretrains language representations on large corpora before fine-tuning them for downstream tasks such as named entity recognition and relation extraction.
- Word and sentence embeddings represent biomedical semantics but require complicated neural networks for effective downstream use.
- Context-dependent representations allow a word’s meaning to change with its surrounding context, and biomedical and clinical variants have been developed.Examples include BioBERT, BioELMo, and a clinical BERT model trained on MIMIC-III.
- Prior models are difficult to compare fairly because studies often evaluate different datasets or datasets with slightly different example sizes.
- BLUE differs by covering biomedical and clinical genres, including document classification, and providing code to reconstruct datasets without removing instances.
3 Tasks
BLUE organizes five biomedical text-mining tasks around established datasets, standard splits, task-specific annotations, and conventional evaluation metrics. The tasks cover sentence similarity, named entity recognition, relation extraction, multilabel classification, and inference across biomedical and clinical data.
- BLUE contains five tasks with ten corpora spanning broad data quantities and difficulty levels, using preexisting datasets widely used as BioNLP shared tasks.
- Sentence similarity: Sentence similarity predicts similarity scores for sentence pairs and is evaluated with Pearson correlation coefficients.BIOSSES uses curator scores from 0 to 4 and a random 80% training and 20% testing split, while MedSTS uses expert scores from 0 to 5 and standard shared-task splits.
- Named entity recognition: Named entity recognition predicts mention spans and uses strict precision, recall, and F1-score against annotated spans.spaCy tokenizes datasets lacking original token information; BC5CDR contains 1,500 PubMed titles and abstracts, and ShARe/CLEF contains 299 deidentified clinical notes.
- Relation extraction: Relation extraction predicts typed relations between entity pairs and is evaluated with standard micro-average precision, recall, and F1-score.The benchmark includes DDI, ChemProt, and i2b2 corpora with expert or practitioner annotations and shared-task data splits or subsets.
- Multilabel classification and inference: Multilabel classification predicts multiple labels from texts, while inference predicts whether a premise entails or contradicts a hypothesis using overall accuracy.HoC contains abstracts annotated with ten cancer hallmarks, and MedNLI contains clinically sourced sentence pairs with entailment, contradiction, or neutral labels.
4 Baselines
The study evaluates BERT and ELMo baselines for BLUE tasks, using domain-specific pre-training and task-specific fine-tuning procedures.
- BERT baseline: BERT is a contextualized word representation model pre-trained with masked language modeling and bidirectional Transformers.
- BERT baseline: The authors continued pre-training BERT initialized from Devlin et al. on PubMed abstracts and MIMIC-III clinical notes.
- Task adaptation: BERT fine-tuning packs sentence pairs into one sequence, predicts BIO tags for named entities, and masks relation-extraction entities with predefined tags.
- Task adaptation: For multi-label tasks, BERT predicts sentence-level labels and combines them at the document level for comparison with gold standards.
- ELMo baseline: ELMo uses concatenated outputs from both layers as token inputs, while task models use averaged or combined embeddings for downstream prediction.
5 Benchmark results and discussion
Across BLUE, domain-pretrained BERT models perform strongly, with BERT-Base using PubMed and MIMIC-III achieving the best overall results despite dataset- and task-specific variation.
- Overall results: BERT-Base (P+M) achieved the best results across five tasks, although it was only slightly better than BERT-Base (P).
- Model size: BERT-Base generally outperformed BERT-Large, except in relation extraction and document classification tasks.
- Model size: BERT-Large (P) performed best in the multilabel task, while BERT-Large (P+M) performed worse overall than other models.
- Sentence similarity: BERT-Base (P+M) achieved the best results on both sentence similarity datasets.
- Sentence similarity: BIOSSES contains only 16 test sentence pairs, making BERT performance unstable across runs.
- Named entity recognition: BERT-Base (P) performed best on two biomedical named-entity datasets, whereas BERT-Base (P+M) performed best on the clinical dataset.
6 Conclusion
The paper introduces BLUE as a resource for evaluating and analyzing biomedical natural language representation models.
- BLUE provides resources for evaluating and analyzing biomedical natural language representation models.
- The authors report that BERT models pre-trained on PubMed abstracts and clinical notes outperform most state-of-the-art models.
- The benchmark is intended to evaluate models’ understanding of biomedical text and inform future development of biomedical language representations.
A Appendices
The appendix defines abbreviations used for evaluation metrics and classification outcomes.
- TP, FP, and FN denote true positive, false positive, and false negative, respectively.
- P, R, and F1 denote precision, recall, and F1-score, respectively.
A.1 Named Entity Recognition
The named entity recognition section covers disease, chemical, and clinical concept datasets, each reported with TP, FP, FN, precision, recall, and F1.
- A.1 Named Entity Recognition: BC5CDR-disease is evaluated using TP, FP, FN, precision, recall, and F1.
- A.1 Named Entity Recognition: BC5CDR-chemical is evaluated using TP, FP, FN, precision, recall, and F1.
- A.1 Named Entity Recognition: ShARe/CLEFE is evaluated using TP, FP, FN, precision, recall, and F1.
A.2 Relation extraction
The relation extraction section covers DDI, Chem-Prot, and i2b2 datasets, each reported with TP, FP, FN, precision, recall, and F1.
- A.2 Relation extraction: DDI is evaluated using TP, FP, FN, precision, recall, and F1.
- A.2 Relation extraction: Chem-Prot is evaluated using TP, FP, FN, precision, recall, and F1.
- A.2 Relation extraction: i2b2 is evaluated using TP, FP, FN, precision, recall, and F1.