Source-linked AI summary
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, Jaewoo Kang
TL;DR
Biomedical text mining needs domain-adapted language models because general-domain pre-training does not fully address biomedical word distributions. BioBERT continues BERT pre-training on biomedical corpora and outperforms previous models across NER, relation extraction, and question answering.
Problem
General-domain NLP models face limitations on biomedical text because biomedical and general corpora have substantially different word distributions.
Method
BioBERT initializes BERT and continues pre-training on PubMed and PMC biomedical corpora before fine-tuning across NER, relation extraction, and question answering.
Results
BioBERT outperforms previous models across NER, relation extraction, and question answering, including a 2.80 higher average micro F1 score in relation extraction.
Takeaways & Limitations
Pre-training BERT on biomedical corpora is crucial for biomedical text mining while requiring minimal task-specific architectural modification.
Takeaways & Limitations
The experiments use only BERTBASE because BERTLARGE was computationally too complex.
Abstract
from arXiv · showhide
Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.
1 Introduction
The rapidly expanding biomedical literature creates a growing need for effective biomedical text mining. However, general-domain language representations may transfer poorly because biomedical and general corpora have different word distributions, motivating biomedical-domain pre-training.
- Motivation: More than 3000 peer-reviewed biomedical articles are published daily, while PubMed contained 29M articles as of January 2019.The literature also continually adds reports describing new discoveries and insights.
- Background: Deep learning advances in NLP, including LSTM and CRF, have improved biomedical named entity recognition.Other deep learning-based models have also improved biomedical text mining performance.
- Problem: General-domain representations such as Word2Vec, ELMo, and BERT may perform poorly on biomedical text because their training data differ from biomedical corpora.The passage identifies substantial differences between general and biomedical word distributions as a central limitation.
- Motivation: The study hypothesizes that state-of-the-art representations such as BERT should be trained on biomedical corpora to work effectively for biomedical text mining.Biomedical corpora contain specialized terms and expressions often absent from general-domain corpora.
2 Approach
BioBERT adapts BERT to biomedical text by initializing from general-domain weights and further pre-training on PubMed and PMC corpora. The approach uses nearly unchanged architecture while delivering state-of-the-art performance across multiple biomedical text mining tasks and releasing its resources publicly.
- Pre-training and fine-tuning: BioBERT initializes from BERT weights trained on English Wikipedia and BooksCorpus, then pre-trains on PubMed abstracts and PMC full-text articles.This adapts a general-domain language model to biomedical corpora.
- Contributions: BioBERT was pre-trained for 23 days on eight NVIDIA V100 GPUs as a domain-specific BERT model for biomedical corpora.The paper presents BioBERT as the first domain-specific BERT-based model pre-trained on biomedical corpora.
- Contributions: 0.62 F1 improvement in biomedical NER, 2.80 F1 improvement in biomedical RE, and 12.24 MRR improvement in biomedical QA demonstrate gains over current state-of-the-art models.The reported improvements follow pre-training BERT on biomedical corpora.
- Contributions: BioBERT achieves state-of-the-art performance across biomedical text mining tasks with only minimal architectural modifications and makes datasets, pre-trained weights, and fine-tuning code publicly available.This contrasts with prior models that mainly focused on a single task such as NER or QA.
3 Materials and methods
BioBERT retains BERT’s architecture while adapting its pre-training to biomedical corpora and applying minimal task-specific changes across biomedical text mining tasks. Its methods combine masked-language-model pre-training, WordPiece tokenization, and fine-tuning for NER, relation extraction, and question answering.
- Model design: BioBERT largely preserves BERT’s structure and uses minimal architectural modification for downstream biomedical text mining tasks.The model is described alongside BERT before its pre-training and fine-tuning procedures.
- BERT background: BERT uses a masked language model with bidirectional transformers to predict randomly masked words in a sequence.This formulation avoids the limitation of combining separate left-to-right and right-to-left language models.
- Pre-training BioBERT: BioBERT is pre-trained on PubMed abstracts and PubMed Central full-text articles because biomedical texts contain domain-specific proper nouns and terminology.The paper contrasts these biomedical corpora with BERT’s general-purpose pre-training on English Wikipedia and BooksCorpus.
- Pre-training BioBERT: BioBERT uses cased WordPiece tokenization to represent unseen biomedical words through frequent subwords and mitigate out-of-vocabulary issues.The supplied passage gives “Immunoglobulin” as an example of subword decomposition and reports slightly better downstream performance with cased vocabulary.
- Fine-tuning BioBERT: BioBERT is fine-tuned for biomedical named entity recognition, relation extraction, and question answering using task-specific output layers or classifiers.Relation extraction uses a [CLS]-based sentence classifier and anonymized target entities, while QA predicts answer-span start and end locations; approximately 30% of BioASQ factoid questions were excluded as unanswerable.
4 Results
The experiments evaluate BERT and BioBERT across biomedical named entity recognition, relation extraction, and question answering tasks, using domain-specific pre-training and task-specific fine-tuning. BioBERT consistently improves over BERT and state-of-the-art models, with especially strong gains in relation extraction and question answering.
- 4.2 Experimental setups: BioBERT v1.0 was trained on PubMed and PMC for 470K steps, while its single-corpus versions used 200K PubMed steps or 270K PMC steps.BERTBASE was pre-trained on English Wikipedia and BooksCorpus for 1M steps.
- 4.2 Experimental setups: Only BERTBASE was used because BERTLARGE was computationally complex.This limited the comparison to the base BERT architecture.
- 4.3 Experimental results: BioBERT outperformed the state-of-the-art models on six out of nine biomedical NER datasets and achieved higher scores than BERT on all datasets.BERT’s micro-averaged F1 score was 2.01 lower than that of the state-of-the-art models.
- 4.3 Experimental results: 2.80 higher: BioBERT v1.0 (þ PubMed) obtained a higher average micro F1 score than the state-of-the-art models in biomedical relation extraction.BioBERT achieved the highest F1 scores on 2 out of 3 biomedical datasets.
- 4.3 Experimental results: 38.77 strict accuracy, 53.81 lenient accuracy, and 44.77 mean reciprocal rank: BioBERT v1.1 (þ PubMed) achieved these micro-averaged QA scores.All BioBERT versions significantly outperformed BERT and the state-of-the-art models.
5 Discussion
The discussion examines how biomedical pre-training data affects BioBERT and illustrates qualitative gains over BERT in downstream predictions. Larger PubMed pre-training corpora improve performance, while BioBERT better recognizes biomedical entities and answers biomedical questions.
- Corpus-size effects: Pre-training on 1 billion words of PubMed is quite effective across the NCBI Disease, BC2GM, and BC4CHEMD NER datasets.BioBERT v1.0 (þ PubMed) used 200K pre-training steps while varying PubMed corpus size.
- Prediction analysis: BioBERT recognizes biomedical named entities that BERT misses and identifies their exact boundaries.This comparison was based on sampled predictions from BERT and BioBERT v1.1 (þPubMed).
- Prediction analysis: BioBERT answers simple biomedical questions correctly when BERT often answers incorrectly.BioBERT can also provide longer named entities as answers.
6 Conclusion
The conclusion presents BioBERT as a biomedical language representation model and emphasizes that biomedical pre-training is crucial for applying BERT in this domain. With minimal architectural modification, BioBERT outperforms previous models across biomedical text-mining tasks including NER, RE, and QA.
- BioBERT is introduced as a pre-trained language representation model for biomedical text mining.
- Pre-training BERT on biomedical corpora is crucial for applying it effectively to the biomedical domain.
- With minimal task-specific architectural modification, BioBERT outperforms previous models on biomedical NER, RE, and QA tasks.