Source-linked AI summary
SciFive: a text-to-text transformer model for biomedical literature
Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, Grégoire Altan-Bonnet
TL;DR
Biomedical text-generation methods underperform on tasks needed to process rapidly growing literature, motivating a domain-specific approach. SciFive adapts T5 using biomedical pretraining and text-to-text task formulations, achieving state-of-the-art or near-state-of-the-art results across several biomedical NLP tasks. The findings support applying domain-specific text generation to more difficult outputs, although the evaluation and demonstrated scope remain limited.
Problem
Existing methods generally underperform on biomedical text-generation tasks such as summarization and question answering, despite the need to process large literature collections.
Method
SciFive is a domain-specific T5 model pretrained on biomedical corpora and fine-tuned by converting biomedical NLP tasks into text-to-text generation problems.
Results
SciFive achieved state-of-the-art or near-state-of-the-art results across NER, relation extraction, natural language inference, document classification, and question answering.
Takeaways & Limitations
The results support further study of domain-specific text-generation models on more difficult tasks such as document summarization and abstract generation.
Takeaways & Limitations
For question answering, SciFive was evaluated only with lenient accuracy because its generated text cannot be assessed using BioBERT’s strict-accuracy procedure.
Abstract
from arXiv · showhide
In this report, we introduce SciFive, a domain-specific T5 model that has been pre-trained on large biomedical corpora. Our model outperforms the current SOTA methods (i.e. BERT, BioBERT, Base T5) on tasks in named entity relation, relation extraction, natural language inference, and question-answering. We show that text-generation methods have significant potential in a broad array of biomedical NLP tasks, particularly those requiring longer, more complex outputs. Our results support the exploration of more difficult text generation tasks and the development of new methods in this area
1 Introduction
Biomedical literature is increasingly accessible, but current methods underperform on text-generation tasks such as summarization and question answering. Domain-specific biomedical NLP models are therefore needed to handle this growing information volume.
- Biomedical databases provide rapid access to millions of journal articles for researchers.
- Text-generation tasks such as document summarization and question answering can help researchers quickly obtain information from large paper collections.
- Current methods generally underperform on these biomedical text-generation tasks.
- Biomedical language is challenging for NLP models because its vocabulary differs substantially from standard-language corpora.
2 Approach
SciFive adapts T5 into a domain-specific text-to-text model for biomedical literature. The model targets both conventional classification tasks and tasks requiring extended outputs, including question answering.
- SciFive is a pretrained, domain-specific adaptation of T5 designed for biomedical-literature tasks.
- The model achieves state-of-the-art results on common biomedical classification tasks, including named entity recognition and relation extraction.
- SciFive supports tasks requiring extended outputs and achieves superior BioASQ question-answering results compared with BioBERT.
3 Unlabeled Dataset
SciFive is pretrained on biomedical corpora chosen to provide broad domain coverage. The data combine PubMed abstracts with PMC full-text articles to support biomedical language modeling and generalization.
- SciFive uses two biomedical language corpora to generalize its pretrained model within the biomedical domain.
- PubMed contributes abstracts from a database containing more than 32 million biomedical-literature citations and abstracts.
- PMC provides free full-text biomedical and life-sciences articles for pretraining.
- The authors hypothesize that full-text training can improve biomedical-context learning while retaining a generalized natural-language representation.
4 Methods
SciFive retains T5’s encoder-decoder text-to-text framework and is pretrained and fine-tuned to generate task-specific biomedical outputs. Its methods include span masking, task-specific representations, subword vocabulary, and multitask training.
- 4.2 SciFive: SciFive retains the original T5 structure and parameters within a sequence-to-sequence encoder-decoder architecture.
- 4.2 SciFive: T5’s decoder enables sequence generation, while span-based language masking provides the self-supervised pretraining objective.
- 4.2 SciFive: SciFive was initialized from pretrained base T5 weights and retrained on combinations of C4, PubMed abstracts, and PMC full-text articles.
- 4.3 Input/Output Representation: During span-based masking, contiguous text spans are replaced by sentinel tokens and the model predicts the masked spans with those sentinels.
- 4.3 Input/Output Representation: Supervised training supplies text inputs and targets, generating marked entities for NER and answer text for question answering.
- 4.3 Input/Output Representation: SentencePiece supplies subword units that preserve semantic content while avoiding the need for an immense word-level vocabulary.
- 4.3 Input/Output Representation: Task-specific input tokens and teacher forcing support multitask learning across the biomedical NLP tasks.
- 4.1 Tasks: The study fine-tuned SciFive across five biomedical NLP task categories using both multitask and single-task learning.
5 Results
SciFive was evaluated across biomedical NER, relation extraction, inference, document classification, and question answering using task-specific datasets and comparisons with established baselines. The results report state-of-the-art performance across most evaluated task groups, while QA required lenient accuracy and expert assessment because SciFive generates text rather than answer probabilities.
- Evaluation used established biomedical datasets, baseline preprocessing, and task-specific metrics including F1, precision, recall, and lenient QA accuracy.
- SciFive’s text-to-text QA output prevented strict accuracy evaluation against BioBERT, so correctness was counted when an answer was correct for one or more contextual snippets.
- Expert assessment found that SciFive’s full-sentence answers were often scientifically correct despite differing from exact BioASQ reference phrasing, and examples showed clearer, more complete answers than BioBERT.
- SciFive achieved SOTA results on 3/7 NER tasks, 2/2 RE tasks, 1/1 NLI tasks, and 3/3 question answering tasks, with a near-SOTA document-classification result.
- Compared with BERT-based models, SciFive provided competitive classification results while also achieving SOTA performance on text-generation tasks such as question answering.
6 Discussion
SciFive achieves state-of-the-art results across biomedical NLP tasks, including question answering, but further evaluation is needed on more complex generation tasks and biomedical corpora.
- SciFive achieves state-of-the-art results on a variety of biomedical NLP tasks, particularly question answering.
- SciFive and T5 significantly outperform BioBERT on question answering, with SciFive producing clearer and more complete answers in examples.
- Question answering remains relatively simplistic compared with other text-generation tasks, such as document summarization and abstract generation.
- State-of-the-art results are split among SciFive versions, indicating that the optimal biomedical pretraining corpus requires further study.
7 Conclusion
SciFive is a domain-specific text-to-text model for biomedical literature that performs effectively across several NLP tasks. The results support broader investigation of domain-specific text generation for more difficult outputs.
- SciFive is a domain-specific text-to-text model trained specifically for tasks involving biomedical literature.
- SciFive achieves state-of-the-art or near-state-of-the-art results on named entity recognition, relation extraction, natural language inference, and question answering.
- The results support text-to-text models as versatile and broadly applicable within domain-specific contexts, including tasks requiring longer output sequences.
- Further study should examine domain-specific text generation models on more difficult tasks such as document summarization and abstract generation.