PaperlayerView exact revision on arXiv

Source-linked AI summary

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, Tie-Yan Liu

arXiv:2210.10341v3cs.CLcs.AI

TL;DR

Biomedical pre-trained models have largely emphasized discriminative understanding despite the need to mine and generate knowledge from rapidly growing literature. BioGPT addresses this gap with a generative Transformer trained on biomedical text, achieving state-of-the-art results on four benchmarks and improved biomedical generation ability.

  • Problem

    Prior biomedical pre-trained models mainly target discriminative understanding tasks, while biomedical literature also requires generation and mining capabilities.

  • Method

    BioGPT is a GPT-2-based generative Transformer trained from scratch on 15M PubMed abstracts with an in-domain vocabulary and task-specific sequence formats.

  • Results

    BioGPT achieves state-of-the-art results on BC5CDR, KD-DTI, DDI, and PubMedQA, and demonstrates better biomedical text generation than standard GPT-2.

  • Takeaways & Limitations

    BioGPT provides a generative pre-trained model for biomedical literature text generation and mining across relation extraction, question answering, classification, and generation tasks.

  • Takeaways & Limitations

    The text-generation task reports no objective evaluation metric and instead provides examples for demonstration.

Abstract

from arXiv · show

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e., BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large scale biomedical literature. We evaluate BioGPT on six biomedical NLP tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms. Code is available at https://github.com/microsoft/BioGPT.

Introduction

Biomedical literature contains extensive knowledge relevant to drug discovery and clinical research, creating an urgent need for automatic mining. BioGPT addresses the limited biomedical generation capability of prior domain-specific pre-trained models with a generative Transformer evaluated across six tasks.

  • Motivation: More than 30M PubMed articles support important biomedical applications, making automatic knowledge mining an urgent demand.Applications include named-entity recognition, interaction mining, question answering, and text generation.
  • Pre-trained Models: BERT-like models mainly support understanding tasks, whereas GPT-like models mainly support language generation.The two branches differ in their principal downstream capabilities.
  • Research Gap: Directly applying general-domain models to biomedicine suffers from domain shift, motivating biomedical-domain pre-training.BioBERT and PubMedBERT improve biomedical benchmark performance, but prior work primarily focuses on BERT-like understanding models.
  • BioGPT: BioGPT is a domain-specific generative Transformer pre-trained from scratch on 15M PubMed abstracts and evaluated on six biomedical NLP tasks.The tasks include relation extraction, question answering, document classification, and text generation.
  • Results: Experiments show BioGPT performs better than baseline and other strong methods across all evaluated tasks.The introduction reports broad gains across the six-task evaluation.

Related Work

Related work establishes the value of domain-specific pre-training for biomedical language processing while highlighting a relative lack of biomedical generative models. The paper frames downstream biomedical tasks as text generation or mining problems.

  • General Pre-training: Pre-training transfers representations learned from large unlabeled corpora to downstream tasks through fine-tuning.BERT uses masked language modeling and next-sentence prediction to learn contextualized representations.
  • GPT Models: GPT models use autoregressive Transformer language modeling for generation tasks and later versions improve performance with larger models and corpora.Examples include translation and summarization.
  • Biomedical Pre-training: Biomedical BERT models improve over general-domain models through continued or from-scratch pre-training on biomedical literature.Domain-specific vocabulary and data are emphasized, but these works do not target biomedical generation tasks.
  • Biomedical GPT: General-domain GPT models perform poorly on biomedical tasks, while prior biomedical GPT pre-training used only 0.5M PubMed abstracts for data augmentation.This motivates larger-scale biomedical generative pre-training.
  • Downstream Tasks: The paper formulates relation extraction, question answering, and document classification as text generation or mining tasks.Relation extraction methods include pipeline, joint, sequence-labeling, table-filling, and text-generation approaches.

Pre-training Method

BioGPT is trained from scratch on in-domain PubMed data using a learned biomedical vocabulary and a GPT-2-based Transformer decoder. Training minimizes the standard autoregressive language-model negative log-likelihood.

  • Dataset: BioGPT uses 15M nonempty PubMed items updated before 2021, each containing a title and abstract.The corpus was collected from the official PubMed site and filtered to remove title-only items.
  • Vocabulary: The model learns a 42,384-token byte-pair encoding vocabulary from the collected biomedical corpus.The vocabulary is trained in-domain rather than inherited from GPT-2.
  • Model: BioGPT adopts GPT-2medium as its backbone, with 24 layers, 1024 hidden units, and 16 attention heads.The backbone has 355M parameters, while BioGPT has 347M because its vocabulary changes embedding and output-projection sizes.
  • Architecture: Multihead attention splits Q, K, and V across heads, concatenates the head outputs, and applies an affine transformation.The resulting attention output is fed to a feed-forward layer to form a Transformer block.
  • Training Objective: The training objective minimizes the negative log-likelihood of token sequences under standard autoregressive language modeling.Each token is modeled conditionally on preceding tokens.

Fine-tuning Method

BioGPT adapts heterogeneous biomedical tasks to generative modeling by converting labels and structured outputs into natural-language target sequences. Task-specific prompts then guide generation during downstream fine-tuning and inference.

  • General Adaptation: Downstream labels are converted into sequences so task formats match BioGPT’s sequence-generation pre-training objective.Natural-language labels are chosen because they are semantically smoother than special-token formats.
  • End-to-end Relation Extraction: Relation extraction represents triplets as natural-language sentences using subject-verb-object, is-of, or relation-is forms.Multiple triplets are ordered by document appearance and concatenated with semicolons.
  • End-to-end Relation Extraction: Natural-language relation sentences can be converted back into triplets with regular expressions, while customized formats remain possible.The examples encode drug-target interactions in sentence form.
  • Question Answering: For question answering, the source prefixes the question and context, while the target states the answer label in a natural-language template.The supported labels are yes, no, and maybe.
  • Document Classification: For document classification, BioGPT generates a target sentence containing the document’s category label.The template is “the type of this document is label”.
  • Prompting: Naively concatenating source and target sequences makes inference difficult because the model lacks task-specific output guidance.Prompting addresses this generation problem by adding task-specific instructions.
  • Prompting: BioGPT mainly uses learned soft prefix prompts, placing continuous virtual tokens before the target sequence and training them end-to-end for each task.This differs from prefix-tuning that places virtual tokens at the very beginning of the source input.

Experiments

BioGPT is evaluated against GPT-2 and prior biomedical methods across relation extraction, question answering, document classification, and text generation. It achieves strong results across these experiments, including state-of-the-art performance on four of six tasks.

  • Experimental setup: BioGPT is compared with general-domain GPT-2 and prior task-specific methods across six biomedical NLP tasks.The evaluations cover relation extraction on BC5CDR, KD-DTI, and DDI; PubMedQA; HoC classification; and text generation.
  • KD-DTI: 38.42% F1 is BioGPT’s KD-DTI result, improving over Transformer + PubMedBERT-attn, GPT-2medium, and REBEL by 14.23%, 9.97%, and 8.03%.It also surpasses REBELpt by 5.1% without additional relation-extraction pre-training.
  • DDI: 40.76% micro-F1 is BioGPT’s DDI result, improving over GPT-2medium and REBEL by 16.08% and 12.49%.BioGPT also surpasses REBELpt, which uses an additional large relation-extraction dataset for two-stage pre-training.
  • Question answering and classification: 78.2% accuracy on PubMedQA improves over the previous best performance by 6.0%, establishing a new state of the art.BioGPT also achieves 85.12% accuracy on HoC, surpassing general-domain GPT-2 and several biomedical models.
  • Text generation: BioGPT generates more specific and professional biomedical descriptions than GPT-2, especially for uncommon and domain-specific names.For very uncommon names, GPT-2 fails to generate informative text while BioGPT remains able to produce related descriptions.

Ablation Study

The ablation study examines target-sequence formats and prompt designs for downstream BioGPT tasks. Natural-language target formats, especially rel-is, and continuous soft prompts perform best in the reported experiments.

  • Target Sequence Format: BioGPT uses a unified module, so natural-language target formats better match the format consistency between inputs and generated answers.Previous encoder-decoder methods could separately process natural-language inputs and structured outputs, whereas BioGPT encodes context and generates answers with one module.
  • Target Sequence Format: Natural-language formats outperform structured formats on KD-DTI, with rel-is performing best by F1.The rel-is format provides a semantically smooth and clear description.
  • Target Sequence Format: 42.85 and 38.60 F1 are obtained by the structured format on BC5CDR and DDI, respectively.The passage contrasts these scores with the corresponding rel-is results, whose values are truncated in the supplied text.
  • Prompt Design: Continuous embeddings with length 13 virtual tokens are the best-performing prompt on KD-DTI.The experiment compares manually designed hard prompts with continuous embedding soft prompts while fixing the target format to rel-is.

Conclusion

The paper concludes that BioGPT is a generative Transformer language model pretrained on biomedical abstracts and evaluated across several biomedical NLP tasks. It achieves state-of-the-art results on four benchmarks and better biomedical text generation than GPT-2.

  • Conclusion: BioGPT is a generative pre-trained Transformer language model for biomedical text generation and mining.The model uses GPT-2 as its backbone and is pretrained from scratch on 15M PubMed abstracts.
  • Conclusion: BioGPT is applied to end-to-end relation extraction, question answering, document classification, and text generation tasks.The conclusion lists these as the downstream biomedical NLP applications.
  • Conclusion: BioGPT demonstrates better biomedical text generation ability than GPT-2.The paper also studies prompt design and target-sequence design, finding natural-language-semantic targets better than structured prompts.
  • Conclusion: BioGPT achieves state-of-the-art results on BC5CDR, KD-DTI, DDI, and PubMedQA.The first three are end-to-end relation extraction benchmarks, while PubMedQA is a question answering benchmark.

Scaling to Larger Size

The paper scales BioGPT to a larger model variant and evaluates it on downstream tasks. BioGPT-Large is based on GPT-2 XL and contains 1.5B parameters.

  • Scaling to Larger Size: BioGPT-Large is built from the GPT-2 XL architecture and has 1.5B model parameters.Its downstream fine-tuning and evaluation results are reported in Table 11.
Loading 2210.10341v3…