PaperlayerView exact revision on arXiv

Source-linked AI summary

BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model

Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, Sheng Yu

arXiv:2204.03905v2cs.CL

TL;DR

Biomedical NLG lacks in-domain generative models and systematic benchmarks despite its importance. The paper adapts BART into BioBART through PubMed pretraining, collates biomedical generation tasks, and finds improved performance over BART, while sentence permutation degrades downstream results.

  • Problem

    Biomedical NLG is important but lacks in-domain generative language models and systematic generative downstream benchmarks.

  • Method

    The paper continuously pretrains BART on PubMed abstracts, collates biomedical NLG tasks, and evaluates BioBART across them.

  • Results

    BioBART achieves competitive or superior results across biomedical benchmarks compared with BART and current state-of-the-art methods.

  • Takeaways & Limitations

    The released benchmarks, code, and weights provide baselines and resources for future biomedical generative-language research.

  • Takeaways & Limitations

    BioBART pretrained on PubMed scientific articles still faces domain shift on clinical datasets and metrics.

Abstract

from arXiv · show

Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.

1 Introduction

Biomedical NLG is important but lacks generative biomedical models and systematic benchmarks. BioBART adapts BART to PubMed abstracts, evaluates diverse biomedical NLG tasks, and reports stronger performance than BART.

  • Research gap: Biomedical NLG is important, but existing biomedical pretrained models are predominantly encoder-only and cannot generate language autoregressively.The passage states that decoder-based generation is necessary for autoregressive language generation.
  • Approach: BioBART adapts BART to the biomedical domain through continuous pretraining on PubMed abstracts using text infilling.BART provides the generative pretrained backbone for biomedical domain adaptation.
  • Benchmarking: The study collates existing biomedical NLG tasks and their data and experimental settings for evaluation and release.The archived biomedical tasks are intended to support future research.
  • Findings: Sentence permutation in BART pretraining degrades performance on biomedical NLG tasks.This finding comes from an ablation analysis of BART’s pretraining tasks.
  • Findings: BioBART models demonstrate superior performance over BART across evaluated biomedical NLG tasks.The authors also plan to release code and weights for reproducibility.

2 Related Work

Related work established domain-specific biomedical language models and generative architectures, but biomedical research emphasized NLU more than NLG. BioBART instead uses BART and evaluates a broader set of biomedical generation tasks.

  • Auto-regressive Language Models: BERT and RoBERTa are encoder-only autoencoding transformers, whereas BART and T5 use encoder-decoder architectures for language generation.BART combines bidirectional input encoding with left-to-right autoregressive decoding.
  • Biomedical Pretraining: Biomedical domain adaptation has used PubMed, PMC, and EMR corpora with models including BioBERT, BlueBERT, ClinicalBERT, SciBERT, and PubMedBERT.These approaches include both continuous pretraining and training from scratch.
  • Biomedical Generative Models: SciFive supports biomedical NLG through T5, but its evaluation focuses on NLU rather than a broad set of generation tasks.BioBART chooses BART and emphasizes biomedical NLG evaluation to leverage decoder-based generation.
  • Biomedical NLG Benchmarks: Biomedical NLG benchmarks include dialogue, medical summarization, and figure-caption tasks, but remain less established than biomedical NLU benchmarks.BLUE and CBLUE are cited as archived biomedical NLU benchmarks.
  • Generative Reformulation: Generative methods have achieved state-of-the-art results for biomedical entity linking and named entity recognition.The passage identifies GENRE, Yuan et al., and BARTNER as examples of autoregressive approaches.

3 Biomedical Domain Pretraining

BioBART continuously pretrains BART on biomedical text through denoising, while examining how BART’s corruption choices affect biomedical downstream tasks.

  • BART Pretraining: BART uses a bidirectional encoder and left-to-right autoregressive decoder in a sequence-to-sequence architecture.Its pretraining denoises corrupted input documents.
  • Corruption Tasks: Text infilling replaces sampled token spans with single mask tokens, while sentence permutation randomly shuffles document sentences.These are the two corruption methods retained for the described pretraining setup.
  • Biomedical Adaptation: BioBART continuously pretrains BART on biomedical-domain corpora rather than training a new model from scratch.The authors revisit corruption methods because sentence permutation may help summarization but slightly degrade other tasks.

4 Generative Downstream Task

The study evaluates BioBART on biomedical dialogue, summarization, entity linking, and named entity recognition as generative sequence-to-sequence tasks.

  • Dialogue: Biomedical dialogue systems generate replies autoregressively from patient descriptions and dialogue histories.The task is trained and evaluated in a sequence-to-sequence fashion.
  • Summarization: Biomedical summarization generates concise summaries of knowledge-rich documents, with BioBART encoding inputs and decoding summaries autoregressively.The evaluation follows the BART summarization setup.
  • Entity Linking: Entity linking maps entity mentions in text to standard entity concepts and can use autoregressive generation of concept names.The study includes generative entity-linking methods to evaluate BioBART.
  • Named Entity Recognition: Biomedical NER extracts biomedical entities, including nested and discontinuous entities, from text.BARTNER formulates this setting as sequence-to-sequence generation of entities and types.

5 Experiments

BioBART is pretrained on PubMed abstracts using text infilling, with PubMed selected over combined PubMed and PMC corpora based on prior performance findings.

  • Pretraining Corpora: PubMed abstracts are selected as the sole pretraining corpus because combining PubMed and PMC previously slightly degraded performance.The corpus contains about 41 GB of biomedical research paper abstracts.
  • Pretraining Setup: Training uses a 0.02 warm-up ratio, linear decay, and learning rate 1e-4 on 16 40GB A100 GPUs across 2 DGX systems.The base model trains for about 100 hours and the large model for 168 hours using DeepSpeed.

5.2 Dataset for Downstream Task

The downstream benchmark spans biomedical dialogue, summarization, entity linking, and named entity recognition, using English datasets from clinical, research, and social-media sources.

  • Dialogue: CovidDialog contains 603 COVID-19-related consultations and 1232 utterances collected from online healthcare forums.Each consultation begins with a patient medical-condition description followed by doctor–patient dialogue.
  • Summarization: iCliniq and HealthCareMagic contain 31,062 and 226,405 patient–doctor dialogue samples, respectively, each paired with a summary.HealthCareMagic summaries are more abstractive and formal, whereas iCliniq summaries are patient-written.
  • Summarization: MeQSum provides 1000 verbose patient questions annotated with expert-written question summaries.The dataset targets question summarization because verbose questions challenge question-answering systems.
  • Summarization: MEDIQA-ANS contains 156 health questions, answers, and expert-created abstractive and extractive answer summaries.MEDIQA-QS uses MeQSum for training, while MEDIQA-MAS uses MEDIQA-ANS for training and targets multi-answer summarization.
  • Entity Linking: Entity-linking benchmarks include MedMentions, BC5CDR, NCBI, COMETA, and AskAPatient, covering PubMed, Reddit, and social-media text.MedMentions contains over 350,000 linked mentions, while BC5CDR includes 1500 annotated PubMed abstracts.
  • Named Entity Recognition: NER benchmarks cover discontinuous adverse drug events in ShARe13, ShARe14, and CADEC, plus biological entities in GENIA.The datasets are English, and GENIA entities may be nested; fine-grained GENIA types are combined into five coarse-grained types.

5.3 Fine-tuning details

Fine-tuning uses sequence-to-sequence generation for dialogue and summarization, generative concept-name prediction for entity linking, and BARTNER for NER.

  • Dialogue: Dialogue fine-tuning feeds dialogue history to the encoder and generates responses autoregressively using negative log-likelihood.The model is trained for 20 epochs and uses beam search with size 5 during inference.
  • Summarization: Summarization encodes documents and decodes summaries with a log-likelihood objective and beam search.Beam size is 5 without length penalty; base and large models use learning rates 5e-5 and 1e-5 for 6 epochs.
  • Entity Linking: Entity linking marks mention positions in encoder inputs and decodes corresponding knowledge-base synonyms directly.Evaluation uses Recall@1 and Recall@5, without the knowledge-base-guided pretraining used by Yuan et al. (2022a).
  • Named Entity Recognition: NER uses BARTNER, outputs the first BPE of each entity word, and is evaluated with entity-level F1.The model is fine-tuned for 30 epochs using parameters selected by Yan et al. (2021).

5.4 Main Result

BioBART is evaluated against BART and existing state-of-the-art systems across biomedical dialogue, summarization, entity linking, and NER. It generally improves performance, while BERTScore remains difficult to interpret because of possible biomedical domain-shifting in RoBERTa.

  • Dialogue: 1.71 and 0.03: Large BioBART exceeds BART on CovidDialog by these amounts on Rouge-2 and Rouge-L, respectively.Evaluation also surpasses the current state-of-the-art BLEU score by 4.45.
  • Summarization: BioBART shows consistent summarization improvements across datasets except iCliniq and HealthCareMagic, with MeQSum large exceeding BART large by 1.93/1.31/2.1 on Rouge-1/2/L.The authors attribute the exceptions to domain shifting from PubMed scientific articles to clinical corpora.
  • Evaluation Caveat: BERTScore changes only slightly across models, possibly because implemented RoBERTa suffers biomedical domain-shifting and may not quantify performance accurately.This caveat affects dialogue and summarization evaluations.
  • Entity Linking: 0.4, 1.67, and 1.36 Recall@1 points: BioBART surpasses discriminative state-of-the-art methods on AAP, BC5CDR, and COMETA, respectively.BioBART-based models have better performance across all reported entity-linking tasks.
  • NER: BioBART significantly improves entity-level F1 on ShARe13, ShARe14, and GENIA, but gains are mediocre on CADEC and remain behind the current state-of-the-art NER method.For large models, improvements are 1.06 on ShARe13 and 1 on GENIA.

5.5 Ablation Study on Pretraining Task

The ablation study compares BART pretraining with and without sentence permutation. Text infilling alone performs best, while sentence permutation degrades biomedical NLG performance.

  • Ablation Result: The model pretrained with isolated text infilling performs best in the ablation study.The comparison uses models fine-tuned on downstream tasks after pretraining with different task configurations.
  • Ablation Result: Sentence permutation downgrades performance on biomedical NLG tasks, including generative summarization and dialogue.The result contrasts with BART’s original use of sentence permutation for summarization.

5.6 Generated example

Qualitative examples compare outputs from BART and BioBART on biomedical dialogue and summarization tasks. BioBART produces more accurate responses and summaries in the presented cases.

  • CovidDialog: BioBART correctly responds to the diabetes-influence example, whereas BART generates an erroneous instruction.The example is drawn from CovidDialog.
  • MeQSum: BioBART generates an accurate and concise summary when BART misunderstands that sugar alcohol is not the cause of dry mouth.This example is from the MeQSum task.
  • MEDIQA-ANS: For a long MEDIQA-ANS document, BioBART addresses information that BART fails to extract completely.The authors summarize these examples as improvements in biomedical common sense and document understanding.

6 Conclusions

The paper presents BioBART, a biomedical generative language model, and collects benchmarks for biomedical generative tasks. Experiments report strong benchmark performance, while future work targets broader vocabularies and clinical corpora.

  • Contributions: BioBART is pretrained as a biomedical-domain generative language model, and the authors collect publicly available benchmarks for biomedical generative tasks.The benchmark collection is intended to prompt future research.
  • Results: BioBART achieves competitive or superior results over current state-of-the-art methods across different benchmarks.The authors also report that continuous pretraining on PubMed abstracts helps domain adaptation.
  • Resources: The released pretraining and fine-tuning code is intended to facilitate reproducibility and future research.The paper states that codes and weights are released.
  • Future Work: Future studies will explore in-domain vocabularies trained from scratch and clinical corpora such as EMRs in MIMIC-III or PMC-Patients.These directions extend beyond the PubMed-abstract pretraining setting.
Loading 2204.03905v2…