PaperlayerView exact revision on arXiv

Source-linked AI summary

SciBERT: A Pretrained Language Model for Scientific Text

Iz Beltagy, Kyle Lo, Arman Cohan

arXiv:1903.10676v3cs.CL

TL;DR

Scientific NLP lacks large-scale annotated data, while existing pretrained models use general-domain corpora. SciBERT pretrains a BERT-based model on scientific text and significantly outperforms BERT-Base, achieving new state-of-the-art results on several tasks.

  • Problem

    Scientific NLP lacks large-scale annotated data, and existing pretrained models are trained on general-domain corpora rather than scientific text.

  • Method

    SciBERT pretrains a BERT-based language model on 1.14 million scientific papers comprising 3.17 billion tokens.

  • Results

    +2.11 F1 with finetuning and +2.43 F1 without, compared with BERT-Base on scientific tasks, with new state-of-the-art results on many tasks.

  • Takeaways & Limitations

    SciBERT provides a scientific-text pretrained resource that achieves new state-of-the-art results on several scientific NLP tasks.

  • Takeaways & Limitations

    SciBERT performs slightly worse than state of the art on three datasets, partly because competing systems use additional training data or features.

Abstract

from arXiv · show

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.

1 Introduction

Scientific NLP needs large-scale knowledge extraction, but its task-specific annotations are difficult to obtain, while existing pretrained models rely on general-domain corpora. SciBERT addresses this gap with a BERT-based model pretrained on scientific text and evaluated across scientific NLP tasks.

  • Motivation: The growing volume of scientific publications makes NLP essential for large-scale knowledge extraction and machine reading, but scientific annotations are difficult to obtain.Deep neural models often require large amounts of labeled data, which can be crowdsourced more readily in general domains than in scientific domains.
  • Motivation: Unsupervised pretraining improves many NLP tasks, yet existing pretrained models such as BERT and ELMo are trained on general-domain corpora.Their contextualized token embeddings can be passed into minimal task-specific neural architectures, making pretraining useful when scientific annotations are scarce.
  • Contributions: SciBERT is a BERT-based pretrained language model trained on a large corpus of scientific text and demonstrated to improve performance across scientific NLP tasks.The resource is released to address the lack of high-quality, large-scale labeled scientific data.
  • Contributions: The study compares finetuning with task-specific architectures atop frozen embeddings and examines the effect of an in-domain vocabulary.These experiments investigate how adaptation strategy and vocabulary domain affect scientific NLP performance.
  • Contributions: SciBERT is evaluated on a suite of scientific-domain tasks and achieves new state-of-the-art results on many of them.The evaluation spans multiple scientific NLP tasks rather than a single benchmark.

2 Methods

SciBERT retains BERT’s multilayer bidirectional Transformer architecture and pretraining objectives while adapting pretraining to scientific text. Its methods use a new 30K scientific WordPiece vocabulary and a 3.17B-token corpus of 1.14M full-text papers spanning computer science and biomedicine.

  • Background: SciBERT follows BERT’s multilayer bidirectional Transformer architecture and is pretrained on scientific text.BERT uses masked-token prediction and next-sentence prediction as its two pretraining tasks.
  • Vocabulary: SCIVOCAB is a new 30K-token WordPiece vocabulary built on the scientific corpus in cased and uncased versions.Its token overlap with BERT’s BASEVOCAB is 42%.
  • Corpus: SciBERT is trained on a random sample of 1.14M Semantic Scholar papers containing 3.17B tokens from full-text articles.The corpus averages 154 sentences and 2,769 tokens per paper, comparable in size to BERT’s 3.3B-token training corpus.
  • Corpus: The training corpus comprises 18% computer science papers and 82% papers from the broad biomedical domain.Sentences are split using ScispaCy.

3 Experimental Setup

The experiments cover five scientific NLP tasks using BERT-Base and four SCIBERT variants, with task-specific architectures, optimization settings, and frozen-embedding analyses. Datasets span entity and relation extraction, citation intent, and seven scientific fields.

  • Tasks: Experiments cover named entity recognition, PICO extraction, text classification, relation classification, and dependency parsing.PICO extracts clinical-trial Participants, Interventions, Comparisons, and Outcomes; REL predicts relations between two entities marked with special tokens.
  • Datasets: The newer datasets include EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field, spanning clinical trials, computer science, citation intents, and seven study fields.Each Paper Field study field has approximately 12K training examples.
  • Models: SCIBERT matches BERT-Base in configuration and size, with four cased or uncased, BASEVOCAB or SCIVOCAB variants.BASEVOCAB models are finetuned from corresponding BERT-Base models, whereas SCIVOCAB models are trained from scratch.
  • Casing: Models use cased versions for NER and parsing and uncased versions for other tasks, although uncased models sometimes perform slightly better even on NER.This casing choice follows Devlin et al. (2019).
  • Finetuning: Finetuning uses task-specific linear classifiers, dropout 0.1, Adam, cross-entropy loss, batch size 32, and 2–5 epochs with selected learning rates.The best learning rate and epoch count are chosen on development data; 2 or 4 epochs and 2e-5 work best across most datasets and models.
  • Frozen embeddings: The study also trains frozen-BERT embedding models with task-specific BiLSTM, MLP, CRF, or dependency-parsing components.These models use dropout 0.5, early stopping with patience 10, batch size 32, and learning rate 0.001.

4 Results

SCIBERT outperforms BERT-Base across scientific, biomedical, computer science, and multidomain tasks, with gains both with and without finetuning. It achieves new state-of-the-art results on many datasets, while remaining slightly below SOTA on three biomedical datasets and matching BIOBERT comparably on others.

  • SCIBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without, achieving new SOTA results on many tasks.
  • On biomedical tasks, SCIBERT surpasses BERT-Base by +1.92 F1 with finetuning and +3.59 F1 without, achieving new SOTA on BC5CDR, ChemProt, and EBM-NLP.
  • SCIBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, where competing systems use additional datasets, biomedical pretraining, or POS features.SCIBERT nevertheless outperforms BIOBERT on BC5CDR and ChemProt and performs similarly on JNLPBA despite using a substantially smaller biomedical corpus.
  • SCIBERT outperforms BERT-Base on computer science tasks by +3.55 F1 with finetuning and +1.13 F1 without, with new SOTA on ACL-ARC and SciERC NER.

5 Discussion

Finetuning improves results more than task-specific architectures atop frozen embeddings, with the largest gains in computer science and biomedical tasks. An in-domain vocabulary provides additional F1 gains, but the authors suspect scientific-corpus pretraining contributes most to SciBERT’s benefits.

  • Finetuning: Finetuning improves average performance by +3.25 F1 with SCIBERT and +3.58 with BERT-Base over frozen embeddings and task-specific architectures.The largest finetuning effects occur in computer science and biomedical tasks, while multidomain tasks show the smallest effects.
  • Finetuning: +5.59 F1 with SCIBERT and +3.17 F1 with BERT-Base are the largest finetuning gains in computer science, versus +2.94 and +4.61 F1 in biomedical tasks.Multidomain tasks show the smallest effects: +0.7 F1 with SCIBERT and +1.14 F1 with BERT-Base.
  • Finetuning: On every dataset except BC5CDR and SciCite, finetuned BERT-Base outperforms or performs similarly to models using frozen SCIBERT embeddings.This comparison indicates that finetuning can outweigh the benefit of using frozen scientific embeddings.
  • Scientific vocabulary: +0.60 F1 is the average improvement from using SCIVOCAB, with gains of +0.76 F1 for biomedical, +0.61 F1 for computer science, and +0.11 F1 for multidomain tasks.The authors conclude that an in-domain vocabulary is helpful, but suspect SCIBERT benefits most from scientific-corpus pretraining.

6 Related Work

SciBERT extends domain-adapted BERT models such as BioBERT and ClinicalBERT by pretraining on a broad scientific corpus and using an in-domain vocabulary. Unlike those models’ source-specific training data and original BERT vocabulary, SciBERT uses 1.14M biomedical and computer science papers with SCIVOCAB.

  • 6 Related Work: BioBERT and ClinicalBERT are recent examples of domain adaptation for BERT.BioBERT is trained on PubMed abstracts and PMC full-text articles, while ClinicalBERT is trained on clinical text from MIMIC-III.
  • 6 Related Work: 1.14M biomedical and computer science papers from the Semantic Scholar corpus train SciBERT, distinguishing it from BioBERT and ClinicalBERT.BioBERT uses PubMed abstracts and PMC full-text articles, whereas ClinicalBERT uses clinical text from MIMIC-III.
  • 6 Related Work: SciBERT uses the in-domain SCIVOCAB, while the other models use BERT’s original BASEVOCAB.The vocabulary choice further distinguishes SciBERT from the related domain-adapted models.

7 Conclusion and Future Work

The authors released SciBERT, evaluated it across scientific-domain tasks and datasets, and found significant improvements over BERT-Base with new state-of-the-art results on several tasks. Future work will extend SciBERT’s scale and domain coverage while pursuing a single resource useful across multiple domains.

  • Contributions: SciBERT significantly outperformed BERT-Base and achieved new state-of-the-art results on several scientific NLP tasks.The results included comparisons with some reported BioBERT results on biomedical tasks.
  • Future Work: Future work will release a BERT-Large-analogous SciBERT and test different proportions of papers from each domain.The authors aim to build one resource useful across multiple domains because these language models are costly to train.
Loading 1903.10676v3…