PaperlayerView exact revision on arXiv

Source-linked AI summary

The Diminishing Returns of Masked Language Models to Science

Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Kyle Chard, Ian Foster

arXiv:2205.11342v2cs.CLcs.LG

TL;DR

Prior scaling results had not been tested systematically on science-focused models, despite the potential for language models to automate scientific knowledge extraction. The paper evaluates 14 BERT-based models across 12 scientific tasks and finds that scaling does not consistently improve information-extraction performance.

  • Problem

    How model size, training data, and compute affect science-focused models had not been investigated, despite the potential for language models to automate scientific knowledge extraction.

    Source: §1 · p. 1

  • Method

    The study compares 14 BERT-based models, including SCHOLARBERT, across 12 scientific downstream tasks while varying model size, pretraining data, and pretraining and finetuning time.

    Source: §1 · p. 1, §3.2 · p. 2, §5 · p. 4

  • Results

    Increasing model size, pretraining data, pretraining time, or finetuning time did not significantly or consistently improve downstream scientific information-extraction performance.

    Source: §4.3 · p. 3, §5 · p. 4

  • Takeaways & Limitations

    For information extraction, performance gains from larger models, more data, and longer training depend strongly on the individual task.

    Source: §1 · p. 1, §4.3 · p. 3

    The 12 labeled test datasets cover only five domains plus two multidisciplinary datasets, and the largest model has 770M parameters, which may be too small to demonstrate language-model scaling laws.

    Source: Limitations · p. 4

Abstract

from arXiv · show

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770M-parameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model sizes, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if at all, in scientific information extraction tasks and offered possible explanations for the surprising performance differences.

1 Introduction

Scientific publications impose a cognitive burden, and general-corpus BERT differs from scientific literature in context, terminology, and writing style. This study tests whether scaling effects extend to scientific tasks and finds gains on information extraction are task-dependent.

1 Introduction

  • Scientific publication growth burdens researchers, while general-corpus BERT differs from scientific literature in context, terminology, and writing style.Language models could potentially alleviate this burden by automating scientific knowledge extraction.Source: §1 · p. 1
  • Previous work had not investigated how model size, training-corpus size, and compute time affect science-focused models.Source: §1 · p. 1
  • SCHOLARBERT models trained on a 225B-token multidisciplinary corpus examine model size, data size, and pretraining and finetuning epochs; information-extraction gains are highly task-dependent.The authors make the models and a sample of the training corpus publicly available.Source: §1 · p. 1

2 Related Work

Prior scaling studies examined general language models, while domain-specific scientific models were developed to improve science-task performance. How pretraining parameters affect those models remained unstudied.

2 Related Work

  • Kaplan et al. (2020) and Brown et al. (2020) studied scaling in GPT, while Devlin et al. (2019) and Liu et al. (2019) reported benefits from larger or longer-trained masked language models.Source: §2 · p. 1
  • SciBERT, PubMedBERT, BioBERT, and MatBERT were pretrained on domain-specific text to improve tasks requiring domain knowledge.Source: §2 · p. 2
  • Prior studies had not examined how pretraining parameters affect the downstream task performance of science-focused models.Source: §2 · pp. 1–2

3 Data and Methodology

The study compares existing BERT models with SCHOLARBERT variants pretrained on Public Resource dataset subsets. Its multidisciplinary corpus and model variants support comparisons across corpus size and model characteristics.

3.1 The Public Resource Dataset

  • The Public Resource dataset contains extracted text from approximately 75M articles, with biomedical, technology, physical-sciences, social-sciences, and arts-and-humanities content.It was constructed from 85M journal-article PDFs, and some extractions failed.Source: §3.1 · p. 2

3.2 Models

  • The comparison includes 14 BERT models, including seven SCHOLARBERT variants; SCHOLARBERT and SCHOLARBERT-XL have 340M and 770M parameters.The models vary by architecture, pretraining method, pretraining corpus, and casing.Source: §3.2 · p. 2
  • The study frames multidisciplinary SCHOLARBERT as an alternative to expensive domain-specific pretraining, with models adaptable through finetuning.Source: §3.2 · p. 2

3.3 SCHOLARBERT Pretraining

  • SCHOLARBERT models were pretrained on random 1%, 10%, or 100% samples of the Public Resource dataset using the RoBERTa pretraining procedure.Source: §3.3 · p. 2

4 Experimental Results

Across 12 scientific tasks, increasing model size, data, or training time did not reliably yield large performance gains. Results varied by task, while finetuning and evaluation scope offer possible explanations.

4.1 Sensitivity Analysis

  • Pretraining loss fell rapidly through about 10,000 iterations, while extending training to roughly 33,000 iterations brought only small further decreases.Source: §4.1 · p. 2
  • More finetuning raised the 0.9k-iteration models’ downstream F1 scores by 8%, but their 75-epoch average remained 19.9 points below the 33k-iteration model finetuned for five epochs.For the 33k model, 75 epochs yielded only 0.39 percentage points more average performance than five epochs.Source: §4.1 · pp. 2–3

4.2 Finetuning

  • With the same training data, BERT-Base was within 1 percentage point of BERT-Large on 6/12 tasks, and SB_1 was within 1 point of SB-XL_1 on 7/12.Source: §4.2 · p. 3, Table 2 · p. 4, Table 3 · p. 4
  • At the same model size, SB_1 was within 1 percentage point of SB_100 on 8/12 tasks; SB-XL_1 was within 1 point of SB-XL_100 on 10/12.Source: §4.2 · p. 3, Table 2 · p. 4, Table 3 · p. 4
  • Biomedical tasks were the only domain where the on-domain model consistently outperformed off-domain or general models by more than 1 percentage point F1.This was not consistent for computer-science, materials, or multidisciplinary tasks.Source: §4.2 · p. 3, Table 2 · p. 4, Table 3 · p. 4

4.3 Discussion

  • The authors suggest task type matters more than model size, with consistent greater-than-1-point gains from larger models only for relation extraction; they also suggest finetuning can fill pretraining knowledge gaps when corpus characteristics match.They note that the Public Resource corpus may saturate even the largest model.Source: §4.3 · pp. 3–4

5 Conclusions

Across 14 BERT-based models and 12 scientific tasks, the experiments evaluate how pretraining parameters relate to performance. Increasing model size, pretraining data, pretraining time, or finetuning time does not consistently improve scientific information-extraction performance; the models and a subset of the training corpus will be made available.

5 Conclusions

  • The study compares 14 existing and newly developed BERT-based models across 12 scientific downstream tasks.It evaluates model size, pretraining dataset size and breadth, and pretraining and finetuning lengths.Source: §5 · p. 4
  • Performance on scientific information-extraction tasks is not significantly or consistently improved by increasing model size, pretraining data, pretraining time, or finetuning time.The authors attribute this pattern to finetuning strength, evaluation-dataset limitations, and, for SCHOLARBERT, model sizes small relative to the pretraining corpus.Source: §5 · p. 4
  • All pretrained SCHOLARBERT models and a subset of the Public Resource Dataset will be made freely available online.The full Public Resource Dataset cannot be shared.Source: §5 · p. 4

Limitations

The evaluation covers 12 labeled test datasets from five domains and two multidisciplinary areas, with five datasets from biomedicine. The authors identify this domain imbalance as a limit and plan to extend evaluation beyond the tasks studied.

Limitations

  • The 12 labeled test datasets span five domains plus two multidisciplinary areas, and five are biomedical, limiting the evaluation's domain coverage.The authors say the imbalance reflects varied adoption of NLP methods across domains.Source: Limitations · p. 4
  • The authors aim to extend experiments beyond NER, relation extraction, and text classification to scientific question-answering and textual entailment.Source: Limitations · p. 5

A Extant BERT-based models

Prior work describes BERT-based models pretrained on general or scientific-domain corpora, including reported task-specific gains. The tokenization account also documents and corrects an integer-overflow issue affecting the PRD_100 vocabulary.

A Extant BERT-based models

  • SciBERT reports average finetuning gains over BERT-Base of 1.66% on biomedical tasks and 3.55% on computer-science tasks.Source: A Extant BERT-based models · p. 7
  • BioBERT reports gains over BERT-Base of 0.62% on biomedical NER, 2.80% on biomedical relation extraction, and 12.24% on biomedical question answering.Source: A Extant BERT-based models · p. 7
  • PubMedBERT, pretrained exclusively on PubMed and PubMedCentral text, outperforms BERT-Base by 1.16% on the Biomedical Language Understanding and Reasoning Benchmark; MatBERT and BatteryBERT report task-specific gains as well.MatBERT reports ∼10% higher F1 than BERT-Base for specified materials-science entities, while BatteryBERT's F1 is around 5% higher than BERT-Base on battery-specific question answering.Source: A Extant BERT-based models · p. 7

B.1 Tokenization

  • An integer overflow in token-frequency counts omitted common words from the PRD_100 vocabulary; after switching to 64-bit counts, the corrected-vocabulary models showed comparable downstream performance.The missing subword tokens also prevented words such as “the” from appearing in the vocabulary.Source: §B.1 · p. 7

B.2 RoBERTa Optimizations

The study adopts RoBERTa pretraining methods with three stated exceptions: shorter samples are included, sentences are packed from a single document, and WordPiece replaces BPE. The cited figure marks evaluation checkpoints on SCHOLARBERT pretraining-loss plots.

B.2 RoBERTa Optimizations

  • RoBERTa uses single-phase training at length 512, dynamic masking, no Next Sentence Prediction, contiguous sentences, batch size 8192, BPE, a 50,000-token vocabulary, and up to 500,000 iterations.It does not randomly use shortened sequences; BERT does, and uses static masking and 31,000 iterations for BERT-Base.Source: §B.2 · p. 7
  • Figure 1 plots SCHOLARBERT pretraining losses, with vertical dashed lines marking approximate iteration checkpoints selected for evaluation.Source: Figure 1 · p. 8
  • The study departs from RoBERTa by adding shorter samples, packing sentences from a single document, and using WordPiece instead of BPE.The stated rationales are that downstream tasks often use shorter sequences and that RoBERTa's authors noted slightly better performance for single-document packing and slightly worse downstream performance for BPE.Source: §B.2 · p. 8

B.3 Hardware and Software Stack

SCHOLARBERT pretraining used distributed GPU training, with additional DeepSpeed resources for the XL models. The variants shared hyperparameters, and checkpoints were evaluated through convergence.

B.3 Hardware and Software Stack

  • All SCHOLARBERT variants used the same hyperparameters, including batch size 32,768, about 33,000 iterations, and initial learning rate 0.0004.Training took roughly 1,000 node-hours, or 8,000 GPU-hours, per model.Source: Table 5 · p. 8, §B.3 · p. 9
  • DeepSpeed optimizations reduced training time and memory use, enabling larger models to train in roughly the same time as smaller ones.Source: §B.3 · p. 9
  • Models were trained past convergence, with checkpoints taken throughout training to measure performance as a function of training time.Source: §B.3 · p. 9

C Evaluation Tasks

Evaluation covered eight scientific NER datasets and four sentence-level tasks spanning biomedical, computer science, materials, and multidisciplinary settings.

C Evaluation Tasks

  • The first four NER datasets covered chemical, disease, and molecular-biology entities, plus entities, relations, and coreference in AI abstracts.They were BC5CDR, JNLPBA, SciERC, and NCBI-Disease.Source: C Evaluation Tasks · p. 9
  • The remaining NER datasets covered chemical mentions and seven materials-entity types, as well as entities from science exams and sociology articles.ChemDNER contains 19,980 unique chemical mentions; ScienceExam has 133K entities, and Coleridge has 13,588.Source: C Evaluation Tasks · p. 9
  • The four sentence-level tasks were ChemProt and SciERC relation extraction, plus Paper Field and Battery sentence classification.Source: C Evaluation Tasks · p. 9
  • Paper Field maps titles to seven fields; Battery classifies abstracts as battery or non-battery using semi-automated labels.ChemProt uses expert-annotated chemical-protein interactions; SciERC provides 4,716 relations.Source: C Evaluation Tasks · p. 9

D Extended Results

Across the reported tasks, overlapping error bars indicate that on-domain pretraining often offered only marginal advantage over other-domain or general-corpus pretraining.

D Extended Results

  • Significant error-bar overlap for NCBI-Disease, SciERC NER, Coleridge, SciERC sentence classification, and ChemProt supports only marginal on-domain pretraining advantage.Table 6 and Figure 2 report NER F1; Table 7 and Figure 3 report sentence-classification F1.Source: D Extended Results · p. 10, Figure 2 · p. 11, Table 6 · p. 11
  • Table 6 compares 14 models across eight NER datasets, reporting mean F1 and standard deviation over five runs.Source: Table 6 · p. 11
  • Table 7 and Figure 3 show sentence-classification F1 scores with standard-deviation error bars.Source: Table 7 · p. 12, Figure 3 · p. 12

Related papers

Loading 2205.11342v2…