PaperlayerView exact revision on arXiv

Source-linked AI summary

SpanBERT: Improving Pre-training by Representing and Predicting Spans

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy

arXiv:1907.10529v3cs.CLcs.LG

TL;DR

SpanBERT addresses the limits of token-level pre-training for tasks requiring reasoning over text spans. It masks contiguous spans and trains boundary representations to predict their contents, consistently outperforming BERT baselines, especially on span-selection tasks.

  • Problem

    BERT masks individual tokens, while many NLP tasks require reasoning about relationships between two or more spans of text.

  • Method

    SpanBERT masks contiguous random spans and uses a span-boundary objective to predict each masked span from its boundary representations.

  • Results

    SpanBERT consistently outperforms BERT baselines across tasks, with substantial gains on span-selection tasks including question answering and coreference resolution.

  • Takeaways & Limitations

    The results show that designing pre-training tasks and objectives can substantially improve performance across span-selection and other benchmarks.

Abstract

from arXiv · show

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT-large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0, respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6\% F1), strong performance on the TACRED relation extraction benchmark, and even show gains on GLUE.

1 Introduction

SpanBERT adapts BERT for span-level reasoning by masking contiguous spans and predicting their contents from boundary representations. It consistently outperforms BERT baselines, with especially large gains on span-selection tasks.

  • SpanBERT targets span-level reasoning, which is central to tasks such as extractive question answering and coreference resolution.Predicting a multi-token span such as “Denver Broncos” is harder than predicting an individual token when its neighboring token is known.
  • The method masks random contiguous spans instead of individual tokens and adds a span-boundary objective that predicts the entire masked span from its boundary tokens.These changes are designed to better represent and predict spans without relying on the individual token representations inside the masked span.
  • The tuned implementation also uses single-sequence pre-training rather than two half-length segments with next sentence prediction.The authors report that this baseline change considerably improves performance on most downstream tasks.
  • 94.6% and 88.7% F1 are achieved on SQuAD 1.1 and 2.0, respectively, reducing error by as much as 27% compared to the tuned BERT replica.Similar gains are observed on five additional extractive question answering benchmarks.
  • 79.6% F1 establishes a new state of the art on OntoNotes coreference resolution, exceeding the previous top model by 6.6% absolute.The method also improves TACRED and GLUE, including tasks without explicit span selection.
  • The results support designing stronger pre-training tasks and objectives as an important source of improvement beyond adding data or increasing model size.

2 Background: BERT

BERT pre-trains transformer encoders with self-supervised objectives on unlabeled text. Its masked language model predicts randomly selected missing tokens, while next sentence prediction classifies whether one sequence continues another.

  • BERT pre-trains a deep transformer encoder before fine-tuning it for a downstream task.
  • BERT optimizes masked language modeling and next sentence prediction using only a large collection of unlabeled text.
  • The encoder produces a contextualized vector representation for each token in an input sequence.
  • Masked language modeling predicts original tokens from modified inputs after selecting 15% of tokens, with most replaced by [MASK].In BERT’s implementation, 80% are masked, 10% replaced randomly, and 10% left unchanged.
  • BERT selects masked tokens independently, whereas SpanBERT defines them by randomly selecting contiguous spans.
  • Next sentence prediction takes two sequences and predicts whether the second is the direct continuation of the first.The sequences are separated by [SEP], and [CLS] represents the prediction target.
  • BERT combines masked language modeling and next sentence prediction with uniformly random word-piece masking and bi-sequence sampling.

3 Model

SpanBERT pre-trains span representations by masking contiguous full-word spans and predicting their contents from boundary representations. It also removes NSP through single-sequence training.

  • 3.1 Span Masking: SpanBERT masks contiguous spans of full words rather than individual tokens, sampling span lengths from a geometric distribution.The masking budget remains 15%, with span-level replacement rather than token-level replacement.
  • 3.2 Span Boundary Objective: The span-boundary objective predicts every token in a masked span using only the representations of its external boundary tokens.Each target also uses a relative position embedding, and the prediction uses a two-layer feed-forward representation function.
  • 3.2 Span Boundary Objective: SpanBERT sums the span-boundary and masked-language-model losses for tokens in each masked span.The target input embedding is reused in both objectives.
  • 3.3 Single-Sequence Training: Single-sequence training without NSP is generally better than BERT’s two-sequence setting, potentially because it provides longer contexts or avoids unrelated-document noise.The paper presents these explanations as conjectures.
  • 3 Model: SpanBERT combines geometric full-word span masking, the span-boundary objective, and a single-sequence data pipeline.These are the three components listed in the paper’s summary of the pre-training method.

4 Experimental Setup

The experiments evaluate SpanBERT across question answering, coreference resolution, relation extraction, and GLUE, using established task-specific architectures and controlled BERT comparisons.

  • 4.1 Tasks: The evaluation covers seven question answering tasks, coreference resolution, nine GLUE tasks, and relation extraction.The paper expects span-selection tasks, especially question answering and coreference resolution, to benefit most.
  • Extractive Question Answering: Extractive question answering selects a contiguous answer span from a passage given a question.The evaluation includes SQuAD 1.1, SQuAD 2.0, and five MRQA datasets.
  • Coreference Resolution: Coreference resolution clusters text mentions that refer to the same real-world entities and is evaluated on the document-level CoNLL-2012 shared task.The implementation encodes documents in independently processed, non-overlapping segments.
  • Relation Extraction: TACRED predicts one of 42 relation types, including no_relation, for a subject and object span pair in a sentence.The setup masks entities with their NER tags before classification.
  • GLUE: GLUE provides sentence-level classification tasks spanning acceptability, sentiment, paraphrase or similarity, and natural language inference.WNLI is excluded for a fair comparison, and the setup adds only a linear classifier over [CLS].
  • Baselines and Training: The comparison includes Google BERT, an improved BERT reimplementation, and a single-sequence BERT reimplementation without NSP.The models use BERT-large configuration and the same BooksCorpus and English Wikipedia pre-training corpus.

5 Results

Across 17 benchmarks, SpanBERT outperforms BERT on almost every task, with especially large gains on extractive question answering and strong results on coreference, relation extraction, and GLUE.

  • Extractive Question Answering: 2.9% F1 improvement is observed on average across five MRQA extractive question answering tasks over the BERT reimplementation.The gains range from 2.0% on Natural Questions to 4.6% on TriviaQA.
  • Coreference Resolution: 79.6% F1 establishes a new state of the art on OntoNotes coreference resolution, versus a previous best result of 73.0%.The model exceeds the previous top model by 6.6% absolute.
  • Relation Extraction: 3.3% F1 improvement over the BERT reimplementation is achieved on TACRED, leaving SpanBERT 0.7 point behind BERTEM + MTB.Most of the gain, +2.6%, stems from single-sequence training; span masking and the span boundary objective contribute 0.7%.
  • GLUE: +6.9% improvement is reported on RTE, accounting for most of the rise in SpanBERT’s GLUE average.The main gains on GLUE occur in the SQuAD-based QNLI dataset (+1.3%) and RTE.
  • Overall Trends: SpanBERT outperforms BERT on almost every task, performing better than all baselines in 14 of 17 benchmarks.It is on-par with single-sequence BERT on MRPC and QQP, while Google BERT performs better on SST-2 by 0.4% accuracy.
  • Overall Trends: Single-sequence training without NSP works considerably better than bi-sequence training with NSP across a wide variety of tasks.The reported comparison concerns BERT’s choice of sequence lengths and two-half-length-sequence processing.

6 Ablation Studies

The ablations show that random span masking is generally competitive or superior to linguistically informed alternatives, while the span-boundary objective adds substantial gains beyond span masking alone.

  • Ablation design: The experiments compare random span masking with linguistically informed masking and evaluate the effects of SBO against BERT’s NSP objective.
  • Masking schemes: Random span masking is preferable to other masking strategies on most tasks, except coreference resolution.The comparison uses BERT-style bi-sequence training with NSP and development-set evaluations.
  • Masking schemes: Noun-phrase masking matches random spans on NewsQA but underperforms on TriviaQA by 1.1% F1.
  • Masking schemes: Random subword masking is preferable to every span-masking variant for coreference resolution before adding the span-boundary objective.
  • Auxiliary objectives: Single-sequence training typically improves performance, and adding SBO yields a +2.7% F1 gain on coreference resolution over span masking alone.Unlike NSP, SBO does not appear to have adverse effects.

7 Related Work

SpanBERT differs from prior pre-training work by explicitly modeling span representations with a span-boundary objective, while related methods target masking, generation, knowledge integration, or broader pre-training improvements.

  • Span representations: SpanBERT pretrains span representations used in question answering, coreference resolution, and other tasks, unlike work focused mainly on dropping or masking words for generation.
  • Masking and objectives: Random geometrically distributed span masking works as well as, and sometimes better than, linguistically coherent masking, while SBO explicitly models spans.
  • Related objectives: Other related methods integrate knowledge bases or combine directional and sequence-to-sequence objectives for generation tasks.
  • Concurrent work: RoBERTa studies BERT’s hyperparameters and training-data size, whereas XLNet combines an autoregressive loss with Transformer-XL and also masks short spans.
  • Span-boundary objective: Unlike pair2vec’s limited word-pair context, SBO produces boundary representations that encode context during both pre-training and fine-tuning.

8 Conclusion

SpanBERT extends BERT with contiguous span masking and boundary-based prediction, outperforming BERT baselines broadly and especially on span selection tasks.

  • Contribution: SpanBERT masks contiguous random spans and trains span-boundary representations to predict their entire masked content without individual masked-token representations.
  • Conclusion: The resulting models outperform all BERT baselines across varied tasks, with substantially stronger performance on span selection tasks.

A Pre-training Procedure

SpanBERT pre-training samples contiguous corpus blocks, masks 15% of word pieces using span masking, and optimizes MLM together with SBO for each masked token.

  • Data sampling: The corpus is divided into single contiguous blocks containing up to 512 tokens.
  • Data sampling: At each pre-training step, batches of blocks are sampled uniformly at random.
  • Masking: 15% of word pieces in each batch are masked using the span-masking scheme.
  • Objectives: For each masked token x_i, the objective combines masked language modeling and the span-boundary objective: L(x_i) = L_MLM(x_i) + L_SBO(x_i).

B Fine-tuning Hyperparameters

Fine-tuning uses shared hyperparameters across methods, with task-specific settings for extractive question answering, coreference resolution, and TACRED/GLUE.

  • All methods, including baselines, use the same fine-tuning hyperparameters.
  • Extractive Question Answering: Extractive question answering uses max_seq_length = 512, a sliding window of 128 for longer inputs, four epochs, and tuned learning rates and batch sizes.
  • Coreference Resolution: Coreference resolution encodes document chunks independently, fine-tunes for 20 epochs, and uses batch size = 1 with separately tuned BERT and task-specific learning rates.
  • TACRED/GLUE: TACRED and GLUE use max_seq_length = 128, 10 epochs, and tuned learning rates and batch sizes, except CoLA uses four epochs to avoid severe overfitting.
Loading 1907.10529v3…