Source-linked AI summary
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, Luo Si
TL;DR
BERT achieves strong NLU performance but does not fully exploit underlying language structures. StructBERT extends BERT pre-training with word- and sentence-level structural objectives, achieving state-of-the-art results across varied downstream tasks, including an 89.0% average GLUE score.
Problem
BERT achieves strong NLU performance but does not make full use of underlying language structures.
Method
StructBERT extends BERT pre-training with word-level and sentence-level ordering objectives that explicitly model language structures.
Results
StructBERT achieves new state-of-the-art results across varied NLU tasks, including an 89.0% average GLUE score.
Takeaways & Limitations
Structural pre-training improves contextual representations by encoding dependencies between words and sentences, supporting generalizability and adaptability.
Takeaways & Limitations
The SQuAD v1.1 test result was unavailable at submission because the CodaLab evaluation server crashed.
Abstract
from arXiv · showhide
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.
1 Introduction
StructBERT extends BERT pre-training with word- and sentence-level structural objectives because BERT does not fully exploit underlying language structures. The resulting contextual representations improve performance across diverse NLU tasks.
- Motivation: BERT provides bidirectional contextual representations for NLU but does not make the most of underlying language structures.Its representations condition on both left and right context while predicting masked words.
- Motivation: Word order and sentence order are essential aspects of language fluency and relevant to many NLP tasks.Prior work found recurrent language models sensitive to word-order regularities and effective for word ordering without explicit syntax.
- Approach: StructBERT adds word-level and sentence-level structural objectives to BERT pre-training through two linearization strategies.The objectives target inner-sentence and inter-sentence structures alongside the existing masking strategy.
- Approach: Structural pre-training encodes dependencies between words and sentences in contextualized representations, providing better generalizability and adaptability.The paper presents this as the effect of incorporating structural information during pre-training.
- Results: StructBERT significantly advances state-of-the-art results across GLUE, SNLI, and SQuAD v1.1.The reported gains span benchmark evaluation, natural language inference, and extractive question answering.
- Results: The approach is reported to outperform published state-of-the-art models across semantic textual similarity, sentiment analysis, textual entailment, and question answering.These applications illustrate the breadth of the claimed downstream improvements.
2 StructBERT Model Pre-training
StructBERT extends BERT with structural pre-training objectives that model word order within sentences and sentence order across pairs. It combines these objectives with masked language modeling in a unified Transformer-based model.
- 2.1 Input Representation: StructBERT builds on BERT’s bidirectional Transformer encoder to produce contextualized representations for input tokens.Inputs may contain one sentence or a sentence pair, with token, positional, and segment embeddings.
- 2.3 Pre-training Objectives: The model jointly pre-trains masked language modeling with word- and sentence-structural objectives to exploit inner-sentence and inter-sentence information.The auxiliary objectives are trained together with the original masked LM objective in a unified model.
- 2.3.1 Word Structural Objective: The word structural objective reconstructs the correct positions of intentionally shuffled word tokens, addressing BERT’s limited explicit modeling of word order and higher-order dependencies.StructBERT masks tokens, shuffles selected unmasked trigrams, and predicts the original tokens and their order.
- 2.3.1 Word Structural Objective: The word objective shuffles three-word subsequences because trigrams balance reconstructing longer sequences against robustness to disturbed input.The selected subsequences use K = 3, and the word objective is jointly learned with masked LM using equal weights.
- 2.3.2 Sentence Structural Objective: The sentence structural objective predicts whether a paired sentence is the next, previous, or a random sentence, modeling sentence order bidirectionally.The three classes distinguish following text, preceding text, and a sentence sampled from another document.
- 2.4 Pre-training Setup: Pre-training uses English Wikipedia and BookCorpus with maximum sequence length 512, and reports StructBERTBase and StructBERTLarge configurations.StructBERTBase has 12 layers, 768-dimensional hidden vectors, 12 attention heads, and 110M parameters; StructBERTLarge has 24, 1024, 16, and 340M, respectively.
3 Experiments
StructBERT was evaluated across GLUE, SNLI, and SQuAD, with structural pre-training producing strong results across sentence-level and sentence-pair tasks. Ablations further examined how word- and sentence-order objectives contribute to downstream performance.
- Evaluation setup: Experiments evaluated StructBERT on GLUE, SNLI, and extractive question answering with SQuAD v1.1.The evaluation covered a variety of downstream language-understanding tasks.
- GLUE benchmark: 89.0% was achieved on the average GLUE score by the StructBERTRoBERTa ensemble, creating a new state-of-the-art result at submission.The model was submitted under the name ALICE and improved language models in addition to BERT.
- SNLI: StructBERT reached 91.7% accuracy on SNLI, improving 0.4% over SJRC and 0.9% over BERT.Because the architecture was identical to BERT, the passage attributes the improvement to the new pre-training objectives.
- Extractive question answering: StructBERT was superior to all published SQuAD models except XLNet+DA on the dev set without additional data augmentation.XLNet+DA outperformed StructBERT when using data augmentation and a large pre-training corpus.
- Effect of Different Structural Objectives: Removing either structural objective generally degraded downstream performance, although the word objective was an exception on SNLI.The ablation used one-objective removals to examine their influence across tasks.
- Effect of Different Structural Objectives: Sentence structure helped sentence-pair tasks, whereas word structure was most important for single-sentence tasks and improved CoLA by over 5%.The passage links sentence structure to understanding relationships between sentences and word-order reconstruction to judging sentence acceptability.
4 Related Work
Prior work develops contextualized representations and studies linearization as recovery of original word order. StructBERT builds on BERT by adding word- and sentence-structure objectives, including sentence ordering for inter-sentence understanding.
- Contextualized language models represent word meaning using surrounding context rather than a single global token representation.
- BERT and RoBERTa use bidirectional Transformer representations but do not fully exploit underlying language structures.
- StructBERT extends BERT by introducing word and sentence structures into pre-training tasks for deep language understanding.
- Linearization recovers the original order of words from a shuffled sentence and motivates structural modeling of language.
- Sentence ordering addresses relationships between text sentences by arranging shuffled sentences according to their original sequence.
5 Conclusion
The paper proposes structural pre-training for BERT using word- and sentence-level objectives. Experiments report new state-of-the-art results across GLUE, SNLI, and SQuAD v1.1.
- StructBERT adds word structural and sentence structural objectives to BERT pre-training for deep natural-language understanding.