PaperlayerView exact revision on arXiv

Source-linked AI summary

Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees

Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, Yunhai Tong

arXiv:2103.04350v1cs.CL

TL;DR

The paper asks how syntax trees can be incorporated effectively and efficiently into pre-trained Transformers, given the gap between implicitly learned syntax and expert-created trees. It proposes Syntax-BERT, a plug-and-play framework that decomposes attention by syntactic relations and aggregates the resulting representations. Experiments report consistent improvements across pre-trained backbones, including a higher GLUE score for T5-Large.

  • Problem

    The paper asks how to incorporate syntax trees effectively and efficiently into pre-trained Transformers, where implicitly learned syntax differs from expert-created trees.

    Source: §1 · p. 1

  • Method

    Syntax-BERT decomposes self-attention into syntax-relation sub-networks and uses task-specific topical attention to aggregate their representations in a pre-trained checkpoint.

    Source: §1 · p. 1, §3.3 · p. 5

  • Results

    86.3 to 86.8 GLUE overall score for T5-Large; experiments also report consistent improvement across BERT, RoBERTa, and T5 backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Syntax-BERT is presented as a plug-and-play way to use syntax trees with pre-trained Transformer checkpoints, and the experiments support their usefulness across the evaluated backbones.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. Meanwhile, syntactic information has been proved to be crucial for the success of NLP applications. However, how to incorporate the syntax trees effectively and efficiently into pre-trained Transformers is still unsettled. In this paper, we address this problem by proposing a novel framework named Syntax-BERT. This framework works in a plug-and-play mode and is applicable to an arbitrary pre-trained checkpoint based on Transformer architecture. Experiments on various datasets of natural language understanding verify the effectiveness of syntax trees and achieve consistent improvement over multiple pre-trained models, including BERT, RoBERTa, and T5.

1 Introduction

The introduction asks whether explicit syntax trees improve pre-trained Transformers and how they can be incorporated without pre-training from scratch. It presents Syntax-BERT as a plug-and-play method and reports consistent improvements across backbones.

1 Introduction

  • The paper asks whether explicit syntax trees help in pre-training and whether they can be added to a pre-trained checkpoint efficiently.It frames the question against a gap between syntax implicitly learned by Transformers and expert-created syntax trees.Source: §1 · p. 1
  • 86.3 to 86.8 GLUE overall score for T5-Large; Syntax-BERT also consistently outperforms vanilla BERT and LISA-enhanced models across BERT, RoBERTa, and T5 backbones.The paper attributes the T5-Large improvement to only a few extra parameters.Source: §1 · p. 2
  • The framework is designed to plug into an existing checkpoint without pre-training from scratch.The introduction describes this as an efficient approach for a specific downstream application.Source: §1 · p. 2
  • Syntax-BERT decomposes self-attention into syntax-relation sub-networks and uses topical attention to weight their task relevance.The sub-networks represent ancestor, offspring, and sibling relations at different hops and share parameters with the pre-trained network.Source: §1 · pp. 1–2
  • Syntax-BERT can be combined with LISA for further enhancement, with the paper describing the two algorithms as complementary.The introduction presents this alongside consistent improvement across multiple pre-trained backbones.Source: §1 · p. 2

2 Related Work

The related work reviews pre-trained Transformer models and prior ways to use syntax in neural networks. It distinguishes Syntax-BERT's checkpoint-compatible approach from syntax-enhanced Transformers designed to train from scratch.

2.1 Pre-trained language models

  • BERT, RoBERTa, and T5 are presented as pre-trained model families, with differences in their training objectives, data, and text-to-text task format.The section describes BERT's MLM and NSP objectives, RoBERTa's dynamic masking and larger training corpus, and T5's unified text-to-text format.Source: §2.1 · p. 2

2.2 Syntax-aware models

  • Prior syntax-aware work includes tree-structured recurrent models and Transformer methods that incorporate syntactic features or tree-guided attention.Examples include Tree-LSTM, Syntax-Infused Transformer, LISA, and hierarchical accumulation approaches.Source: §2.2 · p. 2
  • The section contrasts earlier approaches designed to train Transformers from scratch with Syntax-BERT's plug-and-play use of pre-trained checkpoints.It also notes concurrent work on dependency structures in pre-trained language models and probing methods for syntactic information.Source: §2.2 · p. 3

3 Syntax-BERT

Syntax-BERT changes information flow in a pre-trained Transformer through syntax-aware self-attention. It uses syntax-derived sub-networks and task-oriented attention to aggregate representations for downstream tasks.

3 Syntax-BERT

  • Syntax-BERT is a pre-trained Transformer variant that changes information flow through a syntax-aware self-attention mechanism.The section introduces the architecture, syntax-tree masks, and syntactic attention layers.Source: §3 · p. 3

3.1 Architecture

  • Syntax-derived sub-networks encode grammatical relationships and distances, while topical attention aggregates task-oriented representations from them.The sub-networks share parameters with the global pre-trained self-attention layer.Source: §3.1 · p. 3, §3.2 · p. 4, §3.3 · p. 5

3.2 Masks induced by syntax tree

  • Masks represent parent, child, and sibling relations at different tree distances; pairwise inference adds links between tokens from different sentences.Sibling masks use the nodes’ lowest common ancestor and their distance; pairwise masks do not encode distance.Source: §3.2 · p. 4

3.3 Syntactic attention layers

  • Masked self-attention applies sparse syntax masks, and topical attention uses a trainable task query to emphasize task-oriented knowledge in the final representation.The task query weights outputs from the different sub-networks.Source: §3.3 · pp. 4–5

4 Experiments

The experiments evaluate Syntax-BERT on sentiment classification, natural language inference, GLUE, ablations, and structural probes across pretrained backbones. Syntax-enhanced models improve on their corresponding baselines across the reported benchmarks, while the ablations and probes examine the roles of syntactic structure and attention design.

4 Experiments

  • Experiments evaluate Syntax-BERT on SST, SNLI, MNLI, and GLUE, using dependency and constituency trees unless specified, with 90 subnetworks for single-sentence tasks and 92 for pairwise inference.Backbones include Transformer, BERT-Base and BERT-Large, RoBERTa-Base and RoBERTa-Large, and T5-Large; LISA is also a comparison.Source: §4 · p. 5

4.1 Stanford Sentiment Treebank

  • On SST, Syntax-Transformer improves over comparable vanilla Transformers by 4.8 and 4.9 absolute points, and combining it with LISA boosts results further.The SST setup trains on phrases longer than three words and evaluates only full sentences.Source: §4.1 · p. 5, Table 2 · p. 6

4.2 Natural Language Inference

  • On SNLI and MNLI, Syntax-Transformer improves vanilla Transformer accuracy by 1.9 and 2.7 points, respectively.LISA also outperforms vanilla Transformer, but its accuracy improvement is smaller; BERT-Large syntax-enhanced models show little gain over their vanilla counterparts.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 GLUE Benchmark

  • Syntax-enhanced models consistently outperform corresponding GLUE baselines; Syntax-RoBERTa-Base scores 82.1 average GLUE, 1.3 points above RoBERTa-Base.The reported improvement also holds marginally for T5-Large; the authors note syntax is especially useful when training data or computation is limited.Source: §4.3 · pp. 6–7, Table 4 · p. 7

4.4 Ablation Study

  • Ablations show that all datasets benefit from syntax, parent and child masks matter more than sibling masks, and topical attention is crucial.The pairwise mask is important on STS-B, supporting the benefit of cross-sentence attention.Source: §4.4 · p. 7, Table 5 · p. 7

4.5 Structural Probe

  • Structural probes score syntax-aware models higher than their corresponding baselines on both undirected attachment and word-pair distance correlation.The probe assesses whether models encode Stanford Dependencies syntax.Source: §4.5 · p. 7

5 Discussion

Syntax-BERT adds a topical attention layer while inheriting most parameters from the pretrained model, and its time complexity matches vanilla BERT when the number of subnetworks is small. A CoLA case study illustrates a syntax model identifying an ungrammatical usage that vanilla Transformer misclassifies.

5.1 Complexity analysis

  • Syntax-BERT-Base adds 1.0 million topical-attention parameters, while LISA-BERT-Base adds 0.6 million parameters.The paper reports that both additions leave the original model's space complexity unaffected.Source: §5.1 · pp. 7–8
  • With a small number of subnetworks, Syntax-BERT has the same time complexity as vanilla BERT, O(MD_QN^2), with sparse implementation reducible to O(MD_QE).Here N is the sentence length, M the number of subnetworks, and E the average number of edges in a subnetwork, with E ≪ N^2.Source: §5.1 · p. 8

5.2 Case Study

  • On the CoLA example, Syntax-Transformer correctly classifies the ungrammatical sentence, while vanilla Transformer predicts incorrectly.The syntax model highlights the (“anyone”, “.”) relation and assigns its subnetwork a high topical-attention score; vanilla attention is described as less meaningful.Source: §5.2 · p. 8

6 Conclusion

Syntax-BERT plugs into pre-trained Transformer checkpoints to incorporate syntax-tree priors, and evaluation across BERT, RoBERTa, and T5 supports its effectiveness; future work includes adding further syntactic and semantic knowledge.

6 Conclusion

  • Syntax-BERT is a plug-and-play framework for pre-trained Transformer checkpoints that automatically underlines task-relevant syntactic knowledge.The paper evaluates it with BERT, RoBERTa, and T5 backbones.Source: §6 · p. 8
  • The empirical results support Syntax-BERT's effectiveness and the usefulness of syntax trees across BERT, RoBERTa, and T5 backbones.Source: §6 · p. 8
  • Future work aims to incorporate additional syntactic and semantic knowledge, including dependency-parser relation types and knowledge-graph concepts.Source: §6 · p. 9

A Detailed settings

The paper provides detailed reproduction settings and says its open-source code will be released after official publication.

A Detailed settings

  • The authors provide detailed reproduction settings and plan to release the open-source code when the paper is officially published.Source: A Detailed settings · p. 10

A.1 Stanford Sentiment Treebank

For Stanford Sentiment Treebank, the raw Transformer uses 12 layers, 512-dimensional intermediate layers, 0.1 dropout, and a 2000-dimensional final fully connected layer.

A.1 Stanford Sentiment Treebank

  • The raw Transformer uses 12 layers, 512-dimensional intermediate layers, 0.1 dropout, and a 2000-dimensional final fully connected layer.GloVe glove.840B.300d4 embeddings are fine-tuned, using Adam with an initial learning rate of 1e-4.Source: §A.1 · p. 10

A.2 Natural Language Inference

For natural language inference, the raw Transformer uses 12 layers, 512-dimensional intermediate layers, 0.15 dropout, and a 2000-dimensional fully connected layer before Softmax.

A.2 Natural Language Inference

  • The raw Transformer uses 12 layers, 512-dimensional intermediate layers, 0.15 dropout, and a 2000-dimensional fully connected layer before Softmax.Adam uses an initial learning rate of 5e-4 with exponential learning-rate decay of 0.9.Source: §A.2 · p. 10

B Connection to GNN

A Transformer layer can be viewed as a complete-graph GNN, and prior work has explored sparse architectures. Syntax-BERT decomposes this graph into subnetworks, using syntax trees to provide the inductive bias.

B Connection to GNN

  • A Transformer layer can be viewed as a GNN whose word nodes form a complete graph.This framing motivates introducing sparsity by decomposing the complete graph network.Source: B Connection to GNN · p. 10
  • Prior sparse architectures include Sparse Transformer, which separates full self-attention across several steps, and Star-Transformer, which uses ring and radial connections.The passage also notes sparse architectures as a way to improve training speed and generalization ability.Source: B Connection to GNN · p. 10
  • Syntax-BERT decomposes the complete graph network into multiple subnetworks, with syntax trees supplying its inductive bias for NLP tasks.A random decomposition also yields moderate performance enhancement, and similar phenomena have been reported for GCNs in image classification.Source: B Connection to GNN · p. 10

Related papers

Loading 2103.04350v1…