SG-Net: Syntax-Guided Machine Reading Comprehension

Zhuosheng Zhang; Yuwei Wu; Junru Zhou; Sufeng Duan; Hai Zhao; Rui Wang

Source-linked AI summary

SG-Net: Syntax-Guided Machine Reading Comprehension

Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, Rui Wang

arXiv:1908.05147v3Aug 14, 2019cs.CL

TL;DR

Machine reading comprehension models can attend inaccurately to dispensable words in lengthy passages. SG-Net guides Transformer attention with syntactic dependencies and achieves new state-of-the-art or comparative results on SQuAD 2.0 and RACE.

Problem
Traditional attentive models attend to all words without explicit constraints, risking inaccurate focus on dispensable words in lengthy passages.
Method
SG-Net fuses original self-attention with syntax-guided self-attention based on each word’s dependency ancestors and itself.
Results
Experiments with BERT on SQuAD 2.0 and RACE achieve new state-of-the-art or comparative results across span extraction and multi-choice inference.
Takeaways & Limitations
The results support using syntactic structural information to guide attention learning in challenging reading comprehension tasks.
Takeaways & Limitations
The evaluation compares single models rather than ensemble models to focus on syntactic advances and maintain simplicity.

Abstract

from arXiv · show

For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.

1 Introduction

SG-Net addresses noisy, unfocused attention in lengthy, multi-sentence machine reading comprehension passages by guiding self-attention with dependency syntax. It fuses original and syntax-guided self-attention and substantially improves a BERT baseline on challenging MRC tasks.

Motivation: Lengthy, detailed passages and long questions reduce MRC accuracy because noise and vague attention make relevant text components difficult to identify.Current attentive models often process every word equally and treat passages as undifferentiated sequences, ignoring sentence-internal linguistic structure.
Motivation: SG-Net explicitly guides attention toward syntactically important words and their related subsets, reducing noise from lengthy sentences.The approach uses dependency-structure clues to produce more accurate attentive signals for each concerned word.
Method: SG-Net fuses the original self-attention network with an SDOI-based self-attention network to produce more linguistically inspired representations.The design extends the Transformer encoder’s self-attention mechanism with explicit syntax-guided constraints.
Method: Syntactic dependency of interest (SDOI) assigns each word its ancestors and itself in a pre-trained dependency parse tree.This structure identifies syntax-related nodes for syntax-guided self-attention.
Contribution: The proposed design integrates syntactic relationships into Transformer attention, applies SG-Net to pre-trained BERT (Devlin et al. 2018), and substantially boosts the strong baseline on challenging MRC tasks.The paper presents this integration as the first of its kind for enhancing a state-of-the-art Transformer self-attention network.

2 Related Work

MRC has progressed from cloze-style tests to span-extraction and multiple-choice tasks, alongside attentive and pretrained language models. Recent work shows syntactic information benefits language understanding, motivating syntax-guided representations that improve Transformer self-attention.

MRC Tasks: MRC evolved from cloze-style tests to span-based answer extraction and multiple-choice tasks, with the latter two forming this work’s focus.
Attentive Models: Attentive MRC models include Attention Sum Reader, Gated attention Reader, Self-matching Network, Attention over Attention Reader, and Bi-attention Network.
Pretrained Language Models: Deep contextual language models such as ELMo, OpenAI GPT, and BERT (Devlin et al. 2018) learn universal representations from unlabeled data and achieve strong NLU results.
Syntactic Information: Neural dependency parsing has reached state-of-the-art results, enabling neural models to gain accuracy by leveraging syntactic information rather than ignoring it.
Syntax-Guided Representations: Dependency trees encode linguistic relations, but prior work often concatenated dependency-label embeddings with word representations; this work instead introduces syntax-guided representations for Transformer SAN improvement.The approach updates each word representation with information from dependency-tree neighbors under explicit syntactic constraints, which is linguistically motivated.

3 Syntax-Guided Network

SG-Net augments a Transformer encoder with syntax-guided self-attention that restricts attention to syntactically important ancestor words, then fuses syntax-guided and original representations for task-specific predictions. The architecture is implemented with BERT for span-based SQuAD 2.0 and multi-choice RACE reading comprehension.

General Architecture: SG-Net fuses syntax-guided SAN representations with the original Transformer SAN output before task-specific prediction layers.The model takes SAN-based encoder representations, builds a syntax-guided SAN, and aggregates both contextual representations.
Syntax-Guided Self-Attention: The SDOI mask restricts each token’s attention to itself and its ancestor head words in the syntactic dependency tree.A dependency parser produces each token’s ancestor set and an n × n mask encoding which syntactically important tokens can be attended to; special BERT tokens attend only to themselves.
Syntax-Guided Self-Attention: The syntax-guided attention layer projects Transformer outputs into multi-head queries, keys, and values, applies dependency-masked attention, and uses feed-forward layers with residual layer normalization.The dependency structure is incorporated inside multi-head attention to produce syntax-guided token representations.

4 Experiments

Experiments evaluate SG-Net against fine-tuned whole-word-masking BERT on span-based SQuAD 2.0 and multi-choice RACE. SG-Net significantly improves over BERT and other baselines on both benchmarks, while using a fixed preprocessing parser and single-model comparisons.

Datasets and baselines: Experiments use fine-tuned cased whole-word-masking BERT as the baseline on SQuAD 2.0 and RACE, covering span-based and multi-choice machine reading comprehension.SQuAD 2.0 requires answering supported questions and abstaining when the passage provides no answer; RACE contains middle- and high-school subsets and uses accuracy.
Implementation: The dependency parser is applied during preprocessing and remains fixed during MRC training, achieving 97.00% UAS and 95.43% LAS on the PTB test set.The parser jointly learns constituent parsing with BERT as its sole input.
Implementation: The MRC implementation uses wordpiece tokenization, a maximum input length of 384, dual-context aggregation weight α = 0.5, and BERT-matched multi-head self-attention.Learning rates, batch sizes, and epoch limits are selected from specified ranges according to task.
SQuAD 2.0: SG-Net significantly outperforms the BERT baseline on SQuAD 2.0 (p-value < 0.01), surpasses all published works, and ranked second on the leaderboard at submission.The comparison uses single models rather than ensembles; Table 1 reports Exact Match and F1 scores.
RACE: SG-Net significantly outperforms BERT and all listed baselines on RACE accuracy (p-value < 0.01), verifying the effectiveness of syntax enhancement.The comparison includes DCMN, OCN, RSM, and GPT; Table 2 reports single-model test accuracy.

5 Discussions

SG-Net is more robust than the baseline on lengthy questions, while syntax-guided attention focuses on syntactically relevant answer-bearing words. Ablations show that combining syntax-guided and vanilla attention through dual contextual attention performs best.

Question-Length Analysis: SG-Net remains robust on long SQuAD questions, whereas baseline performance drops sharply beyond 20 words.The study groups SQuAD development questions into 20 equal-range subsets and compares exact-match accuracy by question length.
Attention Visualization: Syntax-guided attention highlights question keywords and passage words that identify the correct answer.For the example, it emphasizes “name,” “legislation,” and “1850,” along with “Missouri” and “Compromise.”
Ablation Study: Dual contextual attention yields the best performance among vanilla-only, syntax-guided-only, and combined variants.The comparison evaluates adding each extra attention layer after BERT, with the combined design adopted in SG-Net.
Ablation Study: Vanilla attention alone adds no improvement, while syntax-guided attention alone is complementary to vanilla attention because it excludes noisy but diverse information.This complementarity motivates integrating both representations in the dual contextual layer; alternative merging operations were also compared, with dual contextual attention performing best.

6 Conclusion

SG-Net enhances Transformer-based encoders by incorporating syntactic constraints into attention and fusing original and syntax-guided representations. With pre-trained BERT, it achieves new state-of-the-art or comparative results on SQuAD 2.0 and RACE.

6 Conclusion: SG-Net adopts a dual contextual architecture that fuses original SAN representations with syntax-guided SAN representations.The framework incorporates syntactic constraints into the attention mechanism to produce linguistically motivated word representations.
6 Conclusion: The work presents syntax-guided text modeling as a framework for enhancing strong Transformer-based encoders.Its empirical results support the effectiveness of syntactic structure for machine reading comprehension.
6 Conclusion: Experiments with pre-trained BERT yield new state-of-the-art or comparative results on SQuAD 2.0 and RACE.The benchmarks cover span-based answer extraction and multi-choice inference.