Source-linked AI summary
Dual Co-Matching Network for Multi-choice Reading Comprehension
Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, Xiang Zhou
TL;DR
Multi-choice reading comprehension requires choosing among candidate answers and prior matching methods may miss bidirectional or some pairwise relationships. The paper uses bidirectional co-matching with gated fusion, reporting new state-of-the-art results on RACE and ROCStories, including a single model that outperforms human turkers on RACE full.
Problem
Prior matching methods are often unidirectional and may omit pairwise relationships among passage, question, and answer, despite all three being important in multi-choice MRC.
Source: §1 · p. 1Method
The model matches passage–question, passage–answer, and question–answer representations bidirectionally and uses gating to fuse the two directions.
Source: §1 · p. 1, §2.2 · p. 2Results
2.6% on RACE and 1.2% on ROCStories over previous state-of-the-art results; the single model also outperforms human turkers on RACE full.
Source: §4 · p. 4Takeaways & Limitations
The single-model result on RACE full is reported as the first time a model outperformed human turkers on that dataset.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Multi-choice reading comprehension is a challenging task that requires complex reasoning procedure. Given passage and question, a correct answer need to be selected from a set of candidate answers. In this paper, we propose \textbf{D}ual \textbf{C}o-\textbf{M}atching \textbf{N}etwork (\textbf{DCMN}) which model the relationship among passage, question and answer bidirectionally. Different from existing approaches which only calculate question-aware or option-aware passage representation, we calculate passage-aware question representation and passage-aware answer representation at the same time. To demonstrate the effectiveness of our model, we evaluate our model on a large-scale multiple choice machine reading comprehension dataset (i.e. RACE). Experimental result show that our proposed model achieves new state-of-the-art results.
1 Introduction
Multi-choice MRC requires selecting among candidate answers, which may not appear in the passage, and therefore calls for deeper understanding. DMN models passage, question, and answer relationships bidirectionally, and reports new state-of-the-art results on RACE and ROCStories.
1 Introduction
- Multi-choice answers may not appear in the passage, allowing question types such as commonsense reasoning and passage summarization.The task requires deeper understanding of the passage and question.Source: §1 · p. 1
- Earlier matching methods are usually unidirectional and can omit pairwise relationships, including question–answer matching.The paper argues that passage, question, and answer are equally important in multi-choice MRC.Source: §1 · p. 1
- DMN models passage–question, passage–answer, and question–answer relationships simultaneously and bidirectionally.It uses a gated mechanism to fuse representations from the two directions.Source: §1 · p. 1
- The gated fusion is reported to be more effective than simple concatenation.The introduction contrasts gating with a simple concatenation strategy.Source: §1 · p. 2
- 2.6% on RACE and 1.2% on ROCStories over previous best results; the single model also outperforms human turkers on RACE full.The paper reports these as new state-of-the-art results.Source: §1 · p. 2
2 Model
The model selects an answer from candidates using BERT-encoded passage, question, and answer sequences. It builds bidirectional representations for their pairwise relationships, fuses them with a gate, and combines them for candidate scoring.
2 Model
- Multi-choice reading comprehension selects the correct answer from candidate answers given a passage and question.Source: §2 · p. 2
2.1 Encoder
- BERT's last-layer outputs encode the passage, question, and candidate answer as token-level sequence representations.Their sequence lengths are |P|, |Q|, and |A|, with hidden-state dimension l.Source: §2.1 · p. 2
2.2 Matching
- Matching builds all three pairwise relationships bidirectionally, uses symmetric representations, and fuses directions with a gate reported to outperform concatenation.The passage-answer procedure also applies to passage-question and question-answer pairs.Source: §2.2 · pp. 2–3
2.3 Objective function
- For each candidate triplet, the three pairwise matching representations are concatenated into C_i ∈ R^3l for the objective function.Source: §2.3 · p. 3
3 Experiment
Experiments on RACE and ROCStories report state-of-the-art performance, with the single model exceeding Amazon Turkers on RACE full dataset.
3 Experiment
- RACE has 27,933 passages and 97,687 questions across middle- and high-school difficulty subsets; ROCStories contains 50k five-sentence stories with a two-option ending task.Source: §3 · p. 3
3.1 Experiment Setup
- The BERTlarge setup uses a 512-token maximum sequence length, 0.1 dropout in every BERT layer, 0.3 matching-layer dropout, and trains for 10 epochs with batch size 4.Source: §3.1 · p. 3
3.2 Comparison with Baselines
- The single model outperforms all baselines and is the first reported to exceed Amazon Turkers on RACE full dataset.Source: Table 2 · p. 4, §3.2 · p. 4
- The comparison introduces bidirectional matching and a gated mechanism to model pairwise passage-question-answer relationships.Source: §3.2 · p. 4
3.3 Ablation Study
- 1.5% decrease follows unidirectional matching, 0.5% replacing gating with concatenation, and 0.4% removing question-answer matching; ROCStories ablations also support these findings.Source: §3.3 · p. 4
3.2 Comparison with Baselines
- 1.2% improvement over BERTlarge gives DMN state-of-the-art accuracy on ROCStories.Source: Table 3 · p. 4, §3.2 · p. 4
4 Conclusions
The paper concludes that Dual Co-Matching Network models bidirectional relationships among passage, question, and candidate answer sequences. With BERT, it reports state-of-the-art results on ROCStories and RACE, and a single model exceeding human turkers on RACE full dataset.
- Dual Co-Matching Network models bidirectional sequence relationships among passage, question, and candidate answer, and with BERT achieves state-of-the-art results on ROCStories and RACE.Source: §4 · p. 4