Source-linked AI summary
Retrospective Reader for Machine Reading Comprehension
Zhuosheng Zhang, Junjie Yang, Hai Zhao
TL;DR
MRC must both answer questions supported by a passage and abstain from answering unanswerable questions, but recent modeling has emphasized pre-trained encoders over verifier design. This paper proposes Retro-Reader, which combines sketchy and intensive reading for staged verification. It achieves new state-of-the-art results on SQuAD2.0 and NewsQA and significantly outperforms strong ELECTRA and ALBERT baselines.
Problem
MRC with unanswerable questions requires answerability verification in addition to reading comprehension, while recent readers largely focus on pre-trained encoders.
Method
Retro-Reader combines sketchy reading for an initial judgment with intensive reading for answer verification and final prediction.
Results
Retro-Reader achieves new state-of-the-art results on SQuAD2.0 and NewsQA and significantly outperforms strong ELECTRA and ALBERT baselines.
Takeaways & Limitations
The results support verifier-oriented reader design as an important component even when powerful pre-trained language models are used as encoders.
Abstract
from arXiv · showhide
Machine reading comprehension (MRC) is an AI challenge that requires machine to determine the correct answers to questions based on a given passage. MRC systems must not only answer question when necessary but also distinguish when no answer is available according to the given passage and then tactfully abstain from answering. When unanswerable questions are involved in the MRC task, an essential verification module called verifier is especially required in addition to the encoder, though the latest practice on MRC modeling still most benefits from adopting well pre-trained language models as the encoder block by only focusing on the "reading". This paper devotes itself to exploring better verifier design for the MRC task with unanswerable questions. Inspired by how humans solve reading comprehension questions, we proposed a retrospective reader (Retro-Reader) that integrates two stages of reading and verification strategies: 1) sketchy reading that briefly investigates the overall interactions of passage and question, and yield an initial judgment; 2) intensive reading that verifies the answer and gives the final prediction. The proposed reader is evaluated on two benchmark MRC challenge datasets SQuAD2.0 and NewsQA, achieving new state-of-the-art results. Significance tests show that our model is significantly better than the strong ELECTRA and ALBERT baselines. A series of analysis is also conducted to interpret the effectiveness of the proposed reader.
1 Introduction
MRC systems must answer questions from passages while recognizing when questions are unanswerable, yet recent progress has emphasized powerful encoders over verifier design. Retro-Reader addresses this gap with sketchy and intensive reading stages, and reports substantial gains over strong baselines.
- Motivation: MRC requires systems to answer passage-based questions and abstain when the passage provides no answer.
- Motivation: Recent MRC readers primarily focus on pre-trained language-model encoders, while decoder and verifier design receives less attention.
- Motivation: Unanswerable-question MRC requires both accurate answers for answerable questions and effective discrimination of unanswerable questions.
- Proposed approach: Retro-Reader uses sketchy reading for an initial passage-question judgment followed by intensive reading to verify the answer and produce the final prediction.
- Contributions: The proposed reader performs answer verification rather than simply stacking a verifier, with experiments reporting substantial improvements and new state-of-the-art benchmark results.
2 Related Work
Prior MRC work developed attention mechanisms and increasingly relied on pre-trained language-model encoders, while answer verification remained comparatively underdeveloped. Retro-Reader differs by using a two-stage reading design for verification instead of simply stacking verifier components.
- Encoder development: MRC research progressed from attention-based passage-question interactions to pre-trained language-model encoders that improve contextual representations and benchmark performance.
- Decoder and verification: Decoder research has reached a bottleneck amid powerful encoders, motivating focus on passage-question interactions and especially answer verification.
- Answer verification: Existing unanswerable-MRC approaches commonly add verification layers, jointly train answer-span and verification objectives, or set answerability thresholds.
- Retro-Reader: Retro-Reader adopts a two-stage humanoid design based on a survey of existing answer-verification solutions rather than simply stacking a verifier module.
3 Our Proposed Model
Retro-Reader uses parallel sketchy and intensive readers to verify answerability and answer spans before producing a final answer. Its design combines external and internal verification, rear verification, and optional question-aware matching within the intensive module.
- 3 Our Proposed Model: The model uses sketchy reading for preliminary answerability judgment and intensive reading for candidate spans, answerability verification, and final prediction.The implementation uses two parallel modules, although the intended process is staged.
- 3.1 Sketchy Reading Module: The external front verifier classifies the question as answerable or unanswerable from the pooled [CLS] representation.Its external verification score is scoreext = logitna − logitans.
- 3.2 Intensive Reading Module: The intensive reader predicts answer spans and uses an internal front verifier to assess answerability.The internal verifier can use cross-entropy, binary cross-entropy, or regression-style mean squared error losses.
- 3.2 Intensive Reading Module: Question-aware matching offers cross-attention and matching-attention alternatives, but it is used only for ablation and omitted from test submissions.Matching attention produces representations aligned between passage and question hidden states.
- 3.3 Rear Verification: Rear verification combines external and internal verification scores to decide whether to output the predicted span or a null string.The final decision uses v = β1scorediff + β2scoreext and compares v with threshold δ.
4 Experiments
The experiments evaluate Retro-Reader on SQuAD2.0 and NewsQA using pretrained-language-model baselines, official metrics, and significance testing. Results show improvements over strong baselines, state-of-the-art performance, and benefits from answer verification.
- Experimental Setup: The experiments use pretrained BERT, ALBERT, and ELECTRA encoders with published hyperparameters and threshold-based answerable verification for baseline comparisons.The reported implementations use official pretrained weights and denote baseline verification with +TAV.
- Experimental Setup: Retro-Reader is evaluated on the SQuAD2.0 and NewsQA benchmark MRC challenges.SQuAD2.0 includes answerable and adversarially written unanswerable questions, while NewsQA contains longer news-based paragraphs and unanswerable questions.
- Experimental Setup: Exact Match (EM) and token-level F1 are the two official evaluation metrics.EM requires an exact prediction, whereas F1 measures average token overlap with the ground-truth answer.
- Experimental Setup: McNemar’s test uses EM as a binary correctness measure to assess statistical significance at the conventional p = 0.05 reliability level.The authors select EM rather than F1 because answer-length variation can bias model comparisons.
- Verification Analysis: Combining both rear verification methods performs best, while either front verifier improves the baselines.The ablation compares different answer-verification methods on the SQuAD2.0 development set.
- Main Results: Retro-Reader significantly outperforms ALBERT and ELECTRA baselines with p-value < 0.01 and achieves new state-of-the-art results on SQuAD2.0.The implemented baselines obtain EM and F1 scores similar to the original reported values, supporting the comparison.
- Main Results: Retro-Reader delivers consistent improvements over baselines and new state-of-the-art results on NewsQA.This result supports effectiveness across both evaluated benchmark datasets.
5 Ablations
The ablations examine answer-verification designs, parameter-matched alternatives, interaction methods, and prediction behavior. They show that combining verification mechanisms and using distinct two-stage objectives improves robustness beyond simply adding parameters or layers.
- 5.1 Evaluation on Answer Verification: Combining the front verifiers E-FV and I-FV as rear verification gives the best performance among the tested answer-verification methods.Final predictions rely on threshold-search answerability scores, creating a trade-off between HasAns and NoAns accuracy.
- 5.1 Evaluation on Answer Verification: Retro-Reader improves substantially on the 5,945 unanswerable SQuAD2.0 development questions, especially on F1 and accuracy.
- 5.2 Comparisons with Equivalent Parameters: Retro-Reader outperforms two ensembled baseline models with equivalent parameters, supporting complementary learning from span prediction and answer-verification objectives.The two parallel modules use independent parameters and share Transformer encoder design while optimizing different training objectives.
- 5.3 Question-aware Matching: Merely adding layers produces no noticeable improvement, whereas answer verification yields consistent and substantial gains on SQuAD2.0 and NewsQA.The matching-interaction methods are evaluated on the development sets of both datasets.
- 5.4 Comparison of Predictions: A prediction example shows Retro-Reader better judges answerability and avoids selecting a plausible answer unsupported by the passage.
6 Conclusion
The paper presents Retro-Reader as a verifier-oriented design for MRC with unanswerable questions. Across SQuAD2.0 and NewsQA, it reports state-of-the-art results and significance-test improvements over strong baselines, while identifying verifier design as important beyond powerful encoders.
- 6 Conclusion: Retro-Reader addresses unanswerable-question MRC through two-stage sketchy and intensive reading, focusing on verifier-oriented task-specific design.
- 6 Conclusion: The model achieves new state-of-the-art results on SQuAD2.0 and NewsQA and significantly outperforms strong ELECTRA and ALBERT baselines.
- 6 Conclusion: The conclusion identifies verification mechanisms as significantly affecting MRC performance and treats the verifier as an indispensable component even with powerful pre-trained encoders.
- 6 Conclusion: Future work will investigate additional decoder-side problem-solving techniques that cooperate with strong encoders.