Source-linked AI summary
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, Dongsheng Li
TL;DR
Unanswerable-question comprehension requires models to abstain when no answer can be inferred, while prior systems do not verify predicted-answer legitimacy. The paper proposes a read-then-verify system with auxiliary reader losses and three verifier architectures, achieving 74.2 F1 on SQuAD 2.0 test data.
Problem
Prior systems detect unanswerable questions with no-answer probabilities but do not verify whether predicted answers are legitimate.
Method
The system combines a reader with independent span and no-answer losses and an answer verifier that compares answer sentences with questions using three architectures.
Results
74.2 F1 was achieved on the SQuAD 2.0 test set, reported as state-of-the-art at submission.
Takeaways & Limitations
The approach validates candidate-answer legitimacy after extraction to support abstention on questions without answers.
Takeaways & Limitations
Training uses plausible answers annotated for every unanswerable SQuAD 2.0 question.
Abstract
from arXiv · showhide
Machine reading comprehension with unanswerable questions aims to abstain from answering when no answer can be inferred. In addition to extract answers, previous works usually predict an additional "no-answer" probability to detect unanswerable cases. However, they fail to validate the answerability of the question by verifying the legitimacy of the predicted answer. To address this problem, we propose a novel read-then-verify system, which not only utilizes a neural reader to extract candidate answers and produce no-answer probabilities, but also leverages an answer verifier to decide whether the predicted answer is entailed by the input snippets. Moreover, we introduce two auxiliary losses to help the reader better handle answer extraction as well as no-answer detection, and investigate three different architectures for the answer verifier. Our experiments on the SQuAD 2.0 dataset show that our system achieves a score of 74.2 F1 on the test set, achieving state-of-the-art results at the time of submission (Aug. 28th, 2018).
Introduction
The paper addresses unanswerable questions by adding answer-legitimacy verification to neural reading comprehension. Its read-then-verify system combines auxiliary reader losses, an answer verifier, and final prediction aggregation, achieving 74.2 F1 on SQuAD 2.0 test data.
- Motivation: Existing systems predict candidate answers and no-answer probabilities but do not further verify whether predicted answers are legitimate.Legitimacy means that the extracted text is supported by the passage and question.
- Approach: The proposed system first extracts a candidate and estimates no-answer probability, then verifies answer legitimacy before producing the final prediction.The pipeline aggregates the reader and verifier outputs for the final decision.
- Reader Enhancements: Two auxiliary losses separately improve answer extraction and no-answer detection in the reader.The independent span loss supports candidate extraction, while the independent no-answer loss focuses on detecting unanswerable questions.
- Answer Verification: The answer verifier compares the answer sentence with the question to find local entailment supporting the extracted answer.Three verifier architectures are investigated: sequential, interaction-based, and hybrid models.
- Results: 74.2 F1 was achieved on the SQuAD 2.0 test set, reported as state-of-the-art at submission.The submission date was Aug. 28th, 2018.
Background
Unanswerable reading comprehension requires models to answer supported questions while detecting when no answer exists. Standard readers represent passages and questions, predict answer spans, and add a no-answer score normalized jointly with span scores.
- Task Motivation: Earlier reading-comprehension models generally assume that every question has a correct answer in the passage.This assumption leads models to select plausible text spans rather than first checking answer existence.
- Task Definition: The task requires predicting an answer span for answerable questions or an empty string when no answer exists.Passages and questions are represented as token sequences, and answer boundaries define the extracted span.
- Answer Extraction: Readers encode passage–question interactions and use pointer networks to produce scores for answer start and end positions.Attention mechanisms build interdependent passage and question representations before span scoring.
- No-Answer Detection: Previous approaches add a no-answer score and jointly normalize it with answer-span scores to detect unanswerable questions.A normalized no-answer score exceeding a threshold triggers an unanswerable prediction.
Approach
The proposed system reads a passage to extract a candidate answer and detect no-answer cases, then verifies whether the candidate is legitimate. It improves the reader with independent auxiliary losses and evaluates sequential, interactive, and hybrid verifier architectures.
- System overview: The read-then-verify system combines a neural reader for candidate extraction and no-answer detection with an answer verifier for checking answer legitimacy.The verifier compares the answer sentence with the question to assess supporting local entailment.
- Auxiliary losses: The independent span loss trains the reader to extract plausible candidates for unanswerable questions, which downstream verification requires.Human-annotated plausible answers are used as gold answers for unanswerable cases in SQuAD 2.0.
- Auxiliary losses: The independent no-answer loss encourages confident no-answer predictions without the shared normalization that can conflict with span scoring.It uses a sigmoid-based objective for the no-answer score.
- Answer verifier: The verifier compares an answer sentence with a question to recognize local textual entailment supporting the extracted answer.The answer sentence contains either a gold answer or a plausible answer, and three architectures are explored.
- Answer verifier: The three verifier designs are sequential, interactive, and hybrid, respectively using ordered encoding, interdependent sentence representations, or merged representations.The sequential model adapts a pretrained Transformer decoder, while the interactive model captures cross-sentence interactions and self-correlations.
Experimental Setup
The system is evaluated on SQuAD 2.0, which combines answerable SQuAD 1.1 questions with 53,775 unanswerable questions. Testing uses a reader-verifier pipeline and compares approaches on the hidden test set.
- Dataset: SQuAD 2.0 combines answerable SQuAD 1.1 questions with 53,775 unanswerable questions about the same passages.The benchmark tests whether models know when they do not know an answer.
- Training: The reader is trained on context passages, while the answer verifier is trained on oracle answer sentences.Model-I additionally uses unsupervised language-model pretraining followed by supervised fine-tuning; Model-II is trained directly with supervised loss.
- Inference: At test time, the verifier combines its sentence-level probability with the reader’s passage-level no-answer probability to detect unanswerable questions.The mean of the two probabilities is compared with a threshold tuned for development-set F1.
Evaluation
On SQuAD 2.0, the proposed auxiliary losses and answer verifier improve no-answer detection and overall performance across reader and verifier configurations. The best system achieves state-of-the-art test performance, while error analysis identifies remaining classification challenges.
- Main Results: 71.7 EM and 74.2 F1 are achieved on the SQuAD 2.0 test set, establishing state-of-the-art results at submission.
- Ablation Study: Removing both auxiliary losses decreases overall F1 by more than 1.5 points with or without ELMo embeddings.The independent span loss mainly supports answer-boundary identification, while the independent no-answer loss chiefly supports NoAns ACC.
- Ablation Study: 76.2 no-answer accuracy is achieved by Model-III, whose combined architectures outperform the other verifier designs.Adding ELMo does not further improve verifier performance.
- Ablation Study: Adding Model-III raises no-answer accuracy from 73.1 to 77.1 for RMR, a 4-point absolute increase.The gains remain consistent when ELMo embeddings are used.
- Ablation Study: With DocQA, Model-III increases no-answer accuracy from 69.1 to 75.2, while DocQA + ELMo improves from 70.6 to 76.1.
- Ablation Study: RMR + ELMo + Verifier has the best precision below 80 recall, but RMR + ELMo becomes slightly better above 80 recall.Removing both auxiliary losses degrades the precision-recall curve overall, though it remains above the baseline.
- Error Analysis: The verifier improves answerable-case detection, whereas auxiliary-loss training mainly improves performance on unanswerable cases.The error analysis reports a 6.5% Case3 answer-extraction error for RMR + ELMo + Verifier and larger no-answer classification errors.
Related Work
Earlier reading-comprehension datasets generally guaranteed that the context contained an answer, limiting evaluation of abstention. Later approaches introduced negative examples and no-answer probabilities, while this work extends that line with independently enhanced extraction and no-answer detection.
- Reading Comprehension Datasets: Earlier large-scale reading-comprehension datasets generally guaranteed that each context contained an answer.
- Reading Comprehension Datasets: Recent datasets and methods introduced negative examples or no-answer probabilities to address unanswerable questions.
- Neural Networks for Reading Comprehension: The proposed no-answer reader adds two auxiliary losses that separately enhance answer extraction and no-answer detection.
Conclusion
The proposed read-then-verify system combines auxiliary reader losses with answer verification to abstain on unanswered questions and achieved state-of-the-art SQuAD 2.0 results at submission.
- The system uses two auxiliary losses to focus separately on answer extraction and no-answer detection.
- An answer verifier validates the legitimacy of predicted answers, with three verifier architectures investigated.
- The system achieved state-of-the-art results on SQuAD 2.0 at the time of submission.