Source-linked AI summary
What Makes Reading Comprehension Questions Easier?
Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa
TL;DR
MRC datasets may include questions answerable through superficial cues, leaving unclear how much they test language understanding. The paper uses simple heuristics to split questions into easy and hard subsets; hard subsets show degraded baseline performance, and their questions demand more complex reasoning.
Problem
It remains unclear to what extent MRC questions require understanding of their texts rather than superficial cues.
Source: §1 · pp. 1–2Method
The study applies simple heuristics to split 12 MRC datasets into easy and hard subsets and compares two baseline models on those subsets.
Source: §1 · p. 2Results
Hard-subset baseline performance degrades markedly, and hard questions require more knowledge inference and multiple-sentence reasoning than easy questions.
Source: §1 · p. 2Takeaways & Limitations
The findings suggest that baseline ability may be overestimated and emphasize analyzing and reporting new datasets’ properties.
Source: §7 · p. 9The authors caution that removing questions solvable by their heuristics to create a hard subset may generate new biases.
Source: §3.3 · p. 6
Abstract
from arXiv · showhide
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.
1 Introduction
The paper asks whether MRC questions genuinely test language understanding or can be solved through superficial cues. It investigates this across 12 datasets and three question styles using heuristics and question annotations.
1 Introduction
- Some questions may be answered using entity-type cues or word overlap with a single related sentence, without complex language understanding.Source: §1 · pp. 1–2
- The study uses simple heuristics to split datasets into easy and hard subsets and compares two baseline models on each subset.Source: §1 · p. 2
- The study investigates 12 recent MRC datasets spanning answer-extraction, description, and multiple-choice questions.Source: §1 · p. 2
- Questions sampled from each subset are manually annotated for validity and required reasoning skills to examine what distinguishes easy from hard questions.Source: §1 · p. 2
- Baseline performance on hard subsets degrades markedly, and those questions require more knowledge inference and multiple-sentence reasoning than easy questions.Source: §1 · p. 2
- Hard questions require knowledge inference and multiple-sentence reasoning compared with easy questions.Source: §1 · p. 2
- Multiple-choice questions tend to require a broader range of reasoning skills than answer-extraction and description questions.Source: §1 · p. 2
- The findings suggest that recent MRC advances may be overestimated and that question sourcing should account for simple answer-seeking heuristics.Source: §1 · p. 2
2 Examined Datasets and Baselines
The study examines 12 MRC datasets spanning answer extraction, description, and multiple choice. It evaluates style-appropriate baselines, using different systems for multiple-choice datasets because some questions lack textual overlap with the context.
2.1 Datasets
- The study covers 12 datasets selected to vary in genre, context length, and question-sourcing method, across answer extraction, description, and multiple choice.The authors note that additional datasets could be analyzed using the same method.Source: §2.1 · pp. 2–3
2.2 Baseline Systems
- BiDAF was used for answer extraction and description, while the Gated-Attentive Reader was used for multiple choice because some questions lack textual overlap to identify a context answer span.BiDAF models bidirectional attention; the Gated-Attentive Reader uses multi-hop attention.Source: §2.2 · p. 3
- Performance was measured with EM/F1 for answer extraction, Rouge-L for description, and accuracy for multiple choice.For description datasets, the authors selected in advance the context span with the highest Rouge-L against the human-generated gold answer.Source: §2.2 · p. 3
- The implementations matched or outperformed official baseline results on most datasets, but fell short on TriviaQA, MCTest, RACE, and ARC-E.The authors attribute the differences to architecture or unavailable reported hyperparameters.Source: §2.2 · p. 3
3 Two Filtering Heuristics
The paper tests whether simple question cues and answer location relate to baseline performance, then uses those heuristics to define easy and hard subsets. Hard-subset performance drops in almost all examined datasets.
3 Two Filtering Heuristics
- The study tests entity-type recognition and attention heuristics by examining baseline performance on questions filtered by each cue.The two heuristics target answers recoverable from question tokens and answers located in the sentence most similar to the question.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
3.1 Entity Type-based Heuristic
- As k decreased, baseline performance fell on all datasets in Table 2 except QAngaroo, while QAngaroo and multiple-choice performance degraded less strongly.For QAngaroo, the full-set versus k = 1 score difference was 1.8; in most multiple-choice datasets, k = 1 scores exceeded random-choice scores.Source: §3.1 · p. 4
3.2 Attention-based Heuristic
- Most datasets had a high proportion of questions whose answers appeared in the most similar sentence, and answer extraction and description baselines generally did better on those questions.The pattern did not hold for QAngaroo; for the first three multiple-choice datasets, performance differences were marginal or reversed.Source: §3.2 · p. 5
- Removing all but the most similar sentence raised AddSent and QAngaroo scores by more than 20 F1, which the authors interpret as evidence that other context sentences distract the baselines.The AddSent observation is consistent with its manually injected distracting sentences.Source: §3.2 · p. 5
3.3 Performance on Hard Subsets
- Hard subsets excluded questions with a positive score at k = 2 and questions whose correct answer appeared in the most similar sentence; the remaining questions formed the easy subsets.The split was intended to investigate performance gaps, not to create a replacement dataset by removing questions.Source: §3.3 · pp. 5–6
- Baseline performance on hard subsets dropped remarkably in almost all examined datasets, suggesting previously reported baseline ability may be overestimated.The authors emphasize using the heuristics when sourcing questions rather than removing heuristic-solvable questions, which could generate new biases.Source: Table 2 · p. 4, §3.3 · pp. 5–6, Table 3 · p. 5
4 Annotating Question Validity and Required Skills
The authors annotated sampled questions for validity and required reasoning skills, then compared these labels across easy and hard subsets. Hard subsets more often involved knowledge and multiple-sentence reasoning, while reasoning-skill patterns also differed by question style and baseline.
4.1 Annotation Specifications
- Annotators labeled the minimum skills needed to choose the correct answer, assuming solvers knew the entity or event type entailed by the question.The labels included detailed validity, such as candidate-answer counts and ambiguity, and treated multiple-sentence reasoning as compatible with other skills.Source: §4.1 · p. 6
4.2 Annotation Results
- TriviaQA, QAngaroo, and ARCs had relatively high unsolvability, which the authors linked to questions and contexts being unrelated or insufficiently informative.Unsolvable questions were an exception among the valid questions reported for the datasets.Source: §4.2 · pp. 6, 8
- Few questions had only one answer candidate, with fewer in AddSent than SQuAD; this supports the claim that adversarial examples increase possible candidates and degrade baseline performance.Source: §4.2 · p. 7
- Ambiguous questions had multiple correct spans, which can make exact-match and F1 scoring unstable.Source: §4.2 · p. 7
- In 10 of 12 datasets, knowledge was more pertinent to hard subsets, and multiple-sentence reasoning also correlated more with hard subsets.NewsQA showed an inverse tendency for word matching, knowledge, and multiple-sentence reasoning; the authors suspect annotation variance and filtering ambiguous questions contributed.Source: §4.2 · p. 8
- Meta/whole and math/logic skills were less needed in answer-extraction and description datasets, and more pertinent in multiple-choice datasets.Source: §4.2 · p. 8
- BiDAF performed well on word-matching questions and relatively poorly on knowledge questions; the same trend was not observed for GA.The authors report statistically significant correlations for six annotation labels despite the small annotated samples.Source: §4.2 · p. 8
5 Discussion
The discussion weighs the trade-offs among answer-extraction, description, and multiple-choice questions. It interprets the proposed heuristics as identifying questions that require multiple hypotheses and complex reasoning to distinguish.
5 Discussion
- Answer extraction is easy to generate at scale but rarely demands meta/whole or math/logic skills, and may not include every possible correct span.Source: §5 · p. 8
- Description and multiple-choice styles allow answers beyond context spans, but description evaluation is difficult with Rouge-L and BLEU, while generating reasonable multiple-choice options takes considerable effort.Source: §5 · p. 8
- Under the paper’s entailment framing, easier questions need one hypothesis whose premises directly describe it; more realistic questions need multiple hypotheses, complex reasoning, and external knowledge.Source: §5 · p. 8
6 Related Work
The related work frames the study around unintended biases in question answering and evaluation overfitting. It contrasts dataset-specific task evaluation with skill-oriented analysis and notes that the study uses a less fine-grained reasoning classification.
6 Related Work
- The study’s heuristic analysis is motivated by unintended-bias concerns, while its annotation is motivated by evaluation-overfitting concerns.The cited motivations are Levesque (2014) and Whiteson et al. (2011), respectively.Source: §6 · p. 8
- Prior work highlights pattern-matching shortcuts in NLU; in SQuAD, 92% of answerable questions could be answered using only one context sentence.Source: §6 · pp. 8–9
- Task-oriented evaluation tests systems on specific datasets, whereas skill-oriented evaluation studies relationships between tasks and skills and can encourage more realistic NLU systems.Source: §6 · p. 9
- The study did not use finer-grained reasoning classifications, including taxonomies developed for toy tasks, MRC, entailment, and science exams.Source: §6 · p. 9
7 Conclusion
The study applies two heuristics to 12 MRC datasets, dividing questions into easy and hard subsets and annotating their validity and required reasoning skills. Baseline performance degraded on hard questions, which more often required knowledge inference and multiple-sentence reasoning.
7 Conclusion
- Across 12 MRC datasets, the heuristics split questions into easy and hard subsets, where baselines performed worse on hard questions requiring more knowledge inference and multiple-sentence reasoning.The authors caution that these results suggest baseline abilities may be overestimated.Source: §7 · p. 9
- The authors identify annotation cost as a limitation and propose exploring automatic classification of reasoning types.They state that this could enable system evaluation through a detailed organization of datasets.Source: §7 · p. 9
- The conclusion emphasizes analyzing and reporting new datasets’ properties when they are released.Source: §7 · p. 9
A Hyperparameters of the Baseline Systems
The baseline systems use dataset-specific hyperparameters to accommodate differences such as context length, while retaining settings from prior implementations.
A Hyperparameters of the Baseline Systems
- Hyperparameters for both baseline systems vary by dataset, reflecting differences such as context length.Source: A Hyperparameters of the Baseline… · p. 12
- Bidirectional Attention Flow settings include batch size, hidden-layer size, and document and question size thresholds; other settings follow its original implementation.TriviaQA uses the dataset preparation method from Joshi et al. (2017).Source: Table 8 · p. 12
- Gated-Attentive Reader settings include batch size, hidden-layer size, attention layers, dropout, and learning rate; other settings follow Lai et al. (2017).Source: Table 9 · p. 12