Source-linked AI summary
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks
Divyansh Kaushik, Zachary C. Lipton
TL;DR
The paper addresses poorly characterized difficulty in reading-comprehension benchmarks, especially whether questions, passages, and full context are necessary. It evaluates question-only and passage-only baselines and context ablations, finding surprisingly strong reduced-information performance and comparable results from CBT’s last sentence alone.
Problem
Reading-comprehension benchmark difficulty is poorly characterized because performance without questions or passages, and the amount of necessary passage text, is often unreported.
Method
The paper analyzes popular reading-comprehension datasets and models using question-only, passage-only, and corrupted-data baselines, plus CBT sentence-level context ablations.
Results
Question-only and passage-only models often perform surprisingly strongly, with some matching or surpassing full models; on CBT, the last sentence can yield comparable or better performance than all 20 sentences.
Takeaways & Limitations
Benchmarks should report baselines omitting questions and passages and test how much context models truly require.
Takeaways & Limitations
The paper speculates that publishing incentives may discourage unfavorable ablations and encourage unsubstantiated explanations of architectural effectiveness.
Abstract
from arXiv · showhide
Many recent papers address reading comprehension, where examples consist of (question, passage, answer) tuples. Presumably, a model must combine information from both questions and passages to predict corresponding answers. However, despite intense interest in the topic, with hundreds of published papers vying for leaderboard dominance, basic questions about the difficulty of many popular benchmarks remain unanswered. In this paper, we establish sensible baselines for the bAbI, SQuAD, CBT, CNN, and Who-did-What datasets, finding that question- and passage-only models often perform surprisingly well. On $14$ out of $20$ bAbI tasks, passage-only models achieve greater than $50\%$ accuracy, sometimes matching the full model. Interestingly, while CBT provides $20$-sentence stories only the last is needed for comparably accurate prediction. By comparison, SQuAD and CNN appear better-constructed.
1 Introduction
The paper argues that popular reading-comprehension benchmarks have poorly characterized difficulty, because models may perform well while ignoring questions, passages, or much of the available context.
- 1 Introduction: Reading comprehension benchmarks often lack reported question-only and passage-only baselines, leaving the necessity of each input unclear.Some passages may also contain answer-relevant information in predictable locations, such as the first or last sentence.
- 1 Introduction: The authors evaluate popular datasets and models using question-only and passage-only inputs.These experiments test whether strong performance requires combining questions with passages.
- 1 Introduction: Question-only and passage-only models obtain surprisingly strong results, sometimes outperforming models given both questions and passages.The paper presents this pattern as evidence that some benchmark tasks may be easier than their standard formulation suggests.
- 1 Introduction: The study addresses weaknesses in reading-comprehension benchmarks alongside related concerns previously raised for visual question answering and natural language inference.The authors position their analysis as part of broader scrutiny of benchmark artifacts.
2 Datasets
The paper reviews datasets with different passage sources, answer formats, and candidate structures, then creates corrupted inputs to isolate question- and passage-only information.
- 2 Datasets: CBT uses 20-sentence children’s-book passages, with the 21st sentence forming a cloze question and the missing word as the answer.It includes Named Entities, Common Nouns, Verbs, and Prepositions categories, with 10 part-of-speech-matched candidates per question.
- 2 Datasets: CNN contains over 1 million news articles whose anonymized entities prevent models from relying on entity identity across examples.Each question removes an entity from a highlight sentence; passages contain 26 entities on average and over 500 possible candidates.
- 2 Datasets: Who-did-What pairs articles about the same events and removes a person’s name from one article’s first sentence for prediction from the paired article.Unlike CNN, it does not anonymize entities and averages 3.5 candidate answers per question.
- 2 Datasets: bAbI contains 20 synthetically generated tasks with restricted clause diversity and a 150-word vocabulary, unlike CNN’s vocabulary of close to 120,000 words.The dataset was designed to help identify and rectify reading-comprehension system failings.
- 2 Datasets: SQuAD contains over 100,000 crowd-sourced questions about 536 article passages, with answers represented as variable-length spans.Its passages are shorter than those in CNN and Who-did-What.
- 2 Datasets: The authors corrupt datasets by randomly assigning questions or passages while preserving passage-answer correspondence, and by inserting candidates into gibberish passages when needed.These manipulations remove information from one input while retaining the surrounding model architecture.
3 Models
The study evaluates three recently proposed reading-comprehension architectures, applying each to datasets suited to its input and answer-prediction design.
- 3 Models: The experiments use key-value memory networks, gated attention readers, and QA Nets as representative recently proposed models.The authors provide references and implementation decisions intended to support reproduction.
- 3 Models: The Key-Value Memory Network is applied to bAbI and CBT using bag-of-words keys and values or candidate-centered windows.The implementation uses three hops and 128-dimensional embeddings.
- 3 Models: The Gated Attention Reader performs multiple passage-reading hops and predicts over candidate answers through an attention-sum module.It is applied to Who-did-What, CNN, CBT-NE, and CBT-CN.
- 3 Models: QA Net separately encodes passages and questions with depth-wise separable convolutions and global self-attention before passage-question attention and stacked encoders.Its outputs are used to predict an answer span inside the passage.
4 Experimental Results
Across benchmarks, question-only and passage-only baselines can perform surprisingly well, sometimes matching or approaching full models, while SQuAD shows a much larger dependence on both inputs.
- bAbI tasks: On bAbI tasks 2, 7, 13, and 20, passage-only models exceed 80% accuracy with randomly assigned questions.
- bAbI tasks: On bAbI tasks 3, 13, 16, and 20, passage-only models match full-dataset performance, while task 18 reaches 91% question-only versus 93% full-model accuracy.
- Children’s Books Test: On CBT, question-only KV-MemNets nearly match full accuracy for Named Entities and Common Nouns, and outperform full models on Verbs and Prepositions.
- Children’s Books Test: Using only the last CBT sentence achieves comparable or better performance than using all 20 sentences on most subtasks.
- CNN: On CNN, question-only and passage-only models reach 25.6% and 38.3% accuracy, versus 77.8% on the full task.
5 Discussion
The paper argues that reading-comprehension benchmarks need stronger baselines and ablations to reveal whether questions, passages, and full context are genuinely necessary. It also cautions that programmatic dataset creation and publishing incentives may allow benchmark weaknesses and unsupported explanations to persist.
- Provide rigorous RC baselines: Rigorous baselines should test performance with questions or passages omitted, because otherwise improvements may reflect passage classification rather than question-passage matching.
- Test that full context is essential: Context ablations should measure how much of each passage a model truly requires, since CBT can yield comparable performance using only its last sentence.
- Caution with cloze-style RC datasets: Cloze-style datasets deserve caution because programmatic construction may involve little manual inspection and leave weaknesses overlooked.
- A note on publishing incentives: Publishing incentives may favor omitting unfavorable ablations or offering unsubstantiated explanations over rigorous studies of model behavior.