Source-linked AI summary
Who did What: A Large-Scale Person-Centered Cloze Dataset
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, David McAllester
TL;DR
Reading-comprehension datasets often rely on summaries, anonymization, or easily exploited regularities, limiting their challenge for person-centered understanding. Who-did-What constructs over 200,000 named-entity cloze problems from independent Gigaword articles and filters simple baselines. Humans achieve 84%, while standard readers perform substantially worse than on CNN, supporting the dataset as a more challenging benchmark.
Problem
Existing comprehension datasets can rely on summary points, anonymized entities, or simple baselines, motivating a more challenging person-centered reading-comprehension benchmark.
Method
Who-did-What forms cloze questions from one Gigaword article, retrieves an independent overlapping passage, uses person entities as choices, and suppresses simple baselines.
Results
84% of humans solved Who-did-What questions, while Attentive Reader and Attention Sum Reader achieved 55% and 59%, respectively.
Takeaways & Limitations
Who-did-What provides a large-scale, flexible extension to machine-comprehension training and test data and a challenging benchmark for neural methods.
Abstract
from arXiv · showhide
We have constructed a new "Who-did-What" dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus. The WDW dataset has a variety of novel features. First, in contrast with the CNN and Daily Mail datasets (Hermann et al., 2015) we avoid using article summaries for question formation. Instead, each problem is formed from two independent articles --- an article given as the passage to be read and a separate article on the same events used to form the question. Second, we avoid anonymization --- each choice is a person named entity. Third, the problems have been filtered to remove a fraction that are easily solved by simple baselines, while remaining 84% solvable by humans. We report performance benchmarks of standard systems and propose the WDW dataset as a challenge task for the community.
1 Introduction
Who-did-What is a person-centered cloze dataset built from independent news articles, designed to require deeper semantic analysis than existing comprehension tasks. Its filtered problems widen the human–machine gap and reduce reader performance relative to CNN.
- Dataset and task design: Who-did-What forms each cloze question from one article and selects an independent, overlapping passage from another article, with person named entities as answer choices.The approach avoids summary-based question formation and supports collections without manually written summaries.
- Dataset and task design: Questions from two distinct articles reduce syntactic similarity between questions and relevant passage sentences, increasing the need for deeper semantic analysis.
- Dataset difficulty: 49.9% of questions were removed to suppress simple baselines, including first-person, most-frequent-person, n-gram, and unigram strategies.The filtering is intended to produce problems requiring deeper semantic analysis.
- Dataset difficulty: 84% of humans solved Who-did-What questions, compared with 75% on CNN and 82% on the CBT named-entities task.The dataset therefore yields a larger human–machine performance gap than existing datasets.
- Dataset difficulty: Attentive Reader accuracy fell from 63% on CNN to 55% on Who-did-What, while Attention Sum Reader accuracy fell from 70% to 59%.
2 Related Work
Who-did-What extends prior machine-comprehension datasets with large-scale newswire cloze problems focused on named people. Earlier datasets differ in size, synthetic structure, narrative demands, or use of article summaries.
- Prior datasets: MCTest contains 660 fictional stories with four multiple-choice questions each, making it too small to train systems for general reading comprehension.
- Prior datasets: bAbI uses synthetic passages describing simulated actions, for which a logical algorithm can solve problems exactly and generate ground-truth answers.
- Prior datasets: CBT contains 113,719 named-entity cloze problems from children’s stories and tests story completion rather than reading comprehension.Because future story events may be surprising and undetermined, CBT human performance is lower than on Who-did-What: 82% versus 84%.
- Prior datasets: CNN and Daily Mail provide the closest existing comparison, combining 1.4 million cloze questions from approximately 300,000 news articles.
3 Dataset Construction
The dataset construction pipeline converts Gigaword news articles into person-centered cloze problems, retrieves independent passages, filters ambiguity and duplication, and suppresses simple baselines. The final splits are organized chronologically by question-article date, with a relaxed training variant.
- Question generation: Questions begin with the first sentence of a randomly selected Gigaword article, from which Stanford NER and PCFG parsing identify person entities and their noun phrases.
- Question generation: Deleting an entire person noun phrase removes descriptors and local contextual cues, increasing problem difficulty; about 700,000 question sentences are generated.
- Passage and choice construction: An independent Gigaword passage is retrieved with Lucene using the question sentence, while requiring the deleted named entity to appear in the returned article.
- Passage and choice construction: Candidate choices are person named entities from the passage, with subset and ambiguous-name cases discarded.
- Filtering and splits: The pipeline removes duplicate problems using bag-of-words overlap and suppresses simple first-person, frequency, n-gram, and unigram baselines through an optimization procedure.The suppression removes 49.9% of questions while limiting baseline performance toward a target.
- Filtering and splits: Validation and test sets are random splits of the most recent 20,000 problems by question-article date, and a relaxed training set uses less baseline suppression.
4 Performance Benchmarks
Standard readers perform consistently worse on Who-did-What than on CNN, and the system ranking changes. Frequency-sensitive Attention Sum and Gated-Attention readers are especially affected by suppressing the most-frequent-person baseline.
- Evaluated systems: The benchmark includes overlap, sliding-window, distance, semantic-feature, Attentive, Stanford, Attention Sum, and Gated-Attention systems.Attention and Stanford Readers use anonymized entity IDs for this evaluation.
- Performance comparison: Attentive and Stanford Readers drop by up to 10%, while Attention Sum and Gated-Attention Readers drop by up to 17% when moving from CNN to Who-did-What.
- Performance comparison: Attention Sum and Gated-Attention Readers explicitly exploit answer frequency, a heuristic that benefits CNN and Daily Mail but is targeted by Who-did-What’s suppression.
5 Conclusion
The paper presents Who-did-What as a large-scale person-centered cloze dataset designed for neural methods and extending existing machine-comprehension training and test data.
- Who-did-What is a large-scale person-centered cloze dataset.
- The dataset’s scalability and flexibility make it suitable for neural methods.
- Who-did-What differs from existing large-scale cloze datasets and extends machine-comprehension training and test data.
6 Appendix
The appendix specifies algorithms for forming cloze questions and selecting passages, using named entities, parsing, ranked retrieval, and validity constraints.
- Question Formation: Question Formation returns a cloze question and deleted person named entity when the article’s first sentence meets length and named-entity conditions.The procedure uses Stanford NER and parsing, deletes overlapping entities, and selects a removable noun-phrase constituent.
- Passage Selection: Passage Selection ranks articles containing the deleted person entity, published within a specified time window, and retrieved by Apache Lucene.The selected passage must discuss the question’s entities while avoiding excessive lexical overlap and copied context around the blank.
- Passage Selection: A valid passage shares neither more than 78% of its words nor five-word context sequences with the question and contains a person named entity from the question.