Source-linked AI summary
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, Matt Gardner
TL;DR
Reading-comprehension benchmarks rarely test complex coreference, despite its importance for understanding longer texts. The paper introduces QUOREF, a crowdsourced Wikipedia dataset and adversarial collection process designed to target coreferential reasoning. Models perform substantially below humans, with the best system at 70.5 F1 versus estimated human performance of 93.4 F1.
Problem
Most reading-comprehension benchmarks contain few complex coreferential phenomena and therefore provide limited evaluation of models’ ability to resolve coreference.
Method
The paper crowdsources over 24K questions from 4.7K English Wikipedia paragraphs and uses worker instructions plus an uncased base BERT adversary to avoid lexical shortcuts.
Results
70.5 F1 is the best model performance, compared with estimated human performance of 93.4 F1 on QUOREF.
Takeaways & Limitations
QUOREF provides a benchmark for evaluating coreference-aware reading comprehension, and its results show that current state-of-the-art models perform significantly worse than humans.
Takeaways & Limitations
The dataset’s adversary was selected qualitatively as the most effective among models with acceptably low latency, and the large BERT model did not fit available GPU memory.
Abstract
from arXiv · showhide
Machine comprehension of texts longer than a single sentence often requires coreference resolution. However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to resolve coreference. We present a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia. Obtaining questions focused on such phenomena is challenging, because it is hard to avoid lexical cues that shortcut complex reasoning. We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues. We show that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark---the best model performance is 70.5 F1, while the estimated human performance is 93.4 F1.
1 Introduction
QUOREF addresses the lack of long-distance coreference questions in reading-comprehension benchmarks with a crowdsourced Wikipedia dataset and adversarial question-collection process. It contains over 24K questions in 4.7K paragraphs, with best system performance of 70.5% F1 versus estimated human performance of 93.4%.
- Existing reading-comprehension datasets largely test local predicate-argument structure, with few questions requiring long-distance entity tracking.
- Coreference-focused question collection is difficult because workers may disagree about coreference and questions may contain lexical giveaways.
- QUOREF introduces questions requiring coreferential reasoning over paragraphs from a diverse set of English Wikipedia articles.
- The annotation process uses worker instructions and an uncased base BERT adversary to discourage questions answerable through obvious lexical cues.
- More than 24K questions span 4.7K Wikipedia paragraphs, while the best system reaches 70.5% F1 compared with estimated human performance of 93.4%.
2 Dataset Construction
QUOREF questions were crowdsourced over selected English Wikipedia paragraphs, with workers required to connect co-referring spans and an adversarial QA model screening submissions.
- Paragraphs were scraped from Wikipedia articles about English movies, art and architecture, geography, history, and music.
- Movie plot summaries were required to contain at least 40 tokens, and about 40% of sampled paragraphs came from the movie category.
- Workers found two or more co-referring spans and wrote questions requiring recognition that those spans referred to the same entity.
- Workers selected one or more answer spans, which did not have to be the co-referring spans that motivated the questions.
- An uncased base BERT model trained on SQuAD 1.1 attempted answers in real time, allowing submission only when its answer did not meet the required criterion.
3 Semantic Phenomena in QUOREF
Manual analysis found that most sampled QUOREF questions require coreference resolution, spanning pronominal and nominal references, with some also requiring commonsense reasoning.
- Requirement of coreference resolution: The remaining 22% involved coreference-related phenomena but did not require resolving them to answer the question.
- Types of coreferential reasoning: QUOREF questions require resolving both pronominal and nominal mentions of entities.
- Types of coreferential reasoning: The pronominal and nominal categories are not disjoint, with 32% of questions requiring both types of reasoning.
- Types of coreferential reasoning: 10% of analyzed questions require some form of commonsense reasoning.
4 Baseline Model Performance on QUOREF
QUOREF evaluates reading-comprehension systems with span-selection questions designed to require coreferential reasoning. XLNet QA performs best, but remains substantially below human performance, while error analyses reveal exploitable dataset artifacts.
- 70.5 F1: XLNet QA achieves the best test-set performance among the evaluated systems.
- More than 20 F1 points: XLNet QA remains below estimated human performance on QUOREF.Human performance was estimated from the authors’ answers to 400 test-set questions, scored with the same metric used for systems.
- 66.4 F1: BERT QA trained on QUOREF under-performs XLNet QA, while BERT QA trained on SQuAD would have achieved an F1 score of 0.
- Passage-only baselines under-perform all other systems and usually predict the most frequent entity in a passage.Their relatively low performance indicates that many questions likely require coreferential reasoning.
- 51% versus 12%: BERT QA’s correctly answered development questions more often selected the first or most frequent entity than its incorrect answers; XLNet shows 48% versus 11%.
- 17% versus 4%: Correct XLNet QA answers were more often nearest entities than incorrect answers; BERT QA showed 17% versus 6%.The analysis tests whether systems exploit answer proximity despite questions often requiring distant entity tracking.
5 Related Work
QUOREF differs from traditional coreference datasets and common reading-comprehension benchmarks by targeting selected cross-reference relations through crowdsourced questions.
- Unlike traditional coreference datasets, QUOREF requires understanding relations between only a few spans rather than complete coreference clusters.
- QUOREF’s less comprehensive and less conservative coreference notion permits questions about relations not marked in OntoNotes annotations.
- Crowdsourcing is more feasible because QUOREF’s less strict notion does not require linguistic expertise from annotators.
- Most reading-comprehension datasets principally test local predicate-argument structure, whereas QUOREF also queries anaphoric references and requires tracking entities throughout the text.
6 Conclusion
The paper presents QUOREF as a focused benchmark for evaluating coreference-aware reading comprehension. Its construction and evaluation provide evidence that the benchmark targets this capability and exposes a gap between models and humans.
- QUOREF evaluates whether reading-comprehension models can resolve coreference.
- Crowdsourced questions over Wikipedia paragraphs were manually analyzed, confirming that most cannot be answered without coreference resolution.
- State-of-the-art reading-comprehension models perform significantly worse than humans on QUOREF.
A Crowdsourcing Logistics
Crowdworkers were instructed to create questions connecting references to the same entities while avoiding questions solvable without understanding the paragraph.
- Workers identified anaphoric expressions and their referents, then wrote questions connecting two mentions in a paragraph.
- The instructions covered traditional coreference and more nebulous referential phenomena.
- An AI system attempted to answer each written question, and workers were told to treat answerable questions as too easy.The instructions cautioned that an AI system’s incorrect answer did not necessarily indicate a good question.
A.2 Examples of Good Questions
Good questions test whether readers connect different references to the same entity, including names, descriptions, pronouns, and repeated noun phrases.
- Good questions ask readers to identify entities linked by multiple references in a paragraph.The examples connect names and descriptions such as Unas and the ninth and final king of the Fifth Dynasty, or Pyramid of Unas and the pyramid.
- Questions can require resolving pronouns, such as identifying the person whose pyramid was built in North Saqqara.
- Other questions resolve references to objects, asking what was significant because of the discovery of Pyramid Texts.
- The examples also ask readers to identify antecedents of plural references, such as what was repurposed as underlay for the causeway.
- Questions can link descriptive references to the same person, such as asking which king had afterlife spells incised in his pyramid.
A.3 Examples of Bad Questions
Bad questions either allow answers through direct lexical matching, create ambiguity, or fail to require coreference resolution.
- Questions that can be answered by directly locating an explicit name do not require coreference resolution.One example asks who withdrew from the race in June 2004, whose answer is stated directly in the paragraph.
- Questions may be bad when several surface forms could plausibly answer them, producing ambiguous responses.The example about the memoir Dreams from My Father permits Barack Obama, Obama, or Senator Obama.
- Training workers with detailed instructions and selecting a qualified pool were used to support high-quality question production.Qualification and final tasks used multiple paragraphs per HIT, with counts varying by domain.
B Experimental Setup Details
The experiments largely follow published baseline procedures, with specified preprocessing, training, truncation, and passage-only comparison settings.
- Unless otherwise noted, the experiments use the original published procedures and hyperparameters for each baseline.
- BERT QA and XLNet QA use sequence length 512, stride 128, batch size 10, AdamW, learning rate 3−5, and 10 training epochs.The reported checkpoint is the one achieving the best development-set performance.
- QANet truncates training paragraphs and questions to 400 and 50 tokens, and evaluation paragraphs and questions to 1000 and 100 tokens.
- The passage-only baseline retains the BERT QA and XLNet QA training hyperparameters while replacing questions with empty strings.