Source-linked AI summary
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, Benjamin Van Durme
TL;DR
Existing MRC datasets often underrepresent commonsense and multi-sentence reasoning, motivating a dataset that tests these abilities directly. The paper introduces ReCoRD, a large-scale news-derived benchmark with automatically generated and filtered cloze-style examples. Humans achieve 91.69 F1 on ReCoRD, compared with 46.65 F1 for the best automatic method, leaving a substantial human–machine gap.
Problem
Existing MRC datasets often permit pattern matching and lack many questions requiring commonsense reasoning or understanding across multiple sentences.
Method
ReCoRD automatically mines news articles to create passage-query-answer examples, filters queries easy for SAN, and validates the data with crowdworkers.
Results
91.69 F1 for humans versus 46.65 F1 for DocQA with ELMo shows a significant human–machine reading-comprehension gap on ReCoRD.
Takeaways & Limitations
ReCoRD presents a challenge for future research in machine reading comprehension with commonsense reasoning.
Abstract
from arXiv · showhide
We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning. Experiments on this dataset demonstrate that the performance of state-of-the-art MRC systems fall far behind human performance. ReCoRD represents a challenge for future research to bridge the gap between human and machine commonsense reading comprehension. ReCoRD is available at http://nlp.jhu.edu/record.
1 Introduction
ReCoRD addresses limitations in existing machine reading comprehension datasets by providing large-scale news-derived examples that often require commonsense reasoning. On ReCoRD, humans substantially outperform state-of-the-art MRC models.
- Motivation: Existing MRC datasets often allow questions to be answered by matching question and answer-sentence patterns, while ReCoRD targets commonsense and multi-sentence reasoning.The paper identifies these reasoning types as underrepresented in prior datasets.
- Dataset: Over 120,000 ReCoRD examples are designed for reading comprehension, most requiring deep commonsense reasoning.ReCoRD stands for Reading Comprehension with Commonsense Reasoning Dataset.
- Dataset: All ReCoRD queries and passages are automatically mined from news articles, reducing human elicitation bias and making data collection cost-efficient.Further analysis shows that a large portion of the dataset requires commonsense reasoning.
- Results: 91.69 F1 for humans versus 46.65 F1 for state-of-the-art MRC models demonstrates a large commonsense reading-comprehension gap.The reported comparison is from experiments on ReCoRD.
2 Task Motivation
The task evaluates whether machines can use implicit commonsense knowledge to select likely consequences from passages. ReCoRD formalizes this challenge with cloze-style queries whose missing spans must be chosen from passage entities.
- Task Motivation: Commonsense reasoning requires inferring likely consequences or details from what a passage states together with existing knowledge, rather than unlikely ones.The paper motivates this formulation because commonsense knowledge is massive and mostly implicit.
- Task Definition: In ReCoRD, a machine reads passage p, marked spans E, and cloze query Q(X), then selects e ∈ E maximizing P(Q(e) | p, M).Filling the query produces a consequence or detail that is factually supported by the passage with some likelihood.
3 Data Collection
ReCoRD is constructed from news articles through automated generation and model-based filtering, followed by human validation. The resulting collection retains examples intended to test reasoning beyond immediate pattern matching.
- Data Collection: The four collection stages curate CNN/Daily Mail articles, generate passage-query-answer triples, filter model-easy queries, and remove queries ambiguous to humans.These stages are summarized as the overall ReCoRD data-collection framework.
- News Article Curation: News articles provide event summaries for passages, later details or consequences for queries, and editor bullet points as supplemental passage sources.This structure motivates using news as the dataset source.
- News Article Curation: The pipeline sampled 148K CNN and Daily Mail articles and used annotated named entities and coreference information during generation.The annotations were produced by a Google NLP pipeline.
- Passage-Query-Answers Generation: Selected sentences were constrained by length, non-question status, limited 3-gram overlap, named-entity presence, and coreference links to passage entities.These constraints aim to produce inferable, connected cloze queries rather than paraphrases or disconnected statements.
- Passage-Query-Answers Generation: 770k passage-query-answer triples were generated by replacing one entity with X and using its passage coreference precedents as reference answers.The process was designed to require reasoning beyond immediate pattern matching.
- Filtering: SAN filtered triples that were easy for a competitive MRC architecture, while crowdworkers validated the remaining data for noise and ambiguity.Human validation used Amazon Mechanical Turk under worker-quality requirements.
- Human Filtering: 20k triples were retained after two-worker validation, then split into 10k development and 10k test examples.The retained triples were those answered correctly by both workers.
4 Data Analysis
Analysis of 100 randomly sampled development examples shows that ReCoRD primarily tests commonsense reasoning rather than simple paraphrasing. The queries span conceptual knowledge, causal reasoning, naïve psychology, and other commonsense types.
- Reasoning Types: 75% of sampled ReCoRD queries require commonsense reasoning, unlike the predominantly paraphrase-based demands of existing datasets.
- Reasoning Types: 3% of sampled queries can be answered by paraphrasing, reflecting the effect of machine filtering.
- Reasoning Types: 6% of queries are ambiguous, indicating a limited proportion of unclear examples after human filtering.
- Reasoning Types: 10% of queries can be answered through partial clues, partly because named-entity recognition during curation was incomplete.
- Reasoning Types: The analysis categorizes reasoning needs across 100 random development-set samples, including conceptual, causal, psychological, social, planning, and spatial types.
5 Evaluation
ReCoRD evaluates extractive and cloze-style MRC systems on queries designed to require commonsense reasoning, comparing them with human readers. Humans substantially outperform automatic methods, while analyses identify weaknesses in commonsense, multi-sentence, and partial-clue reasoning.
- Methods: ReCoRD evaluates cloze-style and extractive MRC models against human performance on its train/dev/test splits.The evaluated systems include DocQA, DocQA with ELMo, QANet, SAN, ASReader, and a language-model baseline.
- Results: 91.69 F1 is achieved by humans, compared with 46.65 F1 for the best automatic method, DocQA with ELMo, on the test set.The corresponding EM scores are 91.31 for humans and 45.44 for DocQA with ELMo.
- Analysis: About 8% of development and test queries are answered incorrectly by humans, often because two candidate entities fit without specific background knowledge.Human readers commonly narrow the candidates but cannot identify a unique answer.
- Analysis: ELMo gives DocQA the largest overlap with human correct predictions, at 92.6% among the three analyzed methods.The analysis attributes some gains to language-model knowledge incorporated by ELMo.
- Analysis: All three analyzed methods produce about 6% out-of-candidate-entities predictions, suggesting candidate-entity information could potentially raise performance by 6%.The other extractive models do not use the marked candidate entities when searching passage spans.
- Analysis: The three methods perform poorly on commonsense reasoning, multi-sentence reasoning, and partial-clue queries.QANet performs better than DocQA on multi-sentence and paraphrased queries, while ELMo improves DocQA on paraphrased queries and commonsense reasoning with presumed knowledge.
6 Related Datasets
ReCoRD connects reading-comprehension datasets with commonsense-reasoning datasets while differing in how examples, contexts, and answers are constructed. Its news-derived, automatically generated design contrasts with crowdsourced or language-modeling-oriented alternatives.
- Research connections: ReCoRD is related to both reading-comprehension and commonsense-reasoning dataset research.The comparison discusses CNN/Daily Mail, CBT, LAMBADA, SQuAD, and NewsQA.
- Reading Comprehension: CNN/Daily Mail queries mostly require limited reasoning, whereas ReCoRD targets deeper reasoning through its construction.CNN/Daily Mail queries are constructed from bullet points, while ReCoRD uses a different dataset-generation process.
- Reading Comprehension: CBT lacks machine or human filtering for quality, so only a small portion of its examples probes contextual understanding.CBT uses 21 consecutive book-excerpt sentences with one word removed from the final sentence.
- Reading Comprehension: LAMBADA is a language-modeling problem whose answers are not necessarily in the context, unlike ReCoRD’s extractive formulation.LAMBADA was filtered to be human-guessable in broader context rather than designed as extractive MRC.
- Reading Comprehension: ReCoRD resembles SQuAD and NewsQA because candidate answers are passage spans, but its questions are automatically generated rather than curated from crowdworkers.The passage notes that many crowdsourced questions in existing datasets can be answered by word matching or paraphrasing.
7 Conclusion
ReCoRD is a large-scale reading comprehension dataset requiring commonsense reasoning, and its baselines remain substantially below human competence. The authors position it as a challenge for future commonsense MRC research.
- ReCoRD contains a large portion of queries that require commonsense reasoning, unlike existing MRC datasets.
- Human competence exceeds the performance of ReCoRD baselines, including top performers on existing MRC datasets.
- ReCoRD is intended to spur research that bridges the gap between human and machine commonsense reading comprehension.
A.1 Case Study
The case studies illustrate both ambiguity that can challenge human readers and contextual patterns that help models narrow candidate answers. They also show concrete extractive-answer examples from news passages.
- Ambiguous queries can prevent humans from selecting a unique answer when multiple passage entities satisfy the question.The example concerns two experts, Vanessa Van Edwards and Adam Grant, who both fit the query about advice to ambiverts.
- In the ambivert example, the passage contrasts quiet behavior at work with being a party animal among friends and describes situational personality shifts.It also associates ambiverts with gaining trust, making points heard, and mirroring others’ personalities.
- A second query asks who advised ambiverts to read each situation more carefully and consider what would make them happiest or most successful.The reference answer is Adam Grant.
- ELMo helps DocQA reject “New Zealand” as a candidate because the context “I’m ...” usually precedes a person name rather than a location.The case study’s reference answer is William Scott Chalmers.
A.2 HIT Instructions
The paper documents the instructions used for its Amazon Mechanical Turk HITs. These instructions are presented in Figure 8.
- The data-collection process used Amazon Mechanical Turk HITs.
- Figure 8 presents the instructions for the Amazon Mechanical Turk HITs.
- The figure is titled “Amazon Mechanical Turk HIT Instructions.”