Source-linked AI summary
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord
TL;DR
Existing QA datasets often allow surface-level retrieval, leaving deeper reasoning under-tested. This paper introduces ARC, a challenging science-question benchmark with supporting corpus and baselines, and finds that none significantly outperform random guessing on its Challenge Set.
Problem
Existing QA datasets emphasized retrieval-style questions, limiting evaluation of reasoning, commonsense knowledge, and deeper text comprehension.
Method
ARC constructs a Challenge Set by filtering questions answered incorrectly by two baseline solvers and releases a science corpus and neural baselines.
Results
None of the tested baseline systems significantly outperformed a random baseline on the Challenge Set, including neural models successful on SNLI and SQuAD.
Takeaways & Limitations
Progress on ARC would represent an impressive achievement because its Challenge Set remains difficult for retrieval and co-occurrence methods.
Takeaways & Limitations
The underlying retrieval methods favor sentences closely matching questions, missing partial matches that collectively explain answers through multi-fact chaining.
Abstract
from arXiv · showhide
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
Introduction
ARC is designed to shift question answering beyond retrieval-style tasks toward questions requiring reasoning, commonsense knowledge, and deeper text comprehension. It provides a 7,787-question natural-science dataset, partitions difficult questions into a Challenge Set, and releases supporting resources for research.
- ARC targets limitations of retrieval-focused QA datasets, which often allow surface-level cues to identify answers without encouraging reasoning or commonsense knowledge.
- The dataset contains 7,787 natural science questions authored for standardized tests.Standardized tests involve varied linguistic and inferential phenomena, differing difficulty levels, and measurable evaluation.
- The Challenge Set contains 2,590 questions missed by both retrieval-based and word-co-occurrence algorithms, while the Easy Set contains the remaining 5,197 questions.
- Challenge questions cannot be solved reliably through simple retrieval or word correlation because relevant Web formulations may be absent and incorrect options can be strongly correlated with the question.The examples require more advanced question-answering methods.
- ARC releases a science text corpus and two baseline neural models to help the research community engage with the dataset.
1. The ARC Corpus,
The ARC Corpus contains 14M science-related sentences with knowledge relevant to ARC, covering a sampled 95% of Challenge questions. Its use for the Challenge is optional.
- The ARC Corpus: The ARC Corpus contains 14M science-related sentences with knowledge relevant to ARC.The corpus provides supporting scientific knowledge for the task.
- The ARC Corpus: A sampled analysis suggests the corpus mentions knowledge relevant to 95% of Challenge questions.Corpus use for the Challenge is optional.
Related Work
Prior QA datasets often relied on explicit answers or synthetic generation, while newer benchmarks sought multistep reasoning and greater challenge. Human standardized-test datasets offered another direction but faced potential size limitations.
- Earlier datasets such as MCTest, SQuAD, NewsQA, and CNN/DailyMail often allowed answers to be determined from surface-level cues because they were explicitly stated.
- Synthetic QA datasets, notably bAbI, used simulated worlds and language generation to create 20 tasks and stimulate multistep reasoning with memory networks.
- Large synthetic QA datasets risk rewarding neural methods for reverse-engineering generation processes or dataset idiosyncrasies without advancing language understanding or reasoning.
- Human standardized-test datasets have supported AI research, but they can be small—sometimes only hundreds of questions—because public real-world data are scarce.
The ARC Dataset
The ARC Dataset contains 7,787 non-diagram, multiple-choice science questions spanning grades 3–9, partitioned into Easy and Challenge Sets. The Challenge Set operationalizes difficulty as failure by both retrieval-based and co-occurrence-based baseline solvers.
- Dataset composition: 7,787 non-diagram, typically four-way multiple-choice science questions comprise ARC, with 2,590 Challenge and 5,197 Easy questions.The questions use 6,329 distinct stemmed words and come from varied sources.
- Grade levels: Questions target students from 3rd through 9th grade, although each grade contains a mixture of easy and difficult questions.Grade levels correspond to students typically aged 8 through 13, with substantial overlap in difficulty.
- Challenge Set: Challenge questions are those answered incorrectly by both the IR and PMI baseline solvers, providing a practical filter for difficult questions.The definition approximates, rather than exactly captures, the informal notion of hardness.
- Challenge Set: The Challenge Set includes questions that defeat corpus retrieval and co-occurrence methods because relevant evidence may be absent or misleading.For example, a mineral question is missed because corpus evidence is lacking and incorrect options have stronger associations.
The ARC Corpus
The ARC Corpus is a 14M-sentence, 1.4GB collection of Web-mined science text designed to provide relevant knowledge for ARC questions, including through indirect and distributed evidence. Its coverage is substantial, but using it effectively still requires identifying, reasoning over, and supplementing the retrieved knowledge.
- Corpus overview: The ARC Corpus contains 14M science-related sentences totaling 1.4GB of text, and its use is optional.It is intended as a starting point for attacking the ARC Challenge, though exploiting indirect mentions is nontrivial.
- Corpus construction: Approximately 75% of a random sample of 805 collected documents were judged science relevant.Queries were automatically generated from ∼100 templates covering 80 elementary- and middle-school science topics.
- Coverage: 99.8% of the ARC question vocabulary is mentioned in the ARC Corpus.The corpus was augmented with the AristoMini corpus, including Wiktionary definitions, science-tagged Simple Wikipedia articles, and additional Web sentences.
- Baseline evidence: Replacing Waterloo with the ARC Corpus changes the IR solver’s Challenge Set score from near zero to a score similar to random guessing.The result suggests a limitation of the IR methodology rather than inadequate ARC Corpus coverage; the score was slightly lower than random guessing.
- Reasoning challenges: Examples show that answering ARC questions may require combining distributed evidence across sentences rather than finding one sentence that states the answer directly.The paper notes that correctly identifying and reasoning with this knowledge, along with injecting unstated commonsense knowledge, remains challenging.
Baseline Performance
Baseline systems performed near random on ARC’s Challenge Set, while reaching 55–65% on the Easy Set. Retrieval bias helps explain the weak results and motivates retrieval methods that combine multiple facts.
- Challenge Set: IR and PMI scored near zero on the Challenge Set because occasional tied answers provided partial credit.These solvers were designed as dataset-definition methods, and tied answers explain their slightly above-zero scores.
- Challenge Set: Challenge-set scores were not significantly higher than the random baseline, within a 95% confidence interval of ±2.5%.This was the paper’s most striking observation about the Challenge Set.
- Easy Set: Easy-set performance generally ranged between 55% and 65%, contrasting with the Challenge Set’s difficulty.The contrast highlights that the two sets have different difficulty and character.
- Error Analysis: Most non-IR solvers first retrieve sentences with simple IR, creating retrieval bias toward text similar to the question.TableILP is an exception because it uses nonsentential knowledge, though its knowledge coverage is low.
- Limitations: Because individually partial sentences may jointly explain an answer through chaining, ARC requires advanced retrieval and information-combination methods.The authors identify multi-fact questions as a motivation for improving both retrieval and combination strategies.
Conclusion
The paper presents ARC to encourage progress beyond QA datasets where surface cues suffice, introducing a new question set, text corpus, and baselines. The ARC resources and leaderboard are publicly available.
- Conclusion: ARC addresses the limitations of factoid QA datasets that discourage progress on questions requiring reasoning or other advanced methods.The authors motivate ARC as a way to help the field move toward more difficult tasks.
- Conclusion: ARC consists of a new question set, text corpus, and baselines designed to support more difficult question-answering research.
- Conclusion: The ARC Dataset, Corpus, three baseline neural models, and leaderboard are available from the ARC website.The listed models are DecompAttn, BiDAF, and DGEM.
Appendix A: Question Sources
Appendix A identifies the question sources used to assemble ARC and refers readers to Table 7 for a brief description of them.
- Appendix A: Question Sources: The appendix concerns the question sources used to assemble ARC.The passage frames these sources as part of ARC’s assembly.
- Appendix A: Question Sources: Table 7 provides a brief description of the ARC question sources.The table is the referenced location for source descriptions.
- Appendix A: Question Sources: The question-source descriptions are presented in Table 7.The passage directs attention to Table 7 for this information.
Appendix B: Question Answering via Entailment
Appendix B describes answering multiple-choice questions by identifying which answer choice is entailed by a sentence in a large corpus such as the ARC Corpus. The method converts each question–answer pair into an assertion, scores its entailment from corpus sentences, and returns the best-supported choice.
- The approach identifies the answer choice entailed by a sentence in a large corpus, such as the ARC Corpus.
- It converts the question and each answer choice into an assertion, treating the assertion as the entailment hypothesis and corpus sentences as premises.
- It computes entailment scores for each assertion from corpus sentences and returns the answer with the highest support.
Question to Assertion
The method converts each question–answer pair into an assertion by turning the question into a fill-in-the-blank statement and inserting the answer choice.
- Question to Assertion: Questions are converted to fill-in-the-blank statements by retaining existing FITB forms or replacing wh-words with blanks.Phrases common to multiple-choice questions, such as “of the following” and “of these,” are removed.
- Question to Assertion: The answer choice fills the blank to generate the assertion hq,a.For example, a FITB statement such as “Plants produce ” becomes an assertion after answer insertion.
Entailment Computation · Answer Scoring
The method retrieves candidate supporting sentences for each question–answer hypothesis, then applies neural entailment models to score support. Each answer choice receives the maximum entailment score among its retrieved premises.
- Entailment Computation: The entailment model evaluates support for the generated hypothesis hq,a rather than scoring every corpus sentence.Scoring every sentence in a large corpus is computationally infeasible.
- Entailment Computation: Information retrieval finds relevant premises by querying an ElasticSearch index with the question and answer choice.Retrieved sentences must contain the answer choice, while long sentences, negated sentences, and other unsuitable matches are filtered.
- Entailment Computation: 0. . . The entailment score for each question, answer choice, and retrieved premise is computed as entailsg(pq,a, hq,a).The paper presents results with Decomposable Attention and the Decomposed Graph Entailment Model (DGEM).
- Entailment Computation: DGEM additionally extracts graph structure from the hypothesis using Open IE and a chunker.The released code can be extended to use other neural entailment models implemented in AllenNLP.
- Entailment Computation: The released implementation supports substituting other neural entailment models available in AllenNLP.This extension is stated alongside the specific Decomposable Attention and DGEM implementations.
- Answer Scoring: The answer-choice score is the maximum entailment score across its supporting premises: scorec(q, a) = max scoree(q, a, pq,a).This selects the strongest retrieved supporting sentence for each answer choice.