Source-linked AI summary
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy
TL;DR
Reading-comprehension benchmarks need evaluations that better test human-like understanding and reasoning than existing answer formats and datasets provide. RACE constructs a large, human-authored examination dataset with flexible candidate answers and broad coverage; its models reach about 43% accuracy versus 94.5% human performance. The dataset is intended for research and evaluation in machine comprehension.
Problem
Existing benchmarks can restrict answers to passage spans, while examination-style resources may be too small for training powerful deep neural networks.
Method
RACE collects English examination passages, questions, and answers for Chinese students aged 12–18, retaining human-designed four-option questions at large scale.
Results
43.3% and 44.1% accuracy were achieved by Stanford AR and Gated AR on RACE, compared with 94.5% human performance.
Takeaways & Limitations
RACE provides a large, broad-coverage benchmark whose questions require substantially more reasoning than other datasets and leave substantial room for model improvement.
Takeaways & Limitations
Because RACE targets Chinese students learning English as a foreign language, its vocabulary and language constructs are simpler than those in news and Wikipedia QA datasets.
Abstract
from arXiv · showhide
We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.
1 Introduction
RACE addresses limitations in existing reading-comprehension benchmarks with a large, human-authored examination dataset designed to test reasoning and broad comprehension. Its flexible answer options, reasoning-heavy questions, and diverse coverage support more realistic evaluation of machine reading comprehension.
- RACE contains 27,933 passages and 97,687 four-option questions collected from English examinations for Chinese middle- and high-school students.Questions and candidate answers are human generated and need not appear as text spans in the passage.
- Passages cover diverse domains and writing styles, making RACE suitable for evaluating generic rather than domain- or style-specific comprehension.
- Human-generated questions and options are designed to test reading comprehension and provide objective machine–human comparison under shared metrics.
- RACE includes a substantially larger proportion of reasoning questions than existing datasets, including passage summarization and attitude analysis.
- Unlike span-restricted benchmarks, RACE permits candidate answers to be expressed in any words, enabling richer question types but increasing task difficulty.
2 Related Work
Prior reading-comprehension datasets differ in scale, question construction, answer format, and domain coverage. RACE extends examination-based evaluation at large scale while emphasizing more demanding, human-oriented comprehension.
- 2.1 MCTest: MCTest uses four candidate answers with one correct answer but contains only 500 stories and 2,000 questions, limiting training of advanced models.
- 2.2 Cloze-style datasets: Cloze-style datasets formulate questions by removing a word or entity, and their automatic generation can introduce substantial noise.
- 2.2 Cloze-style datasets: CNN/Daily Mail contains 1.4M questions, yet the cited discussion reports limited reasoning requirements and machine performance close to human performance.
- 2.2 Cloze-style datasets: CBT and BT use contiguous children’s-book sentences for cloze questions, with BT reported as 60 times larger than CBT and models matching human performance on CBT.
- 2.3 Datasets with Span-based Answers: Span-based datasets restrict answers to text spans, a format described as potentially unintuitive for humans and yielding human performance of 80.3% on SQUAD and 46.5% on NEWSQA.
- 2.4 Datasets from Examinations: RACE is presented as the first large-scale examination-based dataset designed to evaluate reading-comprehension performance under conditions resembling human assessment.
3 Data Analysis
RACE analyzes reading-comprehension questions across student levels, reasoning categories, and dataset characteristics. Its questions emphasize reasoning and include broader language and topic coverage than many existing datasets.
- Dataset Statistics: RACE-M covers middle-school exams, while RACE-H covers high-school exams with larger passage lengths and vocabulary sizes.The dataset distinguishes the two groups because their examinations have substantially different difficulty levels.
- Dataset Statistics: RACE’s vocabulary and language complexity are simpler than those of news and Wikipedia articles in other question-answering datasets.The exams target Chinese students learning English as a foreign language.
- Reasoning Types: Questions are categorized as word matching, paraphrasing, single-sentence reasoning, multi-sentence reasoning, or insufficient/ambiguous.The categories are ordered by ascending difficulty in the paper’s analysis.
- Reasoning Subdivisions: Two crowdworkers labeled the reasoning type of each of 500 sampled questions, producing 1000 labels.The sample contained 100 passages, split evenly between RACE-M and RACE-H.
- Reasoning Types: 59.2% of RACE questions require single-sentence or multi-sentence reasoning, compared with 21% for CNN, 20.5% for SQUAD, and 33.9% for NEWSQA.Word matching accounts for 15.8% of RACE questions, the lowest ratio among the compared categories.
- Reasoning Subdivisions: RACE includes detail reasoning, whole-picture understanding, passage summarization, attitude analysis, and world knowledge questions.A question may belong to multiple reasoning subdivisions; summarization and attitude analysis were not introduced by existing large-scale datasets to the authors’ knowledge.
4 Collection Methodology
RACE was constructed from English-examination materials collected from three large free public websites in China. The authors cleaned the raw data, including OCR-based answer processing, and excluded copyrighted exam examples.
- Raw Data: The raw collection contained 137,918 passages and 519,878 questions from Chinese English examinations.The source exams were designed by teachers and included middle-school and high-school groups.
- Cleaning: The cleaned RACE dataset contains 27,933 passages and 97,687 questions.The final size followed filtering and answer-processing steps.
- Cleaning: Answers stored as images were processed with Tesseract and ABBYY FineReader, and entries were removed when the OCR systems disagreed.The OCR task recognized printed A, B, C, and D answers in a standard font.
- Data Scope: The dataset excludes example exam questions with copyright, including SSAT, SAT, TOEFL, and GRE materials.This exclusion is stated in a collection footnote.
5 Experiments
The experiments compare rule-based and neural reading-comprehension baselines with human performance, including across datasets and question types. Results show RACE remains difficult for models, with substantial gaps from human ceilings and weaker gains from simple matching.
- Methods for Comparison: Accuracy is the evaluation metric used to compare reading-comprehension models and human performance.
- Methods for Comparison: The Sliding Window baseline concatenates each question with an answer and selects the article window with the highest TF-IDF-style matching score.The window size is selected using training and development performance.
- Methods for Comparison: Stanford AR encodes passages and questions with bidirectional GRUs, attends to relevant passage content, and matches each option against the summarized passage.Option representations are also produced with bidirectional GRUs.
- Methods for Comparison: Gated AR uses query-specific document representations with multi-hop attention, then applies the bilinear scoring method to obtain the output.The implementation omits the final Attention Sum layer and character-level embeddings.
- Human Evaluation: 94.5% of questions were judged valid, setting the ceiling human performance; RACE-M and RACE-H ceilings were 95.4% and 94.2%.Turkers achieved 85% on RACE-M and 70% on RACE-H, but the authors note that careful completion could not be guaranteed.
- Main Results: 51.5% versus 37.3%: Sliding Window accuracy was higher on MCTest than RACE, indicating greater reasoning demands in RACE.
- Main Results: 28.6%: Sliding Window improved over the random baseline on RACE, below the corresponding improvements reported for CBT-N, CBT-C, and WDW.The comparison uses improvement over random because candidate counts differ across datasets.
- Main Results: 37.3% and 30.4%: Sliding Window accuracy on RACE-M and RACE-H indicates simpler matching-based performance for middle-school questions.
6 Conclusion
RACE is a large, high-quality reading-comprehension dataset designed to examine human ability, with broad domains and rich question formats. Its reasoning demands exceed those of other datasets, leaving a substantial gap between state-of-the-art models and human performance.
- RACE is a large, high-quality dataset carefully designed to examine human reading-comprehension ability.
- RACE covers broad domains and styles while supporting a rich variety of question formats.
- RACE requires substantially more reasoning than other datasets, as shown by a significant machine–human performance gap.
- The authors hope RACE will stimulate development of more advanced machine-comprehension models.
A.1 Example Question of Passage Summarization
The example passage gives practical advice for maintaining weight and fitness during holidays. Its title question tests whether readers can summarize the passage's central recommendation rather than identify a single detail.
- The passage recommends enjoying holidays without overeating to maintain normal weight.
- It advises eating small portions, beginning with soup or produce, and avoiding high-fat foods.
- It recommends maintaining physical activity, including a 20-minute walk to burn extra calories.
- The best-title question asks readers to identify the passage's overall advice about staying slim and fit.