ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

1 INTRODUCTION

• Pre-trained language models have nearly saturated performance on many popular reading-comprehension datasets, motivating evaluation on more challenging tasks.The introduction frames harder tasks as a step toward more comprehensive analysis and reasoning over text.Source: §1 · p. 1
• Existing reading-comprehension datasets contain little or no logical reasoning, while natural-language inference focuses on three simple relationships at the sentence level.ReClor is presented as a way to study more complicated reasoning at the passage level.Source: §1 · p. 1
• ReClor contains 6,138 logical-reasoning reading-comprehension questions selected from standardized exams such as the GMAT and LSAT.Each question has a context, a question, and four options with one correct answer.Source: §1 · pp. 1–2
• State-of-the-art models perform well on ReClor’s EASY set but struggle on HARD, while humans perform similarly on both sets.The authors attribute the model results to capturing dataset biases without understanding the text and reasoning across connections between lines.Source: §1 · p. 2, Figure 1 · p. 2
• The authors identify biased test examples with option-only-input baselines trained using different random seeds, grouping them as EASY and the remaining examples as HARD.The split is intended to support a more comprehensive evaluation of models’ logical reasoning ability.Source: §1 · p. 2

Reading Comprehension Datasets

• Existing reading comprehension datasets address varied reasoning skills, but the authors report no dataset targeting logical reasoning in reading comprehension.Sugawara & Aizawa (2016) found that none of the MCTest questions and only 1.2% of SQuAD questions required logical reasoning; Lai et al. (2017) found around 70% of sampled RACE questions involved word matching, paraphrasing, or single-sentence reasoning. Later datasets address iterative, multi-hop, commonsense, and numerical reasoning, leaving the gap ReClor aims to fill.Source: §2 · pp. 2–3

Logical Reasoning in NLP

• Unlike sentence-level inference datasets and warrant-only argument comprehension, ReClor integrates diverse logical reasoning types into passage-level reading comprehension.Natural language inference datasets such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) focus on sentence relationships; Habernal et al. (2018) address passage-level reasoning but only warrant identification.Source: §2 · p. 3

Datasets from Examinations

• Among datasets derived from standardized examinations, ReClor distinguishes itself by targeting logical reasoning.Related examples include RACE (Lai et al., 2017), university entrance exam datasets, the grade-school science questions in ARC (Clark et al., 2018), and the English-exam dialogue dataset DREAM.Source: §2 · p. 3

3.1 DATA COLLECTION

• ReClor uses four-option questions with one correct answer, sourced from standardized tests such as GMAT and LSAT because such questions require complex logical reasoning.The source problems had five choices; the authors shuffled their order and randomly removed one wrong option to produce four choices, and restricted the dataset to non-commercial research use.Source: §3.1 · pp. 3–4

3.2 DATA ANALYSIS

• ReClor contains 6,138 examples (4,638/500/1,000 train/validation/test), 91.22% from actual GMAT/LSAT exams; its shorter contexts emphasize reasoning over retrieval.The remaining examples come from high-quality practice exams; the authors state that every context sentence matters in ReClor, unlike the redundant context sentences often found in RACE.Source: §3.2 · p. 4
• The authors group test questions into 17 logical-reasoning categories, including assumptions, strengthening or weakening arguments, inference, flaws, and argument structure.The listed categories reflect those in the GMAT and LSAT logical-reasoning module.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 DATA BIASES IN THE DATASET

• Wrong options average 21.82 words versus 23.06 for right options, and token-label correlations provide an additional lexical-bias signal.Because each item shares its context and question across all four options, the authors analyze option-token correlations and option lengths; they lowercase options and use BERTBASE WordPiece tokens.Source: §3.3 · pp. 4–5

4.1 BASELINE MODELS

• The experiments challenge neural and pre-trained language-model baselines, including RoBERTa, on ReClor.The paper directs readers to its appendices for baseline-model and implementation details.Source: §4.1 · p. 7

4.2 EXPERIMENTS TO FIND BIASED DATA

• Using five baselines on answer options alone, the authors united items each model got right across four seeds, yielding 440 EASY test items and defining the remainder as HARD.Requiring four correct predictions reduces the chance of an item being included through guessing alone to 0.39%.Source: §4.2 · pp. 7–8

4.3 TRANSFER LEARNING THROUGH FINE-TUNING

• Because ReClor is multiple-choice, the authors selected RACE as the source dataset for their fine-tuning study.They note that ReClor is comparable in size to ARC and DREAM but much smaller than RACE.Source: §4.3 · p. 8

4.4 RESULTS AND ANALYSIS

• After RACE pretraining and ReClor fine-tuning, RoBERTaLARGE improved especially on HARD and surpassed graduate students overall, while the strongest baselines scored around 50% on HARD, below graduate students and far from ceiling performance.Source: §4.4 · p. 9
• Transformer models achieved around 75% accuracy on EASY but only around 30% on HARD, while humans performed relatively consistently across both sets.The authors caution that the observed difference in graduate students’ EASY and HARD scores may reflect the small number of participants.Source: §4.4 · p. 9
• Adding questions to answer options did not significantly improve performance over options alone, whereas adding context produced a significant boost.The authors suggest question wording can signal a reasoning type without providing much additional information.Source: §4.4 · p. 9
• On HARD, BERTLARGE, XLNetLARGE, and RoBERTaLARGE struggled with STRENGTHEN, WEAKEN, and ROLE questions but did relatively better on CONCLUSION/MAIN POINT and MATCH STRUCTURES.The authors characterize the latter types as more straightforward.Source: §4.4 · p. 9, Figure 4 · p. 10
• For XLNetLARGE, RACE fine-tuning improved HARD performance on CONCLUSION/MAIN POINT and MOST STRONGLY SUPPORTED, while EASY accuracy stayed similar across most question types.The authors suggest these types require less complex reasoning and may have analogues in RACE; more methods are needed for harder types.Source: §4.4 · p. 10, Figure 4 · p. 10

• Transformer-based models exploit dataset biases on EASY data but struggle on non-biased HARD data, performing close to or only slightly above random guess.The findings point to substantial remaining work to equip deep learning models with real logical reasoning abilities.Source: §5 · p. 10
• ReClor targets passage-level logical reasoning, extending research beyond sentence-level tasks and simple reasoning to multiple complicated forms.The paper proposes splitting test data into EASY and HARD groups for biased and non-biased examples, respectively.Source: §5 · p. 10
• Fine-tuning first on the large-scale RACE dataset and then on ReClor significantly improves model performance, showing transfer learning's potential for reasoning tasks.Source: §5 · p. 10
• The authors hope future work will adopt similar dataset-splitting techniques and evaluation schemes when reporting model performance.Source: §5 · p. 10

• FastText (Joulin et al., 2017) represents sentences as bags of n-grams, scores each answer independently, and selects the highest-scoring option.Source: A BASELINE MODELS · p. 15
• The two-layer bidirectional LSTM uses GloVe embeddings and max-pools its second layer’s final hidden state before a fully connected scoring layer.Source: A BASELINE MODELS · p. 15
• GPT and GPT-2 use language-model-pretrained transformers; fine-tuning scores options from the final-token representation, with GPT-2 using the smallest proposed model (Vaswani et al., 2017).Source: A BASELINE MODELS · p. 15
• BERT uses BooksCorpus and Wikipedia pretraining with MLM and NSP, then scores options from the [CLS] representation through two fully connected layers (Devlin et al., 2019; Vaswani et al., 2017).Source: A BASELINE MODELS · p. 15
• XLNet uses permutation language modeling without NSP and additional pretraining corpora, then scores options from the final <cls> representation through two fully connected layers (Yang et al., 2019).Its additional corpora include Giga5, ClueWeb 2012-B, and Common Crawl.Source: A BASELINE MODELS · p. 15
• RoBERTa modifies BERT pretraining by training longer with larger batches and more data, removing NSP, and adding two fully connected layers to score the final <s> representation (Liu et al., 2019).Source: A BASELINE MODELS · p. 15

B IMPLEMENTATION DETAIL

• Pre-training models are fine-tuned for 10 epochs with batch size 24, and all models use a maximum input length of 256 tokens.The pre-training implementations modify Hugging Face Transformers code for ReClor.Source: B IMPLEMENTATION DETAIL · p. 15
• Bi-LSTM uses two bidirectional layers, 300d GloVe embeddings, max-pooling, and a fully connected layer, trained for 100 epochs with batch size 64 and learning rate 0.1.The learning rate decays by 0.5 every 10 epochs.Source: B IMPLEMENTATION DETAIL · p. 15
• fastText uses its Python library with ReClor converted to the required format and the library’s default hyperparameters; Adam is used by all models.Source: B IMPLEMENTATION DETAIL · p. 15

C EXAMPLES

• Tables 10–11 define and exemplify necessary-assumption and sufficient-assumption questions.These are the two assumption-related reasoning types highlighted in the section.Source: Table 10 · p. 16, Table 11 · p. 16
• Tables 12–17 cover strengthen, weaken, evaluation, implication, conclusion/main point, and most-strongly-supported questions.Each table is labeled as providing a definition and an example of its reasoning type.Source: Table 12 · p. 17, Table 13 · p. 17, Table 14 · p. 17
• Tables 18–22 cover explain-or-resolve, principle, dispute, technique, and role questions, each with a definition and example.The examples broaden coverage beyond assumptions and argument evaluation to other question types.Source: Table 18 · p. 19, Table 19 · p. 19, Table 20 · p. 19
• Tables 23–26 add identify-a-flaw, match-flaws, match-the-structure, and other questions, alongside the role-question example in Table 22.Each table is labeled as presenting a definition and example for its question type.Source: Table 23 · p. 20, Table 22 · p. 20, Table 24 · p. 21

D CONSISTENCY OF DIFFERENT MODELS

• Table 27 reports overlap for each pair of models after intersection among four random seeds.Source: Table 27 · p. 22

• Figure 5 reports the accuracy of all baseline models on the overall testing set.Source: Figure 5 · p. 23
• Figures 6 and 7 report baseline-model accuracy on the EASY and HARD testing sets, respectively.Source: Figure 6 · p. 24, Figure 7 · p. 25
• Figure 8 presents BERTLARGE and RoBERTaLARGE performance on the EASY and HARD testing sets.Source: Figure 8 · p. 26