PaperlayerView exact revision on arXiv

Source-linked AI summary

ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

Weihao Yu, Zihang Jiang, Yanfei Dong, Jiashi Feng

arXiv:2002.04326v3cs.CLcs.AIcs.LG

TL;DR

ReClor addresses the shortage of challenging passage-level logical reasoning benchmarks in reading comprehension. The paper constructs an exam-derived dataset, separates biased from non-biased examples, and finds that powerful models succeed on EASY cases but perform poorly on HARD cases, near random-guess performance.

  • Problem

    Existing reading comprehension datasets contain little logical reasoning, creating a need for more challenging passage-level evaluation.

  • Method

    The authors introduce ReClor from standardized exams and use option-only-input baselines to split testing examples into EASY biased and HARD non-biased sets.

  • Results

    State-of-the-art models perform well on EASY examples but struggle on HARD examples, with performance close to or slightly above random guess.

  • Takeaways & Limitations

    ReClor shows that current pre-trained language models can exploit dataset biases but have difficulty understanding and reasoning over non-biased data.

Abstract

from arXiv · show

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

1 INTRODUCTION

ReClor addresses the limited logical reasoning in existing reading comprehension datasets by introducing exam-derived questions and separating biased from non-biased test examples. The paper finds that current models perform well on EASY examples but struggle on HARD ones.

  • Background: Machine reading comprehension requires models to understand a text and answer a question related to its context.The paper positions ReClor as a more challenging MRC benchmark.
  • Motivation: Existing reading comprehension datasets contain little data requiring logical reasoning, motivating a passage-level dataset with more complicated reasoning.Reported examples include 0% in MCTest and 1.2% in SQuAD.
  • Dataset: ReClor uses multiple-choice logical reasoning questions selected from standardized exams such as GMAT and LSAT.Each problem contains a context, question, and four options with one correct answer.
  • Evaluation design: The authors identify biased test examples using option-only-input baselines, assigning them to EASY and the remaining examples to HARD.The procedure removes the context and question to find problems answerable from options alone.
  • Findings: State-of-the-art models perform well on EASY but struggle on HARD, indicating strong bias exploitation but limited real logical reasoning.The paper presents this split as a way to evaluate reasoning more comprehensively.

2 RELATED WORK

Prior work spans reading comprehension datasets, increasingly complex reasoning tasks, and examination-derived benchmarks. ReClor targets a broader combination of passage-level logical reasoning types than closely related tasks.

  • Reading comprehension datasets: Earlier reading comprehension datasets include fictional-story, Wikipedia, and examination-based benchmarks such as MCTest, SQuAD, RACE, and ARC.These datasets vary in source material, scale, and reasoning demands.
  • Reading comprehension datasets: Analyses found that logical reasoning is rare in MCTest and SQuAD, while many RACE examples involve word matching, paraphrasing, or single-sentence reasoning.The cited analysis reports 0% logical-reasoning questions in MCTest and 1.2% in SQuAD.
  • More complex reasoning: Newer datasets address iterative story reasoning, multi-hop reasoning, commonsense knowledge, and numerical reasoning.These efforts broaden reading comprehension beyond direct retrieval and simple matching.
  • Logical reasoning in NLP: Natural language inference classifies a few logical relationships between sentence pairs, whereas argument reasoning comprehension focuses on identifying implicit warrants.The related tasks are narrower than ReClor in reasoning scope or type.
  • Logical reasoning in NLP: ReClor combines passage-level reading comprehension with multiple logical reasoning types, extending beyond sentence-level inference and single-type warrant identification.This positions ReClor within both logical-reasoning and examination-derived dataset research.

3 RECLOR DATA COLLECTION AND ANALYSIS

ReClor is a multiple-choice dataset built from standardized logical reasoning exams and analyzed for composition and answer-option biases. Its design supports evaluation across diverse reasoning types while exposing shortcut signals.

  • Data collection: ReClor uses the standard multiple-choice format of a context, question, and four answer options with one correct choice.The problems are sourced from standardized tests because crowdsourcing complex logical questions is difficult.
  • Data collection: The dataset contains 6,138 questions, with five original options reduced to four by shuffling and deleting one wrong option.The resulting format has one right option and three wrong options.
  • Dataset statistics: 91.22% of ReClor examples come from actual GMAT and LSAT exams, and the splits contain 4,638 training, 500 validation, and 1,000 testing examples.The dataset is comparable in size to some multiple-choice MRC datasets and has a relatively large vocabulary.
  • Data biases: The authors analyze biases because human-designed examination data may allow models to perform well without truly understanding the text.This analysis motivates separating biased and non-biased examples for evaluation.
  • Reasoning types: ReClor covers varied reasoning types, including necessary assumptions, sufficient assumptions, strengthening, weakening, evaluation, implication, and conclusion or main point.The listed categories are accompanied by percentages and operational descriptions in the dataset analysis.

4 EXPERIMENTS

Experiments evaluate baseline models on ReClor, including bias-controlled EASY and HARD sets, transfer learning from RACE, and performance across input settings and reasoning types. Models perform well on EASY but struggle more on HARD, while RACE fine-tuning improves results, especially for some reasoning types.

  • Experiments to Find Biased Data: 440 testing-set data points predicted correctly across four seeds form EASY, with the remainder assigned to HARD.The EASY set is identified using answer options only, reducing the effect of random guessing.
  • Baseline Models: FastText performs better than random guess, indicating that word correlation can improve performance to some extent.This result provides a non-pretrained baseline for comparison with transformer-based models.
  • Results and Analysis: Around 75% accuracy on EASY contrasts with around 30% on HARD for transformer-based models, while humans remain relatively consistent across both sets.The authors interpret this pattern as strong bias exploitation and weaker performance on non-biased examples.
  • Results and Analysis: Adding context significantly improves performance, whereas adding questions to answer options alone does not produce significant improvement.Some question stems correspond to recurring reasoning types and therefore provide limited information without the context.
  • Results and Analysis: On HARD questions, models perform poorly on STRENGTHEN, WEAKEN, and ROLE but relatively better on CONCLUSION/MAIN POINT and MATCH STRUCTURES.The latter types are described as more straightforward, while the former require extensive logical reasoning.

5 CONCLUSION

ReClor evaluates logical reasoning in reading comprehension by separating biased and non-biased test items into EASY and HARD sets. Models exploit biases effectively but struggle to understand and reason over non-biased data.

  • ReClor targets logical reasoning from sentence-level to passage-level and from simple to multiple complicated reasoning.
  • The testing set is split into EASY and HARD groups for biased and non-biased data, respectively.
  • Transformer-based pre-trained models exploit dataset biases well but have difficulty understanding and reasoning over non-biased data.
  • Performance on non-biased data is low, close to or slightly better than random guess.
  • Fine-tuning first on RACE and then on ReClor yields significant improvement.

A BASELINE MODELS

The baseline models include fastText, Bi-LSTM, and several transformer-based pre-trained language models. Their inputs concatenate context, question, and answer options, with model-specific representations used to score choices.

  • FastText models sentences as bags of n-grams and independently predicts each answer option's probability.The option with the highest score is selected.
  • The LSTM baseline uses a randomly initialized two-layer bidirectional LSTM with GloVe embeddings and max-pooling.The pooled representation is passed to a fully connected layer for output scoring.
  • GPT and GPT-2 are transformer models pre-trained with an unsupervised language-modeling objective.GPT uses BooksCorpus, while GPT-2 uses WebText and the smallest proposed model serves as the baseline.
  • BERT, XLNet, and RoBERTa use transformer representations with additional fully connected layers to score answer options.Their pre-training procedures and aggregate-token choices differ across models.
  • The model input format concatenates context, question, and option token sequences.The notation || denotes concatenation.

B IMPLEMENTATION DETAIL

All models use Adam, while fastText and Bi-LSTM receive specified implementation settings for training and optimization. The Bi-LSTM is trained for 100 epochs with scheduled learning-rate decay.

  • Adam is used as the optimizer for all models.
  • fastText uses its Python library after ReClor is converted to the required format, retaining default hyperparameters.
  • The Bi-LSTM uses two bidirectional layers, 300-dimensional GloVe embeddings, max-pooling, and a fully connected layer.
  • The Bi-LSTM is trained for 100 epochs with batch size 64 and learning rate 0.1.The learning rate is decayed by 0.5 every 10 epochs.

HYPERPARAM GPT GPT-2 BERTBASE BERTLARGE XLNetBASE XLNetLARGE RoBERTaBASE RoBERTaLARGE

Table 9 lists the hyperparameters used to fine-tune the pre-trained language models on ReClor. The supplied passage identifies the table but does not provide its parameter values.

  • Table 9 presents hyperparameters for fine-tuning pre-trained language models on ReClor.

C EXAMPLES

This section provides definitions and examples for a broad set of logical reasoning types, including assumption, argument, inference, structure, and miscellaneous categories.

  • Assumptions: Assumption questions include Necessary Assumptions and Sufficient Assumptions.The cited tables present definitions and examples for both types.
  • Argument and inference: Argument questions include Strengthen, Weaken, Evaluation, Implication, Conclusion/Main Point, and Most Strongly Supported.Each cited table gives the definition and an example of one argument-focused reasoning type.
  • Other reasoning types: Additional reasoning types include Explain or Resolve, Principle, Dispute, Technique, and Role.These tables likewise provide a definition and an example for each named type.
  • Structural and miscellaneous types: Structural and miscellaneous categories include Identify a Flaw, Match Flaws, Match the Structure, and Others.The corresponding tables define and exemplify each category.

D CONSISTENCY OF DIFFERENT MODELS

This section examines consistency across different models by measuring pairwise model overlap after intersection across four random seeds.

  • Model consistency: Table 27 reports overlap for each pair of models after intersection among 4 random seeds.The table is explicitly labeled as a pairwise overlap analysis across multiple random seeds.

E RESULTS WITH RESPECT TO DIFFERENT QUESTION TYPES

This section presents baseline-model accuracy on the overall, EASY, and HARD testing sets, including a comparison of BERTLARGE and RoBERTaLARGE across those subsets.

  • Overall testing set: Figure 5 presents accuracy for all baseline models on the overall testing set.The figure is labeled as an overall testing-set evaluation.
  • EASY testing set: Figure 6 presents accuracy for all baseline models on the EASY testing set.The figure isolates the EASY subset of the testing data.
  • HARD testing set: Figure 7 presents accuracy for all baseline models on the HARD testing set.The figure isolates the HARD subset of the testing data.
  • Model and subset comparison: Figure 8 compares BERTLARGE and RoBERTaLARGE on EASY and HARD testing sets.BERTLARGE results appear at the top and RoBERTaLARGE results at the bottom; EASY is on the left and HARD on the right.
Loading 2002.04326v3…