LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

1 Introduction

• Logical reasoning is a fundamental aspect of reading comprehension, but relatively few modern NLP datasets target it.Earlier and newer machine reading tasks include factual answering, multi-hop evidence integration, and commonsense questions.Source: §1 · p. 1
• LogiQA questions combine natural-language understanding with logical inference, relying less on lexical overlap and external knowledge than factual and commonsense QA.The paper frames them as an opportunity to investigate neural approaches to logical reasoning alongside formal semantic parsing and symbolic reasoning.Source: §1 · pp. 1–2
• LogiQA contains 8,678 paragraph-question pairs with four candidate answers, selected to cover categorical, conditional, disjunctive, and conjunctive reasoning.Problems were manually selected from expert-written public logical examination papers, filtering out figure-, chart-, and mathematics-heavy problems.Source: §1 · p. 2
• In baseline experiments, several state-of-the-art neural reading-comprehension models achieved 35.31% accuracy on LogiQA, compared with 96.00% human ceiling performance, demonstrating a significant gap between machine and human results.Source: §1 · p. 2

2 Related Work

• Unlike many reading-comprehension datasets built around answers found in the passage, LogiQA requires reasoning beyond text matching.Related datasets target challenges such as evidence integration, multi-step reasoning, numerical reasoning, and dialogue reasoning.Source: §2 · p. 2
• Unlike commonsense datasets that draw on external background knowledge, LogiQA focuses on logical reasoning with most necessary facts in the passage.The authors also distinguish its expert-written examination questions from crowd-sourced datasets.Source: §2 · p. 2
• Compared with argument-reasoning and family-relation datasets, LogiQA is a larger, four-choice machine-reading test of general deductive reasoning.The paper reports 8,678 instances versus 1,970 for Habernal et al.'s dataset, which has two choices; CLUTRR targets inductive reasoning in a fixed family-relations domain.Source: §2 · p. 2
• LogiQA is presented as the first large-scale dataset of varied logical problems drawn from human-expert-designed exams.Unlike exams focused on general language understanding, the cited comparison targets logical skills as well as language skills.Source: §2 · p. 2

3.1 Data Collection and Statistics

• The cleaned dataset contains 8,678 paragraph-question pairs after removing non-four-choice, non-self-contained, duplicated, and otherwise difficult-to-reproduce items.The raw collection comprised 13,918 paragraph-question-choice triples with correct answers.Source: §3.1 · p. 3
• Five professional English speakers translated the Chinese dataset, with three proofreaders reviewing translations and requesting revisions when they rejected an instance.The paper also releases a Chinese version for Chinese reasoning-based reading-comprehension research.Source: §3.1 · p. 3
• The authors describe LogiQA's average paragraph length as relatively small because these logical reasoning problems do not heavily rely on complex context..Source: §3.1 · p. 3

3.2 Reasoning Types of the Dataset

• The 867-question test set is categorized into categorical, sufficient conditional, necessary conditional, disjunctive, and conjunctive reasoning, and one question may receive multiple types.These are deductive forms for which a definite conclusion can be derived from premises.Source: §3.2 · pp. 3–4

Human Performance

• The evaluation includes rule-based, deep learning, and pretrained reading-comprehension models, alongside human and ceiling performance.Three postgraduate students answer 500 randomly selected test instances; a question counts toward the ceiling if any student answers correctly.Source: §4 · p. 4

Rule-Based Methods

• Rule-based baselines select answers by unigram overlap or by TF-IDF-style n-gram matching scores.Word matching uses the paragraph-question pair; sliding window extracts n-gram features from that pair.Source: §4 · p. 4

Deep Learning Methods

• Deep learning baselines match paragraph-question text with candidate answers using attentive, multi-hop, or co-matching architectures.The co-matching network also matches paragraph-question and paragraph-candidate pairs.Source: §4 · p. 4

Pre-trained Methods

• Pretrained models score four concatenated paragraph-question-candidate inputs and select the highest-scoring answer.BERT uses [CLS] A [SEP] B [SEP], while RoBERTa substitutes its pretrained model; both use the [CLS] state for MLP-plus-softmax scoring.Source: §4 · p. 4

Implementation Details

• The study re-implements rule-based baselines, uses released implementations for deep learning models, and selects hyperparameters on development sets.Pretrained models follow the HuggingFace implementation and are fine-tuned during training.Source: §4 · p. 4

Main Results

• 86.00% human performance and 95.00% ceiling performance contrast with algorithmic models that perform significantly worse than humans.The dataset is randomly split into 80% training, 10% development, and 10% testing.Source: §5 · p. 4, Table 3 · p. 5

Chinese Results

• Chinese dataset results are reported at the same level as English dataset results.The comparison is reported for the study's results across the two datasets.Source: §5 · p. 4

Rule-Based Results

• 28.37% and 22.51% accuracy for the two rule-based methods show that lexical matching alone performs poorly.The latter score is below the random-guess baseline.Source: §5 · p. 4

Pre-trained Results

• 35.31% is the best RoBERTa result on LogiQA, still much lower than human performance.Pretrained models perform better than methods without contextualized embeddings.Source: §5 · p. 5

6.1 Ablation Tests

• 35.31% to 28.92% is the accuracy drop when removing the paragraph; omitting the question hurts less, supporting the conclusion that dataset bias is weak.The authors attribute the relatively smaller question-related drop as likely reflecting lower question diversity.Source: §6.1 · p. 5

6.2 Transfer Learning

• RACE- or COSMOS-pretrained RoBERTa underperforms LogiQA-only; further LogiQA tuning improves source baselines but still does not surpass LogiQA-only, with RACE near random.Source: Table 4 · p. 5, §6.2 · p. 5

6.3 Performance Across Different Lengths

• RoBERTa accuracy is not negatively associated with total instance length (paragraph, question, and candidates combined), suggesting logical challenge is independent of verbosity.Source: Table 5 · p. 5, §6.3 · p. 5

6.4 Lexical Overlap

• 28.37% gold-standard Overlap Ratio is below every model’s; word matching reaches 100%, while RoBERTa has the lowest matching accuracy and leads on best-match-correct cases.Source: Table 6 · p. 6, §6.4 · p. 6

6.5 Reasoning Types

• RoBERTa performs best on categorical reasoning but lags on the other four types; cases expose ignored age constraints, negation, and conjunctive/disjunctive enumeration.Source: Table 7 · p. 6, §6.5 · p. 6

• LogiQA is a large-scale dataset for logical reasoning in machine reading comprehension.Source: §7 · p. 6
• The dataset can serve as a benchmark for re-examining logical AI in the deep-learning NLP era.Source: §7 · p. 6
• State-of-the-art machine readers still fall far behind human performance on this challenging reading-comprehension test.Source: §7 · p. 6