PaperlayerView exact revision on arXiv

Source-linked AI summary

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral

arXiv:2404.15522v2cs.CLcs.AI

TL;DR

Logical reasoning in LLMs remains underexplored because prior evaluations often cover few inference rules or a single logic type. LogicBench systematically evaluates 25 patterns across propositional, first-order, and non-monotonic logics using single-rule QA tasks, and finds that current models perform poorly, especially on complex contexts and negations. The study supports broader evaluation while showing that the benchmark remains limited to single-rule reasoning and English.

  • Problem

    Logical reasoning remains underexplored, while existing evaluations often cover only a few inference rules or a single logic type.

    Source: §1 · p. 1

  • Method

    LogicBench evaluates 25 reasoning patterns across propositional, first-order, and non-monotonic logics using binary and multiple-choice questions focused on single inference rules.

    Source: §1 · p. 1

  • Results

    LLMs do not perform well on LogicBench, particularly on complex reasoning and negation-related instances.

    Source: §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    LogicBench provides a systematic basis for evaluating and improving LLM logical reasoning across diverse inference patterns.

    Source: §5 · p. 9

    The study evaluates only single inference rules, is limited to English, and could be extended with additional rules, logic types, and multi-step reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied. However, the crucial skill pertaining to 'logical reasoning' has remained underexplored. Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic. Addressing the above limitation, we comprehensively evaluate the logical reasoning ability of LLMs on 25 different reasoning patterns spanning over propositional, first-order, and non-monotonic logics. To enable systematic evaluation, we introduce LogicBench, a natural language question-answering dataset focusing on the use of a single inference rule. We conduct detailed analysis with a range of LLMs such as GPT-4, ChatGPT, Gemini, Llama-2, and Mistral using chain-of-thought prompting. Experimental results show that existing LLMs do not fare well on LogicBench; especially, they struggle with instances involving complex reasoning and negations. Furthermore, they sometimes overlook contextual information necessary for reasoning to arrive at the correct conclusion. We believe that our work and findings facilitate future research for evaluating and enhancing the logical reasoning ability of LLMs. Data and code are available at https://github.com/Mihir3009/LogicBench.

1 Introduction

LogicBench addresses limited evaluation of LLM logical reasoning by isolating single inference rules across diverse logic types. The dataset supports systematic testing, and experiments find substantial weaknesses, especially for complex contexts and negations.

Motivation

  • LogicBench targets the gap left by datasets that mix reasoning types or cover only one logic and few inference rules.The authors position logical reasoning as important for question answering and conversational agents.Source: §1 · p. 1

Dataset

  • LogicBench covers 25 reasoning patterns across propositional, first-order, and non-monotonic logics.It includes inference rules such as hypothetical and disjunctive syllogism and several dilemma forms.Source: §1 · p. 2

Evaluation

  • The benchmark uses binary and multiple-choice question answering to measure whether conclusions are entailed or logically appropriate.These formats provide direct accuracy comparisons across models.Source: §1 · p. 2

Construction

  • LogicBench is constructed through sentence generation, context-question generation, and task-specific answer-triplet generation.The first stage includes varied ontologies and corresponding negations.Source: §1 · p. 2

Findings

  • LLMs struggle with complex logical contexts and inference rules involving negations, despite needing only a single inference rule.The experiments evaluate GPT-4, ChatGPT, Gemini-Pro, Llama-2-7B-Chat, and Mistral-7B-Instruct with chain-of-thought prompting.Source: §1 · p. 2

2 Related Work

Prior benchmarks either mix logical reasoning with other reasoning forms or restrict evaluation to limited logic types and inference rules. LogicBench is designed to evaluate logical reasoning in isolation across a broader range of rules and logic types.

Existing benchmarks

  • LogiQA and ReClor contain mixed forms of reasoning, whereas LogicBench isolates logical reasoning.This distinction motivates a dedicated benchmark for logical reasoning ability.Source: §2 · pp. 2–3
  • Earlier logic-focused datasets commonly provide narrower coverage, such as Ruletaker’s limited synthetic deductive setting and LogicNLI’s automatically generated first-order logic expressions.Other datasets address inductive reasoning, temporal logic, or soft logical rules rather than LogicBench’s full scope.Source: §2 · p. 2

Comparison

  • ProofWriter supports multihop proofs, FOLIO offers diverse first-order expressions, and ProntoQA provides explanations but focuses on modus ponens in first-order logic.These differences help motivate LogicBench’s broader inference-rule coverage.Source: §2 · pp. 2–3

LogicBench

  • LogicBench compares inference rules and logic coverage with existing datasets while evaluating 25 patterns across propositional, first-order, and non-monotonic logics.Table 2 identifies rules including modus ponens, modus tollens, syllogisms, dilemmas, and quantifier axioms.Source: Table 2 · p. 3

3 LogicBench

LogicBench evaluates logical reasoning through 25 patterns across propositional, first-order, and non-monotonic logics, using systematically generated natural-language tasks based on single inference rules. Its creation pipeline generates sentences and negations, converts formal patterns into narratives, constructs BQA and MCQA instances, and validates the resulting dataset.

3.1 Logics Types

  • LogicBench covers nine propositional inference rules, quantified extensions, two first-order axioms, and non-monotonic reasoning patterns involving defaults, unknown expectations, and priorities.Non-monotonic examples are adapted from Lifschitz (1989).Source: §3.1 · pp. 3–4, Table 3 · p. 4

3.2 Data Creation

  • The three-stage data-creation procedure generates sentences and negations, converts logical templates into natural-language narratives, and produces task-specific context-question-answer triplets.The procedure is illustrated schematically in Figure 2.Source: §3.2 · p. 4, Figure 2 · p. 5, §3.2.2 · p. 5

3.2.1 Sentence Generation

  • Sentence generation uses GPT-3.5 to create coherent sentences across diverse ontologies, while negations are generated separately and later combined with logical connectives.The objective at this stage is linguistic diversity rather than generating complete logical sentences.Source: §3.2.1 · p. 4

3.2.2 NL Conversion

  • Formal reasoning expressions are first converted into connective-specific templates, then GPT-3.5 transforms them into natural narrative contexts while preserving the intended logical relationships.The conversion covers formulations such as implication, conjunction, and disjunction, with examples spanning propositional and first-order rules.Source: §3.2.2 · pp. 4–5

3.2.3 Task Instance Generation

  • LogicBench creates binary-question and multiple-choice instances from shared narrative contexts, varying binary questions and generating three incorrect options for each multiple-choice item.MCQA items contain one correct option among four, whereas BQA uses semantically preserving and inverted question-answer variations.Source: §3.2.3 · p. 5

3.3 Statistics and Qualitative Analysis

  • LogicBench(Eval) contains 1,520 BQA samples and 500 unique MCQA samples, while LogicBench(Aug) contains 12,908 BQA samples including variations; generated narratives undergo logical, grammatical, and diversity validation.The evaluation set has 520 “yes” and 1,000 “no” BQA labels, and augmentation provides 150 unique samples per rule for training.Source: §3.3 · pp. 5–6, Table 5 · p. 6

4 Results and Analysis

The authors evaluate multiple LLMs on LogicBench using zero-shot chain-of-thought prompts across BQA and MCQA, finding persistent weaknesses in single-rule logical reasoning. Performance varies by model size, logic type, negation, rule length, and task format.

4.1 Experimental Setup

  • BQA labels whether a conclusion is entailed by a narrative context, whereas MCQA asks models to select the option containing the correct conclusion.BQA reports label-wise A(Y es) and A(No), while MCQA reports overall accuracy.Source: §4.1 · pp. 6–7
  • The evaluation uses zero-shot chain-of-thought prompts across GPT-4, ChatGPT, Gemini-Pro, Llama-2-7B-Chat, and Mistral-7B-Instruct, measuring accuracy on BQA and MCQA.Results are averaged across three chain-of-thought prompts; few-shot results are reported separately in Appendix G.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 63.98% A(Y es) is GPT-4’s average performance on propositional-logic BQA, while ChatGPT achieves 48.04%, showing that classical logical reasoning remains challenging.Models generally perform worse on propositional-logic rules than on first-order and non-monotonic rules, and perform better on No than Y es labels.Source: §4.2 · p. 7

4.3 Analysis and Discussion

  • Humans exceed approximately 85% accuracy across logic types, whereas all models remain below human performance; non-monotonic MCQA challenges both groups.Human evaluation covered 153 BQA pairs and 50 MCQA instances, with interannotator agreement of 0.785 for BQA and 0.813 for MCQA.Source: §4.3 · p. 8
  • LLM performance decreases for longer inference rules and rules containing negations, with negated premises especially difficult in PL and FOL.The affected PL and FOL patterns include DD and MT, while longer dependencies challenge rules such as BD, CD, and DD; DRS is difficult in non-monotonic reasoning.Source: §4.2 · p. 7, §4.3 · pp. 8–9
  • Models sometimes hallucinate information or overlook context, affecting BQA more than MCQA, although this pattern is not dominant.The authors identify this through manual examination of reasoning chains for instances sharing the same contexts.Source: §4.3 · p. 9
  • Larger models generally achieve higher logical-reasoning performance, with Yi-34B improving over Llama-2-7B and GPT-4 outperforming Yi-34B.The authors interpret these results as evidence that larger model capacity is associated with better logical reasoning.Source: §4.3 · p. 9
  • BQA and MCQA expose different difficulty patterns: PL performance is higher in MCQA than BQA, whereas NM performance is lower in MCQA than BQA.For PL, CT and DD contribute to the task gap; for NM, DRO, RAP, and DRD contribute to it.Source: §4.3 · p. 9

5 Conclusions

The paper introduces LogicBench to evaluate LLM logical reasoning across 25 single-rule patterns and reports weaknesses even on these limited reasoning tasks. Augmenting LogicBench for training improves performance on existing logic datasets.

5 Conclusions

  • LogicBench evaluates GPT-4, ChatGPT, Gemini-Pro, Llama-2, and Mistral on 25 inference rules spanning propositional, first-order, and non-monotonic logic.The benchmark provides both BQA and MCQA tasks focused on applying a single inference rule.Source: §5 · p. 9
  • LLMs perform poorly on LogicBench despite needing only one inference rule, while LogicBench(Aug)-trained models show improved logical reasoning and performance on existing logic datasets.The conclusion presents augmentation and training as a demonstrated use of the benchmark.Source: §5 · p. 9

Limitations

LogicBench covers 25 rules across three logic types but remains bounded in rule coverage, reasoning depth, and language scope. The authors identify multi-step, additional-logic, and multilingual evaluation as future extensions.

Limitations

  • LogicBench can be extended with additional inference rules and logic types, although first-order and quantified logics may contain infinitely many possible rules.The study covers substantially more rules than previous work but does not exhaust the space of logical patterns.Source: Limitations · p. 9
  • The study evaluates only single-inference-rule reasoning, leaving combinations of rules and deeper multi-step reasoning for future work.The authors explicitly propose increasing reasoning depth by combining inference rules.Source: Limitations · p. 9
  • The evaluation is limited to English and could be extended to multilingual scenarios.This is stated as a language-scope limitation and future direction.Source: Limitations · p. 9

Ethics Statement

The paper documents AI-assisted editing, describes prompt-based sentence generation, and situates LogicBench among datasets for logical reasoning, prompting, instruction learning, and NLI.

Ethics Statement

  • The authors used Grammarly and ChatGPT to address grammatical errors and rephrase sentences.Source: Ethics Statement · p. 10

A Example Prompt for Sentence Generation

  • The sentence-generation prompt combines a reasoning-pattern definition, examples, and formatting instructions to guide coherent outputs with relevant ontologies.The prompt is illustrated schematically in Figure 3 and exemplified for modus tollens in Figure 4.Source: A Example Prompt for Sentence Generation · p. 14, Figure 3 · p. 14

B Extended Related Work

  • LogicBench evaluates individual inference rules beyond NLI, contrasting with datasets and methods focused on temporal semantics, monotonicity, entailment, or adversarial NLI attacks.The related work also reviews prompt learning and instruction-learning approaches used for zero-shot and cross-task generalization.Source: B Extended Related Work · pp. 14–15

C Examples of Data Instances

This section illustrates LogicBench instances across propositional, first-order, and non-monotonic reasoning, then details narrative conversion and task-question generation.

C Examples of Data Instances

  • LogicBench examples span propositional, first-order, and non-monotonic reasoning patterns, with contexts, questions, and answers organized around individual inference rules.The examples include rule-specific variants and logical conclusions represented as yes/no answers or multiple-choice options.Source: C Examples of Data Instances · p. 15, §C.2 · p. 15, §C.3 · p. 16

C.1 Word Cloud

  • The dataset’s word cloud combines logical terms such as ‘if’, ‘normally’, and ‘then’ with varied ontology terms including ‘cat’, ‘car’, and ‘garden’.Source: §C.1 · p. 15, Figure 5 · p. 16

C.2 Propositional Logic (PL)

  • Propositional examples generate affirmative and negated question variants from rules such as Hypothetical Syllogism, labeling only conclusions supported by the context as “Yes”.For Hypothetical Syllogism, variants include ¬p →r, p →¬r, and ¬p →¬r, which are labeled “No”.Source: §C.2 · pp. 15–16, Table 9 · p. 17

C.3 First-Order Logic (FOL)

  • First-order examples cover inference rules and axioms such as Existential Generalization and Universal Instantiation, with questions testing generalized and specific instances.Existential Instantiation is illustrated as ∃xP (x) ⇒P (a), with a negated variant labeled “No”.Source: §C.3 · p. 16, Table 10 · p. 18

C.4 Non-Monotonic (NM) Reasoning

  • Non-monotonic examples use reasoning patterns such as Default Reasoning with Irrelevant Information and provide correct and incorrect conclusion variants.Source: §C.4 · p. 16, Table 11 · p. 19

D Examples of NL Conversion

  • Templatized logical contexts are converted into narratives using separate prompts for propositional, first-order, and non-monotonic logic while preserving rule-specific sentence structure.The conversion uses sentence and negation pairs for PL and FOL, and targeted negation patterns for NM.Source: D Examples of NL Conversion · pp. 16, 20

E Task Instance Generation

  • Task 1 generates yes/no question variations by negating rule components, whereas Task 2 selects questions and adds three prompted incorrect options around one correct conclusion.MCQA uses the same narrative as BQA but presents the derivable conclusion among four options.Source: §E.1 · p. 21, §E.2 · p. 21

F Mitigation of Errors in LogicBench

The authors manually validate LogicBench narratives and multiple-choice options, correcting leakage, missing premises, and ambiguous distractors.

F Mitigation of Errors in LogicBench

G Few-shot Experiments

Few-shot prompting produces mixed model-specific effects across LogicBench reasoning types, with improvements concentrated in particular models and logic categories.

G Few-shot Experiments

  • The few-shot case study evaluates Llama-2, Mistral, Gemini, ChatGPT, and GPT-4 across inference rules using average A(Y es) performance on BQA.Source: G Few-shot Experiments · p. 22
  • Llama-2 consistently outperforms its zero-shot baselines by large margins with in-context examples, while Mistral degrades across all logic types.Source: G Few-shot Experiments · p. 22
  • ChatGPT improves on non-monotonic reasoning, which the authors associate with better comprehension of terms such as “usually” and “typically”.Source: G Few-shot Experiments · p. 22
  • Gemini improves on propositional logic and GPT-4 improves on first-order logic, while both remain competitive on non-monotonic reasoning.Source: G Few-shot Experiments · p. 22

H.1 Extended Discussion on Experiments

The extended experiments document model versions, prompt averaging, and prompt formats used for LogicBench evaluation and related T5-large experiments.

H.1 Extended Discussion on Experiments

  • Zero-shot evaluations used January 2024 versions of GPT-4, ChatGPT, and Gemini-Pro, alongside Llama-2 7B-Chat and Mistral 7B-Instruct-v0.2.The authors caution that regularly updated APIs may produce variations when reproducing the reported results.Source: §H.1 · p. 22
  • Experiments on other logic datasets fine-tuned T5-large for 10 epochs with batch size 16, maximum input length 1024, learning rate 5e −05, and AdamW.Source: §H.1 · p. 22

H.2 Prompts

  • Zero-shot results average performance across three prompts that combine task descriptions, formatting instructions, step-by-step reasoning, and constrained answer formats.Source: §H.2 · pp. 22–23
  • MCQA results likewise average three prompts requiring step-by-step reasoning and answers restricted to one of four choices.Source: §H.2 · p. 23

I Further Discussion on Results

Further analyses examine transfer effects, reasoning behavior, contextual oversights, and differences across logic types and task formats. Together, these results characterize where LogicBench performance varies and how models approach its inference patterns.

Effect on other logic datasets

  • LogicT5 has a greater impact on datasets focused primarily on logical reasoning, while remaining competitive on datasets that combine logic with other reasoning forms.The comparison concerns single-task and multitask accuracy against baseline T5-large.Source: I Further Discussion on Results · p. 23, Table 12 · p. 24

LLMs sometimes overlook contextual information

  • Approximately 1% of analyzed PL and NM BQA reasoning chains and 0.8% of MCQA chains introduced unrelated information and produced incorrect predictions.The observed pattern involved GPT-4 deviating from the provided context during a reasoning chain.Source: I Further Discussion on Results · pp. 23–24
  • In a Modus Tollens example, the model incorrectly inferred that not catching a cold established that James remembered to wear a sweater.The reasoning chain acknowledged alternative explanations for not catching a cold but still returned the wrong conclusion relative to the gold answer.Source: I Further Discussion on Results · p. 24

PL vs. FOL

  • A prompt sample contained 13 FOL-style and 7 propositional sentences, while deeper story-based PL examples were reported as easier than generalized FOL contexts.The authors connect this pattern to simpler FOL performance and changing difficulty as reasoning depth increases.Source: I Further Discussion on Results · p. 24

Performance of BQA vs. MCQA

  • For PL, LLMs often use elimination in MCQA, whereas for NM they struggle to distinguish options and hallucinate assumptions, lowering performance.The analysis compares reasoning chains for conditional transitivity and disjunctive cases across BQA and MCQA.Source: I Further Discussion on Results · p. 24

Additional analyses

Related papers

Loading 2404.15522v2…