Source-linked AI summary
Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives
Aheli Poddar, Saptarshi Sahoo, Sujata Ghosh
TL;DR
The study asks how well LLMs reason about syllogisms and whether context shapes their judgments. It evaluates 14 models on syntactic validity and natural-language believability, finding that top-tier models can excel at syntax while performing at chance on natural-language understanding.
Problem
Method
The authors evaluate 14 LLMs using a dual-ground-truth framework that distinguishes whether conclusions logically follow from whether they are believable.
Source: §1 · p. 1Results
99.6% syntactic accuracy and 52% natural-language-understanding accuracy characterized top-tier models.
Source: §5 · p. 7Takeaways & Limitations
Near-perfect syntactic performance does not imply natural-language understanding in the tested models.
Source: §5 · p. 7The evaluation focuses on categorical syllogisms, so its findings may not generalize to structures with nested quantifiers or modal operators.
Source: §6 · p. 7
Abstract
from arXiv · showhide
We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.
1 Introduction
The study asks whether stronger logical reasoning makes LLMs more human-like, examining syllogisms through formal validity and natural-language believability. Across 14 models, it finds belief bias and a gap between formal accuracy and plausibility judgments.
1 Introduction
- The study asks whether developing excellent logical reasoning in LLMs makes them more like human reasoners, given that logical skill need not mimic human reasoning.The authors motivate this question through syllogistic reasoning from formal and natural-language viewpoints.Source: §1 · p. 1
- The study distinguishes syntactic validity from natural-language believability, allowing formal reasoning and language understanding to be assessed separately.The two dimensions may align or conflict.Source: §1 · p. 1
- Most evaluated models show belief bias, while their syntactic accuracy exceeds their natural-language plausibility accuracy, opposite to the reported human pattern.The study evaluates 14 models across prompting strategies and temperature settings.Source: §1 · p. 1
- Few-shot prompting degrades performance relative to zero-shot, and reasoning capability depends more on architectural choices than raw parameter count.The authors raise the question of whether LLMs are becoming formal reasoning engines rather than reflecting human-like reasoning biases.Source: §1 · p. 2
2 On Syllogisms
The section introduces syllogisms as arguments with two premises and a conclusion, then distinguishes logical validity from believability. It outlines categorical forms, figures, and validity and belief-bias categories.
2 On Syllogisms
- A syllogism has two subject-predicate premises and a conclusion, whose validity asks whether it follows from the premises.The conclusion relates the two non-shared terms.Source: §2 · p. 2
- Categorical statements use four forms: universal affirmative (A), universal negative (E), particular affirmative (I), and particular negative (O).The section mostly concentrates on categorical syllogisms while also considering a few other types.Source: §2 · p. 2
- The common variable absent from the conclusion determines the syllogism's figure through its placement in the premises.A, E, I, and O are called moods, and the variables can change order.Source: §2 · p. 2
- A syllogism is valid when true premises imply a true conclusion; validity can be checked in first-order language or by enumerating cases with Venn diagrams.For a new tuple, checking validity reduces to matching it to an already defined case.Source: §2 · p. 2
- Believability concerns whether the conclusion is actually true, independent of the argument's logic; the study distinguishes four validity-believability combinations.The categories are valid-believable, valid-unbelievable, invalid-believable, and invalid-unbelievable.Source: §2 · p. 2
3 Experiments
The experiments systematically evaluate syllogistic reasoning across models, prompting strategies, temperatures, and content variations. The design includes 168 unique configurations.
3 Experiments
- The evaluation covers 168 unique configurations: 14 models × 4 prompting strategies × 3 temperatures.These configurations support analysis of factors influencing syllogistic reasoning performance.Source: §3 · p. 2
- The study examines how prompting strategies, temperature settings, and content variations relate to logical inference accuracy.The evaluation is described as systematic and spans diverse language models.Source: §3 · p. 2
- The experimental design analyzes factors influencing LLM syllogistic reasoning performance across the evaluated configurations.The passage identifies prompting strategy, temperature, and content variation as the factors examined.Source: §3 · p. 2
3.1 Models
The study evaluates 14 models from eight organizations, selected to span organizational, scale, and architectural diversity while supporting reproducibility.
3.1 Models
- The evaluation includes 14 large language models spanning eight organizations, with parameter scales from 1B to 671B.Gemini models were accessed through Google AI Studio APIs; other models used the HuggingFace Inference API with cheapest routing.Source: §3.1 · p. 2
- Model selection prioritized organizational diversity, parameter scale, architectural variety, and API reproducibility.The architectural range included dense transformers and Mixture-of-Experts systems.Source: §3.1 · p. 2
- Gemini models were accessed through Google AI Studio APIs, while the remaining models used HuggingFace Inference API with cheapest routing.The routing enabled automatic provider selection.Source: §3.1 · p. 2
3.2 Data and Methodology
The benchmark combines 160 syllogisms with separate annotations for logical validity and natural-language believability. Content and premise-order variants enable tests of reasoning across congruent and incongruent cases.
Dataset Construction
- The 160-syllogism benchmark expands 40 handcrafted base items into normal, nonsense, premise-order-switched, and combined variants.The variants are designed to test logical reasoning without meaningful predicates and sensitivity to premise order.Source: Dataset Construction · pp. 2–3
Dual Ground Truth
- Each item has independent labels for whether its conclusion follows logically and whether it is intuitively plausible given real-world knowledge.The dataset contains 76 valid and 84 invalid items; believability labels cover 38 believable and 122 unbelievable or abstract conclusions.Source: Dual Ground Truth · p. 3
Belief Bias Categories
- Congruent items align logical validity with believability, while incongruent items pit them against each other to test whether models resist semantic plausibility.Accepting a factually correct but logically invalid conclusion, such as affirming the consequent, is identified as diagnostic of belief-bias susceptibility.Source: Belief Bias Categories · p. 4
3.3 Prompting Schema
The prompting schema varies demonstrations and reasoning scaffolding while keeping the requested answer format constant. Inference is deterministic at zero temperature and uses repeated sampling with early stopping at nonzero temperatures.
Prompting Schema
- Four strategies vary scaffolding: ZS and OS use zero or one example, FS uses four balanced examples including a belief-bias trap, and ZS CoT requests intermediate reasoning.All strategies request the same one-word “correct” or “incorrect” response.Source: §3.3 · p. 4
Strategy Specifications
- The system prompt defines the syllogistic judgment task, while the user prompt formats the premises and conclusion and optionally includes demonstrations.Only ZS CoT adds “Think through step by step” to the system prompt.Source: Strategy Specifications · p. 4
Adaptive Stopping Strategy
- At temperature zero, the procedure queries the model once and returns the parsed prediction with full confidence.The procedure takes a syllogism, prompting strategy, and temperature as inputs and outputs a validity prediction and confidence score.Source: Adaptive Stopping Strategy · p. 4, §3.3 · p. 4
- At nonzero temperatures, the procedure draws up to 10 independent predictions and stops after five unanimous samples.This self-consistency procedure uses temperature-based sampling and early stopping to reduce API calls when predictions are unanimous.Source: Adaptive Stopping Strategy · p. 4
- For stochastic sampling, the final validity label is determined by majority vote, with ties defaulting to “invalid.”The algorithm tracks valid and invalid prediction counts across samples.Source: Adaptive Stopping Strategy · p. 4
3.4 Evaluation Methods
Evaluation measures classification performance, tests validity and believability separately, quantifies belief bias through congruent–incongruent accuracy differences, and assesses consistency across syllogism variants.
Primary Metrics
- Accuracy is the primary metric, alongside precision, recall, and F1, given the near-balanced valid and invalid classes.The dataset is 47.5% valid and 52.5% invalid.Source: Primary Metrics · p. 4
Dual Evaluation Framework
- Dual evaluation compares each response independently with syntactic-validity and natural-language-believability labels.This separates judgments of logical structure from judgments of natural-language content.Source: Dual Evaluation Framework · pp. 4–5
Belief Bias Effect
- Belief bias is the accuracy difference between congruent and incongruent syllogisms, with positive values indicating better performance when logic and semantics align.The within-model comparison controls for differences in overall reasoning ability.Source: Belief Bias Effect · p. 5
Consistency Metric
- Consistency is measured across four variants, with pairwise comparisons isolating robustness to predicate content and premise order.The content comparison contrasts meaningful with nonsense predicates; the order comparison uses matched content types.Source: Consistency Metric · p. 5
4 Results
Across 14 models and 26,880 evaluations, syllogistic accuracy varies sharply; syntax accuracy exceeds natural-language believability, while prompting, temperature, bias, and model stability show distinct patterns.
4.1 Overall Performance
- Six of 14 models exceed 95% syntax accuracy, while five score below 70%; Gemini 2.5 Flash reaches 99.6% and Llama 3.2 1B Instruct scores 51.9%.The mean is 81.7% (SD = 17.1%), with a 47.7% gap between top and bottom performers; Table 3 aggregates results across all 12 configurations.Source: §4.1 · p. 5, Table 3 · p. 6
- Top-tier models have precision and recall both above 97%, while Qwen3-Next 80B A3B Thinking has 99.2% precision and 42.8% recall, and Gemma 3 27B IT has 93.1% recall and 61.0% precision.The latter profiles indicate, respectively, frequent rejection of valid syllogisms and over-acceptance of conclusions.Source: §4.1 · p. 5
- Syntax accuracy is 81.7% versus 56.2% for NLU; top models show syntax–NLU gaps of 47.9 pp for Gemini 2.5 Flash and GPT-OSS-20B, and 47.4 pp for Gemini 2.5 Pro.These models correctly judge logical validity independently of content believability, while some lower-tier models show negative or minimal gaps.Source: §4.1 · pp. 5–6, Figure 3 · p. 5
4.2 Prompting Strategy Effects
- Few-shot prompting averages 79.1% accuracy versus 82.7% for zero-shot; its mean decline survives Holm-Bonferroni correction, but the overall four-strategy effect is not significant.McNemar’s test finds different instances solved by the strategies: zero-shot solves 786 that few-shot misses, while few-shot solves 546 that zero-shot misses; effects appear model-specific.Source: §4.2 · p. 6
4.3 Temperature and Belief Bias Effects
- Temperature has no significant accuracy effect with adaptive stopping, while 12 of 14 models show higher accuracy on congruent than incongruent problems.Mean belief bias is +10.81 pp (SD = 16.32; paired t-test p = 0.0280); adaptive majority voting normalizes stochastic variation.Source: §4.3 · p. 6, Table 4 · p. 6
4.4 Consistency and Benchmark Correlations
- Syllogistic reasoning accuracy is strongly associated with stability across variants (Spearman ρ = 0.890, p < 0.0001) and with LMArena rank (Spearman ρ = −0.825, p = 0.0010, N = 12).Lower LMArena rank indicates better performance.Source: §4.4 · p. 7, Figure 4 · p. 7
4.5 Statistical Summary
- The few-shot mean decline survives Holm-Bonferroni correction, but its median effect is marginally non-significant after correction (p_adj = 0.0584).Table 5 summarizes the statistical findings, including the significant instance-level error redistribution detected by McNemar’s test.Source: §4.5 · p. 7
- Higher syntax accuracy is associated with smaller belief-bias magnitudes (Spearman ρ = −0.565, p = 0.0353), while syntax and NLU accuracy are moderately negatively correlated (Spearman ρ = −0.543, p = 0.0449).The paper interprets these patterns as indicating that higher-performing models exhibit smaller bias magnitudes and that logical-structure performance may diverge from intuitive believability judgments.Source: §4.5 · p. 7
5 Discussion
Across 14 models, top-tier systems nearly reached perfect syntactic accuracy while remaining at chance on natural language understanding. Most models showed belief bias, although stronger reasoning aligned with smaller bias.
5 Discussion
- 99.6% syntactic accuracy versus 52% natural language understanding: top-tier models excelled at formal logic but remained at chance on semantic plausibility.The study evaluated 40 syllogism instances and variations, totaling 160 data points across 14 models.Source: §5 · p. 7
- +10.81 pp mean effect (p = 0.028) reflected significant belief bias in most models, while bias magnitude declined with reasoning capability (ρ = −0.565, p = 0.035).Source: §5 · p. 7
- Architectural and training choices proved more consequential than raw parameter count by substantial margins.Source: §5 · p. 7
- Few-shot prompting degraded performance compared with zero-shot prompting, suggesting demonstration examples may introduce noise in formal reasoning tasks.Source: §5 · p. 7
- Instruction-following quality correlated strongly with reasoning accuracy (LMArena, ρ = −0.825), which the authors link to precise rule adherence underlying both capabilities.Source: §5 · p. 7
- The top performers appeared more like formal logic engines than human-like reasoners susceptible to natural-language biases, raising questions about alignment with human cognition.Source: §5 · p. 7
6 Limitations
The evaluation is limited in logical scope, model and prompting coverage, and the dimensions used to measure belief bias and robustness.
6 Limitations
- The study focuses mainly on categorical syllogisms, which may not generalize to nested quantifiers or modal operators, and its framework simplifies logic-language interaction in human reasoning.Source: §6 · p. 7
- The 14-model sample is a snapshot rather than exhaustive coverage, and the prompting strategies cover only a limited part of the prompt-engineering space.Source: §6 · p. 7
- Consistency metrics assess stability across content and order variations, but not robustness to adversarial perturbations or systematically manipulated distractors.Source: §6 · p. 7
- Belief bias captures only one dimension of the relationship between real-world beliefs and logical reasoning; response time, confidence calibration, and fine-grained error taxonomies could broaden assessment.Source: §6 · p. 8
7 Future Work
Future work would test whether the observed formal-logic and natural-language divide generalizes beyond categorical syllogisms, and investigate its computational basis and prompting conditions.
7 Future Work
- Evaluation could extend to modal and transitive-closure logics, especially systems with simple formal syntax but complex natural-language semantics.Source: §7 · p. 8
- Mechanistic interpretability could distinguish explicit logical rules, statistical approximations, and hybrid representations behind near-perfect syntactic performance.Source: §7 · p. 8
- Controlled fine-tuning experiments could test whether logical training reduces bias or reduced bias enables better reasoning, an unresolved causal relationship.Source: §7 · p. 8
- Systematic study of when demonstrations help or hinder reasoning could clarify the reported few-shot degradation and inform prompting strategies for logical tasks.Source: §7 · p. 8
- The human-like reasoner versus formal-logic-engine tension bears on model development and appropriate expectations for reasoning-intensive deployments.Source: §7 · p. 8