Source-linked AI summary
Logical Reasoning in Large Language Models: A Survey
Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, Yue Zhang
TL;DR
The survey addresses the gap between LLMs’ demonstrated heuristic reasoning and the question of whether they can perform rigorous logical reasoning. It synthesizes research across reasoning paradigms, benchmarks, and enhancement strategies, concluding that progress depends on hybrid architectures, robust evaluation, and scalable methods.
Problem
Despite LLMs’ impressive heuristic reasoning, rigorous logical inference across deductive, inductive, abductive, and analogical paradigms remains inconsistent amid robustness, generalization, and interpretability limitations.
Source: §7 · p. 7Method
Results
The survey concludes that LLM logical inference remains inconsistent, while identifying hybrid architectures, robust evaluation frameworks, and scalable methods as priorities for future progress.
Source: §7 · p. 7Takeaways & Limitations
Future progress hinges on hybrid architectures unifying neural and symbolic reasoning, robust evaluation frameworks, and scalable methods for cross-domain and multimodal reasoning.
Source: §6 · p. 7, §7 · p. 7LLMs struggle with extended deductive reasoning, hypothetical sub-proofs without examples, generalization, and sensitivity to syntactic variations.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
With the emergence of advanced reasoning models like OpenAI o3 and DeepSeek-R1, large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, their ability to perform rigorous logical reasoning remains an open question. This survey synthesizes recent advancements in logical reasoning within LLMs, a critical area of AI research. It outlines the scope of logical reasoning in LLMs, its theoretical foundations, and the benchmarks used to evaluate reasoning proficiency. We analyze existing capabilities across different reasoning paradigms - deductive, inductive, abductive, and analogical - and assess strategies to enhance reasoning performance, including data-centric tuning, reinforcement learning, decoding strategies, and neuro-symbolic approaches. The review concludes with future directions, emphasizing the need for further exploration to strengthen logical reasoning in AI systems.
1 Introduction
The survey focuses on formal and symbolic logical reasoning in LLMs, distinguishing it from heuristic reasoning and reviewing reasoning paradigms, benchmarks, enhancement strategies, and open challenges.
1 Introduction
- Logical reasoning structures inference in LLMs to support multistep deduction and abstraction, with potential benefits for interpretability, reliability, and generalization to novel scenarios.The introduction notes its growing importance as LLMs enter domains such as legal analysis and scientific discovery.Source: §1 · p. 1
- Post-training LLMs for reasoning has attracted growing interest in industry and research.The passage attributes this attention to the increasing importance of ensuring reasoning correctness and verifiability.Source: §1 · p. 1
- Existing surveys often conflate logical reasoning with general-purpose heuristic strategies, leaving formal and symbolic logic in LLMs without a dedicated review.The survey distinguishes deductive, inductive, abductive, and analogical reasoning and examines benchmarks, enhancement methods, and open challenges.Source: §1 · p. 1
2 Logic in Artificial Intelligence
The section introduces logical reasoning as structured inference and traces its development from classical logic to modern AI. It distinguishes deductive, inductive, abductive, and analogical reasoning by how each supports inference.
2 Logic in Artificial Intelligence
- Logical reasoning applies structured rules to derive conclusions from premises, providing a framework for decision-making and inference.The section presents it as a cornerstone of AI for solving complex problems.Source: §2 · p. 1
2.1 History of Logic Reasoning Research
- The history runs from Aristotle’s syllogisms and Leibniz’s logic-mathematics connection through Boolean algebra, formal logic, knowledge representation, expert systems, and knowledge graphs.The section describes 21st-century neuro-symbolic approaches as merging deep learning with logical inference.Source: §2.1 · pp. 1–2
2.2 Types of Logical Reasoning
- Deduction derives specific conclusions from general premises, induction generalizes from observations, abduction seeks plausible explanations, and analogy transfers inferences across similar situations.Deductive conclusions follow necessarily from true premises and valid reasoning, whereas inductive and abductive conclusions are not guaranteed.Source: §2.2 · p. 2
3 Tasks and Benchmarks
The section surveys logical-reasoning datasets by construction and task, then describes benchmark suites that standardize model evaluation and comparison. It covers NLI, reading comprehension, and suites spanning multiple tasks or evaluation settings.
3 Tasks and Benchmarks
- Logical reasoning datasets are grouped by source into rule-based, expert-designed, and exam-based collections, each offering different routes to scale or carefully crafted evaluation.Rule-based datasets require diversity to avoid repetitive patterns; expert-designed and exam-based datasets draw on expert-crafted problems.Source: §3 · p. 3
3.1 Natural Language Inference (NLI)
- NLI tests whether a hypothesis follows from a premise, using binary or ternary labels; examples include formal-inference datasets and benchmarks spanning multiple logic types.FOLIO uses True/False labels; LogicNLI isolates FOL-based inference, ProofWriter evaluates generalization across linguistic variations, and LogicBench covers 25 reasoning types.Source: §3.1 · p. 3
3.2 Machine Reading Comprehension (MRC)
- MRC asks models to answer questions from a passage through multiple-choice, span-extraction, or free-response formats, with multiple-choice valued for standardization.LogiQA, ReClor, and AR-LSAT are exam-derived multiple-choice datasets; Figure 2 includes a LogiQA reading-comprehension example.Source: §3.2 · p. 3, §3.3 · p. 4, Figure 2 · p. 4
- Other MRC datasets target varied reasoning demands, including kinship inference, mathematical robustness, and pattern generalization in low-resource or extinct languages.CLUTRR combines entity extraction and inference; GSM-PLUS adds mathematical perturbations, while LINGOLY spans over 90 languages.Source: §3.2 · p. 3
3.3 Benchmark Suites
- Benchmark suites standardize evaluation and model comparison, with offerings for low-data generalization, uniform multi-task processing, and accessible dataset-based evaluation.GLoRE includes 17 test-only datasets, LogiGLUE covers 24 tasks, and LogiTorch offers 16 datasets; BIG-bench includes seven logical-reasoning tasks.Source: §3.3 · pp. 3–4
4 Evaluations
The survey reviews evaluation of LLM logical reasoning across deductive, inductive, abductive, and analogical paradigms, alongside benchmarks and metrics. It describes uneven capabilities and argues that evaluation must look beyond conventional task scores.
4.1 Deductive Reasoning
- Deductive reasoning supports theorem proving, but LLMs struggle with extended reasoning, hypothetical sub-proofs without examples, generalization, and syntactic variation.The cited studies report strengths on compositional proofs, standard benchmarks, and encoding entailment relationships [Saparov et al., 2023].Source: §4.1 · p. 4
4.2 Inductive Reasoning
- Inductive reasoning generalizes from instances, yet reported findings range from effective pre-trained reasoners to failures on simple symbolic tasks and fundamental logical principles.Yang et al. [2024b] report effective pre-trained reasoners; Bowen et al. [2024] and Sullivan [2024] report difficulties, including after fine-tuning.Source: §4.2 · p. 4
4.3 Abductive Reasoning
- Abductive reasoning seeks plausible explanations from incomplete information, and studies report challenges generating hypotheses and handling legal-domain tasks.Del and Fishel [2023] and Nguyen et al. [2023] describe these difficulties.Source: §4.3 · p. 4
4.4 Analogical Reasoning
- On analogical reasoning, LLMs struggle as analogy complexity increases, while other findings report learning with little data and question whether relevant examples outperform random ones.The cited work includes the ANALOGICAL benchmark and reports approaching human performance with minimal data [Wijesiriwardene et al., 2023; Petersen and van der Plas, 2023; Qin et al., 2024].Source: §4.4 · p. 4
4.5 Overall Analysis and Metrics
- GPT-4 outperforms ChatGPT on LogiQA and ReClor, but both struggle on out-of-distribution tasks; NeuLR evaluates six dimensions of reasoning.NeuLR’s dimensions are correctness, rigor, self-awareness, proactivity, guidance, and absence of hallucinations [Liu et al., 2023b; Xu et al., 2023].Source: §4.5 · p. 4
- Accuracy and F1 alone are insufficient; evaluation also considers consistency, generalization, and explainability, while BERTScore-based approaches are reported to align with human judgments.BERTScore aligns better with human rankings than LLM evaluators such as GPT-4, and evaluation metrics should account for logical-theory expressivity, particularly for induction [Thatikonda et al., 2025; Liu et al., 2024c; Gandarela et al., 2024].Source: §4.5 · pp. 4–5
5 Enhancement Methods
The survey groups enhancement strategies into data-centric, model-centric, external-knowledge, and neuro-symbolic approaches. These include curating training data, optimizing models and decoding, using external resources, and combining neural systems with symbolic reasoning.
5.1 Data-Centric Approaches
- Data-centric methods optimize curated training data, including expert-assembled resources, synthetic examples, and LLM-distilled reasoning datasets.Examples include FOLIO, LeanDojo, and Symbol-LLM; RuleTaker and FLD×2; and LogiCoT, LogicPro, and PODA, respectively [Han et al., 2024a; Han et al., 2024b; Yang et al., 2023; Xu et al., 2024a; Clark et al., 2021; Morishita et al., 2024; Liu et al., 2023c; Jiang et al., 2024; Wang et al., 2024b].Source: §5.1 · p. 5
5.2 Model-Centric Approaches
- Model-centric approaches jointly optimize model parameters and decoding strategy, with implementations spanning instruction fine-tuning, reinforcement learning, and inference-time decoding.The survey presents these methods as complementary to data-centric approaches [Jiao et al., 2024; Xi et al., 2024].Source: §5.2 · p. 5
Instruction Fine-Tuning
- Instruction fine-tuning uses task-specific instructions, solver-like deduction demonstrations, or symbolic knowledge injection and infusion; contrastive learning and logic-specific architectures are also described.The survey also reports s1’s test-time scaling through fine-tuning on 1,000 long-CoT samples with budget forcing, enabling Qwen2.5-32B-Instruct to extrapolate beyond its performance without test-time intervention [Liu et al., 2023c; Feng et al., 2024; Xu et al., 2024a; Wang et al., 2024b; Wang et al., 2024a; Muennighoff et al., 2025].Source: Instruction Fine-Tuning · p. 5
Reinforcement Learning
- Reinforcement learning is used for planning and reasoning optimization, including outcome-only process supervision, MCTS-based exploration or data generation, and DeepSeek-R1’s cold-start and iterative data-synthesis strategy.R1-Zero uses RL without fine-tuning but faces readability and language-consistency challenges; minimal long-CoT fine-tuning before RL balances usability and reasoning performance [Jiao et al., 2024; Xi et al., 2024; Zhao et al., 2024; Zhang et al., 2024; DeepSeek-AI, 2025].Source: Reinforcement Learning · pp. 5–6
Inference-Time Decoding
- Inference-time decoding scales computation through structured or iterative reasoning workflows, while constrained decoding enforces logical or automata-based constraints.Examples include GoT, Chain of Logic, Maieutic Prompting, Logic-of-Thoughts, DetermLR, Neurologic, and Formal-LLM [Lei et al., 2023; Servantez et al., 2024; Jung et al., 2022; Liu et al., 2024a; Sun et al., 2024; Lu et al., 2021; Li et al., 2024b].Source: Inference-Time Decoding · p. 6
5.3 External Knowledge Utilization
- External-knowledge approaches jointly consider the model and a knowledge-integration strategy, with examples using Lean proofs, knowledge graphs, supergraphs, and selective knowledge acceptance.The survey motivates these approaches by noting hallucinations in complex reasoning tasks [Liu et al., 2024b; Ouyang et al., 2023; Mai et al., 2025].Source: §5.3 · p. 6
5.4 Neuro-Symbolic Approaches
- Hybrid optimization targets both the neural model’s symbolic representations and the solver’s processing, although earlier pipelines often keep the external solver fixed.The model maps inputs to symbolic representations compatible with the solver; advanced practice increasingly uses LLMs in the solver role [Liu et al., 2024a; Toroghi et al., 2024].Source: §5.4 · pp. 6–7
- Neuro-symbolic pipelines can translate natural language into formal logic for external solvers, with work improving translation through training, decoding, verification, or correction.Later systems replace external solvers with rule-guided LLM inference or perform translation, planning, solving, and verification within LLM-driven components [Olausson et al., 2023; Yang et al., 2024a; Ryu et al., 2024; Pan et al., 2023; Liu et al., 2024a; Toroghi et al., 2024; Xu et al., 2024c; Xu et al., 2024b].Source: §5.4 · pp. 6–7
6 Discussion
The discussion identifies tensions in robustness and generalization, interpretability and performance, and evaluation rigor. It points to hybrid systems and more systematic testing as directions for addressing them.
Robustness vs. Generalization
- LLMs show inconsistent structured-reasoning performance: models fine-tuned on FOLIO can excel in controlled settings yet struggle with adversarial perturbations and equivalent rephrasings.The discussion links this inconsistency to surface-level statistical correlations and limited out-of-distribution generalization, while noting uncertainty about real-world applicability.Source: §6 · p. 7
Interpretability vs. Performance
- Symbolic methods can improve interpretability but face scalability bottlenecks, while data-driven methods cover more tasks but may fail to generalize beyond syntactic patterns.Hybrid architectures are presented as promising but introduce computational overhead that limits practical deployment.Source: §6 · p. 7
Evaluation Rigor
- Multiple-choice benchmarks may conflate reasoning with pattern recognition, and common metrics do not assess consistency or soundness of formal proofs.NeuLR seeks to isolate reasoning from domain knowledge but is described as lacking scope for holistic evaluation; the discussion calls for tests of core principles such as transitivity and contraposition.Source: §6 · p. 7
Future Directions
- Proposed directions include dynamically integrated neural-symbolic systems, stress tests using perturbed statements, multimodal reasoning, and interdisciplinary work on uncertainty.The discussion states that high-stakes deployment remains precarious until LLMs reliably disentangle logic from lexicon.Source: §6 · p. 7
7 Conclusion
The survey concludes that rigorous logical inference across reasoning paradigms remains inconsistent despite strong heuristic reasoning. It highlights robustness, evaluation, and scalable hybrid methods as priorities for further progress.
7 Conclusion
- The survey synthesizes advances and challenges across four reasoning paradigms and reviews data-centric tuning, reinforcement learning, test-time scaling, decoding methods, and neuro-symbolic integration.It highlights FOLIO and LogiQA for evaluation and identifies robust frameworks, scalable hybrid architectures, and cross-domain and multimodal inference as future directions.Source: §7 · p. 7