Source-linked AI summary
Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoning
Xinglang Zhang, Yunyao Zhang, ZeLiang Chen, Junqing Yu, Wei Yang, Zikai Song
TL;DR
The paper addresses limited understanding of how increasing logical depth affects LLM reasoning. It introduces LoCM and Neuro-Symbolic Curriculum Tuning, finding universal abrupt accuracy collapse beyond critical complexity thresholds while improving robustness within complexity regimes.
Problem
Existing studies observe substantial degradation as logical reasoning becomes harder, but do not clearly characterize reasoning behavior across increasing logical complexity.
Method
The paper combines the model-agnostic LoCM, a complexity-graded reasoning corpus, and neuro-symbolic curriculum optimization aligned with increasing logical complexity.
Results
LLMs exhibit universal logical phase transitions, with accuracy collapsing toward random guessing beyond critical thresholds; fine-tuning and CoT improve accuracy within regimes but do not shift those thresholds.
Takeaways & Limitations
Logical phase transitions appear to persist under standard tuning and prompting, indicating that current models retain a complexity horizon for logical reasoning.
Takeaways & Limitations
The framework re-annotates existing logical reasoning datasets with complete FOL representations, constraining logical-form diversity to the underlying datasets.
Abstract
from arXiv · showhide
Symbolic logical reasoning is a critical yet underexplored capability of large language models (LLMs), providing reliable and verifiable decision-making in high-stakes domains such as mathematical reasoning and legal judgment. In this study, we present a systematic analysis of logical reasoning under controlled increases in logical complexity, and reveal a previously unrecognized phenomenon, which we term Logical Phase Transitions: rather than degrading smoothly, logical reasoning performance remains stable within a regime but collapses abruptly beyond a critical logical depth, mirroring physical phase transitions such as water freezing beyond a critical temperature threshold. Building on this insight, we propose Neuro-Symbolic Curriculum Tuning, a principled framework that adaptively aligns natural language with logical symbols to establish a shared representation, and reshapes training dynamics around phase-transition boundaries to progressively strengthen reasoning at increasing logical depths. Experiments on five benchmarks show that our approach effectively mitigates logical reasoning collapse at high complexity, yielding average accuracy gains of +1.26 in naive prompting and +3.95 in CoT, while improving generalization to unseen logical compositions. Code and data are available at https://github.com/AI4SS/Logical-Phase-Transitions.
1 Introduction
The paper studies how increasing logical complexity shapes LLM reasoning and identifies abrupt Logical Phase Transitions rather than smooth degradation. It introduces LoCM, a logic-enhanced dataset, and Neuro-Symbolic Curriculum Tuning to analyze and mitigate these collapses.
- Symbolic logical reasoning applies explicit logical rules to structured premises and supports tasks including mathematical proof and commonsense inference.
- Current LLMs perform well on simple logical tasks but degrade as reasoning becomes more challenging, while behavior across logical depth remains poorly characterized.
- The Logical Complexity Metric quantifies logical difficulty through symbolic structure and compositional depth, supported by explicit FOL propositions, premises, and reasoning chains.
- Logical Phase Transitions describe stable performance within LoCM regimes followed by abrupt collapse at critical thresholds, with multiple transitions across the LoCM range.
- Neuro-Symbolic Curriculum Tuning aligns neural representations with symbolic structures and progressively increases logical complexity around phase-transition boundaries.
2 Related Work
The related-work context spans linear, aggregative, and symbolic approaches to LLM logical reasoning. The paper’s overview positions complexity measurement, phase-transition discovery, and curriculum tuning as its central workflow.
- Prior LLM logical-reasoning methods include Linear Reasoning, Aggregative Reasoning, and Symbolic Reasoning frameworks.
- The paper organizes its approach around measuring logical complexity, discovering phase transitions, and applying Neuro-Symbolic Curriculum Tuning.
3 Methodology
The methodology quantifies logical difficulty, identifies abrupt reasoning collapses as complexity increases, and uses neuro-symbolic alignment plus adaptive curriculum scheduling to mitigate them.
- Logical complexity measurement: LoCM assigns each reasoning instance a scalar score capturing logical difficulty from premises, operators, nesting depth, and reasoning hops.The metric uses symbolic-complexity weights and a monotonic transformation to stabilize scale and fit empirical correlations.
- Logical complexity measurement: The NSA-LR dataset pairs natural-language and first-order-logic representations for propositions, premises, and reasoning steps.Its explicit symbolic grounding enables LoCM computation and fine-grained logical-complexity analysis.
- Logical phase-transition discovery: Reasoning accuracy remains stable across some LoCM ranges but drops abruptly within one or more critical intervals, then stabilizes beyond the upper bound.The phenomenon occurs across open- and closed-source LLMs and is termed a Logical Phase Transition.
- Neuro-Symbolic Curriculum Tuning: Neuro-Symbolic Curriculum Tuning combines language-logic alignment with training schedules that progressively increase logical complexity around transition regions.The framework is motivated by evidence that direct exposure to high-complexity samples is ineffective.
- Neuro-Symbolic alignment: Hybrid models interpolate independently tuned natural-language and first-order-logic models, then select the best configuration on validation data.Each interpolated model is further fine-tuned on the alignment dataset before selection.
- Complexity-aware curriculum: Complexity-aware optimization monitors performance in identified critical intervals and advances to higher-complexity samples when gains stabilize.The procedure evaluates accuracy after updates and iteratively adjusts sampling rather than following a fixed easy-to-hard schedule.
4 Experiment
The experiments define the evaluation setting and compare methods using dataset-specific tuning variants and the proposed model, with performance summarized relative to the Original model.
- Experimental organization: The experiments cover the experimental setting, evaluation results, and ablation studies.These components are organized in Sections 4.1–4.3.
- Experimental setting: Qwen2.5-7B is used as the tuning base model with parameter-efficient LoRA tuning, while LPT analysis spans multiple Qwen, Gemma, and closed-source model families.Training uses one GPU with fp16 quantization, a learning rate of 1 × 10^-4, one epoch, batch size 1, and gradient accumulation of 48.
- Evaluation comparison: Table 2 compares each method with the Original model, marking improvements in green, degradations in red, and the best result in bold.The comparison includes dataset-specific tuned variants and the proposed θ∗ model.
CoT
The proposed variant θ∗ achieves the strongest overall results across prompting strategies and heterogeneous reasoning datasets, with particularly large gains in high-complexity settings.
- +1.26 average gain: θ∗ is the only variant improving over the Original model under Naive prompting and scores best on every evaluated dataset.
- +3.95 average performance: θ∗ ranks highest under CoT prompting despite the greater reasoning difficulty.
- θ∗ trained on NSA-LR generalizes across ProntoQA, ProofWriter, FOLIO, and ProverQA, unlike single-dataset fine-tuning methods.
- NSCT improves results with Naive, CoT, ToT, DetermLR, and SymbCoT, indicating robustness across inference-time reasoning strategies.
- High-LoCM gains reach +7.4 under CoT, +6.1 under ToT, and +5.4 under DetermLR, exceeding gains in lower-complexity bands.
2. How sensitive is the method to the interpolation coefficient λ?
Ablations show that moderate NL–FOL mixing and complexity-aware curriculum scheduling are important for performance, while logical phase-transition behavior persists across variants and models.
- Q3: Effect of training under different complexity regimes: All variants show sharp accuracy collapse beyond critical LoCM thresholds, so alignment and curriculum improve matched-complexity accuracy without eliminating LPTs.
- Q2: Impact of balancing FOL and NL supervision (λ): At λ = 0, NL-only supervision surpasses baseline, whereas λ = 1 pure FOL causes a marked accuracy drop by removing natural-language cues.
- Q2: Impact of balancing FOL and NL supervision (λ): Accuracy peaks around λ = 0.3, where NL semantic grounding and FOL operator structure are complementary.
- Q3: Effect of training under different complexity regimes: Single-regime training provides limited gains: easy-only reaches 61.3, medium-only 58.8, and hard-only 61.9 versus the 60.6 mixed baseline.
- LPTs in LLMs: Across models, larger systems degrade later and remain more robust, while the 1B model collapses once LoCM exceeds 8.
- Structured prompting on LPTs: CoT and fine-tuning raise accuracy within complexity bins, but neither extends the complexity horizon or changes transition onset.
5 Conclusion
The paper introduces LoCM and a neuro-symbolic curriculum framework to characterize and mitigate abrupt logical-reasoning collapse. Experiments report improved robustness across benchmarks and prompting settings.
- LoCM is a model-agnostic measure that reveals universal logical phase transitions as abrupt accuracy collapse beyond critical complexity thresholds.
- The proposed curriculum optimization aligns training with complexity progression through a complexity-graded reasoning corpus.
- The approach consistently improves robustness across diverse benchmarks and prompting settings, mitigating post-transition failure in high-complexity reasoning.
Limitations
The paper identifies limitations in how existing logical-reasoning methods and benchmarks characterize complexity, and acknowledges scope and calibration constraints in its framework.
- The framework relies on re-annotated existing datasets and targets first-order-logic-expressible reasoning, excluding direct coverage of commonsense, world-modeling, and probabilistic inference.
- Logical phase-transition boundaries require model-specific empirical calibration even though the phenomenon is observed across open- and closed-source LLMs.
- Current benchmarks provide limited support for studying how reasoning difficulty scales because they often lack complete symbolic representations and verified reasoning chains under one schema.
- Existing methods often improve prompting, reasoning-time scaffolds, or external modules without characterizing robustness across a fine-grained logical-complexity continuum.
A.3 Logical Reasoning Capacity Analysis
The analysis positions existing capacity studies and benchmarks as insufficient for fine-grained complexity analysis, then introduces NSA-LR to align natural language with explicit, verifiable first-order logic.
- Existing analyses often rely on coarse difficulty indicators, limiting insight into how reasoning performance evolves across nuanced complexity levels.
- The evaluation uses synthetic and natural-language benchmarks spanning controlled deduction, expert-curated FOL reasoning, proof supervision, and theorem-proving chains.
- NSA-LR extends ProverGen by translating propositions, premises, and intermediate reasoning steps into explicit first-order logic representations.
- NSA-LR combines scalability, linguistic richness, and fully verifiable symbolic reasoning chains for logic-consistency analysis.
B.3 Example of NSA-LR Data Instance
The appendix directs readers to a high-complexity NSA-LR example and documents supplementary curves, evaluation procedures, quantization, and fine-tuning settings.
- A high-complexity instance at approximately LoCM 7.25 demonstrates alignment between natural language and first-order logic across a multi-step deduction chain.
- Supplementary Figure 6 presents additional-model phase-transition curves with shaded transition intervals and a 1/3 random-guess baseline.
- Evaluation prompts models to return JSON containing reasoning traces and final answers, with post-processing used to recover malformed predictions.
- Open-source models and the fine-tuned checkpoint are evaluated under llama.cpp Q4 quantization for fair comparison.
- Qwen2.5-7B-Instruct is fine-tuned with parameter-efficient LoRA for one epoch on a single Tesla V100S-PCIE-32GB GPU.
C.2 Full Results
Full results show that complexity-related performance collapse is not reliably solved by scale, single-regime training, or standard interventions, while LoCM captures structural difficulty beyond premise count.
- Accuracy systematically degrades from Bin 1 to Bin 9 as task complexity increases across model families.
- 70.2% overall accuracy for Qwen2.5-14B exceeds 68.3% for its 32B counterpart, with the 14B model outperforming it in Bins 5–8.
- Scaling produces consistent gains in Bins 1–3 but diminishing returns or instability as models approach the complexity threshold.
- Training only on high-complexity instances improves that region but substantially degrades low- and medium-complexity performance through catastrophic forgetting.
- FOLIO is an exception to otherwise consistent NSCT gains under CoT, likely reflecting its greater demands for natural-language interpretation and commonsense grounding.
- Supervised fine-tuning and advanced prompting increase accuracy within regimes without substantially delaying the phase-transition interval.
- LoCM captures structural variation beyond premise count, and accuracy decreases with LoCM even when premise-count intervals are held approximately constant.
- Accuracy falls from 94.66% in the shortest completion-length bin to below 40% in the tail, indicating that longer generations more often mark difficult regimes.