PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions

Leonardo Ranaldi, Marco Valentino, Andrè Freitas

arXiv:2502.12616v2cs.CL

TL;DR

CoT reasoning can be brittle because its explanations are vulnerable to content bias, while fully symbolic methods require costly and inflexible complete formalisation. QuaSAR combines natural language with selectively formalised variables and predicates through quasi-symbolic reasoning, improving performance and robustness across natural-language and symbolic tasks, including an average 8% accuracy increase over CoT on GPT-4o.

  • Problem

    CoT explanations are susceptible to content biases, while fully symbolic approaches require complete natural-language formalisation that reduces efficiency and flexibility.

  • Method

    QuaSAR guides LLMs through quasi-symbolic reasoning by formalising relevant predicates, variables, and constants while retaining natural-language content.

  • Results

    QuaSAR outperforms CoT across most natural-language and symbolic tasks, with an average 8% accuracy increase over CoT on GPT-4o.

  • Takeaways & Limitations

    Quasi-symbolic abstractions improve reasoning performance, robustness, and consistency on challenging adversarial variations, including shuffled answer options and lexical changes.

  • Takeaways & Limitations

    Future work is needed to extend QuaSAR to non-English languages and retrieval-augmented settings.

Abstract

from arXiv · show

Chain-of-Though (CoT) represents a common strategy for reasoning in Large Language Models (LLMs) by decomposing complex tasks into intermediate inference steps. However, explanations generated via CoT are susceptible to content biases that negatively affect their robustness and faithfulness. To mitigate existing limitations, recent work has proposed using logical formalisms coupled with external symbolic solvers. However, fully symbolic approaches possess the bottleneck of requiring a complete translation from natural language to formal languages, a process that affects efficiency and flexibility. To achieve a trade-off, this paper investigates methods to disentangle content from logical reasoning without a complete formalisation. In particular, we present QuaSAR (for Quasi-Symbolic Abstract Reasoning), a variation of CoT that guides LLMs to operate at a higher level of abstraction via quasi-symbolic explanations. Our framework leverages the capability of LLMs to formalise only relevant variables and predicates, enabling the coexistence of symbolic elements with natural language. We show the impact of QuaSAR for in-context learning and for constructing demonstrations to improve the reasoning capabilities of smaller models. Our experiments show that quasi-symbolic abstractions can improve CoT-based methods by up to 8% accuracy, enhancing robustness and consistency on challenging adversarial variations on both natural language (i.e. MMLU-Redux) and symbolic reasoning tasks (i.e., GSM-Symbolic).

1 Introduction

CoT improves LLM reasoning by decomposing problems, but its explanations can remain vulnerable to content bias, limiting robustness and faithfulness. QuaSAR addresses this trade-off with quasi-symbolic reasoning that combines selective formalisation with natural language, improving performance across reasoning settings.

  • CoT breaks complex problems into intermediate reasoning steps, but its explanations can contain content biases that harm robustness and faithfulness.
  • QuaSAR guides LLMs to disentangle content from logical reasoning by formalising relevant predicates and variables without completely translating natural language into formal language.
  • QuaSAR uses abstraction, formalisation, explanation, and answering to produce quasi-symbolic reasoning chains, supporting both in-context learning and demonstrations for smaller models.
  • 8% average accuracy increase over CoT was obtained on GPT-4o, with further gains of 6.8% over CoMAT and 8.2% over Faithful CoT.
  • QuaSAR achieves state-of-the-art performance across diverse tasks and improves robustness and consistency on adversarial natural-language and symbolic reasoning variations.

2 QuaSAR: Quasi-Symbolic Abstract Reasoning

QuaSAR integrates symbolic abstractions into natural-language reasoning through a structured sequence that identifies relevant information, formalises it semi-symbolically, explains transformations, and produces an answer. The framework supports in-context prompting and synthetic demonstrations, with symbolic structure intended to improve transparency and verification.

  • Quasi-symbolic abstraction replaces concrete entities and predicates with abstract symbols, helping explanations generalise across problems while retaining natural-language content.
  • QuaSAR extends the conventional (Q, R, A) reasoning structure to (Q, S, R, A), where S contains four instructions for symbolically elicited transformations.
  • Structured decomposition makes each reasoning step more transparent and facilitates systematic verification.
  • The four stages are abstraction, formalisation, explanation, and answering: the model identifies predicates and variables, creates a mixed symbolic-language form, reasons through transformations, and generates the solution.
  • 2.1.2 Formalisation: Formalisation translates natural language into a semi-structured symbolic form that represents problem facts and aims to reduce ambiguity and content effects.
  • QuaSAR operates both as an in-context learning strategy and as a synthetic annotation method for supervised learning.

3 Experiments

QuaSAR is evaluated across mathematical, commonsense, reasoning, and natural-language-understanding tasks using multiple LLMs, exact-match assessment, and comparisons with prompting and tuning baselines.

  • Tasks and datasets: The evaluation covers symbolic tasks including GSM8K, SVAMP, AQuA, MMLU-Redux, and Olympiad Bench across varied mathematical topics and question formats.
  • Tasks and datasets: Natural-language evaluation uses GPQA for difficult open-ended questions and DROP for discrete reasoning over passages.
  • Models: Experiments use GPT-4o, Qwen2, and Llama3, with Llama3 selected partly because smaller releases support additional tuning.
  • Baselines: Comparisons include zero-shot prompting, CoT, Faithful CoT, FLAIRE, and CoMAT under the same greedy decoding strategy with temperature 0.
  • Evaluation setup: QuaSAR is tested both as an in-context learning strategy and for generating tuning demonstrations, using a common prompt and configured instruction-tuning setups.

4 Results & Discussions

QuaSAR generally improves reasoning performance across tasks, with especially strong benefits from structured demonstrations and robustness to adversarial or superficial task variations.

  • Overall results: QuaSAR outperforms CoT in most advanced mathematical, reading-comprehension, and logical-reasoning tasks.
  • In-context learning: 19.1% for GPT-4o, 11.8% for Llama-3-70B, and 17.2% for Qwen2-72B are the reported overall improvements when QuaSAR is used for in-context learning.
  • Smaller models: 5.2% for Llama-3-8B, 13.4% for Llama-3-1B, 10.5% for Qwen2-7B, and 8.3% for Qwen2-1.5B are reported improvements over baseline, although smaller models underperform CoT.
  • Demonstration tuning: QuaSAR-generated demonstrations consistently improve tuned Llama and Qwen2 models over baseline SFT and SFT using CoT demonstrations.
  • Ablation analysis: Removing formalisation or explanation causes a stable drop of more than 3.5 points, while removing abstraction lowers final accuracy by 1.8 on average.
  • Robustness: QuaSAR maintains similar performance with considerably less variation than CoT when answer choices are reordered or mathematical tasks receive superficial perturbations.
  • Training efficiency: QuaSAR demonstrations outperform CoT demonstrations consistently and can surpass larger models with 50% of the training examples for Llama-3-8B and 25% for Qwen2-7B.
  • Additional analyses: Outputs generated via QuaSAR are reported to be easier to correct because their structured rationales simplify correction.

5 Related Work

Related work spans rule-based, neural, hybrid, and symbolic reasoning systems, alongside CoT methods that improve mathematical reasoning but retain limitations in scope or verification.

  • Logical reasoning: Logical-reasoning systems include rule-based methods, neural paradigms, and hybrid frameworks that map natural-language inputs into symbolic syntax for external reasoning tools.
  • Symbolic reasoning: Symbolic reasoning combines natural and symbolic languages to decompose queries into sub-problems solved by symbolic programs and deterministic solvers.
  • Symbolic reasoning: Recent symbolic approaches reduce dependence on symbolic programs but primarily address logical reasoning and rely on verifiers, limiting applicability to complex mathematical tasks.
  • Chain-of-Thought: CoT research has improved mathematical symbolic reasoning through advances in problem understanding, structured formats, and supervision models.

6 Future Works

Future work proposes extending QuaSAR to non-English languages and studying its use with retrieval-augmented LLMs.

  • Multilingual extension: The authors plan to extend QuaSAR to non-English languages, including through a multilingual extension of GSM-Symbolic.
  • Retrieval augmentation: They also plan to investigate QuaSAR in retrieval-augmented LLM scenarios involving knowledge conflicts in retrieved documents.

7 Conclusion

QuaSAR combines natural-language and symbolic abstractions into systematic reasoning paths for complex tasks. The conclusion reports improved performance, robustness, consistency, faithfulness, verifiability, and reliability across diverse benchmarks.

  • 7 Conclusion: QuaSAR uses quasi-symbolic step-by-step reasoning to handle complex tasks requiring both natural-language and symbolic abstractions.The framework is used for in-context learning and for constructing demonstrations for smaller models.
  • 7 Conclusion: QuaSAR improves smaller-model performance when used to construct demonstrations.The paper also evaluates QuaSAR as an in-context learning strategy across diverse benchmarks.
  • 7 Conclusion: QuaSAR delivers transparent and consistent reasoning trajectories while outperforming conventional Chain-of-Thought methods.The conclusion reports state-of-the-art performance and improved robustness across tasks of varying complexity.
  • 7 Conclusion: Evaluation combines exact-match checking with GPT-4o judging when more accurate response assessment is required.The evaluation procedure uses exact match especially for multiple-choice question answering and GPT-4o as a judge in some experiments.

D Data Composition

The evaluation data are divided into training and testing sets to support both in-context prompting and instruction generation. Dataset instances and constructed demonstrations are documented separately.

  • D Data Composition: Datasets were split into training and testing sets because the evaluated tasks often lack dedicated evaluation and training partitions.Existing splits were used where available; otherwise, the authors produced and documented new splits.
  • D Data Composition: Table 7 documents the instances assigned to training and testing for each dataset.The table includes author-produced splits where pre-existing split data were unavailable.
  • D Data Composition: QuaSAR demonstrations were generated through annotation, filtered with Appendix B heuristics, and balanced across tasks.These demonstrations support the instruction-generation configuration.

E Additional Task

The additional-task evaluation reports performance for GPT-4o and Llama-3-8B on MATH, XCOPA, and MGSM. The tables provide a cross-model view over mathematical and language reasoning tasks.

  • E Additional Task: GPT-4o performance is reported on MATH, XCOPA, and MGSM.These tasks cover mathematical and language-oriented reasoning evaluations.
  • E Additional Task: The additional-task tables organize results across two models evaluated on the same three benchmarks.The shared benchmark set comprises MATH, XCOPA, and MGSM.

F Training Setup

The training setup compares QuaSAR, Chain-of-Thought, and standard-output demonstrations for smaller models. It also specifies model choices, fine-tuning conditions, and scaling experiments.

  • F Training Setup: Selected smaller models are trained with QuaSAR, Chain-of-Thought, or standard-output demonstrations for comparison.The Llama-3 models are fine-tuned for 3 epochs with batch size 32, learning rate 3e-5, and weight decay 0.001.
  • F Training Setup: The listed model versions and configurations are tied to repositories and appendix settings.Table 10 records the versions used in the experiments.
  • F Training Setup: Performance is assessed by scaling the number of QuaSAR tuning instructions.Table 11 replicates the experimental settings while changing the amount of tuning data.
  • F Training Setup: The study evaluates GPT-4o, Llama-3 variants, and Qwen2 variants, with GPT-4 accessed through an API.The model selection reflects reproducibility and the cost associated with non-open-source models.

J Error Propagation

The paper analyzes how errors accumulate across reasoning stages and contrasts CoT with QuaSAR-style abstractions on representative GSM-Symbolic problems. The examples show QuaSAR reaching the intended answers where CoT produces incorrect results, while the pipeline’s total failure rate is 36%.

  • Error analysis: 36% is the total failure rate across the full reasoning pipeline in an ablation subset of GSM-Symbolic using GPT-4o.Each step’s error rate was independently assessed through manual verification.
  • GSM-Symbolic examples: CoT produces 135 minutes for the fog problem by disregarding the specified wind-speed change, whereas the QuaSAR reasoning accounts for both phases.The CoT passage explicitly describes its result as a wrong final answer.
  • GSM-Symbolic examples: QuaSAR’s fog solution formalizes relevant variables and predicates, splits the calculation at the wind-speed change, and obtains the target answer of 210 minutes.The target distance is 18 miles; 8 miles are covered in the first 60 minutes and the remaining 10 miles require 150 minutes at the halved speed.
  • GSM-Symbolic examples: QuaSAR’s probability example reaches 50 percentage points, while the corresponding CoT passage reports approximately 30 percentage points.The QuaSAR calculation uses predicates for event probabilities and assumes independent die rolls.
  • Error detection: The self-consistency evaluation reports that CoT and QuaSAR can produce the same final answer, but QuaSAR explanations make errors easier to detect and correct.The comparison is presented as an example of logical steps shown for solving a problem.
Loading 2502.12616v2…