PaperlayerView exact revision on arXiv

Source-linked AI summary

Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Yuting Zeng, Jiaxing Wang, Xingyu Wang, Hailong Yang, Jing Li

arXiv:2409.17539v2cs.CL

TL;DR

Existing neuro-symbolic methods can lose information when extracting logical expressions, contributing to incorrect intermediate reasoning. Logic-of-Thought expands extracted logical information and adds it to original prompts; experiments report improved performance across prompting methods and five logical reasoning datasets.

  • Problem

    Neuro-symbolic methods can omit information when extracting logical expressions, leading to incorrect intermediate reasoning.

    Source: §1 · p. 2

  • Method

    Logic-of-Thought expands logical expressions extracted from context into natural-language descriptions and adds them to the original prompt.

    Source: §1 · p. 2

  • Results

    LoT boosts performance across five logical reasoning datasets, including a +4.35% improvement for CoT on ReClor.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LoT can be seamlessly integrated with existing prompting methods, which the experiments show it boosts across multiple logical reasoning datasets.

    Source: §1 · p. 3, §6 · p. 9

    Hallucinations in LLMs can still cause Logic Extraction failures, including repeated expressions, omitted logical relationships, or deviations in propositions and expressions.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information descriptions and utilizes them as an additional augmentation to original contexts, thereby ensuring information completeness and enhancing logical reasoning ability. LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%, improves Chain-of-Thought with Self-Consistency's performance on the RuleTaker dataset by +3.52%, and boosts performance of Tree-of-Thoughts on the ProofWriter dataset by +8%.

1 Introduction

LoT augments original prompts with expanded logical information to address information loss in neuro-symbolic reasoning, and it can be combined with existing prompting methods. Experiments report gains across five logical reasoning datasets and multiple prompting methods.

1 Introduction

  • Existing neuro-symbolic methods can omit logical expressions during extraction, losing information and producing incorrect intermediate reasoning.The introduction illustrates this with omitted facts that prevent Prover9 from deriving the correct result.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • LoT extracts propositions and logical expressions, expands them using reasoning laws, translates the deductions into natural language, and adds them to the original context.Preserving the full original context is part of the approach to information completeness.Source: §1 · p. 2
  • +8% improvement for Tree-of-Thoughts on ProofWriter is one reported gain from applying LoT across five logical reasoning datasets.Other reported gains include +4.35% for CoT on ReClor and +3.52% for CoT-SC on RuleTaker.Source: §1 · p. 2
  • LoT is designed to integrate with prompting methods including CoT, Self-Consistency, CoT-SC, and ToT.The paper describes LoT as orthogonal to existing prompting methods.Source: §1 · p. 2

2 Preliminary

The preliminary section introduces propositions, connectives, and logical reasoning laws as the paper’s propositional-logic framework. Its setting is deliberately straightforward, with broader connective and law coverage left for future work.

2 Preliminary

  • Propositions are declarative sentences with a truth value and serve as the fundamental elements of logical expressions.The paper uses uppercase letters for specific propositions and lowercase letters for arbitrary propositions.Source: §2 · p. 3
  • The paper focuses on negation, implication, and conjunction as connectives for forming logical expressions.Conjunction is true only when both propositions are true.Source: §2 · p. 3
  • Logical reasoning laws define deduction relations between expressions; the paper uses double negation, contraposition, and transitivity.The paper leaves incorporating more connectives and laws to future work.Source: §2 · p. 3

3 Methodology

LoT adds natural-language descriptions of logically derived information to the original prompt. Its three phases extract logical expressions, extend them with reasoning laws, and translate the results back into natural language.

3 Methodology

  • LoT extracts propositions and logical relations, expands the resulting expressions, then adds natural-language descriptions of the deductions to the original prompt.The framework uses LLMs for extraction and translation, and Python-implemented logical rules for extension.Source: §3 · p. 3, Figure 2 · p. 4
  • Logic Extraction identifies conditional sentences, maps similar meanings to shared propositional symbols, and encodes opposites with ¬ and conditionals with →.Hints such as “if...then...” and “...causes...” guide analysis of logical connections.Source: §3 · p. 3
  • Logic Extension applies logical deduction rules to extracted expressions; from ¬A →¬B and ¬B →¬C, transitivity and contraposition yield C →A.Source: §3 · p. 4
  • Logic Translation converts extended expressions into natural-language descriptions and combines them with descriptions of the propositions to augment the original prompt.For example, C →A is rendered using the descriptions of C and A and the phrase “if...then...”.Source: §3 · p. 4

4 Experiments

The experiments evaluate LoT across five reasoning datasets, prompting baselines, and neuro-symbolic comparisons. Results generally favor adding LoT, with reported gains in prompting performance and ToT exploration, while extraction errors and reduced logical information weaken some conditions.

4.1 Datasets

  • The evaluation uses ReClor, LogiQA, RuleTaker, ProofWriter, and FOLIO, spanning standardized-test, expert-written, automatically generated, rulebase, and human-annotated tasks.The study compares Direct, SC, CoT, CoT-SC, and ToT prompting with SatLM and LINC as neuro-symbolic baselines.Source: §4.1 · p. 4, §4.2 · pp. 4–5

4.4 Main Results

  • LoT+CoT-SC(5) outperforms all other methods on five datasets with GPT-3.5-turbo-0125 and four with GPT-4-0613; LoT significantly improves baselines in 37 of 40 comparisons.Table 1 presents the main results; on GPT-4 LogiQA, LoT+CoT and LoT+CoT-SC instead trail their baselines by 0.57% and 1.38%, respectively, with extraction deviation identified as the primary factor.Source: Table 1 · p. 5, §4.4 · p. 6

4.3 Experiment Setup

  • Main experiments use zero-shot GPT-3.5-turbo-0125 and GPT-4-0613 across five datasets, while SatLM/LINC and ToT analyses use separate settings and subsets.ToT analysis uses few-shot prompting and 100 randomly selected depth-5 ProofWriter validation examples; SatLM is rerun one-shot and LINC retains its original 8-shot setting.Source: §4.3 · p. 5

4.2 Baselines

  • Across the five datasets and two language models, the experiments compare Direct, CoT, SC, CoT-SC, and their LoT combinations.The prompting baselines include majority-vote variants and tree search; neuro-symbolic comparisons use SatLM and LINC.Source: §4.2 · pp. 4–5, §4.3 · p. 5

4.5 Comparison with Neuro-symbolic Methods

  • On ReClor, LoT outperforms SatLM and improves Direct, CoT, and SC accuracy, but LoT+CoT-SC underperforms CoT-SC; LoT also outperforms LINC on FOLIO.The case study reports that LoT's natural-language rendering can correct an earlier logical-expression error, whereas extraction errors in solver-dependent methods can propagate to final answers.Source: §4.5 · pp. 6–7, Figure 4 · p. 7

4.6 In-depth Analysis of ToT with LoT

  • At ProofWriter depth 5, ToT reaches 70% accuracy, 19% above Direct; LoT+ToT adds a further 8% over ToT.Compared with ToT, LoT+ToT has 2.14% more total states, 2% more full reasoning, and 5.06% more successful states.Source: §4.6 · p. 7, Figure 5 · p. 7

4.7 Ablation Study

  • Removing Contraposition Law or retaining only half the generated descriptions reduces FOLIO accuracy; removing Logic Extension lowers accuracy on LogiQA and RuleTaker, though results remain above Direct.The first ablation tests GPT-4-0613; the second compares Logic Extraction without subsequent extension against LoT.Source: §4.7 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5 Related Work

Prior prompting methods expand, branch, or aggregate reasoning, but may still be unfaithful or insufficiently explore logical information. Neuro-symbolic methods add symbolic solvers, yet their reliance on extracted expressions can limit accuracy.

5.1 Prompting Methods for LLMs Reasoning

  • Prompting approaches include stepwise CoT, majority-vote CoT-SC, subquestion decomposition, process feedback, candidate selection, and branching methods such as ToT and GoT.The paper notes that these methods can exhibit unfaithful reasoning and lack in-depth exploration of logical information.Source: §5.1 · p. 8

5.2 Neuro-symbolic Methods

  • Neuro-symbolic systems formalize language for symbolic solvers, but reliance on extracted expressions can lose information and limit accuracy.Examples include Logic-LM, SatLM, and LINC; LReasoner instead extends context with logical reasoning laws.Source: §5.2 · pp. 8–9

6 Conclusion

The paper presents Logic-of-Thought (LoT), a zero-shot prompt augmentation that derives expanded logical information from context. Experiments report performance gains when LoT is combined with existing prompting techniques across multiple logical reasoning datasets.

6 Conclusion

  • LoT derives expanded logical information from input context as an augmentation to prompts and is compatible with widely used prompting techniques.The paper reports that experiments show performance gains across multiple logical reasoning datasets.Source: §6 · p. 9

7 Limitations

LoT is limited to a small set of connectives and reasoning laws, and its Logic Extraction phase remains vulnerable to LLM hallucinations.

7 Limitations

  • LoT supports a limited set of connectives and logical reasoning laws.The authors plan to add more connectives and laws.Source: §7 · p. 9
  • Adding connectives and laws increases prompt-design complexity in Logic Extraction and Logic Translation and makes deduction in Logic Extension more difficult.Source: §7 · p. 9
  • Hallucinations can cause Logic Extraction to repeat expressions, omit logical relationships, or deviate in its propositions and expressions.These are identified as failure modes that still need to be addressed.Source: §7 · p. 9

A Comparative Study of States in ToT and LoT+ToT

In an illustrative state-exploration comparison, LoT+ToT adds successful states beyond ToT alone and reaches the correct result.

A Comparative Study of States in ToT and LoT+ToT

  • LoT's logical description enabled ToT to generate four additional successful states compared with ToT alone, leading to the correct result.The example description is “If things are rough, then things are round.”Source: A Comparative Study of States in ToT… · p. 12
  • The example attributes the added states to a LoT-generated implication used alongside ToT's exploration.The passage characterizes this as expanding the reasoning space and improving ToT reasoning accuracy.Source: A Comparative Study of States in ToT… · p. 12

B An Error Case of LoT

An error case shows how inaccurate logical extraction can propagate into incorrect descriptions and final outcomes.

B An Error Case of LoT

  • Inaccurate extraction of logical relationships produced erroneous expressions and incorrect final outcomes in the example.The reported expressions include D →¬B, B →C, and C →D.Source: B An Error Case of LoT · p. 12

C An Example of Overlap Capabilities between CoT and LoT

The example illustrates overlap between LoT and CoT: both connect conditional statements to support the same inference, though the example presents their reasoning differently.

C An Example of Overlap Capabilities between CoT and LoT

  • LoT links A →B and B →C into A →C, translated as “If a person reads a book, that person becomes smarter.”This added logical information links the book-reading premise to the conclusion in the context.Source: C An Example of Overlap Capabilities… · p. 12

C.1 An Example of Overlap Capabilities Between CoT and LoT

D The in-depth analysis of LoT

The in-depth analysis quantifies LoT errors by phase, while the prompt sections show how logical content is represented, translated, and used for reasoning.

D The in-depth analysis of LoT

  • When extraction was correct, 5.19% of the total sample was correct with the direct method but incorrect with LoT.The tables report proportions of samples whose results changed due to specific errors, relative to the entire sample set.Source: D The in-depth analysis of LoT · pp. 13–14

E.1 Logic Extraction Prompt in LoT

  • The extraction prompts label propositions with uppercase letters, represent negation separately, and encode causal relations only when the context directly supports them.They ask for propositions and causal expressions as output.Source: §E.1 · p. 14

E.2 Logic Translation Prompt in LoT

  • The translation prompt uses the provided propositions to turn each logical expression into a complete sentence.It requests only the resulting sentences in a paragraph.Source: §E.2 · pp. 14–15

D The in-depth analysis of LoT

E.3 SatLM Prompt

  • The SatLM example encodes treatment choices as predicates, counts, and implications, then asks which treatment pair must follow if O is prescribed.The example includes nine treatments grouped into antibiotics, dietary regimens, and physical therapies.Source: §E.3 · p. 15

E.4 ToT Prompt

  • The ToT prompts generate new propositions from premises, validate deductions with logical reasoning rules, check direct sourcing, and use propositions to judge a hypothesis.The prompts also instruct against introducing unsourced common knowledge.Source: §E.4 · pp. 16–17

Related papers

Loading 2409.17539v2…