PaperlayerView exact revision on arXiv

Source-linked AI summary

DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy

Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji-Rong Wen, Rui Yan

arXiv:2310.18659v2cs.AIcs.CL

TL;DR

LLMs still lack human-like advanced reasoning skills for analyzing known conditions and drawing conclusions. DetermLR organizes reasoning from indeterminacy to determinacy, prioritizes premises, and retains reasoning history; it achieves the highest accuracy across multiple logical reasoning benchmarks while using fewer reasoning steps.

  • Problem

    LLMs still lack human-like advanced reasoning skills to rationally analyze known conditions and draw conclusions.

    Source: §1 · p. 1

  • Method

    DetermLR classifies premises as determinate or indeterminate, prioritizes premises by their relationship to the target, and stores historical reasoning details for later steps.

    Source: §1 · p. 2

  • Results

    DetermLR achieves the highest accuracy across multiple logical reasoning benchmarks while requiring fewer reasoning steps; it reaches 54.19 accuracy with GPT-4 on LogiQA.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Across the evaluated logical reasoning benchmarks, DetermLR combines higher accuracy with fewer reasoning steps than previous multi-step reasoning methods.

    Source: §1 · p. 2, §5 · p. 8

    LLMs still struggle to extract implicit conditions and parse lengthy descriptions and intricate nested if-then structures.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have revolutionized the landscape of reasoning tasks. To enhance the capabilities of LLMs to emulate human reasoning, prior studies have focused on modeling reasoning steps using various thought structures like chains, trees, or graphs. However, LLM-based reasoning still encounters the following challenges: (1) Limited adaptability of preset structures to diverse tasks; (2) Insufficient precision in exploiting known conditions to derive new ones; and (3) Inadequate consideration of historical reasoning experiences for subsequent reasoning steps. To this end, we propose DetermLR, a novel perspective that rethinks the reasoning process as an evolution from indeterminacy to determinacy. First, we categorize known conditions into two types: determinate and indeterminate premises This provides an oveall direction for the reasoning process and guides LLMs in converting indeterminate data into progressively determinate insights. Subsequently, we leverage quantitative measurements to prioritize more relevant premises to explore new insights. Furthermore, we automate the storage and extraction of available premises and reasoning paths with reasoning memory, preserving historical reasoning details for subsequent reasoning steps. Comprehensive experimental results demonstrate that DetermLR surpasses all baselines on various logical reasoning benchmarks: LogiQA, ProofWriter, FOLIO, PrOntoQA, and LogicalDeduction. Compared to previous multi-step reasoning methods, DetermLR achieves higher accuracy with fewer reasoning steps, highlighting its superior efficiency and effectiveness in solving logical reasoning tasks.

1 Introduction

DetermLR frames reasoning as a progression from indeterminate premises toward determinate conclusions, addressing limits in preset structures, premise use, and reasoning memory. It combines premise prioritization and memory, and reports higher accuracy with fewer steps across logical reasoning benchmarks.

  • LLM-based reasoning faces preset structures that adapt poorly to diverse tasks, random premise combinations, and inadequate retention of reasoning history.Source: §1 · pp. 1–2
  • DetermLR models reasoning as accumulating determinate information: simple determinate statements contribute to conclusions, while indeterminate premises can encode complex rules among propositions.Source: §1 · p. 2
  • Quantitative measurements prioritize premises relevant to the conclusion and screen supplementary premises likely to interact with the primary premise.Source: §1 · p. 2
  • Reasoning memory stores and extracts premises and reasoning paths, preserving historical details to update reasoning states and inform later steps.Source: §1 · p. 2
  • Across LogiQA, ProofWriter, FOLIO, ProntoQA, and LogicalDeduction, DetermLR achieves the best accuracy and requires fewer reasoning steps than previous multi-step methods.Source: §1 · p. 2

2 Related Work

Prior work covers conventional logical-reasoning approaches and benchmarks, alongside LLM methods that elicit or structure intermediate reasoning. The section also describes DetermLR’s focus on relevant premises and historical reasoning experience.

2.1 Conventional Logical Reasoning

  • The section surveys machine reading comprehension, adversarial pre-training, and contrastive learning, and identifies LogiQA, FOLIO, and ProofWriter among established reasoning benchmarks.Source: §2.1 · pp. 2–3

2.2 LLM-based Logical Reasoning

  • LLM reasoning methods include CoT and least-to-most prompting, expanded with tree- and graph-based structures such as ToT and GoT.Source: §2.2 · p. 3
  • The section presents DetermLR as prioritizing relevant premises and using key historical reasoning experiences for iterative reasoning.Source: §2.2 · p. 3

3 DetermLR

DetermLR frames logical reasoning as iterative derivation from input premises toward a target, moving from indeterminate information to determinate premises. It identifies premise types, prioritizes premise combinations for exploration, and uses memory to carry reasoning experience across iterations.

3.1 Problem Formulation

  • The formulation treats reasoning as deriving intermediate propositions from known premises and deduction rules until reaching the target conclusion.The LLM is instructed to understand the problem and iteratively provide new insights.Source: §3.1 · p. 3

3.2 Premise Identification

  • Premises are classified by their description and connection to the target: determinate premises directly support it, while indeterminate ones may need combination with others.Indeterminate premises can include disjunctions or hypotheses; the Gary example classifies statements mentioning Gary as determinate.Source: §3.2 · pp. 3–4

3.3 Premise Prioritization and Exploration

  • Two-stage scoring selects the determinate premise most relevant to the target, then identifies supplementary premises by semantic similarity and adherence to deduction rules for exploration.The LLM combines the primary premise with selected supplements to generate a new proposition.Source: §3.3 · p. 4
  • A new proposition is retained only if it is logically valid, contributes a useful determinate premise, and adds information beyond the original premises.The checks filter out logically valid but unhelpful propositions and paraphrases.Source: §3.3 · pp. 4–5

3.4 Reasoning Memorization

  • Memory stores successful propositions with their reasoning paths and also retains paths from failed attempts as reasoning experience.The memory is initialized with the input premises before reasoning begins.Source: §3.4 · p. 5
  • In later iterations, extracted history guides premise prioritization, helps avoid repeated mistakes, and supports checking whether current determinate information suffices for the target.The module supports retrospective updates to reasoning states and prospective use of prior experience.Source: §3.4 · p. 5

4 Experiments

Across the benchmark comparisons, DetermLR combines stronger reasoning accuracy with fewer steps. Ablations and further analyses examine its components, reasoning structure, and efficiency trade-offs.

4.1 Experimental Setup

  • Experiments use five benchmarks, including 179 curated LogiQA questions, 600 depth-5 ProofWriter cases, 204 FOLIO validation examples, and 300 LogicalDeduction test examples.The ProofWriter subset uses an open-world assumption; the FOLIO evaluation follows the official split.Source: §4.1 · pp. 5–6

4.2 Main Results

  • DeterminLR achieves superior accuracy with fewer reasoning steps than prior multi-step methods, reaching 54.19 accuracy with GPT-4 and averaging 11.74 steps on LogiQA.The comparison covers LogiQA, ProofWriter, FOLIO, and LogicalDeduction.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • Removing premise identification, prioritization, or memory reduces performance; without memory, accuracy decreases by at least 7.84.The variants remove identification, replace priorities with random sampling, or remove iterative reasoning memory.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 Further Analysis

  • Case A’s 18 premises can be reasoned through in a two-step chain, while Case B’s four complex premises require repeated use, so context size alone can misstate difficulty.The examples argue against presetting reasoning structure before reviewing the problem’s reasoning outcomes.Source: §4.4 · p. 7, Figure 2 · p. 7
  • Increasing the number of generated determinate premises can improve accuracy, but gains become limited as reasoning iterations rise; the experiments set n to 4.The selected value is described as near the inflection point for balancing effectiveness and efficiency.Source: §4.4 · p. 7, Figure 3 · p. 8
  • DetermLR uses the fewest steps per useful proposition across datasets; on LogiQA, CR takes 4.25 steps and DetermLR 2.63, consistent with its greater analytical difficulty.Two volunteers achieved LogiQA accuracy rates of 73 and 59, respectively.Source: §4.4 · p. 8, Table 3 · p. 8
  • Although each DetermLR step takes slightly longer than a ToT or CR step, fewer steps make its overall inference time per case better.The comparison uses the same device for all experiments in the analysis.Source: §4.4 · p. 8, Table 4 · p. 8
  • Current LLM-based reasoning still misses implicit conditions, boundary-condition deductions, and flexible use of logical rules.For example, the reported errors include failing to assert B from A implies B and not-A implies B.Source: §4.4 · p. 8

5 Conclusion

DetermLR frames reasoning as a progression from indeterminate to determinate premises, prioritizes premises, and retains reasoning history. It reports higher accuracy with fewer steps across multiple benchmarks, with larger advances on LogiQA.

5 Conclusion

  • DetermLR models logical reasoning as an evolution from indeterminacy to determinacy.It categorizes premises into determinate and indeterminate types.Source: §5 · p. 8
  • Quantitative premise prioritization and a reasoning memorization module guide premise exploration and preserve historical details across iterations.Source: §5 · p. 8
  • DetermLR achieves the highest accuracy on multiple logical reasoning benchmarks while requiring fewer reasoning steps, with more pronounced advancements on LogiQA.Source: §5 · p. 8

6 Limitation

Despite outperforming baselines across tasks, DetermLR retains challenges in extracting implicit conditions from intricate contexts, including lengthy descriptions and nested conditional structures.

6 Limitation

  • Despite superior performance over baselines, DetermLR still faces challenges in extracting implicit conditions for intricate reasoning tasks.Source: §6 · p. 9
  • LLMs struggle to parse nuanced or implicit information in verbose descriptions and exceedingly lengthy condition descriptions.Source: §6 · p. 9
  • Nested if-then structures linking multiple subjects increase the risk of reasoning errors, making better handling of implicit and complex conditions a stated future research direction.Source: §6 · p. 9

A More Examples on Datasets

The examples section presents a LogicalDeduction ordering problem about seven golfers and asks which golfer finished third.

A More Examples on Datasets

  • The LogicalDeduction example describes seven golfers whose finishing order is constrained by statements about relative positions.Source: §A.1 · p. 12
  • The stated conditions include Ada finishing third-to-last, Mya second-to-last, and relative ordering constraints among the other golfers.Source: §A.1 · p. 12
  • The example asks which of seven answer options identifies the golfer who finished third.Source: §A.1 · p. 12

Conclusion:

The LogicalDeduction example shows DetermLR deriving intermediate ordering premises before concluding Ana finished third. The block also gives a LogiQA example about inferring an individual’s gender and education level.

Conclusion:

  • Additional derived premises state that Mya did not finish first, Ada did not finish first or second, and Amy did not finish third-to-last.Source: Conclusion · p. 13
  • Other derived premises place Joe above Ana, Amy above Eli, Amy second, Joe first, and Ada and Eli outside the top three.Source: Conclusion · p. 13
  • Across six rounds, DetermLR derives premises including Joe above Ana, Amy above Eli, and constraints on Amy, Ada, and Eli’s finishing positions.Source: Conclusion · p. 13
  • DetermLR’s final derivation concludes that Ana finished third, selecting option A.Its reasoning places Joe first, Amy second, Ana third, Eli fourth, and Mel seventh.Source: Conclusion · p. 13

A.2 LogiQA

  • The LogiQA example gives constraints on seven people’s education levels and genders, then asks which listed conclusion follows.Source: §A.2 · p. 14

Conclusion:

The conclusion illustrates DetermLR deriving increasingly specific premises across reasoning rounds, culminating in identifying D as a female PhD holder. A ProofWriter example separately shows a chain-of-thought response judging a hypothesis unknown.

Conclusion:

  • Across rounds 1–4, the example derives bachelor’s degrees for A, B, and C, identifies E, F, and G as male, identifies B and C as female, and infers that D and E hold PhDs.Each new determinate premise is derived from the premises listed for that round.Source: Conclusion · p. 15
  • Rounds 5–7 derive that C is a female undergraduate, E is a male PhD holder, and D is a female PhD holder.The example uses previously derived premises in subsequent rounds.Source: Conclusion · p. 15
  • The reasoning concludes that D is a female PhD holder, using intermediate deductions that include E as a male PhD holder and D as a female PhD holder.The final propositions also identify A, B, and C as undergraduates with bachelor’s degrees, and C as a female undergraduate.Source: Conclusion · pp. 14–15

Conclusion:

This section presents examples of DetermLR deriving determinate outcomes where GPT-4 reasoning leaves hypotheses unknown or uncertain, alongside prompts describing task-specific premise handling. For LogicalDeduction, the prompts guide identifying and prioritizing premises, deriving and validating them, and producing step-by-step answers grounded in premises and propositions while recalling prior reasoning.

A.4 FOLIO

  • For the 1984 FOLIO hypothesis, GPT-4 says the premises cannot establish whether it is not a streaming service, while DetermLR infers it is true.DetermLR combines the conditional that 1984 is a hardcover book if it is streaming with deductions that streaming services are digital and not analog, producing a contradiction.Source: Conclusion · pp. 18–19

B Prompt Templates

  • ProofWriter and LogicalDeduction provide explicit premises, whereas LogiQA and LogicalDeduction require parsing context to extract premises and may need premise transformation.The prompt templates use ProofWriter and LogicalDeduction as representative tasks; further prompt-design details are referenced in the paper’s GitHub repository.Source: B Prompt Templates · p. 20

B.2 LogicalDeduction

  • LogicalDeduction prompts classify premises as determinate using their relation to the hypothesis, then prioritize variables by mention counts, prerequisites, and prior failed inferences.The workflow screens premises related to the selected variable and avoids choosing a variable that failed in earlier inference history.Source: §B.2 · pp. 25–26
  • The LogicalDeduction prompts derive propositions with logical reasoning rules, validate deductions and boundary conditions, and extract or transform premises using boundary conditions.Exploration prompts require propositions useful for resolving the hypothesis; transformation uses one-to-one relationships to infer new premises without duplicating existing ones.Source: §B.2 · pp. 27–31
  • The extraction prompts request the discussion topic, premise, and boundary conditions from the context.They place the extracted fields in a structured response format.Source: §B.2 · p. 31
  • The final-conclusion prompt directs reasoning from premises, propositions, and boundary conditions, avoiding common-sense information and continuing until a single answer is reached if needed.It instructs the model to answer directly when a proposition suffices, and to recall the reasoning history and question options before selecting a judgment.Source: §B.2 · p. 32

B.1 ProofWriter

The prompts implement DetermLR through premise identification, prioritization, exploration, logical validation, and final conclusion. They define a determinate premise, score premise relevance, and carry reasoning history into later steps.

B.1 ProofWriter

  • The prompt design divides reasoning into premise identification, premise prioritization, premise exploration, logical validation, and final conclusion.The final-conclusion prompt also recalls reasoning history and the question and options.Source: §B.1 · p. 20
  • A determinate premise is defined as a premise sharing a noun or adjective with the hypothesis and not having an if/then structure.The identification prompt asks the model to classify premises using this definition rather than unsourced common sense.Source: §B.1 · p. 20
  • Prioritization selects the highest-scoring determinate premise sharing the hypothesis’s subject, then retains related premises scoring above 0.25.Relevance scores add 0.25 for a shared noun, 0.3 for a shared adjective, and 0.25 when one sentence is a hypothetical premise of another, with a cap of 1.Source: §B.1 · p. 21
  • Premise exploration asks for a new proposition derived from at least two premises using specified logical reasoning rules.The prompt requires propositions to be logically correct, non-duplicative, and useful for determining whether the hypothesis is true, false, or unknown.Source: §B.1 · p. 22
  • The logical-validity prompt asks the model to judge whether a deduction from premises to a proposition is valid or invalid.It instructs the model to use the stated Logical Reasoning Rules.Source: §B.1 · p. 23
  • The final-conclusion prompt combines premises, boundary conditions, propositions, and recalled reasoning history to answer the question.The prompt says to continue reasoning with prior propositions when they do not directly determine the answer.Source: §B.1 · p. 24

C Scalability Analysis

On sampled ProofWriter training cases, DetermLR consistently outperforms cumulative reasoning in accuracy and time-efficiency as dataset size increases.

C Scalability Analysis

  • DetermLR consistently outperforms CR in accuracy and time-efficiency as data scales across 1,600 sampled ProofWriter training cases.The comparison concerns performance across varying dataset sizes; the supplied table caption gives no cell values.Source: C Scalability Analysis · p. 33, Table 5 · p. 33

D Generalization in Math Reasoning

The paper evaluates whether DetermLR can extend to math reasoning by constructing a multi-variable-equation dataset. Its prioritization strategy guides equation combination toward progressively more determinate results, and the reported comparison favors DetermLR over CR.

D Generalization in Math Reasoning

  • The math-reasoning evaluation uses a constructed dataset of multi-variable equations to test DetermLR on a task beyond logical reasoning.The authors state that the method can adapt to tasks with available conditions by classifying premises as determinate or indeterminate.Source: D Generalization in Math Reasoning · p. 33
  • The method treats equations relating multiple variables as indeterminate and exact variable values as determinate, then combines equations to eliminate variables and solve the system.The intended progression is from relationships among variables toward exact values for each variable.Source: D Generalization in Math Reasoning · p. 33
  • DetermLR’s equation prioritization is described as producing a streamlined, human-like elimination trajectory, while CR’s random equation selection often yields erroneous or incomplete solutions.The account attributes the direction of the process to prioritization and reducing variables in newly generated equations.Source: D Generalization in Math Reasoning · p. 33
  • In the worked rounds, generated equations include y = 4 and z = 10 as the process moves toward exact variable values.These outputs are labeled as new equations in the NO:5 and NO:6 rounds.Source: D Generalization in Math Reasoning · p. 35
Loading 2310.18659v2…