PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning

Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, Ji-Rong Wen

arXiv:2306.02408v1cs.CL

TL;DR

Existing math-reasoning datasets may not fully evaluate tool manipulation or intermediate reasoning. The paper constructs CARP and proposes DELI, which iteratively checks and refines solution steps; DELI mostly outperforms baselines across CARP and six other datasets.

  • Problem

    Many existing math-reasoning datasets may require few tool invocations or lack formal intermediate-step annotations, limiting systematic evaluation and analysis of tool manipulation and reasoning.

    Source: §1 · p. 2

  • Method

    DELI initializes a step-by-step solution from retrieved exemplars, then iteratively checks and refines it through tool-manipulation and natural-language-reasoning procedures.

    Source: §1 · p. 2

  • Results

    Across CARP and six other computation-intensive math datasets, DELI outperforms baselines and boosts CoT methods; on CARP, it improves accuracy 9.35% over the best baseline.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    CARP’s formulated intermediate-step annotations support systematic evaluation of computation-intensive reasoning and analysis of LLM errors.

    Source: §1 · p. 2

Abstract

from arXiv · show

Chain-of-thought prompting~(CoT) and tool augmentation have been validated in recent work as effective practices for improving large language models~(LLMs) to perform step-by-step reasoning on complex math-related tasks. However, most existing math reasoning datasets may be not able to fully evaluate and analyze the ability of LLMs in manipulating tools and performing reasoning, as they may only require very few invocations of tools or miss annotations for evaluating intermediate reasoning steps. To address the issue, we construct \textbf{CARP}, a new Chinese dataset consisting of 4,886 computation-intensive algebra problems with formulated annotations on intermediate steps. In CARP, we test four LLMs with CoT prompting, and find that they are all prone to make mistakes at the early steps of the solution, leading to wrong answers. Based on this finding, we propose a new approach that can deliberate the reasoning steps with tool interfaces, namely \textbf{DELI}. In DELI, we first initialize a step-by-step solution based on retrieved exemplars, then iterate two deliberation procedures that check and refine the intermediate steps of the generated solution, from the perspectives of tool manipulation and natural language reasoning, until obtaining converged solutions or reaching the maximum turn. Experimental results on CARP and six other datasets show that the proposed DELI mostly outperforms competitive baselines, and can further boost the performance of existing CoT methods. Our data and code are available in \url{https://github.com/RUCAIBox/CARP}.

1 Introduction

Existing math-reasoning datasets may not adequately test repeated tool use or intermediate reasoning, motivating CARP’s annotated computation-intensive problems. The paper also introduces DELI, which iteratively checks and refines solutions; experiments report gains over existing prompting methods.

1 Introduction

  • Existing datasets may require only one-off tool use and often lack formal intermediate-step annotations, limiting systematic evaluation of tool manipulation and reasoning.The paper notes that incorrect intermediate steps can produce wrong answers.Source: §1 · p. 2
  • On CARP, four popular LLMs using CoT cannot solve over half the problems, and all are more likely to err in the first step, with mistakes exceeding 69%.The paper reports that these early errors lead to wrong answers and are not corrected in later steps.Source: §1 · p. 2
  • CARP is a Chinese dataset of 4,886 computation-intensive algebra problems with formulated annotations for all intermediate steps.Its problems require multiple math expressions and arithmetic computations.Source: §1 · p. 2
  • DELI mostly outperforms competitive baselines across CARP and six other datasets and can further boost existing CoT prompting methods.The experiments cover seven computation-intensive math reasoning datasets.Source: §1 · p. 2
  • DELI initializes a step-by-step solution from retrieved exemplars, then iteratively checks and refines it through tool-manipulation and natural-language-reasoning procedures.Iteration stops when the solution converges or a maximum turn is reached.Source: §1 · p. 2
  • The paper also devises fine-grained tool interfaces with practical calculation functions for complex reasoning.These interfaces can also be applied to other math-reasoning datasets to improve tool manipulation.Source: §1 · p. 2

2 CARP Dataset

CARP is a Chinese dataset of computation-intensive middle-school algebra problems with annotated intermediate reasoning steps and fine-grained tool interfaces. Its CoT evaluation shows that LLMs often make errors early in the solution process.

2.1 Dataset Construction

  • CARP starts from Zhixue middle-school solutions, filters for more than one computation and two reasoning steps, then has teachers select examples and annotate and verify EFGs.Earlier datasets may omit intermediate steps or present solutions in poorly formulated natural language [28] [29] [7] [8].Source: §2.1 · pp. 3–4
  • An expression flow graph (EFG) represents a solution as a directed acyclic graph whose expression nodes derive from earlier nodes until reaching the final expression.Natural-language nodes preserve special initial conditions that cannot readily be written as expressions.Source: §2.1 · p. 3

2.2 Dataset Details

  • CARP contains 4,886 problems; each EFG averages 6.0 nodes, 5.7 edges, and 4.7 expression nodes.The expression nodes support evaluation of intermediate-step accuracy.Source: §2.2 · p. 4
  • All tested LLMs score below 50.0 answer accuracy and below 57.0 intermediate-step accuracy on CARP, with first-step mistakes more common than last-step mistakes.The authors attribute early failures to improper mathematical knowledge or incorrect calculations.Source: §2.2 · pp. 4–5, Table 3 · p. 5

2.3 Tool Interfaces

  • CARP provides SymPy-based interfaces for numerical computation, equation solving, and expression transformation, plus a think interface for deriving conclusions around tool use.The interfaces are described in natural language to support model manipulation.Source: §2.3 · p. 5

2.4 Dataset Discussion

  • CARP combines multi-step mathematical reasoning, repeated fine-grained tool use, and intermediate-step metrics, unlike datasets focused mainly on final-answer accuracy [6] [7] [8].The authors say these metrics help quantify mastery of the solution process and provide clues for improving models.Source: §2.4 · p. 5

3 Approach

DELI initializes a step-by-step solution from retrieved exemplars, then iteratively checks and refines it through tool manipulation and chain-of-thought deliberation. The process stops when solutions converge, the two procedures agree on an answer, or the iteration limit is reached.

3.1 Retrieval-Augmented Solution Initialization

  • Retrieved question-and-solution exemplars provide the in-context basis for initializing a step-by-step solution.The retriever selects relevant problems by question-question matching, and the prompt includes their solutions; other prompting methods can also initialize solutions.Source: §3.1 · p. 6

3.2 Iterative Deliberation

  • DELI alternates tool-interface and chain-of-thought deliberation to check and refine intermediate steps in a generated solution.The tool stage produces a formal sequence of interface actions and outputs; the chain-of-thought stage reorganizes it into natural language for review and readability.Source: §3.2 · pp. 6–7
  • During chain-of-thought deliberation, the revised solution integrates the natural-language solution with the tool-generated verification and favors tool results when computations conflict.The exemplars guide the model to verify and revise intermediate steps.Source: §3.2 · p. 7
  • Iteration ends when the solution is unchanged, the two procedures agree on the answer, or the maximum number of turns is reached.At the turn limit, the final answer comes from the last tool-manipulation procedure.Source: §3.2 · p. 7

4 Experiment

Across seven computation-intensive math reasoning datasets, DELI performs better than competitive baselines in most cases. Analyses report gains across existing CoT methods, accuracy increases with more deliberation turns before convergence, and improved intermediate-step measures on challenging problems.

4.1 Main Experiments

  • DELI performs better than competitive baselines in most cases across the seven datasets, while unmodified iterative CoT and ReAct variants underperform their basic methods.Retrieval CoT beats Random CoT on average, ReAct outperforms CoT methods on average, and ReAct performs better than PAL on the benchmark.Source: §4.1 · p. 8

4.2 Analysis

  • DELI improves results across the tested CoT methods on CMA, Algebra, and Count & Prob., including the iterative method PHP.The authors attribute the gains to fine-grained interfaces helping find and fix errors that CoT may not correct, such as complex calculation errors.Source: §4.2 · p. 9
  • DELI accuracy overall increases and eventually converges as the maximum iteration turns increase, whereas Iterative CoT and Iterative ReAct do not generally gain accuracy and can decline.The authors report that DELI’s two deliberation perspectives reduce calculation errors and progressively refine solutions.Source: §4.2 · p. 9, Figure 4 · p. 9
  • On 128 challenging CARP problems answered incorrectly by all evaluated methods, DELI achieves better ExpAcc than CoT and ReAct and yields a larger percentage of near-correct solutions.Fail@where indicates DELI is less likely to generate completely wrong solutions on this subset.Source: §4.2 · p. 9

5 Conclusion

CARP evaluates computation-intensive algebra reasoning with intermediate-step annotations, and the experiments identify early-step errors in CoT solutions. DELI checks and refines reasoning through tool manipulation and natural-language deliberation, outperforming baselines and boosting CoT methods across the evaluated datasets.

5 Conclusion

  • CARP provides formulated intermediate-step annotations for systematically evaluating tool manipulation and math reasoning on computation-intensive algebra problems.The experiments found that popular LLMs using CoT could not solve over half of CARP problems.Source: §5 · p. 9
  • CoT prompting errors often occur in early solution steps, leading to wrong answers.Popular LLMs with CoT could not solve over half of CARP problems.Source: §5 · pp. 9–10
  • DELI iteratively checks and refines solutions through tool manipulation and natural-language reasoning, outperforming baselines and boosting CoT methods.The experiments covered CARP and six other computation-intensive math reasoning datasets.Source: §5 · p. 10

A Interface Definition

The appendix provides definitions for tool interfaces and identifies their main functionality categories.

A Interface Definition

B DELI Algorithem

DELI initializes a solution and repeatedly applies tool-interface and chain-of-thought deliberation, stopping when the solution converges or the iteration limit is reached.

B DELI Algorithem

  • The algorithm takes an input problem, optional retrieval module, and tool interfaces, then initializes a step-by-step solution.The initialization uses retrieved exemplars when available.Source: B DELI Algorithem · p. 13
  • Each iteration deliberates first with tool interfaces and then with chain of thought to update the solution.The process is illustrated in Algorithm 1.Source: B DELI Algorithem · p. 13
  • DELI stops early when a stopping condition is met, including convergence, and returns the last solution after the loop.The algorithm also has a maximum iteration count.Source: B DELI Algorithem · p. 13

C Baselines

The paper compares DELI with CoT, tool-augmented, and iterative prompting baselines, implementing the methods on ChatGPT.

C Baselines

  • All compared methods are implemented on ChatGPT.The baseline set spans CoT, tool-augmented, and iterative prompting methods.Source: C Baselines · pp. 13–14
  • The CoT baselines vary exemplar selection through random sampling, complex-problem sampling, or retrieval of relevant examples.These are Random CoT, Complex CoT, and Retrieval CoT.Source: C Baselines · p. 13
  • PAL converts reasoning into an executable Python program, while ReAct invokes tool interfaces during reasoning when needed.Both are implemented using the paper's proposed tool interfaces.Source: C Baselines · pp. 13–14
  • Iterative baselines include LP and PHP, plus Iterative CoT and Iterative ReAct variants that review prior solutions or invocations with self-generated judgments.LP learns solutions from training sets; PHP uses previous answers as hints.Source: C Baselines · p. 14

D Implementation Details

The implementation uses fixed solver settings, dataset-specific retrieval, and a shared maximum of three deliberation turns. Initialization also varies by dataset and, for PHP, by subset.

D Implementation Details

  • DELI uses gpt-3.5-turbo as solver and reasoning tool, SymPy for computation, temperature 0, top_p 1, and a three-turn maximum.Interface descriptions are specified for each dataset.Source: D Implementation Details · p. 14
  • Similar problems are retrieved with a SimCSE-trained sentence embedding model for MATH and BM25 for CARP.Source: D Implementation Details · p. 14
  • Retrieval CoT initializes most datasets, while GK-Cloze and SAT-Math use Random CoT because they lack training sets.PHP uses Complex CoT on MATH Algebra, Prealgebra, CP, and NT subsets, and DELI's initialization elsewhere.Source: D Implementation Details · p. 14

E Case Study

The CARP case study contrasts errors by Retrieval CoT and ReAct with DELI's iterative tool and reasoning deliberation, which corrects the computation and reaches the right answer.

E Case Study

  • Figure 5 presents the translated case and compares DELI's solving process with Retrieval CoT and ReAct.Source: Figure 5 · p. 14, E Case Study · p. 15
  • Retrieval CoT and ReAct each fail on the case: one expands incorrectly, while the other misreads a condition and collects by the wrong term.The paper attributes these errors to computation and reasoning challenges.Source: E Case Study · p. 15
  • DELI invokes expand and collect in sequence, obtains the correct expanded expression, then solves the resulting equation correctly.Source: E Case Study · p. 15
  • After tool-based correction, revised chain-of-thought fixes the original computation error; agreement with the prior ReAct answer ends this iteration.In general, iteration continues until solutions converge, answers agree, or the iteration limit is reached.Source: E Case Study · p. 15

F Prompts for Two-Stage Deliberation

The prompts provide mathematical tool functions and prescribe a two-stage process: verify and revise an existing solution, then produce a final answer.

F Prompts for Two-Stage Deliberation

  • The tool prompt exposes calculation, equation and inequality solving, substitution, expansion, factoring, collecting, partial solving, and a natural-language think function.The think function is reserved for cases when the listed functions do not apply.Source: F Prompts for Two-Stage Deliberation · p. 15
  • The prompt inputs the question and a possibly erroneous trial solution, and requires a numerical or LaTeX-math final answer.Source: F Prompts for Two-Stage Deliberation · p. 16
  • Each function call is a single Action with arguments and must be followed immediately by one Output; the cycle may repeat.Source: F Prompts for Two-Stage Deliberation · p. 16
  • The chain-of-thought prompt asks the model to verify an existing solution using functions, then revise it based on the verification.When verification and original computed results differ, the verification result is treated as the standard.Source: F Prompts for Two-Stage Deliberation · p. 17
  • The prompt defines verification as transforming the original solution into a function-based process that corrects computational errors and simplifies the process.Source: F Prompts for Two-Stage Deliberation · p. 17
Loading 2306.02408v1…