PaperlayerView exact revision on arXiv

Source-linked AI summary

ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen

arXiv:2309.17452v4cs.CLcs.AI

TL;DR

Complex mathematical reasoning remains difficult for open-source language models. ToRA integrates natural-language reasoning with program-based tool use, training on interactive trajectories through imitation learning and output space shaping. Across 10 mathematical reasoning datasets, ToRA substantially outperforms prior open-source models, while analysis identifies incorrect reasoning and diagram interpretation as remaining challenges.

  • Problem

    Open-source language models still struggle with advanced mathematical reasoning, especially on complex mathematics.

  • Method

    ToRA trains open-source models to interleave natural-language reasoning with program-based tool use using synthesized trajectories, imitation learning, and output space shaping.

  • Results

    13%-19% absolute improvements across 10 tasks show that ToRA consistently surpasses prior open-source models across scales.

  • Takeaways & Limitations

    Tool-integrated reasoning enables strong performance on challenging quantitative tasks and provides a basis for analyzing the benefits and challenges of tool interaction.

  • Takeaways & Limitations

    Incorrect reasoning steps account for 38% of annotated errors, while misinterpreting input diagrams accounts for 21%.

Abstract

from arXiv · show

Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics. In this paper, we propose ToRA a series of Tool-integrated Reasoning Agents designed to solve challenging mathematical problems by seamlessly integrating natural language reasoning with the utilization of external tools (e.g., computation libraries and symbolic solvers), thereby amalgamating the analytical prowess of language and the computational efficiency of tools. To train ToRA, we curate interactive tool-use trajectories on mathematical datasets, apply imitation learning on the annotations, and propose output space shaping to further refine models' reasoning behavior. As a result, ToRA models significantly outperform open-source models on 10 mathematical reasoning datasets across all scales with 13%-19% absolute improvements on average. Notably, ToRA-7B reaches 44.6% on the competition-level dataset MATH, surpassing the best open-source model WizardMath-70B by 22% absolute. ToRA-Code-34B is also the first open-source model that achieves an accuracy exceeding 50% on MATH, which significantly outperforms GPT-4's CoT result, and is competitive with GPT-4 solving problems with programs. Additionally, we conduct a comprehensive analysis of the benefits and remaining challenges of tool interaction for mathematical reasoning, providing valuable insights for future research.

1 INTRODUCTION

ToRA combines natural-language reasoning with program-based tool use to address the complementary weaknesses of each approach. Its training and output-space shaping produce strong gains over open-source baselines on challenging mathematical reasoning tasks.

  • Open-source models still struggle with advanced mathematical reasoning despite broader progress by large language models.
  • Natural-language reasoning supports semantic analysis and planning but struggles with precise computation, symbolic manipulation, and algorithmic processing.
  • ToRA interleaves natural-language reasoning with program-based tool use to combine their complementary strengths.
  • Output space shaping trains models on self-sampled valid trajectories and corrected invalid trajectories to increase plausible reasoning diversity and reduce improper tool use.
  • TORA-7B outperforms WizardMath-70B by 22% absolute on MATH, while TORA-CODE-34B reaches 50.8% versus GPT-4 CoT’s 42.5%.

2 TORA: TOOL-INTEGRATED AGENTS FOR MATHEMATICAL REASONING

TORA generates interactive trajectories that alternate rationales, programs, and tool outputs, then trains models through imitation learning and output space shaping. The pipeline expands valid behavior by sampling trajectories and correcting invalid continuations with a teacher model.

  • TORA alternates natural-language rationales, generated programs, and tool outputs while iteratively updating the reasoning trajectory.
  • Interactive tool-use trajectories are synthesized with GPT-4 from GSM8k and MATH because existing datasets lack such annotations.
  • Prompt curation uses instructions and diverse few-shot examples in an interleaved format to demonstrate interactive tool-use trajectories.
  • The collection process retains correct trajectories, uses nucleus sampling when greedy decoding fails, and produces 16k TORA-CORPUS annotations after filtering.
  • Imitation learning minimizes trajectory negative log-likelihood and yields state-of-the-art mathematical reasoning performance despite the corpus’s small scale.
  • TORA-CORPUS uniquely combines natural-language rationales with program-based tool usage and draws questions from GSM8k and MATH training sets.
  • Output space shaping samples diverse trajectories, retains valid ones, and uses a teacher model to complete plausible prefixes from wrong trajectories.
  • The resulting corpus contains 233k distinct valid samples and 69k corrected samples before selecting up to four trajectories per problem for retraining.

3 EXPERIMENTS

TORA is evaluated across 10 mathematical reasoning datasets and against rationale-only, program-only, and output-shaping alternatives. Results show broad gains from tool-integrated reasoning, while analysis identifies topic-specific benefits and persistent failure modes.

  • Main Results: TORA consistently surpasses prior state-of-the-art open-source models across all scales, with 13% to 19% absolute improvements across 10 tasks.
  • Main Results: TORA-70B reaches 84.3% on GSM8k and 49.7% on MATH, exceeding ChatGPT’s CoT and PAL results on both datasets.
  • Main Results: TORA-CODE-34B scores 50.8% on MATH, 8.3% above GPT-4’s CoT result and close to GPT-4-Code at 51.8%.
  • Format Ablation: Interleaved reasoning outperforms rationale-only and program-only formats, improving LLaMA-2 results by 29.0% and 6.7%, respectively.
  • Output Space Shaping: Output space shaping improves average GSM8k and MATH accuracy by 3.4% and 4.0%, with correction providing up to a 4.5% absolute boost.
  • Subtopic Analysis: TORA outperforms WizardMath by around 45% in Algebra and Number Theory, while interleaving improves Precalculus, Algebra, and Geometry by 8.6% to 18.8%.
  • Remaining Challenges: Manual analysis attributes MATH errors mainly to incorrect reasoning steps at 38%, diagram misinterpretation at 21%, and tool-use errors.

4 CONCLUSION

The paper presents TORA as a tool-integrated reasoning approach that combines natural-language rationales with program-based tools. It reports state-of-the-art performance across diverse mathematical tasks and analyzes both the benefits and challenges of tool interaction.

  • TORA combines natural-language rationale with program-based tool use for mathematical problem solving.
  • The paper situates TORA within efforts to address the difficulty of complex computation and symbolic manipulation in natural-language reasoning.
  • Tool augmentation is presented as a broader approach for alleviating language-model limitations and improving reasoning and generation performance.
  • Learning interactive tool-use trajectories is identified as a promising direction for adapting language models to reasoning tasks.

B EVALUATION DATASETS

The evaluation uses 10 mathematical datasets spanning basic arithmetic through competition-level problems and multiple mathematical domains. Results also include subject-level comparisons for ChatGPT and GPT-4 on MATH.

  • The evaluation dataset collection contains 10 mathematical reasoning datasets spanning basic arithmetic to competition-level problems.
  • MAWPS reports average accuracy across SingleEq, SingleOp, AddSub, and MultiArith in the main results table.
  • The MATH evaluation includes ChatGPT and GPT-4 accuracies broken down by mathematical subject under PAL and TORA prompting.

C ADDITIONAL EXPERIMENTS AND ANALYSIS

TORA’s tool-integrated reasoning method significantly outperforms PAL prompting when applied directly to GPT-4 on MATH, demonstrating benefits from combining natural-language reasoning with program-based tool use.

  • TORA’s tool-integrated reasoning method significantly outperforms PAL prompting on GPT-4’s MATH evaluation.The comparison directly tests the method on a closed-source model.
  • The comparison provides evidence for synergizing natural-language reasoning and program-based tool use.The passage identifies this synergy as the source of the demonstrated benefit.
  • The evaluation uses detailed GPT-4 accuracies on the MATH dataset.These accuracies are presented in Table 6.

C.2 EFFECTS OF # VALID TRAJECTORIES FOR OUTPUT SPACE SHAPING

Increasing the number of additional valid trajectories benefits output space shaping, indicating that richer valid-trajectory information improves the shaping procedure.

  • Figure 7 examines output space shaping across different numbers of additional valid trajectories per question.The figure focuses specifically on the effect of trajectory count.
  • Increasing the number of additional valid trajectories benefits output space shaping.The experiment varies how many additional valid trajectories are used per question.
  • The reported trend supports using more additional valid trajectories when applying output space shaping.This follows the stated benefit of increasing the number of trajectories.

C.3 IMPACT OF OUTPUT SPACE SHAPING IN RELATION TO QUESTION DIFFICULTY

Output space shaping generally improves performance across MATH difficulty levels and model sizes, with larger gains on difficult, long problems; TORA-CODE-34B still trails GPT-4 PAL at Level 5.

  • Impact across difficulty levels: 4.0% average improvement is obtained from output space shaping across difficulty levels and model sizes.The average is reported across different model sizes.
  • Impact across difficulty levels: TORA-CODE-13B gains 5.4% to 5.7% on Level 3 to Level 5 problems, while Level 1 to Level 2 improvements are not significant.The larger gains occur on difficult, long problems.
  • Comparison with GPT-4 PAL: TORA-CODE-34B outperforms GPT-4 PAL from Level 1 through Level 4 after shaping.
  • Comparison with GPT-4 PAL: 27.3% vs. 30.0% at Level 5 leaves TORA-CODE-34B below GPT-4 PAL after shaping.Level 5 problems average about 248.4 characters, require more than 1,000 reasoning characters, and include diagrams about 20% of the time.

D DETAILED INFORMATION OF TORA-CORPUS

TORA-CORPUS construction combines controlled interactive tool-use annotations, quality filtering, and sampled trajectories, with additional analysis of coverage and rationale-guided annotation.

  • Data format and quality control: Few-shot prompting controls the reasoning format and improves data quality during interactive tool-use annotation.Sampling also increases annotation success and provides more comprehensive training-query coverage.
  • Data filtering process: Incorrect-answer paths and samples with intermediate program execution errors are filtered from the constructed data.The filtering prevents models from learning incorrect answers or erroneous intermediate processes.
  • Dataset statistics: TORA-CORPUS-Greedy uses greedy trajectories, whereas TORA-CORPUS-16k combines greedy and sampled trajectories.Table 8 compares their training-set accuracy on GSM8k and MATH.
  • Dataset statistics: Table 9 reports TORA-CORPUS-16k sample counts and trajectory-length statistics.The reported statistics include average, minimum, and maximum trajectory lengths.
  • Rationale as hints: GPT-4’s greedy MATH annotation success rate is below 65%, motivating rationale hints during annotation.With hints, GPT-4 tends to replicate the rationales and ignore tool-use outputs.

E PROMPTS

The prompts instruct ToRA to combine step-by-step natural-language reasoning with Python or symbolic-tool execution, then present simplified, formatted answers. Examples cover coordinate conversion, binary arithmetic, projections, and inequalities.

  • The prompts ask the agent to analyze each question, write argument-free solver functions, and integrate step-by-step reasoning with Python code.They also require a boxed LaTeX final result without units.
  • The coordinate-conversion example uses formulas for spherical coordinates, executes them with SymPy, and adjusts theta into the required interval.The initial output is (6, -pi/2, pi/3), which becomes (6, 3pi/2, pi/3).
  • The binary-arithmetic example converts binary inputs to decimal, performs the requested operations, and converts the result back to base 2.The reported answer is 10001_2.
  • The projection example constructs P = (v * v.T) / (v.T * v)[0] for v = (2, -3), yielding entries 4/13, -6/13, and 9/13.The matrix is simplified and displayed in boxed form.

F.1 SUCCESS CASES

The success cases illustrate ToRA using tools for computation, correcting an initially incorrect perfect-squares solution, and finalizing answers in the requested mathematical form.

  • Success cases: The domain example uses symbolic solving to identify values excluded from a function's domain and reports the output {2}.The surrounding reasoning determines where the denominator is zero.
  • Success cases: The perfect-squares example initially produces an incorrect symbolic result, then factors 133 and finds that (13, 6) gives the smallest sum.The revised computation returns 205, which is presented as the final answer.
  • Success cases: 205 is the smallest possible sum of the two perfect squares whose difference is 133.The answer follows the corrected factor-pair analysis and is boxed in the final presentation.
  • Success cases: The infinite-series example constructs the sum symbolically, but the tool leaves it unevaluated and the response retains the original series.The text says the sum does not converge to a simple expression.

F.2 FAILURE CASES

The failure cases span inappropriate tool use, syntax and reasoning errors, hallucination, and diagram-understanding difficulties, including incorrect outputs despite executable programs.

  • Reasoning Error: Tool-generated solutions can compute incorrect answers when the program encodes the wrong mathematical formulation, even when execution succeeds.For the arithmetic-sequence product, the program returns 50760 while the reference derivation gives 348.
  • Diagram Understanding: Diagram understanding remains difficult because solving geometry and related problems may require visual interaction and auxiliary constructions that SymPy only marginally supports.The paper identifies diagram navigation as important in Geometry, Precalculus, and Intermediate Algebra.
  • Hallucination: TORA also fails on problem interpretation and answer finalization, including hallucinated predictions that disagree with mathematically supported results.Examples include predicting 25 instead of 87 for the complex-number problem and producing a wrong answer for a radical-expression problem.
  • Syntax Error: TORA can produce syntactically invalid programs or exceed the function-call limit, yielding no execution result and an incorrect prediction.One failure outputs a SyntaxError and reaches the maximum function-call limit; another predicts 513 instead of the ground-truth 507.
  • Runtime Error: TORA sometimes applies brute-force code where direct computation is infeasible, causing timeouts instead of falling back to rationale-based solving.This issue is reported for Intermediate Algebra and Precalculus problems requiring complex reasoning over mathematical expressions.
Loading 2309.17452v4…