PaperlayerView exact revision on arXiv

Source-linked AI summary

ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, Wanjun Zhong

arXiv:2504.11536v2cs.CLcs.AI

TL;DR

Reasoning LLMs remain limited on precise numerical and symbolic problems, motivating ReTool, which combines cold-start training with reinforcement learning that interleaves code execution in reasoning. Across AIME2024 and AIME2025, ReTool outperforms text-based RL with fewer training steps, including 67.0% versus 40.0% accuracy on AIME2024.

  • Problem

    Reasoning LLMs remain limited on precise numerical or symbolic tasks, and prompting or supervised fine-tuning may not generalize or adaptively determine when and how to use tools.

    Source: §1 · p. 2

  • Method

    ReTool pairs a cold-start dataset and supervised fine-tuning with reinforcement learning that interleaves code-interpreter execution during rollouts and uses outcome feedback to refine tool-use strategies.

    Source: §1 · pp. 2–3

  • Results

    Across AIME2024 and AIME2025, ReTool outperforms text-based RL with fewer training steps; on AIME2024, accuracy is 67.0% versus 40.0%.

    Source: §1 · p. 2, §3.2 · p. 6, §5 · p. 10

  • Takeaways & Limitations

    The experiments support code-interpreter-integrated RL as a promising approach to more efficient tool-augmented mathematical reasoning.

    Source: §5 · p. 10

Abstract

from arXiv · show

While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic cold-start data generation to produce code-augmented long-form reasoning traces for fine-tuning base models. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on the challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.

1 Introduction

ReTool integrates real-time code execution into LLM reasoning and uses outcome-driven reinforcement learning to teach tool-use strategies. The introduction reports improved AIME2024 accuracy and training efficiency, alongside more adaptive tool use after RL training.

  • Textual reasoning models struggle with precise calculation and symbolic manipulation, while code interpreters enable executable computation, validation, and programmable exploration.Prior prompting and supervised fine-tuning can be limited to curated patterns and may fail to generalize or adaptively decide when and how to use tools; the introduction frames outcome-feedback RL as a way to learn such strategies and recover from execution mistakes.Source: §1 · p. 2
  • ReTool combines a cold-start dataset demonstrating code-interpreter use with RL rollouts that interleave code execution and reasoning, refining tool strategies through outcome rewards.The policy writes code blocks and receives real-time results from a sandbox-style interpreter to support subsequent reasoning.Source: §1 · pp. 2–3
  • 67.0% AIME2024 accuracy in 400 training steps surpassed the text-based RL baseline’s 40.0% in 1080 steps.On Qwen2.5-32B-Instruct, ReTool was evaluated on AIME2024 and AIME2025; the introduction reports these specific accuracy and training-step results for AIME2024.Source: §1 · p. 2
  • 40.9% AIME2024 accuracy from the Qwen2.5-32B-Instruct cold-start model was comparable to text-based RL’s 40.0% and exceeded the untrained model’s 26.7%.With a DeepSeek-R1-Distill-Qwen-32B backbone, ReTool also surpassed QwQ-32B-Preview, s1-32B, and OpenAI o1-preview; no scores for those comparisons are given here.Source: §1 · p. 2
  • RL training was associated with more accurate and complex code, appropriate and adaptive tool invocation, effective call structure, and emergent code self-correction.The introduction also reports adaptive tool selection as an emergent behavior during RL.Source: §1 · pp. 2–3
  • After RL training, response length fell by approximately 40%, while code ratio, code lines, correct code counts, and earlier code invocation showed increasing trends.The introduction presents these trends as evidence of improved code use and strategic tool usage.Source: §1 · p. 3

2 Methodology

ReTool pairs cold-start supervised fine-tuning on code-augmented reasoning traces with reinforcement learning that interleaves natural-language reasoning and real-time code execution.

2.1 Overview

  • The framework has two stages: cold-start supervised fine-tuning initializes the model, followed by tool-using reinforcement learning to improve appropriate tool selection and application during reasoning.The RL stage uses interleaved code-execution rollouts.Source: §2.1 · p. 3

2.2 Cold-start for Tool-Integrated Reasoning Foundation

  • The cold-start pipeline curates diverse math data including OpenThoughts, uses human and Deepseek-R1 [4] checks, then converts and verifies code-augmented traces as DCI for SFT.The conversion replaces manual calculations that benefit from code with snippets and interpreter results; two-stage checks verify format and final-answer correctness.Source: §2.2 · p. 3

2.3.1 Training Algorithm

  • PPO [16] training uses rule-based answer-accuracy rewards, encouraging the model to develop strategies for tool choice and invocation without an executability reward.Answers use a specified format for rule-based verification; the reward focuses on outcomes to alleviate reward hacking and support diverse problem-solving behaviors.Source: §2.3.1 · p. 4
  • The rollout interleaves reasoning, code, and sandbox feedback across turns, returning both execution results and errors so the model can refine its reasoning and tool use.The model marks code with <code></code>; execution output is returned within <interpreter></interpreter> tags before generation continues toward an answer or another code snippet.Source: §2.3.1 · pp. 4–5

2.3.2 Training Details

  • Training uses Qwen2.5-32B-Instruct [15] with PPO on curated cold-start data for two epochs, a 1e-6 learning rate, 16384-token maximum sequence length, batch size 512, and KL coefficient 0.0.Interpreter feedback is masked from loss computation; KV-cache reuse reduces rollout memory, and an asynchronous worker-pool sandbox supports parallel interactions and load balancing.Source: §2.3.2 · p. 5

3 Experiment

The experiments evaluate ReTool on AIME benchmarks and analyze how reinforcement learning changes the model’s code use, self-correction, and reasoning. Results report stronger accuracy than text-based RL, alongside evolving tool-use behaviors.

3.1 Evaluation Setup

  • Evaluation estimates pass@1 by averaging 32 runs on AIME2024&2025, using temperature 1.0 and top-p 0.7; literature baselines are reported as avg@k.The comparison includes competitive models and Text-based RL using Qwen2.5-32B-Instruct.Source: §3.1 · p. 5

3.2 Main Results

  • 67.0% on AIME2024 and 49.3% on AIME2025 after 400 steps, ReTool (Qwen2.5-32B-Instruct) beats text-based RL’s 40.0% and 36.7% after over 1000 steps.ReTool (DeepSeek-R1-Distill-Qwen-32B) scores 72.5% on AIME2024 and 54.3% on AIME2025; the Qwen2.5 ReTool model surpasses s1-32B by 10.3% on AIME2024 and OpenAI o1-preview by 11.4% on AIME2025.Source: Table 1 · p. 6, §3.2 · p. 6
  • 40.9% on AIME2024, the Qwen2.5-32B-Instruct cold-start model approaches text-based RL’s 40.0% and exceeds the base model’s 26.7%.The authors attribute this to curated executable reasoning traces capturing tool-use patterns, with CI-integrated training contributing positively to reasoning performance.Source: §3.2 · p. 6

3.3 Cognitive Analysis

  • During RL, average responses first shorten sharply then rise gently, ending 40% shorter than before RL (10k to 6k); code responses approach 98% and average code lines grow nearly fivefold.The authors attribute the response-length change first to replacing complex computation with concise code, then to more diverse and complex code behavior.Source: §3.3 · pp. 6–7, Figure 3 · p. 7
  • Correct code counts on the test set rise from 1k to 5k, while pass rates stay near 100% for correct responses but decline for incorrect ones, and code invocation advances during RL.The pass-rate analysis concerns the last code in incorrect responses; the authors connect executability with reasoning and the final result.Source: §3.3 · p. 7, Figure 3 · p. 7
  • After interpreter feedback on code that failed because greedy() was undefined, the model defined the needed function and generated executable code without explicit self-correction training data.The authors describe this emergent behavior as evidence that RL can foster metacognitive iterative code refinement for more complex problems.Source: §3.3 · pp. 7–8, Figure 4 · p. 8
  • Calculation and verification dominate code purposes, while post-RL code purposes become more diverse, which the authors link to adaptive tool selection and broader generalizability.Code snippets were classified by contextual primary purpose using Doubao-1.5-pro2.Source: §3.3 · p. 8, Figure 5 · p. 9
  • In a case study, text-only reasoning uses laborious calculations prone to numerical errors, whereas CI-powered reasoning replaces them with concise code and supports holistic reasoning strategies.The passage attributes computational accuracy to assistance from an external code interpreter.Source: §3.3 · p. 8

4 Background and Related Work

The section surveys CoT and PoT reasoning and the shift toward test-time computation, then situates ReTool among tool-integrated methods that learn adaptive code-interpreter use [1] [5] [8].

4.1 LLM Reasoning

  • CoT uses step-by-step natural-language reasoning, while PoT adds external tools such as Python interpreters to simplify and validate complex reasoning [1] [5].Test-time scaling methods also include stepwise preference optimization [7], MCTS, and reinforcement learning [9]; OpenAI-o1 [12] and DeepSeek-R1 [4] exemplify CoT-based reasoning.Source: §4.1 · p. 9

4.2 Tool Integrated Reasoning

  • Earlier tool-integrated approaches combined textual reasoning with code execution or supervised code-integrated CoT, but did not learn adaptive tool-use strategies through reinforcement learning [25] [2].A concurrent RL approach learned tool-use strategies on Qwen2.5-Math models at 1.5B and 7B scales, but its performance remained suboptimal [8].Source: §4.2 · p. 9
  • ReTool scales RL-based tool-use learning and reports significantly outperforming Qwen-Math-72B-TIR [31] and o1-preview on AIME2024 and AIME2025.The section also presents analysis of learned tool-use behaviors and cognitive patterns in code invocation after training.Source: §4.2 · p. 9

5 Conclusion

ReTool uses reinforcement learning to help LLMs improve mathematical reasoning through Code Interpreter use. On AIME2024 and AIME2025, it achieves higher accuracy than text-based RL approaches with fewer training steps.

5 Conclusion

  • ReTool improves mathematical reasoning through reinforcement learning that enables LLMs to use Code Interpreters.The framework aims to let models develop effective computational intervention strategies.Source: §5 · p. 10
  • ReTool achieves higher accuracy than conventional text-based RL approaches on AIME2024 and AIME2025 while converging in fewer training steps.The paper attributes these results to careful data curation and a specialized tool-using pipeline.Source: §5 · p. 10

Appendix · A Appendix

The appendix provides template prompts for ReTool rollouts and data curation.

A Appendix

Related papers

Loading 2504.11536v2…