Source-linked AI summary
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, Wanjun Zhong
TL;DR
Reasoning models perform well on textual chains but remain limited on precise computation and symbolic manipulation. ReTool integrates interleaved code execution with cold-start fine-tuning and outcome-driven RL, achieving higher AIME accuracy with fewer training steps than text-based RL. The approach also yields adaptive tool use and code self-correction.
Problem
Text-based reasoning models remain limited on precise numerical calculation and symbolic manipulation, while supervised tool-use methods may fail to generalize or adaptively invoke tools.
Method
ReTool combines code-integrated cold-start data with reinforcement learning that interleaves sandbox execution and natural-language reasoning, using outcome-based rewards.
Results
67.0% accuracy on AIME2024 with 400 training steps surpassed the text-based RL baseline’s 40.0% with 1080 steps, while training produced adaptive tool use and code self-correction.
Takeaways & Limitations
ReTool demonstrates that outcome-driven code-interpreter integration can improve mathematical reasoning performance and training efficiency while supporting sophisticated tool-use behaviors.
Abstract
from arXiv · showhide
While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic cold-start data generation to produce code-augmented long-form reasoning traces for fine-tuning base models. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on the challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.
1 Introduction
ReTool addresses the limits of text-only reasoning on precise computation and symbolic tasks by integrating code-interpreter use into reinforcement learning. On AIME, it improves accuracy and training efficiency while exhibiting adaptive tool-use behaviors.
- Motivation: Text-based reasoning models remain limited on precise numerical calculation, geometric reasoning, and complex equation solving, where code interpreters provide executable computation and verification.Code interpreters can validate intermediate results and reduce ambiguity in multi-step reasoning.
- Motivation: Supervised tool-use methods can imitate curated distributions but may fail to generalize or adaptively decide when and how to invoke tools.The paper motivates outcome-based RL as a way to explore flexible reasoning trajectories and tool-use strategies.
- Approach: ReTool combines a cold-start dataset demonstrating code-interpreter use with tool-enhanced RL that discovers tool-manipulation strategies from outcome rewards.During long-chain reasoning, the policy can write code and receive real-time sandbox execution results.
- Results: 67.0% accuracy on AIME2024 was achieved in 400 training steps, versus 40.0% for text-based RL in 1080 steps.The comparison uses Qwen2.5-32B-Instruct as the backbone.
- Behavioral analysis: Approximately 40% shorter responses after RL training indicate potential reasoning-token efficiency, alongside increasing code use and earlier code invocation.These trends are reported together with more advanced tool-use development during training.
- Behavioral analysis: RL training produces emergent code self-correction and adaptive tool selection, enabling more sophisticated tool-augmented reasoning patterns.The analysis also reports enhanced code utilization and more appropriate tool invocation.
2 Methodology
ReTool trains models in two stages: cold-start supervised fine-tuning on code-integrated reasoning data, followed by PPO with interleaved sandbox execution and outcome-based rewards. The method supports dynamic tool interaction, feedback-driven refinement, and efficient rollout infrastructure.
- Methodology: ReTool first performs cold-start supervised fine-tuning, then reinforcement learning with interleaved code-execution rollouts.The cold-start stage initializes tool use before specialized RL develops strategic selection and application.
- Cold-start data: The cold-start pipeline transforms mathematical reasoning traces by replacing suitable manual calculations with code snippets and interpreter results.Format and answer verification filter the resulting code-integrated data.
- Cold-start data: Supervised fine-tuning uses the curated dataset to teach when and how to invoke the code interpreter and analyze computational outputs.This provides the model with foundational tool-utilization capability.
- Reward design: A rule-based accuracy reward evaluates whether the predicted answer matches the ground truth, while avoiding a separate code-executability reward.Final-answer formatting enables reliable rule-based verification.
- Interleaved rollout: PPO rollouts interleave natural-language reasoning with multi-turn real-time code execution through collaboration between the policy model and a sandbox.Generated code is delimited with tags, and sandbox results or errors are fed back into subsequent reasoning.
- Interleaved rollout: Successful execution results and interpreter errors provide dynamic feedback for iterative refinement of reasoning and tool-use strategies.The rollout can produce either a final answer or a new code snippet within a hybrid reasoning trajectory.
- Training details: Training uses PPO with Qwen2.5-32B-Instruct as the main backbone and curated cold-start data for two epochs.The implementation uses the VeRL framework and a maximum sequence length of 16384 tokens.
- Training details: Interpreter feedback is masked from loss computation, while KV-cache reuse reduces rollout memory cost and an asynchronous sandbox accelerates parallel environment interactions.These implementation choices target training stability, memory reduction, and reduced bottlenecks.
3 Experiment
ReTool is evaluated on AIME benchmarks against text-based and competitive baselines, while analyses track how code-interpreter behavior changes during reinforcement learning. The results show improved accuracy, shorter responses, increasing code use and complexity, adaptive invocation timing, and emergent code self-correction.
- 3.2 Main Results: 67.0% AIME2024 accuracy and 49.3% AIME2025 accuracy were achieved by ReTool with 400 training steps, versus 40.0% and 36.7% for text-based RL after over 1000 steps.ReTool also surpassed s1-32B by 10.3% on AIME2024 and OpenAI o1-preview by 11.4% on AIME2025.
- 3.3 Cognitive Analysis: Response length ended 40% shorter after reinforcement learning, declining from 10k to 6k tokens despite a later increase.The authors attribute the initial decline to replacing complex computation with concise code and the later rise to more diverse code behaviors.
- 3.3 Cognitive Analysis: Code-containing responses approached 98%, average code lines grew nearly fivefold, and correct code counts increased from 1k to 5k during training.These trends are interpreted as improving code utilization and increasingly complex tool-use strategies.
- 3.3 Cognitive Analysis: Code invocation advanced during training, while code pass rates stayed near 100% for correct responses and declined for incorrect responses.The analysis reports that executable intermediate code is iteratively refined using interpreter feedback and affects final reasoning outcomes.
- 3.3 Cognitive Analysis: After an initially non-executable program failed because “greedy()” was undefined, the model used interpreter feedback to generate an executable revision with the required definitions.The authors describe this as emergent code self-correction and a possible metacognitive capability.
- 3.3 Cognitive Analysis: After reinforcement learning, code purposes became more diverse, while calculation and verification remained dominant purposes of code in CI-powered reasoning.The purpose analysis classified code snippets by contextual role and counted purposes appearing more than once.
4 Background and Related Work
Prior work established chain-of-thought and tool-integrated reasoning, but text-only reasoning remains vulnerable to numerical errors and tool-use approaches may imitate curated patterns without robust adaptation. The paper contrasts these approaches through examples of text-based and CI-powered reasoning.
- 4 Background: Chain-of-thought prompting improves reasoning by eliciting step-by-step natural-language descriptions, supporting recent progress toward metacognitive-like behavior.The passage situates OpenAI o1 and DeepSeek R1 among reasoning models that self-correct and deliberate in text.
- 4 Background: Prompting and supervised fine-tuning can equip LLMs with tool use but may fail to generalize beyond curated data or adaptively decide when and how to invoke tools.The passage notes possible tool misuse and brittle heuristics across diverse problem settings.
- 4.2 Tool Integrated Reasoning: Text-based reasoning can rely on laborious calculation and produce numerical errors, whereas CI-powered reasoning replaces that process with concise code.The case study presents this contrast on the same question before and after reinforcement learning.
5 Conclusion
The paper presents ReTool as reinforcement learning for mathematical reasoning through Code Interpreter utilization. Experiments on AIME2024 and AIME2025 report higher accuracy and convergence with fewer training steps than conventional text-based RL.
- 5 Conclusion: ReTool uses Code Interpreter utilization to help LLMs self-enhance mathematical reasoning and develop computational intervention strategies.The framework combines data curation with a specialized tool-using pipeline.
- 5 Conclusion: ReTool achieves superior accuracy and converges with significantly fewer training steps than conventional text-based RL approaches on AIME2024 and AIME2025.The conclusion characterizes the result as supporting more efficient tool-augmented reasoning.
A Appendix
The appendix provides prompt templates for ReTool’s reinforcement-learning rollout and data curation. These templates specify executable Python interaction with an external sandbox and transformation of reasoning traces into code-integrated data.
- Reinforcement Learning Rollout: The rollout prompt lets the model selectively write executable Python code, receive sandbox output, and use it to reach a final answer.Generated code is returned through interpreter output tags and must be a complete script with necessary imports.
- Data Curation: The data-curation template transforms manual calculation steps into code snippets paired with interpreter execution results.This process creates code-integrated reasoning data for subsequent training.