Source-linked AI summary
ToRL: Scaling Tool-Integrated RL
Xuefeng Li, Haoyang Zou, Pengfei Liu
TL;DR
Tool-integrated reasoning is limited by pure language computation and by supervised methods that constrain tool-use exploration. ToRL trains reinforcement learning directly from base models, yielding stronger mathematical benchmark performance and emergent adaptive tool-use behaviors, while allowing more tool calls introduces substantial computational overhead.
Problem
Pure natural-language reasoning struggles with precise computation, while supervised tool-integrated methods restrict models to predetermined tool-use patterns.
Method
ToRL trains base language models with reinforcement learning in an environment that integrates code execution and feedback, without prior supervised fine-tuning.
Results
43.3% accuracy on AIME problems was achieved by TORL-7B, while TORL consistently outperformed baseline models across tested mathematical benchmarks.
Takeaways & Limitations
Reward-driven tool integration produces strategic invocation, self-regulation of ineffective code, and adaptive switching between computational and analytical reasoning.
Takeaways & Limitations
Allowing more tool calls improves performance but severely reduces training efficiency because tool integration introduces computational overhead.
Abstract
from arXiv · showhide
We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3\% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14\% and the best existing Tool-Integrated Reasoning (TIR) model by 17\%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.
1 Introduction
TORL addresses limitations of supervised tool-use imitation by training directly from base models with reinforcement learning, enabling exploration of tool strategies. Experiments and analyses report improved mathematical performance and emergent behaviors including code use, self-regulation, and adaptive reasoning.
- Traditional natural-language reasoning struggles with complex calculations, equation solving, and precise computation.
- TORL trains reinforcement learning directly from base models, avoiding predetermined tool-use patterns imposed by prior supervised fine-tuning.The framework is designed to let models discover effective tool-utilization strategies through exploration.
- 43.3% accuracy on AIME problems was achieved by TORL-7B, comparable to some 32B models trained with reinforcement learning.
- Emergent behaviors: Code use, syntactic correctness, and executability increase during training, indicating autonomous acquisition of effective tool-utilization strategies.
- Emergent behaviors: Without explicit instruction, models reduce ineffective code patterns, suggesting emerging self-regulation about tool utility.
- Trade-offs: More allowed tool calls improve performance but create severe computational overhead, exposing an efficiency-effectiveness trade-off.
- Emergent behaviors: TORL produces feedback use, computational-analytical cross-checking, and adaptive reasoning-strategy selection through reward-driven learning.
- The implementation and trained models are open-sourced to support further research on tool-augmented language models.
2 Methodology
The methodology combines tool-integrated reasoning with iterative code execution and feedback. Reasoning trajectories interleave natural-language reasoning, generated code, and execution results until a final answer is produced.
- The dataset contains 28,740 final questions distilled from 75,149 verifiable Olympic-level mathematics questions.Sources include NuminaMATH, MATH, and DeepScaleR; proof-based and ambiguously verifiable problems were filtered out.
- Tool-integrated reasoning lets an LLM write code, call an interpreter, and continue reasoning from the execution output.
- TIR improves numerical and logical reasoning by iteratively combining natural-language reasoning with executable code.
- A reasoning trajectory is represented as alternating reasoning, code, and execution-result elements: s_k = r_1, c_1, o_1, ..., r_k, c_k, o_k.
- Each step uses previous execution results to verify intermediate work and correct errors during complex problem solving.
2.3 TORL: Tool Integrated Reinforcement Learning
TORL integrates a code interpreter into reinforcement-learning rollouts from base language models, replacing pure language trajectories with tool-informed reasoning. The rollout framework controls tool calls, handles execution feedback, and uses reward signals for answer correctness and code behavior.
- TORL combines tool-integrated reasoning with reinforcement learning directly from base models without prior supervised fine-tuning.
- The RL environment replaces pure Chain-of-Thought trajectories with paths that interact directly with a Code Interpreter.
- TIR Rollout Framework: When a code terminator appears, the system executes the latest code block, inserts an observation, and resumes reasoning or awaits a final answer.Execution failures return error messages to support subsequent code generation.
- TIR Rollout Framework: The maximum tool-call hyperparameter C limits calls per response and forces pure-text reasoning after the threshold is exceeded.
- Execution environment: Sandbox Fusion was selected over an unisolated Python executor because execution errors could compromise the training process.
- Error processing: Error handling extracts only the final line of verbose tracebacks to preserve relevant information and reduce context length.
- Output masking: Sandbox observations are masked during loss computation to improve stability and discourage memorization of specific outputs.
- Reward Design: Correct answers receive reward 1, incorrect answers receive -1, and non-executable code incurs a -0.5 reduction.
3.1 Experimental Setup
The experiments use Qwen-2.5-Math models trained with RL and evaluate them on challenging mathematical benchmarks. The setup includes answer-correctness and code-executability reward designs.
- The code-executability reward is represented as a separate reward design in the experimental setup.
- RL experiments use the veRL framework, Sandbox Fusion interpreter, GRPO, rollout batches of 128, and 16 samples per problem.Exploration is increased by omitting KL loss and setting temperature to 1; the default maximum tool calls is C = 1.
- Evaluation uses greedy decoding across AIME24, AIME25, MATH500, OlympiadBench, and AMC23.
3.2 Main Results
TORL consistently outperforms baseline models across mathematical benchmarks, with especially strong gains at the 7B scale and on challenging evaluation sets.
- 62.1% average accuracy: TORL-7B exceeds other open-source models with the same base model by 14.7% absolute.
- 48.5% average accuracy: TORL-1.5B surpasses Qwen2.5-Math-1.5B-Instruct at 35.9% and its TIR version at 41.3%.
- 43.3%, 30.0%, and 49.9% accuracy: TORL-7B performs strongly on AIME24, AIME25, and OlympiadBench, respectively.
- Across five mathematical benchmarks, TORL-7B improves consistently over training steps and maintains an advantage over both comparison models.
3.3 Analysis
The analysis examines how tool usage, code-writing behavior, and key TORL parameters affect performance and training efficiency.
- The study quantitatively analyzes tool invocation and code-writing behaviors and examines parameter effects on performance and training efficiency.
3.3 Analysis
During training, the model increasingly uses code, produces more executable code, and improves the effectiveness of that code while execution success correlates with answer correctness.
- Code Ratio: 40% to 80%: the proportion of responses containing code increases during the first 100 training steps.
- Pass Ratio: The proportion of successfully executed code rises continuously, reflecting improved coding capability during training.
- Pass Ratio of Correct to Incorrect Responses: Correct responses exhibit higher code pass rates than incorrect responses, linking execution errors with final-answer accuracy.
- Effective Code Ratio: Executed code ratio excludes code omitted after exceeding C, while before final answer code ratio excludes post-answer correctness checks.
- Effective Code Ratio: Both effective-code metrics increase over time, indicating that model-generated code becomes more effective as training progresses.
3.3 Analysis
Training increases the model’s effective code use and reduces ineffective code, while tool-call limits expose a performance–efficiency trade-off and execution penalties fail to improve performance.
- Code behaviors during training: As training progresses, code use and correct execution increase while the model reduces ineffective code generation.
- Tool-call limit: C=2 increases average accuracy by approximately 2% but substantially reduces training speed compared with C=1.
- Reward shaping: Adding Code Executability Reward does not improve performance and may encourage overly simplistic code, according to the authors’ hypothesis.
- Tool-call limit: Increasing C enhances performance but severely reduces training efficiency, creating a direct performance–efficiency trade-off.
- Emergent cognitive behaviors: The model can correct an indexing error after receiving interpreter feedback and then produce executable code leading to the correct answer.
- Emergent cognitive behaviors: TORL exhibits feedback use, reflection, and cross-checking between code and natural-language reasoning.
4 Conclusion
ToRL enables reinforcement-learning-based tool integration beyond predefined usage constraints, with results showing emergent reasoning behaviors. Models can detect errors through code verification and correct their reasoning.
- Code verification can expose an incorrect natural-language result and trigger subsequent corrections.
- ToRL enables LLMs to integrate tools during reasoning through reinforcement learning, surpassing predefined tool usage constraints.