PaperlayerView exact revision on arXiv

Source-linked AI summary

ToRL: Scaling Tool-Integrated RL

Xuefeng Li, Haoyang Zou, Pengfei Liu

arXiv:2503.23383v1cs.CL

TL;DR

Tool-integrated reasoning is limited by pure language computation and by supervised methods that constrain tool-use exploration. ToRL trains reinforcement learning directly from base models, yielding stronger mathematical benchmark performance and emergent adaptive tool-use behaviors, while allowing more tool calls introduces substantial computational overhead.

  • Problem

    Pure natural-language reasoning struggles with precise computation, while supervised tool-integrated methods restrict models to predetermined tool-use patterns.

    Source: §1 · p. 2

  • Method

    ToRL trains base language models with reinforcement learning in an environment that integrates code execution and feedback, without prior supervised fine-tuning.

    Source: §1 · p. 2, §2.3.1 · p. 4

  • Results

    43.3% accuracy on AIME problems was achieved by TORL-7B, while TORL consistently outperformed baseline models across tested mathematical benchmarks.

    Source: §1 · p. 2, §3.2 · p. 5

  • Takeaways & Limitations

    Reward-driven tool integration produces strategic invocation, self-regulation of ineffective code, and adaptive switching between computational and analytical reasoning.

    Source: §1 · p. 2

    Allowing more tool calls improves performance but severely reduces training efficiency because tool integration introduces computational overhead.

    Source: §1 · p. 2

Abstract

from arXiv · show

We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3\% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14\% and the best existing Tool-Integrated Reasoning (TIR) model by 17\%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.

1 Introduction

TORL addresses limitations of supervised tool-use imitation by training directly from base models with reinforcement learning, enabling exploration of tool strategies. Experiments and analyses report improved mathematical performance and emergent behaviors including code use, self-regulation, and adaptive reasoning.

  • Traditional natural-language reasoning struggles with complex calculations, equation solving, and precise computation.Source: §1 · p. 2
  • TORL trains reinforcement learning directly from base models, avoiding predetermined tool-use patterns imposed by prior supervised fine-tuning.The framework is designed to let models discover effective tool-utilization strategies through exploration.Source: §1 · p. 2
  • 43.3% accuracy on AIME problems was achieved by TORL-7B, comparable to some 32B models trained with reinforcement learning.Source: §1 · p. 2

Emergent behaviors

  • Code use, syntactic correctness, and executability increase during training, indicating autonomous acquisition of effective tool-utilization strategies.Source: §1 · p. 2
  • Without explicit instruction, models reduce ineffective code patterns, suggesting emerging self-regulation about tool utility.Source: §1 · p. 2

Trade-offs

  • More allowed tool calls improve performance but create severe computational overhead, exposing an efficiency-effectiveness trade-off.Source: §1 · p. 2

Emergent behaviors

  • TORL produces feedback use, computational-analytical cross-checking, and adaptive reasoning-strategy selection through reward-driven learning.Source: §1 · p. 2
  • The implementation and trained models are open-sourced to support further research on tool-augmented language models.Source: §1 · p. 2

2 Methodology

The methodology combines tool-integrated reasoning with iterative code execution and feedback. Reasoning trajectories interleave natural-language reasoning, generated code, and execution results until a final answer is produced.

  • The dataset contains 28,740 final questions distilled from 75,149 verifiable Olympic-level mathematics questions.Sources include NuminaMATH, MATH, and DeepScaleR; proof-based and ambiguously verifiable problems were filtered out.Source: §2.1 · p. 2
  • Tool-integrated reasoning lets an LLM write code, call an interpreter, and continue reasoning from the execution output.Source: Figure 2 · p. 3
  • TIR improves numerical and logical reasoning by iteratively combining natural-language reasoning with executable code.Source: §2.2 · p. 3
  • A reasoning trajectory is represented as alternating reasoning, code, and execution-result elements: s_k = r_1, c_1, o_1, ..., r_k, c_k, o_k.Source: §2.2 · p. 3
  • Each step uses previous execution results to verify intermediate work and correct errors during complex problem solving.Source: §2.2 · p. 3

2.3 TORL: Tool Integrated Reinforcement Learning

TORL integrates a code interpreter into reinforcement-learning rollouts from base language models, replacing pure language trajectories with tool-informed reasoning. The rollout framework controls tool calls, handles execution feedback, and uses reward signals for answer correctness and code behavior.

  • TORL combines tool-integrated reasoning with reinforcement learning directly from base models without prior supervised fine-tuning.Source: §2.3 · p. 3
  • The RL environment replaces pure Chain-of-Thought trajectories with paths that interact directly with a Code Interpreter.Source: §2.3 · p. 3

TIR Rollout Framework

  • When a code terminator appears, the system executes the latest code block, inserts an observation, and resumes reasoning or awaits a final answer.Execution failures return error messages to support subsequent code generation.Source: §2.3.1 · p. 4
  • The maximum tool-call hyperparameter C limits calls per response and forces pure-text reasoning after the threshold is exceeded.Source: §2.3.1 · p. 4, §2.3.2 · p. 4

Execution environment

  • Sandbox Fusion was selected over an unisolated Python executor because execution errors could compromise the training process.Source: §2.3.2 · p. 4

Error processing

  • Error handling extracts only the final line of verbose tracebacks to preserve relevant information and reduce context length.Source: §2.3.2 · p. 4

Output masking

  • Sandbox observations are masked during loss computation to improve stability and discourage memorization of specific outputs.Source: §2.3.2 · p. 4

Reward Design

  • Correct answers receive reward 1, incorrect answers receive -1, and non-executable code incurs a -0.5 reduction.Source: §2.3.3 · p. 4

3.1 Experimental Setup

The experiments use Qwen-2.5-Math models trained with RL and evaluate them on challenging mathematical benchmarks. The setup includes answer-correctness and code-executability reward designs.

  • The code-executability reward is represented as a separate reward design in the experimental setup.Source: Table 2 · p. 5
  • RL experiments use the veRL framework, Sandbox Fusion interpreter, GRPO, rollout batches of 128, and 16 samples per problem.Exploration is increased by omitting KL loss and setting temperature to 1; the default maximum tool calls is C = 1.Source: §3.1 · p. 5
  • Evaluation uses greedy decoding across AIME24, AIME25, MATH500, OlympiadBench, and AMC23.Source: §3.1 · p. 5

3.2 Main Results

TORL consistently outperforms baseline models across mathematical benchmarks, with especially strong gains at the 7B scale and on challenging evaluation sets.

  • 62.1% average accuracy: TORL-7B exceeds other open-source models with the same base model by 14.7% absolute.Source: §3.2 · p. 5
  • 48.5% average accuracy: TORL-1.5B surpasses Qwen2.5-Math-1.5B-Instruct at 35.9% and its TIR version at 41.3%.Source: §3.2 · p. 5
  • 43.3%, 30.0%, and 49.9% accuracy: TORL-7B performs strongly on AIME24, AIME25, and OlympiadBench, respectively.Source: §3.2 · p. 5
  • Across five mathematical benchmarks, TORL-7B improves consistently over training steps and maintains an advantage over both comparison models.Source: §3.2 · p. 5, Table 3 · p. 5

3.3 Analysis

The analysis examines how tool usage, code-writing behavior, and key TORL parameters affect performance and training efficiency.

  • The study quantitatively analyzes tool invocation and code-writing behaviors and examines parameter effects on performance and training efficiency.Source: §3.3 · p. 5

3.3 Analysis

During training, the model increasingly uses code, produces more executable code, and improves the effectiveness of that code while execution success correlates with answer correctness.

Code Ratio

  • 40% to 80%: the proportion of responses containing code increases during the first 100 training steps.Source: §3.3.1 · p. 6

Pass Ratio

  • The proportion of successfully executed code rises continuously, reflecting improved coding capability during training.Source: §3.3.1 · p. 6

Pass Ratio of Correct to Incorrect Responses

  • Correct responses exhibit higher code pass rates than incorrect responses, linking execution errors with final-answer accuracy.Source: §3.3.1 · p. 6

Effective Code Ratio

  • Executed code ratio excludes code omitted after exceeding C, while before final answer code ratio excludes post-answer correctness checks.Source: §3.3.1 · p. 6
  • Both effective-code metrics increase over time, indicating that model-generated code becomes more effective as training progresses.Source: §3.3.1 · p. 6

3.3 Analysis

Training increases the model’s effective code use and reduces ineffective code, while tool-call limits expose a performance–efficiency trade-off and execution penalties fail to improve performance.

Code behaviors during training

  • As training progresses, code use and correct execution increase while the model reduces ineffective code generation.Source: §3.3 · p. 7

Tool-call limit

Reward shaping

  • Adding Code Executability Reward does not improve performance and may encourage overly simplistic code, according to the authors’ hypothesis.Source: §3.3.2 · p. 7, Figure 6 · p. 7

Tool-call limit

  • Increasing C enhances performance but severely reduces training efficiency, creating a direct performance–efficiency trade-off.Source: §3.3.2 · p. 7

Emergent cognitive behaviors

  • The model can correct an indexing error after receiving interpreter feedback and then produce executable code leading to the correct answer.Source: §3.3.3 · p. 7, Table 5 · p. 8
  • TORL exhibits feedback use, reflection, and cross-checking between code and natural-language reasoning.Source: §3.3.3 · p. 7

4 Conclusion

ToRL enables reinforcement-learning-based tool integration beyond predefined usage constraints, with results showing emergent reasoning behaviors. Models can detect errors through code verification and correct their reasoning.

  • Code verification can expose an incorrect natural-language result and trigger subsequent corrections.Source: Table 6 · p. 8
  • ToRL enables LLMs to integrate tools during reasoning through reinforcement learning, surpassing predefined tool usage constraints.Source: §4 · p. 8

Related papers

Loading 2503.23383v1…