ToolRL: Reward is All Tool Learning Needs

Cheng Qian; Emre Can Acikgoz; Qi He; Hongru Wang; Xiusi Chen; Dilek Hakkani-Tür; Gokhan Tur; Heng Ji

Source-linked AI summary

ToolRL: Reward is All Tool Learning Needs

Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji

arXiv:2504.13958v1Apr 16, 2025cs.LGcs.AIcs.CL

TL;DR

Tool-integrated reasoning poses an open reward-design challenge because multi-step interactions may invoke multiple tools with carefully specified parameters. The paper systematically analyzes reward design, proposes an algorithm-agnostic framework, and applies it with GRPO, reporting stronger benchmark performance and generalization than base and SFT models.

Problem
Tool-integrated reasoning poses an open reward-design challenge because multi-step interactions may invoke multiple tools with carefully specified parameters.
Method
The paper systematically analyzes reward type, scale, granularity, and dynamics, then proposes an algorithm-agnostic framework tailored for tool-integrated reasoning and demonstrates it with GRPO and PPO.
Results
17% improvement over base models and 15% over SFT models is reported across multiple tool-use and question-answering benchmarks, alongside generalization to unseen scenarios.
Takeaways & Limitations
The results support reward design as a practical route to robust tool selection, application, proactivity, and metacognitive behavior.
Takeaways & Limitations
The step-wise reward formulation assumes training data contains ground-truth tool calls at each step, although the method is later shown to generalize to free-form tool calls.

Abstract

from arXiv · show

Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.

1 Introduction

The paper frames reward design as the central challenge in reinforcement-learning-based tool-integrated reasoning, where multi-step tool interactions require fine-grained guidance. It systematically studies reward strategies and reports improved performance and generalization from a principled design.

Tool-Integrated Reasoning requires models to select tools, interpret intermediate outputs, and adapt trajectories across multi-step interactions.
SFT struggles with generalization, exploration, and adaptability, sometimes over-interpreting tools instead of rejecting inappropriate calls.
17% improvement over base models and 15% over SFT models is reported across multiple tool-use and question-answering benchmarks, with generalization to unseen scenarios.
Reward design is studied across reward type, scale, granularity, and dynamics to identify strategies suited to agentic tool use.
Longer reasoning traces are not inherently better, and length rewards can degrade performance.
Dynamic reward scales support transitions from simple to complex behaviors, while fine-grained decomposition improves learning stability and effectiveness.

2 Related Work

Related work has established tool-integrated reasoning and increasingly explored reinforcement learning for language-model tool use. Existing RL studies remain narrower than this paper’s focus on general-purpose tool selection and application across diverse tools and tasks.

Tool-Integrated Reasoning of LLMs: Tool-integrated reasoning equips LLMs with external tools and benchmarks capabilities including API selection, argument generation, and generalization.
Exploration of RL in LLMs: Earlier tool-use training primarily relied on supervised fine-tuning, while reinforcement learning has emerged as a more scalable and generalizable paradigm.
Exploration of RL in LLMs: GRPO replaces the traditional critic with group-based evaluation and has been applied to mathematics, search interaction, and code generation.
Exploration of RL in LLMs: Prior RL tool-use work focuses narrowly on search tools for question answering or code tools for mathematical problem solving.

3 Method

The method formulates Tool-Integrated Reasoning as sequential tool selection and application, then trains an LLM with GRPO using structured format and fine-grained correctness rewards. Rollouts preserve dialogue history, while group-wise advantage normalization supports stable optimization toward structurally valid and semantically accurate tool calls.

3.1 Task Definition: TIR trajectories contain reasoning, invoked tools, and observations across multiple steps, with each next action selecting parameterized tools to advance the user query.The policy maps the current trajectory to the next reasoning step and tool set.
3.2 TIR Rollout: The rollout prompt elicits reasoning traces and tool calls, parses invocations and responses, and appends tool outputs to the dialogue history.Special tokens mark thoughts, tool calls, and responses; tool outputs are inserted into an observation field.
3.3 Reward Design: The final reward separates binary format compliance from correctness, evaluating required token order and the accuracy of predicted tool calls.Correctness compares predicted and ground-truth tool names, parameter names, and parameter contents.
3.3 Reward Design: Fine-grained correctness scoring captures tool names, parameter schemas, and parameter values instead of reducing tool-call quality to a binary criterion.The design targets the interdependent structure of real-world tool calls.
3.3 Reward Design: The reward combines structural adherence with semantic tool-call accuracy, guiding outputs that are syntactically valid and faithful for downstream execution.The total reward is the sum of format and correctness components.
3.4 RL Training with GRPO: GRPO normalizes rewards within response groups for each query and optimizes the policy with a clipped PPO objective using the resulting advantages.Group-wise normalization is intended to reduce reward variance and stabilize, improve sample-efficient alignment.

4 Experiments

Experiments evaluate the reward design across diverse tool-use benchmarks, model initializations, and RL strategies. GRPO generally delivers the strongest performance and generalizes to unseen scenarios and task goals.

Experiment Setup: The RL dataset combines ToolACE, masked Hammer, and xLAM examples spanning diverse tool-use scenarios and complexity levels.Multi-step trajectories are decomposed into single-step instances while preserving prior dialogue history in prompts.
Evaluation: Evaluation covers BFCL’s single- and multi-step tool-use challenges, API-Bank’s 73 tools across three difficulty levels, and Bamboogle multi-hop QA.Bamboogle measures answer accuracy and tool-call counts rather than tool-use correctness alone.
Main Results: GRPO trained from scratch on Qwen2.5-Instruct generally outperforms baselines on BFCL and API-Bank, with approximately 10% absolute gains over same-volume SFT.LLaMA-3.2-Instruct improves less but remains competitive and outperforms most baselines on API-Bank.
SFT Initialization Impacts: Limited-SFT initialization can outperform full-scale SFT trained on ten times more data, but still underperforms cold-start GRPO.The comparison indicates that more SFT data does not consistently improve subsequent RL initialization.
SFT Initialization Impacts: SFT-initialized models obtain higher training rewards yet generalize worse on the two benchmarks, showing that training reward does not necessarily predict generalization.The authors hypothesize that memorization and overfitting reduce GRPO’s generalization impact.
Reward Design on PPO: GRPO achieves higher correctness rewards and faster format-reward gains than PPO, while PPO benefits more from SFT initialization and is less stable across settings.The reward design is partially effective for PPO but produces more robust and consistent improvements with GRPO.
Generalization Studies: Cold-start GRPO achieves the highest performance on unseen programming-language scenarios and novel irrelevant-tool detection goals.Qualitative examples show proactive clarification of ambiguous intent and direct answering without inappropriate tool use.
Free-form Inference Effectiveness: On Bamboogle, the reward design achieves the highest performance without excessive tool calls despite the setting being absent from training.The result indicates flexible tool invocation and efficient navigation toward correct answers in a free-form QA setting.

5 Analysis

The analysis evaluates reward length, scale, and granularity for tool calling, finding that gradual, fine-grained reward design is more effective than longer reasoning, abrupt scale changes, or coarse signals.

Effect of Length Reward: The length-based reward targets the <think> segment, using an empirically selected target output length of 512.The length component is added alongside format and correctness rewards; dynamic variants increase the target thinking length with training progress.
Effect of Length Reward: Length rewards increase reasoning traces but do not consistently improve performance and may harm smaller models.Dynamic length rewards also failed to improve performance, supporting the view that longer reasoning is not inherently better for tool use.
Effect of Reward Scale: Equalizing correctness and format reward scales causes a slight accuracy drop across most models, except Qwen2.5-3B.These results support assigning greater weight to correctness reward for robust tool-use generalization.
Effect of Reward Scale: Gradual reward-scale adjustment improves benchmarking performance, whereas a two-stage coarse transition reduces it.The dynamic strategy shifts emphasis from format adherence toward correctness as training progresses, with the switching point empirically set to 30 steps.
Effect of Reward Granularity: Fine-grained reward decomposition produces better training outcomes and higher final task performance than increasingly coarse alternatives.As granularity becomes coarser, reward values become harder to increase, suggesting sparser signals and more difficult credit assignment.

6 Conclusion

The paper presents a GRPO-oriented reward design for tool-use tasks and reports consistent outperformance across held-out benchmarks. It also reports generalization to QA settings with robust multi-turn and agentic behaviors.

Conclusion: The proposed reward design for GRPO consistently outperforms SFT-based, SFT-initialized RL, and alternative RL baselines across held-out tool-use benchmarks.The evaluation covers a variety of benchmarks and model-training baselines.
Conclusion: The model generalizes to QA settings while exhibiting robust multi-turn interactions, proactiveness, and metacognitive behaviors.The paper presents these behaviors as traits relevant to efficient and adaptable tool use.
Conclusion: The analysis of reward types, scaling, granularity, and temporal dynamics provides a roadmap for future tool-use RL research.The conclusion connects the empirical reward analysis to broader research on reward shaping and learning behavior.

A User Prompt Details

The user prompt stores the trajectory history used by the system, including intermediate thoughts, tool calls, environment observations, and additional user commands.

A User Prompt Details: The user prompt stores trajectory history, including intermediate thoughts, tool calls, environment observations, and additional user commands.The complete user instruction is presented separately in Figure 12.

B Experiment Details

The experiments use fixed training and SFT data budgets, JSON tool-call representations, and separate GRPO and PPO configurations. SFT supplies format-oriented initialization, while GRPO computes rewards from tool calls without requiring ground-truth thoughts.

Training Data Details: The RL training set contains 4K examples, with additional same-distribution data showing diminishing returns for convergence and final performance.The authors report that increasing the dataset beyond 4K does not noticeably improve results under this setting.
GRPO Setting Details: Tool calls are represented in JSON format, and each GRPO run uses two A100 80G GPUs.The passage introduces the associated GRPO hyper-parameter configuration in Table 8.
Baselines: SFT uses 400 examples with distilled <think> content, whereas GRPO uses tool calls alone to compute rewards.The SFT examples share the RL data distribution but differ in content.
Baselines: SFT initialization helps the raw model learn tool-call format but can overfit when used alone.The authors report that overfitting may ultimately degrade performance.
PPO Setting Details: PPO uses approximately the same parameter settings as GRPO and also runs on two A100 80G GPUs.The associated PPO hyper-parameter configuration is listed in Table 9.

C Additional Results

Additional experiments compare RL initialization from models fine-tuned on 400 versus 4K SFT data points across BFCL, API-Bank, and Bamboogle. The results indicate that more SFT data does not consistently improve RL outcomes and may sometimes hurt held-out-task generalization.

Experimental Setting: Additional evaluations apply GRPO and PPO to SFT-initialized models across BFCL, API-Bank, and Bamboogle.The experiments compare models initialized from 400 or 4K SFT data points; the 4K setting reuses the same data for SFT and RL as a theoretical upper bound.
Experimental Setting: The experiments compare RL training initialized with models fine-tuned on either 400 or 4K SFT data points.Results are reported in Tables 10–12 for BFCL, API-Bank, and Bamboogle, respectively.
Findings: 4K SFT initialization does not consistently outperform initialization from only 400 SFT data points.This finding holds across the additional benchmark results rather than establishing a uniform advantage for the larger SFT initialization set.
Findings: In BFCL, some performance drops below the raw instruct model, possibly because SFT overfitting restricts RL exploration and worsens held-out-task generalization.The proposed explanation is presented as a possible cause rather than a demonstrated mechanism.