PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch

Yirong Zeng, Xiao Ding, Yutai Hou, Yuxian Wang, Li Du, Juyi Dai, Qiuyang Ding, Duyu Tang, Dandan Tu, Weiwen Liu, Bing Qin, Ting Liu

arXiv:2511.01934v2cs.LGcs.AI

TL;DR

Tool-use models often struggle to generalize beyond familiar scenarios because supervised fine-tuning can favor memorized surface patterns over intrinsic reasoning. Tool-Zero applies pure rule-based reinforcement learning with progressively shifting exploration-to-exploitation rewards, and improves performance by 7.14% over ToolACE-8B while consistently generalizing across cross-dataset and intra-dataset evaluations.

  • Problem

    Supervised fine-tuning can encourage trajectory imitation and memorization, limiting intrinsic reasoning and generalization to unseen tool-use scenarios.

  • Method

    Tool-Zero uses pure rule-based reinforcement learning with a dynamic reward strategy that progressively shifts from exploratory behavior to precise tool-use patterns.

  • Results

    7.14% improvement over SFT model ToolACE-8B, with gains consistently replicated across cross-dataset and intra-dataset evaluations.

  • Takeaways & Limitations

    Across diverse benchmarks, Tool-Zero models outperform both SFT and RL-with-SFT baselines, supporting pure RL for scalable, autonomous tool learning.

  • Takeaways & Limitations

    Progressive reward switching adds computational costs during RL training, limiting scalability on resource-constrained hardware without further optimization.

Abstract

from arXiv · show

Training tool-augmented LLMs has emerged as a promising approach to enhancing language models' capabilities for complex tasks. The current supervised fine-tuning paradigm relies on constructing extensive domain-specific datasets to train models. However, this approach often struggles to generalize effectively to unfamiliar or intricate tool-use scenarios. Recently, reinforcement learning (RL) paradigm can endow LLMs with superior reasoning and generalization abilities. In this work, we address a key question: Can the pure RL be used to effectively elicit a model's intrinsic reasoning capabilities and enhance the tool-agnostic generalization? We propose a dynamic generalization-guided reward design for rule-based RL, which progressively shifts rewards from exploratory to exploitative tool-use patterns. Based on this design, we introduce the Tool-Zero series models. These models are trained to enable LLMs to autonomously utilize general tools by directly scaling up RL from Zero models (i.e., base models without post-training). Experimental results demonstrate that our models achieve over 7% performance improvement compared to both SFT and RL-with-SFT models under the same experimental settings. These gains are consistently replicated across cross-dataset and intra-dataset evaluations, validating the effectiveness and robustness of our methods.

1 Introduction

Tool augmentation expands LLM capabilities for complex tasks, but SFT-trained models often fail to generalize beyond familiar tool-use patterns. The paper addresses this limitation with pure RL and dynamic generalization-guided rewards, yielding substantial gains over SFT and RL-with-SFT baselines.

  • Background: Tool augmentation lets LLMs invoke and execute external tools, including APIs, functions, and plugins, enabling complex-task automation and agent applications.The introduction positions tool integration as useful across practical domains, including reasoning-task automation and agent systems.
  • Motivation: SFT-trained models perform well on familiar tool-use distributions but mainly imitate surface patterns, limiting generalization to unfamiliar scenarios.Figure 1 illustrates this limitation through a model that handles natural-language translation but fails to invoke tools for code transpilation.
  • Method motivation: The paper extends rule-based R1-style RL without SFT to tool learning, aiming to elicit intrinsic reasoning abilities for better generalization.Prior studies found that simple rule-based RL can improve complex reasoning, motivating its application to tool-use tasks.
  • Empirical results: 7.14% improvement over SFT ToolACE-8B and 7.18% over RL-with-SFT ToolRL-7B show Tool-Zero-7B’s stronger function-calling performance.The reported gains were observed across diverse function-calling benchmarks; Tool-Zero 7B/32B models significantly outperformed both baseline types.

2 Related Work

Prior work equips LLMs with external tools for complex open-domain tasks, typically through imitation learning on large supervised tool-use datasets. RL offers a more scalable route to reasoning, but existing tool-integrated approaches remain limited by SFT-then-RL training and single-type tool use.

  • Tool learning: External tools such as search engines, calculators, and Python interpreters are increasingly integrated with LLMs to address complex open-domain tasks (Qu et al., 2025; Wang et al., 2024).
  • Tool learning: Imitation learning remains the dominant tool-use paradigm, relying on large-scale supervised datasets labeled for tool-use behavior (Prabhakar et al., 2025; Liu et al., 2024b).
  • RL for reasoning: RL-based methods such as R1-Zero use GRPO (Shao et al., 2024) to unlock reasoning at test time, with successes in mathematics and coding (DeepSeek-AI, 2025; Yu et al., 2025; Pan and Liu, 2025).This paradigm shifts emphasis from train-time scaling to test-time scaling (Muennighoff et al., 2025; Xia et al., 2025).
  • Tool-integrated reasoning: Tool-integrated methods such as Torl (Li et al., 2025) and ReTool (Feng et al., 2025) achieve promising mathematical performance with code tools but use SFT-then-RL and remain limited to single-type tool use.

3 Problem Statement and Analysis

The section formalizes tool-augmented reasoning as action–observation trajectories governed by a policy that selects tools and invocation parameters. Preliminary studies show that SFT improves in-distribution accuracy but weakens generalization to unseen tools, formats, and benchmarks, motivating pure RL that balances exploration and exploitation.

  • Problem Formulation: Tool-augmented reasoning is modeled as a trajectory of action–observation pairs, where each action contains natural-language reasoning and a selected subset of tools.The trajectory is τk = [a1(c1), o1], [a2(c2), o2], . . . , [ak(ck), ok].
  • Problem Formulation: At each step, the policy generates the next reasoning action, selects a tool subset, and formulates parameterized invocations to address the user query.The generalized policy maps τk to ak+1(ck+1) and produces a sequence of action–observation pairs.
  • Preliminary Studies: SFT improves in-distribution tool-use accuracy but weakens generalization to unseen tools, invocation formats, and scenarios across benchmarks.ToolACE-8B and xLAM-7B-r improved Live performance from 61.1 to 78.6 but showed negligible gains or regressions on Non-Live and Multi-Turn subsets, including 9.6 → 7.8; xLAM-7B-fc led BFCL yet had the lowest overall average across benchmarks.
  • Motivation: The proposed response is a pure RL framework that dynamically balances exploration of new tool interactions with exploitation of task-relevant patterns.This directly addresses the trade-off identified in the SFT analysis.

4 Method

The method trains Tool-Zero models with GRPO and a dynamic generalization-guided reward that shifts from exploratory partial credit to strict tool-use correctness. It combines curated and augmented tool-use data, name masking, format validation, and adaptive rewards to improve tool-agnostic reasoning.

  • Training data and preprocessing: Training uses ToolACE and xLAM for tool-selection, compositional, and multi-step interaction learning, with xLAM augmented through dialog combination, tool removal, parameter clarification, and result validation.The data pipeline also filters invalid tool calls and candidate-tool formats before GRPO training.
  • Training data and preprocessing: Function and parameter name masking redirects attention from superficial naming patterns to tool descriptions and argument semantics, improving tool-agnostic reasoning in open-domain settings.Examples include mapping calculate_sum to func_1 and input_list to param_1; the strategy follows Lin et al., 2024.
  • Dynamic generalization-guided reward: GG-GRPO decomposes reward into format correctness and a dynamic tool reward that progresses from broad exploration to precise, structured tool-use behavior.The approach addresses the exploration–exploitation dilemma by first eliciting generalization and then enforcing task-specific correctness.
  • Dynamic generalization-guided reward: The strict reward checks AST-level structural and semantic validity, reinforces multi-tool collaboration with +0.3 reward, and penalizes each invalid parameter value by +0.3.AST comparison verifies API argument validity and multi-tool dependency chains, while the parameter-value signal encourages precise context grounding (Zeng et al., 2025b; Lin et al., 2024).
  • Dynamic generalization-guided reward: A sigmoid-weighted switch blends the general and strict rewards over training, avoiding abrupt reward changes that could destabilize optimization.The transition uses training step t, midpoint m, and steepness κ to move automatically from rgeneral toward rstrict.
  • GRPO optimization: GRPO trains from foundation models without a critic by estimating baselines from group scores, while GG-GRPO replaces rule-based accuracy rewards with adaptive generalization-guided computation (DeepSeek-AI, 2025; Shao et al., 2024).The method removes KL-divergence regularization following Yu et al., 2025.

5 Experiments

Experiments show that Tool-Zero models achieve strong, robust tool-use performance across benchmarks, with pure RL outperforming SFT-then-RL and GG-GRPO benefiting from its progressive, tool-specific reward design. Ablations further show that reward components, gradual shaping, and Base-model training materially affect performance and generalization.

  • Results on BFCL: Tool-Zero models outperform other models across all BFCL metrics, including Live and Multi-Turn evaluations.SFT models perform well on in-distribution Live data but generalize poorly to out-of-distribution metrics such as Multi-Turn; Tool-Zero-7B improves by 13.32 on Live and 19.32 on Multi-Turn.
  • Results on BFCL: Pure RL outperforms SFT-then-RL among R1-like models, with Tool-Zero-7B surpassing DS-R1 by +8.33 and ToolRL-7B by +6.84.The results indicate that RL better elicits intrinsic reasoning from Zero models, while GRPO-trained models are comparable on Live and stronger on Non-live and multi-turn tasks than SFT models.
  • Results on More Benchmarks: Across additional benchmarks, Tool-Zero models show more stable performance than SFT and R1-like models, demonstrating robust cross-benchmark generalization.SFT performance is inconsistent, whereas Hammer2.1-7b is relatively consistent and GPT-4o performs best overall; the supplied Table 4 caption also describes consistent Tool-Zero performance across multiple benchmarks.
  • Ablation Study: GG-GRPO improves performance by +5.26 over GRPO and +6.8 over SFT, with multi-tool and value-error signals contributing more than call-pattern signals and the tool mask.The ablation evaluates SFT and pure GRPO using the same ToolACE data and finds that all GG-GRPO components contribute significantly.
  • Hyperparameter Analysis: A progressive-reward midpoint of tm = 25 performs best, while tm ≥50 degrades performance and lower steepness consistently improves training stability.The results attribute degradation from excessive exploration to possible reward hacking and superficial-cue overfitting, supporting gradual reward shaping.
  • Data and Backbone Generalizability: Training with Base models consistently outperforms training with Instruct models across datasets, showing that stronger instruction following does not necessarily improve tool-use training benefits.The analysis covers different datasets and backbone language models.

6 Conclusion

This study extends pure rule-based RL for tool-augmented models with a dynamic generalization-guided reward and progressive exploration–exploitation strategy. Tool-Zero reduces reliance on task-specific data, improves tool-agnostic adaptability, and outperforms SFT and RL-with-SFT baselines across diverse benchmarks.

  • 6 Conclusion: Tool-Zero uses dynamic generalization-guided rewards and progressive exploration–exploitation to foster intrinsic reasoning while reducing reliance on task-specific data.The approach is designed to address generalization limitations in tool-augmented models and enhance tool-agnostic adaptability.
  • 6 Conclusion: Across diverse benchmarks, Tool-Zero models outperform both SFT and RL-with-SFT baselines.These results support the potential of pure rule-based RL for scalable tool-augmented modeling.

7 Limitation

The study identifies computational and evaluation-dataset limitations that motivate future work. The progressive reward-switching strategy increases RL training costs, while BFCL and API-Bank have known limitations.

  • Computational limitations: Progressive reward switching improves generalization but adds RL training costs, especially for large models such as Tool-Zero-32B.This additional computational burden limits scalability on resource-constrained hardware without further optimization.
  • Evaluation limitations: The evaluation datasets, including BFCL and API-Bank, have known limitations that should be addressed in future work.

A Details of Experimental Setup · A.1 The Implementation Settings

The experiments used MindSpeed-RL3 with QwQ-32B and API-based DeepSeek-R1 baselines, alongside specified exploration–exploitation settings. Tool calls used a hybrid JSON–Python format, while GG-GRPO training ran on Ascend 910b NPUs.

  • A.1 The Implementation Settings: The experiments used the publicly accessible MindSpeed-RL3 framework, an end-to-end reinforcement-learning accelerator based on the Ascend ecosystem.
  • A.1 The Implementation Settings: Baselines used QwQ-32B and DeepSeek-R1 with thinking inference enabled through Alibaba Cloud API calls.
  • A.1 The Implementation Settings: The implementation set κ = 0.1 as the temperature parameter governing the exploration–exploitation trade-off.
  • A.1 The Implementation Settings: The reward schedule used a transition midpoint of 25 to define its inflection point.
  • A.1 The Implementation Settings: Tool calls in the dataset were encoded using a hybrid format combining JSON structure with Python code snippets.
  • A.1 The Implementation Settings: GG-GRPO training used 5*8 Ascend 910b NPUs per run and could be completed within 28 hours.
  • A.1 The Implementation Settings: The RL-training configurations for GG-GRPO are provided in Table 6.

B Benchmark & Metric Details. · C System Thinking Template

The paper evaluates tool use across diverse benchmarks and finds substantial cross-dataset tool variability, motivating dynamic retrieval and generalizable tool-use learning. It also uses a lightweight thinking template that separates reasoning from answers to support adaptation across tool-augmented tasks.

  • B Benchmark & Metric Details.: The evaluation covers BFCL, API-Bank, Nexus Raven, and Tool-Alpaca, spanning known API invocation, candidate retrieval, and function-calling tasks.API-Bank contains 314 dialogues and 753 API calls, while Nexus Raven contains 318 examples across 65 APIs.
  • B Benchmark & Metric Details.: The dynamic-tool analysis tests whether cross-dataset candidate-tool variability is sufficient to require retrieval that scales and adapts to unseen tools.The analysis examines tool distributions and cross-dataset overlap to justify the method’s scalability and adaptability design.
  • B Benchmark & Metric Details.: The overlap rate is computed as the intersection size of two toolsets divided by the size of the smaller toolset.For toolsets A and B, the formula uses |A ∩ B| and min(|A|, |B|).
  • B Benchmark & Metric Details.: Toolsets vary substantially across datasets, so tools relevant to one benchmark are often irrelevant to another, supporting dynamic retrieval for unseen scenarios.The findings favor learning generalizable tool-use patterns rather than memorizing fixed tool identities.
  • C System Thinking Template: The system prompt instructs the model to place intermediate reasoning inside <think>...</think> tags and the final response inside <answer>...</answer> tags.This lightweight schema is designed to elicit tool-use capabilities while explicitly separating reasoning from the answer.
  • B Benchmark & Metric Details.: 47.1% is the maximum toolset overlap between ToolACE and SealTool, while overlap reaches 0% between xLAM and API-Bank.Table 8 reports overlap rates ranging from 0% to 47.1%, indicating minimal-to-nonexistent overlap across many dataset pairs.
  • C System Thinking Template: Allowing freer reasoning articulation is intended to improve generalization across diverse tool integrations and adaptation to complex tool-augmented reasoning tasks.The template draws inspiration from prior work by DeepSeek-AI (2025) and Face (2025).

D Model Scale Analysis

The study evaluates Tool-Zero across base-model scales and compares its reward module under PPO and GRPO. Performance is strongest at 3B and 7B scales, while GRPO outperforms PPO with robust, consistent gains.

  • Model Scale Analysis: Tool-Zero performed strongly with 3B and 7B base models across datasets including BFCL-v3 and API-Bank.The scalability experiment evaluated 1.5B, 3B, 7B, and 32B base models.
  • Algorithmic Effectiveness: The reward module with GRPO outperformed native PPO (Song et al., 2025), demonstrating robust and consistent gains.The comparison is reported as GG-GRPO versus GG-PPO in Table 9.
  • Algorithmic Effectiveness: The algorithmic experiment evaluates whether the reward function transfers across native PPO and GRPO implementations.The compared variants are GG-PPO and GG-GRPO.
Loading 2511.01934v2…