PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, Guilin Liu

arXiv:2505.00024v2cs.CLcs.AI

TL;DR

Tool-calling research has largely relied on SFT over distilled trajectories, motivating methods that improve reasoning without annotated intermediate traces. Tool-N1 applies rule-based RL with binary rewards for output format and tool-call correctness, and its models outperform strong baselines across several benchmarks. The study also finds that SFT-then-RL does not necessarily outperform pure RL.

  • Problem

    Existing tool-calling approaches primarily use SFT on distilled trajectories, which provides limited supervision for the underlying reasoning process.

  • Method

    Tool-N1 uses rule-based reinforcement learning with binary rewards for reasoning-format validity and functional tool-call correctness, without requiring curated reasoning trajectories.

  • Results

    Tool-N1 consistently outperforms existing baselines across BFCL, APIBank, and ACEBench, including GPT-4o comparisons.

  • Takeaways & Limitations

    Pure RL can be a stronger tool-calling training strategy than combinations involving SFT, including the commonly used SFT-then-RL pipeline.

Abstract

from arXiv · show

Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.

1. Introduction

Tool-N1 explores rule-based reinforcement learning for tool-calling models, replacing supervision of distilled reasoning traces with rewards for reasoning format and functional tool-call correctness. Across major benchmarks, the resulting models outperform strong baselines, while pure RL need not be improved by adding SFT first.

  • Motivation: External tools extend LLMs beyond purely textual tasks, supporting applications such as search-based question answering and Python-assisted mathematical problem solving.These capabilities motivate improving models’ ability to use tools reliably.
  • Research gap: Existing tool-calling pipelines mainly synthesize trajectories with advanced models and apply SFT, often supervising tool calls without explicit guidance for underlying reasoning.Reasoning may instead be omitted during training or deferred to inference-time prompting.
  • Approach: Tool-N1 trains tool-calling models with rule-based RL that enforces structured reasoning before tool invocation and uses binary rewards for format and tool-call correctness.The approach allows variation in argument ordering while requiring functional correctness.
  • Results: Tool-N1-14B improves over GPT-4o by approximately 2% on BFCL and 5% on APIBank, while Tool-N1-7B surpasses GPT-4o and Hammer2.1-7B on BFCL by 0.85% and 2.97%, respectively.The reported evaluations cover BFCL, APIBank, and ACEBench.
  • Training-strategy study: The widely adopted SFT-then-RL pipeline does not necessarily outperform pure RL in tool-calling tasks.The study compares SFT, RL, and their combinations using 5,518 distilled reasoning trajectories.

2. Related Work

Related work positions tool learning as a way to extend LLM capabilities through external interfaces and describes a shift toward reinforcement learning for improving complex reasoning.

  • Tool Learning: Tool-learning research integrates LLMs with external systems such as search engines, calculators, vision tools, and Python interpreters.These integrations support complex tasks beyond ordinary text generation.
  • LLM Reasoning and Reinforcement Learning: LLM reasoning research has shifted toward test-time scaling and increasingly uses rule-based reinforcement learning to improve complex problem solving.Earlier approaches often relied on step-level supervision or learned reward models.

3. Problem Formulation

The problem formulation models tool use as sequential decision-making: at each step, an LLM uses prior tool interactions and currently available tools to select one or more parameterized calls and receive observations.

  • Tool representation: Each external tool is represented by its name, natural-language description, and input-parameter instructions.These components define the information available for tool usage.
  • Decision process: At each decision step, the LLM receives historical tool-call and observation context together with the currently available tool set.It must then select the next action under policy π.
  • Actions and observations: An action may contain one or more calls to accessible tools, each paired with its corresponding parameters.The interaction records an observation after the action is taken.
  • Objective: The objective is a generalized policy that addresses user queries by producing a sequence of action-observation pairs.This sequence captures iterative interaction with external tools.

4. Nemotron-Research-Tool-N1

Tool-N1 trains tool-using reasoning LLMs with GRPO and structured tool-call data, combining explicit reasoning-format checks with binary functional rewards. The pipeline cleans trajectories, prompts reasoning before invocation, and verifies executable calls through structured matching.

  • Nemotron-Research-Tool-N1: Tool-N1 applies GRPO to train general-purpose tool-using reasoning LLMs for queries requiring available external tools.The model generates candidate responses containing textual reasoning and an associated action, then optimizes its policy using reward-based GRPO.
  • Data Preparation: Training data is cleaned by removing invalid tool calls, unparsable JSON, and formatting inconsistencies before reinforcement learning.Multi-turn ToolACE trajectories are additionally segmented into single-step prediction instances.
  • Thinking Template: The prompting template separates intermediate reasoning in <think> tags from tool invocations in <tool_call> tags.The lightweight template is intended to avoid overly rigid formatting and reduce reliance on specific prompt patterns.
  • Reward Modeling: Ground-truth tool calls are represented as structured dictionaries so training can verify tool names and argument-value pairs rather than string identity.Dictionary-based checking permits argument-order variation while preserving functional matching.
  • Reward Modeling: The binary reward is 1 only when both required reasoning tags are present and the predicted tool call matches the ground truth in tool name and argument key-value pairs.The reward jointly evaluates format correctness and tool-call correctness.
  • Reward Modeling: Ablation studies find that the simple binary reward scheme is most effective for training tool-calling LLMs.The paper attributes this finding to extensive ablations in Section 5.4.1.

5. Experiments

Experiments evaluate Tool-N1 across tool-calling benchmarks, model scales, backbones, reward designs, and training-data compositions. The results show strong benchmark performance, effective scaling and generalization, and advantages for pure rule-based RL with binary rewards and structured reasoning.

  • Main Results: Tool-N1-7B and Tool-N1-14B outperform GPT-4o and supervised fine-tuning baselines on BFCL.The models also surpass specialized tool-calling systems, including ToolACE and xLAM series models.
  • Main Results: Tool-N1-7B improves over its base model by more than 20% on APIBank and more than 30% on ACEBench, while Tool-N1-14B surpasses GPT-4o by 5.03%.Tool-N1-7B also outperforms GPT-4o-mini on both additional benchmarks.
  • Scalability and Generalizability: At comparable model scales, Qwen models outperform both LLaMA 3 and LLaMA 3.1 variants on BFCL.This evaluates the proposed training approach across alternative backbone families.
  • Scalability and Generalizability: R1-style training scales effectively with model size, producing greater performance gains for larger Qwen2.5-Instruct models than for smaller ones.The study evaluates 0.5B, 1.5B, 3B, 7B, and 14B models against their original instruction-tuned counterparts.
  • SFT or RL?: Pure RL achieves 83.24% under equal data budgets, while SFT-then-RL reaches 83.17%, so adding SFT does not improve performance.Pure RL also outperforms Reason-SFT and No-Reason SFT in the reported training-recipe comparison.
  • Ablations: Binary rewards outperform fine-grained rewards, especially on the Live subset, where performance is 80.38% versus 76.61%.Removing the reasoning-format constraint further reduces performance from 80.38% to 76.24%, indicating that structured reasoning is important for reliable tool use.

6. Conclusion

Nemotron-Research-Tool-N1 uses rule-based reinforcement learning to train tool-using language models without annotated reasoning trajectories. Across multiple benchmarks, it outperforms existing baselines, while analysis finds that SFT-then-RL does not necessarily beat pure RL.

  • Tool-N1 uses rule-based reinforcement learning that supervises final answers and reasoning structure rather than annotated step-by-step trajectories.
  • Tool-N1 consistently outperforms existing baselines across BFCL, APIBank, and ACEBench.
  • 5,518 distilled reasoning trajectories support comparisons of supervised fine-tuning, reinforcement learning, and their combination.
  • SFT-then-RL does not necessarily yield better performance than pure RL.

A. Benchmarks

The paper evaluates tool-calling ability with BFCL, ACEBench, and APIBank, using distinct subsets and execution settings across the benchmarks.

  • BFCL evaluates accurate function calling and includes Live, Non-Live, and Multi-turn categories; this paper focuses on Live and Non-Live data.
  • ACEBench categorizes tool-use evaluation into Normal, Special, and Agent; the study uses Normal-category Atom and Single-turn subsets.
  • APIBank evaluates intent interpretation and local Python tool execution in Call and Retrieve + Call modes; the experiments use Call mode.

B. More Implementation Details

RL experiments use the open-source Verl library and grid-search hyperparameter tuning, with identical training hyperparameters for Tool-N1-7B and Tool-N1-14B.

  • RL experiments use the open-source Verl library and grid search for hyperparameter tuning.
  • Tuning focuses on learning rate, KL coefficient, entropy coefficient, number of rollouts, and batch size.
  • Tool-N1-7B and Tool-N1-14B use the same set of training hyperparameters.

C. More Learning Curves

During training, both Tool-N1 models show rapidly increasing rewards and gradually decreasing entropy loss, indicating stable learning. Their response-length dynamics differ: Tool-N1-7B increases toward Tool-N1-14B, whose length remains relatively stable.

  • Reward increases rapidly and entropy loss decreases gradually for both Tool-N1-7B and Tool-N1-14B during training.
  • Tool-N1-7B’s minimum response length steadily increases and eventually converges to Tool-N1-14B’s value.
  • Tool-N1-14B maintains a relatively stable response length throughout training.
  • The authors interpret the response-length trends as greater reasoning-ability improvement for the smaller model.

D. Examples

The section presents examples of single- and multiple-function-calling queries, including a hair-stylist search request and its corresponding tool-call format.

  • A single-function example asks for a hair stylist in Lafayette, Louisiana and outputs a Services_1_FindProvider call with a city argument.The example shows the request, reasoning, and tool-call sections together.
  • The examples distinguish single-function calling from multiple-function calling scenarios.

E. Prompts

The prompts are kept identical across training, evaluation, and reasoning-trajectory distillation, and instruct the model how to select and format tool calls. The section also points to the full system prompts used during training and inference.

  • Identical prompts are used for model training, evaluation, and reasoning-trajectory distillation from DeepSeek-R1.
  • The prompt instructs the model to make one or more function calls when they serve the question’s purpose.
  • The prompt requires the model to flag unusable functions or missing required parameters and return calls only in tool-call sections.
  • Figure 8 provides the full system prompts used during model training and inference.
Loading 2505.00024v2…