PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, Guilin Liu

arXiv:2505.00024v2cs.CLcs.AI

TL;DR

Existing tool-use training often supervises tool-call steps without explicit reasoning guidance. Tool-N1 uses rule-based reinforcement learning with a binary reward for format and tool-call correctness, and its 7B/14B models outperform GPT-4o across benchmarks.

  • Problem

    Existing tool-use training often supervises tool-call steps from synthetic trajectories without explicit guidance on the underlying reasoning process.

    Source: §1 · p. 1

  • Method

    Tool-N1 uses rule-based reinforcement learning with a binary reward for reasoning-format validity and functional correctness of structured tool calls.

    Source: §4.3 · pp. 5–6

  • Results

    Across benchmarks, Tool-N1-7B/14B outperform GPT-4o; Tool-N1-14B achieves approximately 2% improvement on BFCL over GPT-4o.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The study finds that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.

    Source: §1 · p. 3

Abstract

from arXiv · show

Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.

1. Introduction

Nemotron-Research-Tool-N1 explores rule-based reinforcement learning for tool-calling models, supervising tool-call format and correctness without requiring curated reasoning traces. Its experiments compare Tool-N1 with strong baselines and examine how SFT and RL interact.

1. Introduction

  • External tools extend LLMs beyond text-only tasks, enabling capabilities such as answering questions beyond training data and solving complex math with Python libraries.Source: §1 · p. 1
  • Prior tool-use pipelines commonly fine-tune on synthetic trajectories without explicit reasoning, while distilling reasoning traces can produce pseudo-reasoning that imitates surface patterns.This may leave the model without guidance on the underlying decision process.Source: §1 · p. 1
  • Tool-N1 uses rule-based RL to elicit explicit reasoning before tool calls, with a binary reward evaluating reasoning format and tool-invocation correctness.The reward allows reordered arguments when tool calls remain functionally correct, and training does not require curated reasoning traces.Source: §1 · p. 2
  • Tool-N1-14B improves over GPT-4o by approximately 2% on BFCL and 5% on APIBank, while Tool-N1-7B surpasses GPT-4o and Hammer2.1-7B on BFCL by 0.85% and 2.97%, respectively.The models are built upon Qwen2.5-7B/14B-Instruct.Source: §1 · p. 2
  • A study using 5,518 tool-calling reasoning trajectories finds that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.Source: §1 · pp. 2–3

2. Related Work

Prior work improves tool use through data-dependent model training or prompting, while recent rule-based reinforcement learning offers a way to learn reasoning without explicit intermediate supervision. Tool-enabled RL studies have largely focused on single-tool reasoning, whereas this work targets broader settings with multiple tools and complex arguments.

Tool Learning

  • Tool-learning research either trains the backing LLM with curated datasets and SFT or DPO, or elicits tool use through prompting without changing the model (Liu et al., 2024; Zeng et al., 2025; Kim et al., 2023).Training approaches depend heavily on high-quality, large-scale data; prompting can struggle with intricate reasoning and complex multi-step actions because of base-model limitations.Source: §2 · p. 3

LLM Reasoning and Reinforcement Learning

3. Problem Formulation

The formulation represents accessible tools by their names, descriptions, and input instructions, and models tool use as a policy that answers queries through multiple calls. At each step, the model uses prior call-observation history and currently available tools to select actions and produce observations.

  • Each accessible tool is represented by its name, natural-language description, and input-parameter instructions, while the policy may issue multiple calls to answer a query.Source: §3 · p. 3
  • At each decision step, the model conditions on prior tool-call and observation pairs plus the currently available tools to choose its next action.Source: §3 · p. 3
  • An action comprises one or more tool calls with corresponding parameters; each action yields an observation, and the goal is a generalized policy that addresses queries through action-observation sequences.Source: §3 · p. 4

4. Nemotron-Research-Tool-N1

Nemotron-Research-Tool-N1 applies R1-style GRPO to tool-using reasoning models and studies rule-based reinforcement learning strategies. Its approach combines prepared tool-call data, a lightweight reasoning prompt, and binary rewards for output format and tool-call correctness.

4. Nemotron-Research-Tool-N1

  • Tool-N1 uses R1-style GRPO to train general-purpose tool-using reasoning models and systematically studies rule-based reinforcement learning strategies.The model generates candidate responses containing reasoning and an action, evaluates them with rewards, and optimizes its policy using GRPO.Source: §4 · p. 4

4.1. Data Preparation

  • The training data combines xLAM and a subset of ToolACE, filters invalid or malformed calls, and splits multi-turn ToolACE trajectories into single-step prediction instances.Available and ground-truth tools are parsed into structured dictionaries, and samples with invalid tool names, JSON parsing failures, or formatting inconsistencies are discarded.Source: §4.1 · pp. 4–5

4.2. Thinking Template

  • The prompt separates intermediate reasoning in <think> tags from tool invocations in <tool_call> tags while avoiding overly rigid formatting rules.The template is intended to reduce overfitting to prompt patterns and support integration with more sophisticated prompting strategies.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3. Reward Modeling

  • A binary reward is given only when the output uses both required tags and the predicted tool name and argument key-value pairs match the ground truth.Structured dictionary matching allows argument order to vary; the paper reports this simple binary scheme as most effective in ablations.Source: §4.3 · pp. 5–6

5. Experiments

Experiments find that Tool-N1 models achieve strong results across tool-calling benchmarks. The analyses examine training recipes, scaling, reward design, data composition, and training dynamics.

5.1. Settings

  • Experiments use ToolACE and xLAM training subsets and evaluate single-turn tool calling on BFCL, API-Bank, and ACEBench, primarily with Qwen2.5-7B/14B-Instruct.ToolACE covers multiple candidate tools and parallel calls across 26,507 tools (Liu et al., 2024), while xLAM comprises 60,000 APIGen instances focused on single-turn calling (Zhang et al., 2024). BFCL includes Live and Non-live subsets; API-Bank and ACEBench evaluations exclude multi-turn cases.Source: §5.1 · pp. 6–7

5.2. Main Results

  • Tool-N1-7B and Tool-N1-14B clearly outperform GPT-4o and specialized tool-calling models on BFCL, while R1-style training beats same-data SFT baselines.The reported comparisons include xLAM-2-70B and ToolACE-8B.Source: Table 2 · p. 7, §5.2 · p. 8
  • Over 20% on APIBank and over 30% on ACEBench are Tool-N1-7B’s absolute gains over base models; Tool-N1-14B surpasses GPT-4o by 5.03%.Tool-N1-7B also outperforms GPT-4o-mini on both benchmarks.Source: Figure 3 · p. 7, §5.2 · p. 8

5.3.1. Scalability and Generalizability

  • R1-style training yields greater performance gains for larger Qwen2.5 models than for 0.5B and 1.5B models, while Qwen outperforms LLaMA variants at comparable scales.The scaling comparison covers Qwen2.5-Instruct models from 0.5B to 14B on BFCL Live and Non-Live; the backbone comparison uses LLaMA 3 8B and LLaMA 3.1 8B.Source: Figure 4 · p. 8, §5.3.1 · pp. 8–9, Figure 5 · p. 8

5.3.2. SFT or RL? A Deep Investigation into the Training Recipes

  • 83.24% with 100% RL exceeds 82.71% with 100% Reason-SFT and 81.47% with 100% No-Reason SFT under equal data budgets.Reason-SFT+RL reaches 83.17% versus 83.24% for pure RL; the comparison uses 5,518 filtered reasoning trajectories from DeepSeek-R1 on Qwen2.5-Instruct.Source: Table 3 · p. 9, §5.3.2 · pp. 9–10

5.4. Visualization

  • KL divergence and validation accuracy improve during training without a significant increase in response length; Tool-N1-7B stays relatively flat while Tool-N1-14B’s length declines.The authors suggest that longer reasoning chains are not necessary for better tool-use performance and that a sufficient response length may support accurate tool calling.Source: Figure 6 · p. 10, §5.4 · p. 10

5.4.1. Ablations

  • 80.38% vs. 76.61% on the Live subset shows binary rewards outperform fine-grained rewards; removing the reasoning-format constraint lowers binary-reward performance to 76.24%.The study attributes the fine-grained reward shortfall to reward hacking on formatting or partial matches without execution correctness.Source: Table 4 · p. 10, §5.4.1 · p. 11
  • 6.36% and 1.62% are the respective margins by which Tool-N1-7B trained solely on xLAM and ToolACE subsets outperforms the corresponding SFT model.ToolACE data particularly improves the Live setting, and progressively enriching training data enhances tool-calling proficiency.Source: Table 5 · p. 11, §5.4.1 · p. 11

6. Conclusion

Nemotron-Research-Tool-N1 uses rule-based reinforcement learning with supervision on final answers and reasoning structure, and consistently outperforms baselines across multiple benchmarks. An analysis of 5,518 distilled reasoning trajectories finds that SFT-then-RL does not necessarily outperform pure RL.

6. Conclusion

  • Nemotron-Research-Tool-N1 consistently outperforms existing baselines across BFCL, APIBank, and ACEBench.The conclusion reports results across multiple benchmarks without specifying metric values.Source: §6 · p. 11
  • The models use rule-based reinforcement learning that supervises final answers and reasoning structure, enabling strategies to emerge without step-by-step annotations.This contrasts with approaches relying on supervised fine-tuning with annotated reasoning trajectories.Source: §6 · p. 11
  • Analysis of 5,518 distilled reasoning trajectories finds that SFT-then-RL does not necessarily yield better performance than pure RL.Source: §6 · p. 11

A. Benchmarks

The study evaluates tool use with BFCL, ACEBench, and API-Bank, focusing on selected benchmark categories or modes rather than their full scope.

BFCL

  • BFCL evaluates function-calling accuracy, and this study focuses on its Live and Non-Live categories.Live cases use real user-contributed queries; Non-Live cases are synthetic or researcher-curated.Source: A. Benchmarks · p. 16

ACEBench

  • On ACEBench, the study uses Atom and Single-turn subsets of Normal, covering specific parameter types and sequential or parallel tool calls.ACEBench also categorizes tool-use cases as Special and Agent, but the study concentrates on Normal.Source: A. Benchmarks · p. 16

API-Bank

  • On API-Bank, the study uses Call mode to test dialogue intent interpretation and local Python tool execution against expected outputs.API-Bank also has a Retrieve + Call mode.Source: A. Benchmarks · p. 16

B. More Implementation Details

The RL implementation uses Verl and grid-search tuning, with the same hyperparameters for Tool-N1-7B and Tool-N1-14B.

B. More Implementation Details

  • The authors tune learning rate, KL and entropy coefficients, rollout count, and batch size via grid search in Verl, using identical hyperparameters for both model sizes.The selected training hyperparameters are summarized in Table 6.Source: B. More Implementation Details · p. 16, Table 6 · p. 16

C. More Learning Curves

Training curves show rapidly increasing reward and gradually decreasing entropy loss for both models. Tool-N1-7B’s minimum response length rises to the value achieved by Tool-N1-14B, whose response length stays relatively stable.

C. More Learning Curves

  • Reward rises rapidly and entropy loss decreases gradually for both models, indicating stable learning.Figure 7 reports reward, entropy loss, and minimum response length across training steps.Source: Figure 7 · p. 16, C. More Learning Curves · p. 16
  • Tool-N1-7B’s minimum response length steadily increases to Tool-N1-14B’s value, while 14B’s remains relatively stable, suggesting greater reasoning improvement in the smaller model.The authors interpret the response-length trends as evidence that Tool-N1-7B undergoes greater improvements in reasoning ability than Tool-N1-14B.Source: Figure 7 · p. 16, C. More Learning Curves · pp. 16–17

D. Examples

The examples illustrate both a single function call for a provider search and multiple function calls for a multi-step drone-distance query.

Examples

  • A hair-stylist search in Lafayette, Louisiana, is shown as a single call to Services_1_FindProvider with the city parameter.The example’s tool call supplies the city as “Lafayette, LA.”Source: D. Examples · p. 17
  • A drone’s two successive moves are shown as a multiple-call example using math.hypot to calculate distances between points.The reasoning describes calculating each leg’s distance and summing them for the total traveled distance.Source: D. Examples · p. 17

E. Prompts

The paper uses identical prompts for model training, evaluation, and reasoning-trajectory distillation from DeepSeek-R1. Figure 8 presents the full system prompts used in training and inference.

E. Prompts

  • Identical prompts are used for training, evaluation, and DeepSeek-R1 reasoning-trajectory distillation; Figure 8 shows the full system prompts for training and inference.Source: E. Prompts · p. 17, Figure 8 · p. 17

Related papers

Loading 2505.00024v2…