PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolRM: Outcome Reward Models for Tool-Calling Large Language Models

Mayank Agarwal, Ibrahim Abdelaziz, Kinjal Basu, Merve Unuvar, Luis A. Lastras, Yara Rizk, Pavan Kapanipathi

arXiv:2509.11963v2cs.CL

TL;DR

Tool-use reward modeling is underexplored, and existing reward models often miss signals of tool-based reasoning and execution. The paper introduces FC-RewardBench and trains ToolRM on data synthesized from open-weight LLMs; ToolRM outperforms larger reward models and improves downstream results, including up to 25% average improvement in Best-of-n.

  • Problem

    Existing reward models are designed primarily for natural-language outputs, and no dedicated benchmark evaluates reward models on function-calling tasks.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces FC-RewardBench and trains 1.7B–14B ToolRM outcome reward models using preference data synthesized from permissively licensed, open-weight LLMs.

    Source: §1 · p. 2

  • Results

    ToolRM outperforms much larger reward models and LLMs-as-Judges on FC-RewardBench and achieves up to 25% average improvement across multiple benchmarks with Best-of-n sampling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ToolRM supports inference-time scaling, data-efficient fine-tuning, and policy optimization without ground-truth reward labels.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

As large language models (LLMs) increasingly interact with external tools, reward modeling for tool use has emerged as a critical yet underexplored area of research. Existing reward models, trained primarily on natural language outputs, struggle to evaluate tool-based reasoning and execution. To quantify this gap, we introduce FC-RewardBench, the first benchmark to systematically evaluate reward models in tool-calling scenarios. Our analysis shows that current reward models frequently miss key signals of effective tool use, highlighting the need for domain-specific modeling. We address this by proposing a training framework for outcome reward models using data synthesized from permissively licensed, open-weight LLMs. We introduce ToolRM - a suite of reward models for tool-use ranging from 1.7B to 14B parameters. Across diverse settings, these models consistently outperform general-purpose baselines. Notably, they achieve up to a 25% improvement with Best-of-N sampling, while also improving robustness to input noise, enabling effective data filtering, and supporting RL-training of policy models.

1 Introduction

Existing reward models often miss key signals of successful tool use, motivating a dedicated benchmark and reward models for tool-calling tasks. ToolRM outperforms much larger general-purpose models and supports gains in sampling, data-efficient fine-tuning, and policy training.

1 Introduction

  • FC-RewardBench is introduced as the first benchmark for tool-calling reward models, and the authors report that existing models often miss key signals of successful tool use.The benchmark is derived from BFCL Version 3 and contains 1,500 user inputs paired with correct and incorrect function calls.Source: §1 · pp. 1–2
  • ToolRM outperforms much larger reward models and LLMs-as-Judges on FC-RewardBench.The paper also reports downstream gains from Best-of-n sampling, data filtering, and RL policy training.Source: §1 · p. 2
  • The authors propose training tool-calling outcome reward models on preference data synthesized from permissively licensed, open-weight LLMs, and introduce ToolRM models from 1.7B to 14B parameters.The suite includes ToolRM-1.7B, ToolRM-7B, and ToolRM-14B.Source: §1 · p. 2
  • Up to 25% average improvement across multiple benchmarks is reported with Best-of-n; ToolRM-based filtering also yields stronger fine-tuned models with 50% of the data, and RL policies match models trained with ground-truth-dependent rewards.The RL-trained policies use ToolRM rewards rather than requiring ground-truth labels for reward computation.Source: §1 · p. 2

2 Related Work

Related work covers tool-calling methods and benchmarks, reinforcement learning for tool-use alignment, and reward modeling. The paper identifies tool-call sequences as an underexplored outcome-reward-modeling domain.

2.1 Tool Calling

  • Tool-calling research spans prompting, built-in and fine-tuned tool use, benchmarks, curated datasets, and autonomous tool construction.ReAct is cited as an early prompting-based approach (Yao et al., 2023).Source: §2.1 · p. 2

2.2 RL for Tool-Use Alignment

2.3 Reward Modeling

  • ORMs score final outputs while PRMs evaluate intermediate reasoning steps; the paper positions tool-call sequences as a new ORM domain.The related-work discussion contrasts prior free-text, math, and code settings with tool calling.Source: §2.3 · p. 3

3 Methodology

The methodology introduces FC-RewardBench for evaluating reward models on function-calling tasks, describes pairwise reward modeling, and generates incorrect tool calls for training outcome reward models.

3.1 FC-RewardBench Evaluation Dataset

  • FC-RewardBench contains 1,500 examples and emphasizes subtle tool-call errors that make reward models demonstrate deeper function-calling understanding.Its errors include incorrect parameter values, missing optional parameters, and incorrect numbers of functions.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Reward Modeling

  • The reward-modeling section estimates a neural reward function from preferred and nonpreferred output pairs using maximum likelihood and adds reward centering.The centering term penalizes the squared sum of rewards for each pair, with η as a small positive hyperparameter.Source: §3.2 · p. 4

3.3 ToolRM Training Data Generation

  • Training data is generated by prompting permissively licensed, open-source function-calling models and retaining calls that differ from ground-truth sequences.This process is intended to capture natural variability and error patterns, including subtle failure modes that are difficult to enumerate manually.Source: §3.3 · p. 4

4 Experimental Setup

The experiments use varied function-calling datasets and benchmarks, train reward models on synthesized incorrect outputs, and compare them with reward-model and LLM-judge baselines.

Training Data

  • Training data combines single-turn, multi-turn, and irrelevance examples, with incorrect outputs subsampled to one per user query, yielding 180K samples.The reported split is 85K single and multi-turn data each and 10K irrelevance data.Source: §4 · p. 4
  • Samples are obfuscated by replacing function and parameter names with random strings and reordering schema keys to reduce training-data regurgitation.The models generating data include 11 permissively licensed, open-weight LLMs ranging from 0.5B to 32B parameters.Source: §4 · p. 4

Benchmarks

  • Evaluation spans function-calling benchmarks with varied difficulty, turn structure, nested calls, and real-user or synthetic examples.For API-Bank, the Call and Retrieval+Call splits are included.Source: §4 · p. 5

Models and Baselines

  • FC-RewardBench comparisons include eight reward models sized 3B–70B and six LLM judges sized 70B–685B; downstream tests use xLAM-2 and Qwen3 models sized 0.6B–70B.ToolRM uses Qwen-2.5-Instruct 1.5B, 7B, and 14B variants as its base architecture, with a linear head producing a scalar reward.Source: §4 · pp. 4–5

5 Results

The experiments evaluate ToolRM against existing reward models, then test its use for Best-of-n sampling, data filtering, and policy optimization. ToolRM performs strongly across these settings, including tool-calling evaluation, robustness, and training-data efficiency.

5.1 RQ1: FC-RewardBench evaluation

  • On FC-RewardBench, Themis reaches 45% accuracy, while LLM-as-Judge accuracy exceeds 80% but has high computational cost; ToolRM-7B and 14B outperform other generative and sequential classifiers.ToolRM-1.5B also surpasses gpt-oss-120B and approaches much larger Llama-4 models.Source: §5.1 · p. 5
  • FC-RewardBench's average correlation of 0.84 with downstream Best-of-n scores across five benchmarks makes it a proxy for efficient reward-model evaluation.The reported correlations cover 11 reward models.Source: §5.1 · p. 5

5.2 RQ2: Best-of-n sampling with ToolRM

  • Qwen3-0.6B rises from 39.5% to 64.4% with Best-of-n, exceeding greedy Qwen3-32B (63.8%) and xLAM-2-70B (63.6%); Qwen3-1.7B gains 5.3 points in BFCL-v3 overall accuracy.Qwen3-8B reaches 70.5%, 5.6 points above all greedy baselines; BFCL-v3 gains are 9.6 and 8.7 points on Non-Live AST and Live AST.Source: §5.2 · p. 6, Figure 2 · p. 7, Table 2 · p. 7
  • On RoTBench, Qwen-8B Tool Selection rises from 52.4 to 72.4 on Clean and from 45.7 to 66.7 on Union with Best-of-n; Union@32 also exceeds Clean for Qwen-32B (72.4 vs. 65.7).The reported gains extend across Tool Selection, Parameter Identification, and Content Filling; improvements of 15–25 points are reported for the latter two tasks.Source: §5.2 · p. 6, Table 3 · p. 8

5.3.1 ToolRM for data filtering

  • ToolRM-14B-selected FT-Best-8K reaches 62.5% accuracy, above FT-16K at 61.0% while using half the corpus; random 8K reaches 58.4%.The base model starts at 54.0%; the 16K corpus contains single- and multi-turn examples with no overlap with ToolRM training data.Source: §5.3.1 · pp. 6–7, Table 4 · p. 8

5.3.2 Policy Optimization Using ToolRM

  • ToolRM-14B rewards match or exceed ground-truth-dependent RToolRL on BFCL-v3 across the two evaluated models, without requiring ground truth for reward computation.For Llama-3.2-3B-Instruct, RToolRM has the highest accuracy on both metrics; for Qwen2.5-3B-Instruct, it has the best Live AST accuracy.Source: §5.3.2 · p. 8, Table 5 · p. 8

6 Conclusion

The conclusion presents a framework for tool-calling reward modeling, including FC-RewardBench and outcome reward models that outperform larger existing models in this setting. It identifies inference-time scaling, data-efficient fine-tuning, and ground-truth-free policy optimization as supported uses, and outlines three directions for further work.

6 Conclusion

  • The paper presents FC-RewardBench for systematic assessment and a framework for outcome reward models that outperform significantly larger reward models in tool calling.The proposed framework is for reward modeling in tool-calling scenarios.Source: §6 · p. 8
  • ToolRM supports effective inference-time scaling, data-efficient fine-tuning, and ground-truth-free policy optimization..Source: §6 · p. 8
  • Future directions include generative verifiers with chain-of-thought, training with tool and environment state, and bridging outcome with process reward modeling.The paper suggests these directions may improve robustness and interpretability, support safe recovery from execution failures, and balance scalability with fine-grained reasoning control.Source: §6 · p. 8

A.1 FC-RewardBench benchmark details

The appendix points readers to the benchmark's model inventory and a representative example from its data.

A.1 FC-RewardBench benchmark details

  • Table 7 lists the models included in FC-RewardBench and the number of incorrect tool-call outputs per model.Source: §A.1 · p. 12
  • Figure 3 provides a representative example involving an incorrect player_count parameter; the tool catalog is omitted from the figure for brevity.Source: §A.1 · p. 12, Figure 3 · p. 12

A.2 ToolRM Training Data Details

The appendix lists the models used to generate ToolRM training data, spanning Qwen, IBM Granite, Mistral, and HuggingFaceTB model families.

A.2 ToolRM Training Data Details

  • The IBM Granite models listed are granite-3.3-2b-instruct, granite-3.3-8b-instruct, and granite-20b-functioncalling.Source: §A.2 · p. 12
  • The Qwen models listed are Qwen2.5-14B-Instruct, Qwen2.5-32B-Instruct, Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen2.5-7B-Instruct.Source: §A.2 · p. 12
  • The remaining listed generators are Mistral-7B-Instruct-v0.3, Mistral-Nemo-Instruct-2407, and SmolLM2-1.7B-Instruct.Source: §A.2 · p. 12

A.3 ToolRM prompt

The ToolRM training prompt presents a user query, a tool catalog, and tool calls for judging whether the calls adequately fulfill the request. Its examples contrast correct and incorrect calls, including omissions of a call or a required parameter.

A.3 ToolRM prompt

  • Training samples pair a conversation and tool catalog with correct and incorrect tool-call versions.Source: Figure 4 · p. 13
  • Incorrect examples include a missing tool call or a missing parameter in a tool call.Source: Figure 4 · p. 13
  • The prompt asks the model to judge whether tool calls adequately fulfill a user request, given the query and available tools.Source: §A.3 · p. 14

A.4 FC-RewardBench Experiment Details

The experiment details cover reward-model and judge baselines, evaluation prompts, downstream correlation, error patterns, generalization, GRPO setup, and ablations. FC-RewardBench scores correlate with downstream performance, while the analyses report error reductions, low train–test similarity, and component-level performance differences.

A.4.1 Model Details

  • The evaluation includes 11 reward models—eight from RewardBench and three ToolRM variants—and LLM-as-judge baselines.Source: §A.5 · p. 15, §A.4.1 · p. 15

A.5 Correlation with performance on downstream tasks:

  • 0.84 average Pearson correlation between FC-RewardBench scores and downstream task accuracy was observed across benchmarks and generator models.Across generator models, average correlations ranged from 0.62 to 0.94; Figure 5 presents the correlations.Source: §A.5 · pp. 15, 17, Figure 5 · p. 17

A.4.2 LLM-as-Judge Prompt

  • The LLM-as-judge prompt compares responses against tool specifications and query requirements, including tool relevance, call order, and parameter correctness.It also instructs judges to avoid position and response-length biases.Source: §A.5 · p. 16

A.6 Error Analysis

  • 22.7% relative reduction in total errors, from 742 to 573, was reported with ToolRM-14B in Best-of-32 sampling on BFCL-v3 single-turn splits.Incorrect Parameter Value errors fell 28%, Incorrect Function Name errors 37%, and Wrong Number of Functions errors 57%; Irrelevance Errors rose from 185 to 210.Source: §A.6 · p. 17, Table 8 · p. 17

A.7 ToolRM generalization

  • Median maximum cosine similarity between training and test examples was below 0.6 for both tools and conversations, alongside reported generalization to novel inputs.Figure 6 shows the similarity distributions for tools and conversations.Source: §A.7 · p. 17, Figure 6 · p. 17

A.8 GRPO Experiment Details

  • The GRPO experiments used the verl library and the code base and dataset provided by ToolRL, with hyperparameters listed in Table 9.Source: §A.8 · p. 18, Table 9 · p. 18

A.9 Ablation analysis

  • The full ToolRM-1.5B achieved the highest FC-RewardBench performance among the ablations; removing API-Gen cost 16.4 points and obfuscating tool and parameter names cost 13.63 points.The authors suggest obfuscation encourages attention to other parts of tool specifications; Table 10 summarizes the ablations.Source: §A.9 · p. 18, Table 10 · p. 18

Related papers

Loading 2509.11963v2…