ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

Dawei Li; Yuguang Yao; Zhen Tan; Huan Liu; Ruocheng Guo

Source-linked AI summary

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo

arXiv:2601.12294v1Jan 18, 2026cs.AIcs.SE

TL;DR

Tool-using agents need reliable step-level evaluation because existing PRM benchmarks do not systematically cover their long, structured interactions. ToolPRMBench converts trajectories from diverse tool-use benchmarks into verified decision-step cases and evaluates multiple PRM families, finding clear model differences and benefits from specialized training. The study also identifies scope and evaluation gaps for future work.

Problem
Existing benchmarks do not provide a unified, systematic evaluation of process reward models for tool-using agents.
Method
ToolPRMBench converts trajectories from representative tool-using benchmarks into step-level cases using offline and online sampling, multi-LLM verification, and correct-versus-plausible-incorrect actions.
Results
Tool-specialized PRMs outperform open-source LLMs and general-purpose PRMs, while model scale and general capabilities help but are not sufficient without specialized training.
Takeaways & Limitations
Tool-specific supervision and reinforcement learning are promising directions for improving robustness and generalization in tool process reward modeling.
Takeaways & Limitations
The study focuses mainly on intrinsic PRM discrimination rather than extensive end-to-end evaluation of search-based strategies under heavy training budgets.

Abstract

from arXiv · show

Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.

1 Introduction

ToolPRMBench addresses the lack of systematic evaluation for process reward models in tool-using settings by providing a step-level benchmark and broad empirical analysis. Its experiments reveal differences across model types and identify specialized training and further analysis directions.

ToolPRMBench fills the absence of a unified benchmark for systematically evaluating process reward models in tool-using settings.
The benchmark combines offline and online trajectory sampling to isolate local single-step errors and capture realistic multi-step failures.Samples include interaction history, correct and plausible incorrect actions, and tool metadata.
The dataset uses multi-LLM verification and filtering to improve label reliability and support fine-grained decision-step evaluation.
17 large language models, general PRMs, and tool-specialized PRMs show clear and consistent performance differences across model types.
The work adds meta-evaluation, data synthesis, cost analysis, and case studies to inform future reward-guided trajectory search.

2 Related Work

Tool-using agents interact with external tools to solve tasks requiring execution, while reward-guided search evaluates multiple candidate actions. Process-level rewards are important for guiding intermediate decisions in these long-horizon settings.

Tool-using agents extend language models by interacting with external tools or APIs, enabling tasks that require external actions rather than text generation alone.
Reward-guided search samples and searches over multiple candidate actions to improve model performance at test time.
Long-horizon tool-using systems require accurate step-level process rewards because sparse outcome rewards are often insufficient for intermediate decisions and credit assignment.

3 ToolPRMBench

ToolPRMBench evaluates process reward models by converting tool-using trajectories into step-level action-selection cases. Its construction combines complementary sampling strategies, verification, and specialized model-training variants across diverse tool-use settings.

Benchmark construction: ToolPRMBench converts trajectories from four representative tool-using benchmarks into step-wise samples that test process-level correctness rather than only final task success.The benchmark covers information-seeking, multi-step reasoning, and interactive tool execution.
Benchmark construction: Each sample pairs an interaction history with a chosen correct action, a rejected incorrect action, and tool metadata such as available tool descriptions.The history represents the user instruction and preceding action-observation pairs at a single decision step.
Trajectory sampling: Offline sampling isolates local single-step deviations by preserving the golden trajectory after sampling an alternative action, whereas online sampling captures correlated multi-step failures from freely generated trajectories.Offline samples are discarded when the sampled and golden actions are semantically equivalent; online sampling is harder to analyze but more realistic for error propagation.
Trajectory sampling: Failed online trajectories are converted into step-level preference pairs by identifying the first incorrect step and proposing a corrected action.The resulting history ends before the erroneous action, making trajectory-level failures suitable for PRM evaluation.
Data verification: Multi-LLM majority voting retains unanimously accepted alternatives, discards unanimously rejected ones, and sends mixed cases for additional human verification to reduce label noise.Three LLMs independently judge whether the rejected action is valid given the history.
ToolPRM training: ToolPRM variants range from direct action-label prediction with supervised fine-tuning to explicit reasoning modeling and reinforcement learning refinement.ToolPRM-Base predicts the selected candidate position, ToolPRM-CoT models a reasoning sequence, and ToolPRM-GRPO further refines reasoning and action selection through reinforcement learning.
Dataset statistics: The benchmark contains 984 samples spanning moderate and complex trajectory lengths and common failures including incorrect actions, incorrect arguments, and improper tool use.Its distributions cover four subsets and varying task difficulty rather than only trivial short interactions.

4 Experiment

Experiments compare API-based LLMs, open-source LLMs, general PRMs, and tool-specialized PRMs on ToolPRMBench. Results show benefits from scale and general capabilities, while specialized training and reinforcement learning improve tool-specific performance and generalization.

Models and setup: The benchmark evaluates API-based and open-source LLMs, general PRMs, and tool-specialized PRMs, including ToolPRM-Base, ToolPRM-CoT, and ToolPRM-GRPO.ToolPRM variants are trained on Qwen-3-4B using BFCL and ToolSandbox subsets with a 7:3 training-testing split.
Overall comparison: API-based LLMs consistently achieve the strongest overall results across almost all ToolPRMBench subsets.The comparison includes 17 large language models, general PRMs, and tool-specialized PRMs.
Overall comparison: ToolPRM-GRPO achieves the best average accuracy among non-API models, while tool-specialized PRMs substantially outperform open-source LLMs and general-purpose PRMs.Many open-source LLMs and general PRMs fail to exceed 55% average accuracy, indicating limited transfer from math reasoning or web navigation.
Scaling behavior: Model size shows a positive relationship with ToolPRMBench performance, but even the largest open-source models substantially lag behind ToolPRMs.The results associate larger general model capacity with improved tool process reward modeling while retaining a substantial specialization gap.
Generalization: ToolPRM-Base and ToolPRM-CoT improve in-distribution performance but decline out of distribution by 20.4% and 13.6%, respectively.Both variants use supervised fine-tuning, and the observed drops suggest reliance on distribution-specific patterns.
Generalization: ToolPRM-GRPO improves out-of-distribution performance by 21.8% while also achieving gains in the in-distribution setting.The result supports reinforcement learning as a promising direction for robust ToolPRM learning beyond the training distribution.

5 Further Analysis

Further analyses examine reward-guided search validity, synthetic-data training, efficiency, and tool-use errors. Results show benchmark accuracy tracks search effectiveness, synthetic data is task-dependent, and specialized PRMs offer favorable performance–cost trade-offs.

Meta-evaluation: Models with high ToolPRMBench accuracy consistently produce larger gains in best-of-8 reward-guided search on GTA and BFCL.Models below 50% accuracy often yield negative gains because they misguide exploration and amplify incorrect trajectories.
Synthetic data: Synthetic data improves GTA performance substantially, with ToolPRM-Base-Syn and ToolPRM-GRPO-Syn both exceeding 22% relative gains.The strategy inserts incorrect actions into ground-truth trajectories, avoiding additional rollouts and reducing annotation cost.
Synthetic data: Synthetic-data effects are weaker on ToolTalk: ToolPRM-Base-Syn slightly degrades performance, while ToolPRM-GRPO-Syn shows only marginal improvement.The results indicate that synthetic-data effectiveness depends strongly on the task and environment.
Cost analysis: ToolPRMs achieve competitive accuracy at substantially lower inference cost than API-based LLMs, while outperforming open-source LLMs and general PRMs under similar or lower budgets.API-based LLMs achieve strong benchmark performance but have significantly higher estimated inference costs.
Case study: A BFCL case study shows that valid tool use requires satisfying low-level specifications and filesystem state transitions, not merely matching the user’s high-level intent.Changing the working directory with cd is required before valid cp operations under the tool interface.

6 Conclusion

The paper introduces ToolPRMBench for evaluating process reward models in tool-using agents through diverse benchmarks and offline and online trajectory sampling. Experiments find that scale and general capabilities help but specialized training remains necessary, while reinforcement learning improves robustness and generalization.

ToolPRMBench is a large-scale benchmark for evaluating process reward models in tool-using agent settings.It combines diverse tool-using benchmarks with offline and online trajectory sampling.
Evaluation reveals consistent differences across model families: greater scale and stronger general capabilities help, but specialized training is not dispensable.The conclusion also identifies reinforcement learning as important for robustness and generalization.
Complementary analyses cover reward-guided search, distribution shift, synthetic data, and efficiency, providing practical insights for future tool-using reward modeling.

Limitations

The study’s evaluation scope is limited by its emphasis on intrinsic PRM discrimination and its reliance on a selected set of existing tool-using benchmarks.

The study does not extensively evaluate inference-time scaling strategies with reinforcement learning or include newer MCP-based datasets.The authors cite time, computing, budget, data-collection cost, and environment-setup constraints, and identify efficient RL and MCP environments as future directions.

A More Details of ToolPRMBench

The supplementary details specify the benchmark’s annotation inputs, error-identification criteria, and exact JSON output structures for BFCL and ToolSandbox samples.

ToolPRMBench statistics are documented in Table 3, alongside the prompts used for LLM annotation and multi-LLM verification.
ToolSandbox annotation: The ToolSandbox annotation input includes a sample identifier, trajectory, available tools, milestones, milestone edges, and raw tool feedback.
ToolSandbox annotation: Annotators identify the first incorrect agent step when it violates a user-facing response, milestone ordering, or required milestone dependency.
BFCL annotation: BFCL annotations return the sample identifier, history, rejected and chosen actions, a rationale, and an error type in a single JSON object.

B.1 Training Environment and Infrastructure

Experiments use a single 8-GPU NVIDIA H20 machine with DeepSpeed ZeRO-3 to support full-parameter fine-tuning. Qwen3-4B is fine-tuned through an SFT setup tailored to standard and long-context CoT datasets.

All experiments run on one machine equipped with 8 NVIDIA H20 GPUs, each with 96GB memory.
DeepSpeed ZeRO-3 shards model states, gradients, and optimizer states across all 8 GPUs.This enables full-parameter fine-tuning within the available memory budget.
Qwen3-4B serves as the base model for training.
SFT uses full-parameter fine-tuning for 2 epochs with AdamW, a 1.0 × 10−5 learning rate, cosine scheduling, and 0.1 warmup ratio.
The SFT cutoff length is 4,096 tokens to accommodate long-context reasoning in CoT tasks.
Per-device batch size is 4 for standard tasks and 1 for CoT-enhanced datasets to manage memory consumption.

B.3 Reinforcement Learning (RL)

The RL stage uses GRPO with grouped completions and KL regularization, while the evaluation prompts compare two candidate actions given tool-use context. Additional prompts define CoT judgments and synthetic error construction, with binary rewards based on whether the ground-truth answer appears after </think>.

Reinforcement Learning (RL): GRPO trains the model for 1 epoch with a peak learning rate of 2.0 × 10−6 and generates G = 8 completions per prompt.The completion group is used to compute relative rewards.
Reinforcement Learning (RL): The training configuration uses a maximum prompt length of 2,048 tokens, a maximum completion length of 4,096 tokens, and β = 0.01 for KL divergence.
Reward: The reward is binary: 1.0 when the ground-truth answer appears after the </think> token and 0.0 otherwise.
Evaluation: The evaluation prompt asks which of two actions is the correct intermediate step that could help finish the task, given interaction history and function descriptions.
Evaluation: ToolPRM-CoT prompts require a concise 2–6 sentence rationale explaining the better action and identifying the winning action.
Data Synthesis: The data-synthesis prompt selects an assistant step likely to contain a specified mistake, constructs rejected and chosen actions, and outputs a JSON object.