PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Reasoning Reward Model for Agents

Kaixuan Fan, Kaituo Feng, Manyuan Zhang, Tianshuo Peng, Zhixun Li, Yilei Jiang, Shuang Chen, Peng Pei, Xunliang Cai, Xiangyu Yue

arXiv:2601.22154v2cs.AIcs.CL

TL;DR

Sparse outcome rewards do not distinguish intermediate reasoning quality in long-horizon agentic tasks. The paper introduces Agent-RRM and Reagent integration strategies, with Reagent-U achieving gains across 12 benchmarks, including 43.7% on GAIA.

  • Problem

    Outcome-based rewards provide coarse supervision that fails to distinguish intermediate reasoning quality in long-horizon, multi-step agentic tasks.

  • Method

    Agent-RRM generates rationales, actionable critiques, and holistic scores, which are integrated through Reagent-C, Reagent-R, and Reagent-U.

  • Results

    Reagent models achieve significant gains across 12 diverse benchmarks, with Reagent-U reaching 43.7% on GAIA.

  • Takeaways & Limitations

    Textual critiques support inference-time refinement, while model-based rewards mitigate training signal sparsity and together improve long-horizon reasoning and multi-step tool use.

  • Takeaways & Limitations

    Experiments primarily use 8B-parameter models, leaving scaling to larger models and adaptability to broader, open-ended tool environments unexplored.

Abstract

from arXiv · show

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.

1 Introduction

The introduction identifies sparse outcome-based rewards as inadequate for long-horizon agentic tasks and presents Agent-RRM for reasoning-aware, structured trajectory feedback. It further proposes three integration variants, releases four specialized datasets, and evaluates Reagent models across 12 benchmarks.

  • Motivation: Sparse rewards based only on final correctness cannot distinguish high-quality intermediate reasoning in long-horizon tasks requiring multi-step tool use.The introduction highlights this limitation as a central problem in prior agentic RL methods.
  • Agent-RRM: Agent-RRM generates explicit reasoning traces, targeted critiques, and holistic quality scores to provide transparent, granular supervision for agentic trajectories.Its multi-level signals combine reasoning analysis, refinement guidance, and overall quality assessment.
  • Integration strategies: The paper systematically investigates Text-augmented Refinement (Reagent-C), Reward-augmented Guidance (Reagent-R), and Unified Feedback Integration (Reagent-U).The variants use textual feedback, model-based rewards alongside rule-based rewards, or harmonized multi-source rewards with critique-augmented sampling.
  • Resources: The authors curate and release four specialized, high-quality datasets for agent reasoning and reward-model training.These resources are intended to support research on multi-granular feedback for agentic reinforcement learning.
  • Evaluation: Extensive experiments evaluate Reagent models across 12 diverse benchmarks and report significant performance gains.The experiments are presented as evidence for the efficacy of multi-level reasoning-based feedback in complex agentic tasks.

2 Related Work

Agentic RL enables multi-step information-seeking and tool-use capabilities, while reward models provide learning signals for policy improvement and increasingly incorporate explicit reasoning. In agent settings, existing reasoning-based reward models remain underexplored and largely rely on step-level scalar rewards.

  • Agentic Reinforcement Learning: Agentic RL has demonstrated effectiveness in developing agents with multi-step information-seeking and tool-use proficiencies.The cited studies examine agents operating in dynamic, open-ended environments.
  • Reward Models: Reward models optimize complex reasoning tasks by providing learning signals for policy improvement.Reasoning-aware reward models additionally perform explicit reasoning before assigning rewards for more transparent supervision.
  • Reward Models: Reasoning-based reward models in agent domains remain underexplored.Atom-Searcher uses an untrained Qwen3-30B-A3B to score agent steps, while PPR evaluates trajectory steps using a predefined principle set.
  • Reward Models: Existing agent reward-model methods remain confined to step-level scalar rewards, which are susceptible to reward hacking and provide limited language feedback.The passage contrasts Atom-Searcher and PPR with richer reasoning-based supervision.

3 Method

The method combines GRPO with specialized tools, curated training datasets, and Agent-RRM’s structured reasoning feedback. Three policy variants—Reagent-C, Reagent-R, and Reagent-U—incorporate critiques and scalar rewards through refinement or reinforcement learning.

  • Training Setup: The training pipeline provides six tools and curated datasets spanning mathematical deduction, multimodal understanding, web information seeking, and complex tool use.Filtering, deduplication, and difficulty-aware sampling produce Reagent-RL-709K and Reagent-SFT-55.6K, while the agent uses retrieval, browsing, code execution, file reading, and image description tools.
  • Agent-RRM: Agent-RRM produces a reasoning trace, targeted critique, and scalar score in [0, 1], trained with SFT followed by GRPO.It uses Reagent-RRM-SFT-28K for supervised training and Reagent-RRM-RL-90K for reinforcement learning.
  • Reagent-C: Reagent-C uses Agent-RRM critiques for training-free, in-context refinement while keeping the Qwen3-8B policy frozen.The agent generates an initial response, obtains a targeted critique, and produces a refined output conditioned on that feedback.
  • Reagent-R: Reagent-R combines rule-based correctness with Agent-RRM’s model-based score to provide fine-grained, reasoning-aware rewards.This reduces sparse reward limitations by rewarding logical merit and penalizing reasoning deficiencies independently of final-answer correctness.
  • Reagent-U: Reagent-U jointly integrates scalar rewards and critique-driven refinement by pooling initial and refined trajectories within one RL objective.Critiques are used during training to internalize reasoning capabilities; inference runs as a standard agent without additional critique refinement or external guidance.

4 Experiments

Experiments evaluate Agent-RRM across mathematical, knowledge-intensive, general-agent, and search benchmarks using Reagent-C, Reagent-R, and Reagent-U. Reagent-U achieves the strongest overall results, while textual critiques and holistic rewards each improve agent performance.

  • Benchmarks: Evaluations span mathematical, knowledge-intensive, general-agent, and search-reasoning benchmarks.The benchmark suite includes AIME24, AIME25, GSM8K, MATH500, HotpotQA, 2Wiki, Bamboogle, MuSiQue, GAIA, and WebWalkerQA.
  • Reagent-C: Reagent-C delivers consistent gains across all benchmarks without parameter updates.Improvements are particularly pronounced in Mathematical Reasoning, with additional advances on General Agentic and Knowledge-Intensive tasks.
  • Reagent-R: 72.8% on Bamboogle and 41.0% on xbench: Reagent-R surpasses Reagent w/o Agent-RRM by 11.2 and 9.0 percentage points, respectively.Reagent-R consistently outperforms the rule-based reward baseline across all benchmarks by adding holistic reasoning-level scores.
  • Reagent-U: 43.7% on GAIA (text), 46.2% on WebWalkerQA, 76.8% on Bamboogle, and 60.0% on AIME24: Reagent-U outperforms all baselines while remaining robust across task types.Reagent-U’s unified feedback mechanism combines the strengths of textual and scalar supervision, whereas Reagent-R remains inferior across most tasks.
  • Modality and tool generalization: Reagent-U significantly outperforms baselines on the full GAIA set while maintaining competitive performance on the text subset.The full-set evaluation tests heterogeneous reasoning and tool use beyond web navigation and information retrieval.
  • Reward-weight analysis: Performance plateaus at λ ∈[0.2, 0.4] and slightly declines at λ = 0.5 after initially increasing with the Agent-RRM reward weight.The analysis compares reasoning-reward integration against the baseline at λ = 0.

5 Conclusion

The paper introduces Agent-RRM, a multi-faceted reasoning reward model, and Reagent, a scheme for exploring multi-dimensional feedback in agentic learning. Evaluations show that textual critiques and model-based rewards support long-horizon reasoning and multi-step tool use across complex, multi-modal tasks.

  • Contributions: Agent-RRM provides textual critiques and holistic reasoning-aware rewards for agentic learning.It is designed as a multi-faceted reasoning reward model.
  • Contributions: Reagent explores the efficacy of multi-dimensional feedback in agentic learning.The scheme builds upon Agent-RRM.
  • Findings: Textual critiques provide diagnostic guidance for inference-time refinement, while model-based rewards support the agent’s learning.Together, these signals bolster long-horizon reasoning and multi-step tool-use proficiency.
  • Findings: Evaluations show consistent gains across diverse complex, multi-modal tasks.The gains are associated with improved long-horizon reasoning and multi-step tool-use proficiency.

6 Limitations

The work’s experiments focus primarily on 8B-parameter models, leaving scaling to larger models unresolved, while broader toolsets and more intricate reasoning chains remain important future directions.

  • Model scale: Experiments primarily target 8B-parameter models, so the scheme’s scaling behavior on larger models remains unexplored.Future work could examine whether more powerful base models amplify the benefits of structured reasoning feedback.
  • Task breadth: Extending beyond standardized benchmarks to broader toolsets and more intricate reasoning chains is identified as essential.The paper presents this extension as a direction for future work.

A.1 Agent Training Data Distribution

The study constructs Reagent-RL-709K from 709k publicly available question-answer pairs and samples 100k examples for trajectory collection. Correct trajectories produce the 55.6k-example Reagent-SFT-55.6K dataset for supervised fine-tuning.

  • A.1 Agent Training Data Distribution: 709k question-answer pairs from publicly available datasets form the RL dataset Reagent-RL-709K.The detailed data distribution appears in Figure 1 (bottom).
  • A.1 Agent Training Data Distribution: 100k examples are randomly selected from the RL dataset for trajectory collection with DeepSeekV3.1 and 6 tools.Trajectories reaching the correct final answer are retained for the SFT dataset.
  • A.1 Agent Training Data Distribution: 55.6k high quality trajectories constitute the SFT dataset Reagent-SFT-55.6K.These trajectories are drawn from the collected examples that reach the correct final answer.

A.2 Dataset Selection and Filtering · A.3 Agent-RRM Construction Details

The paper filters datasets for reliable supervision, suitable difficulty, modality-tool compatibility, and manageable inputs, then constructs Agent-RRM with structured reasoning, critique, and scalar evaluation signals. Its annotation prompt emphasizes trajectory-wide reasoning quality and tool-use decisions.

  • A.2 Dataset Selection and Filtering: DeepMath and DeepScaleR samples are removed when generated solutions disagree with one another or with labeled ground truth.DeepMath uses three independently generated solutions; inconsistent final answers create ambiguous supervision.
  • A.2 Dataset Selection and Filtering: SimpleRL-Zoo retains more medium- and hard-level questions and fewer easy ones to encourage non-trivial reasoning.
  • A.2 Dataset Selection and Filtering: MMK12, PixelReasoner, LiveVQA, and ToolVQA are selected around image-based information needs that available OCR, image-description, search, and calculator tools can support.MMK12 uses charts or tables; PixelReasoner targets textual content or visual avatars; LiveVQA retains identifiable image attributes; ToolVQA uses four specified categories.
  • A.2 Dataset Selection and Filtering: SimpleDeepSearcher is converted to a Qwen-compatible action format, AFM-WebAgent is linearized into single-agent trajectories, and LongAudio retains 5–40-second clips.AFM-WebAgent preserves planning, verification, and reflection patterns, while the LongAudio range balances informational content against Whisper-large-v3 computational overhead.
  • A.3 Agent-RRM Construction Details: Agent-RRM training instances are structured for multi-granular supervision, targeting reliable and informative feedback at both semantic and scalar levels.The prompt template is provided in Figure 4.
  • A.3 Agent-RRM Construction Details: Each Agent-RRM judgment contains <think> for logical-consistency reasoning, <critique> for targeted flaws, and <score> for holistic scalar evaluation.The critique focuses on global logic and appropriate external-tool invocation, while the reasoning trace makes the reward judgment interpretable.
  • A.3 Agent-RRM Construction Details: The annotation prompt requires comprehensive trajectory analysis centered on reasoning quality and tool-use behavior across all tool types.It evaluates whether tool calls are appropriately made, penalizing unnecessary reliance and missed calls when tools are needed.

B Training Details … C.2 Tool Configurations

The paper trains Agent-RRM and Reagent with specified supervised- and reinforcement-learning implementations, then evaluates them on defined benchmark subsets using the same tool configurations. The agentic framework integrates search, browsing, image-to-text, and audio-to-text tools with dedicated backends.

  • B.1 Training Codebase: The training codebase implements supervised fine-tuning for both Agent-RRM and Reagent through LLaMA-Factory.The same codebase supports the stated supervised-training setup for both models.
  • B. Training Details: Agent-RRM and Reagent are trained with supervised fine-tuning for 2 epochs using LLaMA-Factory.Both the reward model and agent model receive the same two-epoch training duration.
  • B.1 Training Codebase: Agentic reinforcement-learning training uses rLLM for Reagent and VeRL for Agent-RRM, with 300 training steps.The reinforcement-learning hyperparameters are provided in Table 4.
  • B.2 Tools configuration: The models are trained on 8 NVIDIA A800-80G GPUs.This hardware configuration is stated alongside the implementation details for the integrated tools.
  • B.2 Tools configuration: Search uses the Bing Search API and returns top-k results containing each entry’s URL, title, and content snippet.The agent receives the retrieved results in this structured form.
  • B.2 Tools configuration: Browse retrieves website content with Jina Reader and condenses it using DeepSeek-Chat as the summarization model.The two-stage configuration separates content retrieval from summarization.
  • C.1 Evaluation Benchmarks: Evaluation covers GAIA’s 103-task Text subset and 165-task full set, HLE’s 500-task subset, and xbench-DeepSearch.These reporting choices follow the specified benchmark conventions.
  • C.2 Tool Configurations: Evaluation tools use configurations identical to those specified in Appendix B.2.Thus, the evaluation reuses the documented search, browsing, image-to-text, and audio-to-text setup.

C.3 Agent Evaluation Settings

Benchmark evaluations use fixed decoding and interaction limits, with Qwen2.5-72B-Instruct serving as an automated binary judge against ground-truth answers.

  • Evaluation settings: Evaluations use temperature 0.6, top_p 0.95, a 32,768-token response limit, and at most 30 tool-use steps per query.These settings apply across all benchmark evaluations.
  • Evaluation settings: Qwen2.5-72B-Instruct performs binary scoring by comparing agent predictions with ground-truth answers.This choice follows Dong et al. (2025).

D Case Study

The case studies illustrate how Reagent-C critiques expose reasoning and tool-use problems, enabling agents to refine responses and correct errors. Examples span a GAIA search question and a GSM8K math question.

  • Critique mechanism: Agent-RRM critiques highlight logical inconsistencies and inappropriate tool usage, helping agents self-correct and rectify errors in subsequent responses.The paper presents two Reagent-C case studies to demonstrate this critique mechanism.
  • Case 1: GAIA: The GAIA example required matching a catastrophe’s explosive force with a US nuclear test’s yield and returning the test name’s last word.The ground-truth answer is Bravo.
  • Case 1: GAIA: The initial GAIA response assumed the catastrophe was Tunguska and the yield was approximately 5 megatons without verifying the Phys.org or Britannica sources.It also selected Cannikin among multiple candidate tests despite ambiguous evidence.
  • Case 2: GSM8K: The second case study is a GSM8K question asking how many hours each of four painters worked on a mansion over three weeks.Its ground-truth answer is 189.
Loading 2601.22154v2…