PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Reasoning Reward Model for Agents

Kaixuan Fan, Kaituo Feng, Manyuan Zhang, Tianshuo Peng, Zhixun Li, Yilei Jiang, Shuang Chen, Peng Pei, Xunliang Cai, Xiangyu Yue

arXiv:2601.22154v2cs.AIcs.CL

TL;DR

Agentic RL commonly relies on sparse final-correctness rewards that do not distinguish intermediate reasoning quality. This paper introduces Agent-RRM, which provides rationales, critiques, and scores, and evaluates three Reagent integration strategies. Across diverse benchmarks, Reagent models show significant gains, with Reagent-U reaching 43.7% on GAIA.

  • Problem

    Agentic RL commonly relies on sparse final-correctness rewards, while existing reward models can be costly or provide limited actionable guidance about trajectory quality.

    Source: §1 · p. 1

  • Method

    Agent-RRM generates structured reasoning rationales, actionable critiques, and holistic quality scores, which the paper integrates through three Reagent variants.

    Source: §1 · p. 2

  • Results

    Across diverse benchmarks, Reagent models achieve significant performance gains, with Reagent-U reaching 43.7% on GAIA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Textual critiques provide diagnostic guidance for inference-time refinement, while model-based rewards mitigate signal sparsity during training.

    Source: §5 · p. 8

    Experiments primarily focus on 8B-parameter models, leaving scaling to larger models and adaptability to broader toolsets and open-ended real-world tasks unvalidated.

    Source: §6 · p. 9

Abstract

from arXiv · show

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.

1 Introduction

The introduction motivates Agent-RRM as a response to coarse outcome-only supervision and limitations in existing reward models, then presents structured reasoning feedback and three Reagent integration strategies. It reports Reagent-U’s benchmark results and describes the released datasets and broader evaluation.

1 Introduction

  • Outcome-only rewards obscure successful intermediate reasoning in long-horizon tasks, treating trajectories that fail at the final step as total failures and contributing to suboptimal performance.The introduction notes that RLVR advances have extended to agents handling complex interactions with dynamic environments and external knowledge sources.Source: §1 · p. 1
  • Existing reward-model approaches face costly, hackable step-level annotations and pairwise reasoning preferences that miss fine-grained trajectory quality and actionable refinement guidance.Most also rely on numeric rewards, leaving natural-language critique underexplored.Source: §1 · p. 1
  • Agent-RRM evaluates each trajectory with a reasoning trace, a targeted critique, and a holistic quality score, providing dense supervision without requiring ground truth.The signals combine scalar rewards for global optimization with textual critiques for explicit error correction.Source: §1 · pp. 1–2
  • 43.7% on GAIA and 46.2% on WebWalkerQA are reported for Reagent-U, which the introduction identifies as the strongest integration of Agent-RRM feedback.The paper also reports significant gains for Reagent models across 12 diverse benchmarks.Source: §1 · p. 2
  • The three integration variants are Reagent-C for text-based trajectory refinement, Reagent-R for reward-augmented guidance, and Reagent-U for unified feedback integration.Reagent-U combines multi-source rewards with critique-augmented sampling.Source: §1 · p. 2
  • The authors curate and release four specialized datasets for agent reasoning and reward-model training to support research on multi-level feedback for agentic reinforcement learning.Experiments span 12 diverse benchmarks.Source: §1 · p. 2

2 Related Work

Prior work shows that Agentic RL can support multi-step information seeking and tool use, but sparse outcome rewards limit feedback on intermediate errors. Reasoning-aware reward models offer richer supervision, yet agent-domain methods remain limited in the guidance they provide.

2.1 Agentic Reinforcement Learning

2.2 Reward Modeling

  • Reasoning-aware reward models provide explicit reasoning before scoring; RM-R1 (Chen et al., 2025d), for example, derives rubrics before evaluating candidate responses.In agent settings, Atom-Searcher (Deng et al., 2025) scores steps with an untrained model, while PPR (Xu et al., 2025) evaluates trajectory steps against predefined principles; both use scalar rewards that can be reward-hacked and lack language guidance for complex logic flaws.Source: §2.2 · pp. 2–3

3 Method

Agent-RRM provides structured reasoning traces, critiques, and scores for agent trajectories, which Reagent incorporates through textual refinement, reward guidance, or unified feedback training.

3.1 Preliminaries: GRPO Framework

  • GRPO uses importance-sampling ratios and KL divergence in its objective, with advantages normalized from rewards within each output group.The reward system assigns each output its reward.Source: §3.1 · p. 3

3.2 Agentic Tool Design

  • The agent has six specialized tools for information retrieval, code execution, and multimodal perception.They include Search, Web Browse, Python Code Interpreter, File Reader, Image Descriptor, and Audio Converter.Source: §3.2 · p. 3

3.3 Dataset Construction

  • The authors construct datasets for agent training and Agent-RRM across mathematical deduction, multimodal understanding, web information seeking, and complex tool use.A three-stage quality pipeline filters ambiguous answers, deduplicates across sources, and samples by difficulty. The datasets include Reagent-RL-709K, Reagent-SFT-55.6K, Reagent-RRM-SFT-28K, and Reagent-RRM-RL-90K; the reward-model examples receive structured annotations from GPT-OSS-120B.Source: §3.3 · pp. 3–4

3.4 Agent-RRM: Reward Model Training

  • Agent-RRM produces a reasoning trace, targeted critique, and scalar quality score in [0, 1], trained in two stages to improve evaluative rationales and reward calibration.SFT on Reagent-RRM-SFT-28K establishes the output format and evaluative capabilities; GRPO on Reagent-RRM-RL-90K refines evaluation, including for tasks without ground-truth answers.Source: §3.4 · p. 4

3.5 Reagent: Integrating Reasoning Rewards into Agents

  • Reagent is seeded by fine-tuning on Reagent-SFT-55.6K, then investigates three ways to incorporate Agent-RRM feedback into agent policies.The variants are Reagent-C, Reagent-R, and Reagent-U.Source: §3.5 · pp. 4–5

3.5.1 Textual-augmented Refinement

  • Reagent-C uses Agent-RRM critiques to refine Qwen3-8B responses through in-context prompting, without updating the policy.Reported outputs are the refined responses.Source: §3.5.1 · p. 5

3.5.2 Reward-augmented Guidance

  • Reagent-R combines rule-based answer correctness with Agent-RRM’s scalar score to provide fine-grained trajectory rewards that address sparse correctness feedback.The combined signal can reward logical merit and penalize reasoning deficiencies irrespective of final-answer correctness.Source: §3.5.2 · p. 5

3.5.3 Unified Feedback Integration

  • Reagent-U jointly trains initial-generation and critique-guided refinement, normalizing advantages across both sets of trajectories; critiques are used during training, not inference.At inference, it operates as a standard agent without additional critique refinement or external guidance.Source: §3.5.3 · p. 5

4 Experiments

Reagent-U surpasses compared methods on diverse reasoning and agentic benchmarks, while the experiments also test textual critiques, scalar rewards, cross-modal generalization, and reward-weight sensitivity.

4 Experiments

  • Experiments span 12 benchmarks across mathematical, knowledge-intensive, and general-agent/search reasoning, using SFT then RL with agents and reward models initialized from Qwen3-8B.Evaluation reports pass@1 at temperature 0.6 and top-p 0.95, using Qwen2.5-72B-Instruct as a binary judge.Source: §4 · pp. 5–6

4.3 Does Unified Feedback Synergistically Boost Performance?

  • 43.7% on GAIA (text), 46.2% on WebWalkerQA, 76.8% on Bamboogle, and 60.0% on AIME24; Reagent-U surpasses all compared methods across diverse tasks.The results span general-agent, search, knowledge-intensive, and mathematical reasoning benchmarks.Source: Table 1 · p. 6, Table 2 · p. 7, §4.3 · p. 8

4.2 Does Model-based Reward Improve Learning?

  • Reagent-R's holistic scores beat the rule-based Reagent w/o Agent-RRM baseline across benchmarks, including 72.8% on Bamboogle and 41.0% on xbench, gains of 11.2 and 9.0 points.Reagent-R remains inferior to Reagent-U across most tasks; scalar scores differentiate trajectory quality but lack textual critiques' explicit structural guidance.Source: Table 1 · p. 6, Table 2 · p. 7, §4.2 · pp. 7–8

4.1 Can Textual Critiques Guide Inference-Time Refinement?

  • Reagent-C improves performance across benchmarks without parameter updates, and its critique-guided second response consistently beats the initial response, including without ground-truth access.Agent-RRM critiques diagnose logical fallacies and tool-execution errors, providing actionable refinement guidance.Source: §4.1 · pp. 6–7

4.4 Beyond Text-Only: Cross-Modal Reasoning and Complex Tool Use

  • On the full GAIA set, Reagent-U significantly outperforms baselines while remaining competitive on the text subset, demonstrating generalization beyond text-only tasks.The full set combines open-domain search, multimodal interpretation, Python coding, and file-based reasoning.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 Parameter Analysis on λ

  • Performance on AIME24 and xbench rises as λ increases, plateaus at λ ∈[0.2, 0.4], then declines slightly at λ = 0.5.The results indicate that excessive reasoning-reward weight can over-emphasize intermediate steps at the expense of task completion.Source: Figure 3 · p. 8, §4.5 · p. 8

5 Conclusion

Agent-RRM provides textual critiques and holistic reasoning-aware rewards, while Reagent explores multi-dimensional feedback in agentic learning. The combined signals support inference-time refinement and training, improving long-horizon reasoning and multi-step tool use across complex multimodal tasks.

5 Conclusion

  • Agent-RRM provides textual critiques and holistic reasoning-aware rewards, and Reagent explores multi-dimensional feedback in agentic learning.Source: §5 · p. 8
  • Together, textual critiques and model-based rewards bolster long-horizon reasoning and multi-step tool-use proficiency across diverse complex, multimodal tasks.The evaluations find that critiques provide diagnostic guidance for inference-time refinement, while model-based rewards mitigate training signal sparsity.Source: §5 · pp. 8–9

6 Limitations

Experiments primarily test 8B-parameter models on standardized benchmarks, leaving scaling to larger models and adaptability to broader toolsets and open-ended real-world tasks for future research.

6 Limitations

  • The experiments primarily use 8B-parameter models, leaving the scheme’s scaling behavior on larger models unexplored.Future work could test whether more powerful base models amplify the benefits of structured reasoning feedback.Source: §6 · p. 9
  • The scheme remains to be validated beyond standardized benchmarks, across broader toolsets and intricate reasoning chains in open-ended real-world applications.The passage gives AI for science as an example and notes unpredictable task environments.Source: §6 · p. 9

A.1 Agent Training Data Distribution

The training data comprises a 709k-pair RL dataset and a 55.6k-trajectory SFT dataset generated from a random 100k-example subset.

A.1 Agent Training Data Distribution

  • Reagent-RL-709K contains 709k question-answer pairs, and a random 100k-example subset was used to collect tool-assisted trajectories for SFT.DeepSeekV3.1 generated trajectories using six tools; the 55.6k trajectories that reached the correct final answer formed Reagent-SFT-55.6K.Source: §A.1 · p. 12

A.2 Dataset Selection and Filtering · A.3 Agent-RRM Construction Details

The sections describe filtering and adapting datasets for valid, tool-compatible agent training, then constructing Agent-RRM to provide structured reasoning feedback and scalar rewards from curated training data.

A.2 Dataset Selection and Filtering

  • DeepMath removes questions with inconsistent final answers across three independently generated solutions, and DeepScaleR excludes solutions inconsistent with the labeled ground truth.Source: §A.2 · p. 12
  • SimpleRL-Zoo retains a higher proportion of medium- and hard-level questions and fewer easy ones to encourage non-trivial reasoning.Source: §A.2 · p. 12
  • MMK12 selects chart and table inputs processable with OCR, while PixelReasoner selects image questions about text or visual avatars solvable with OCR and image-description tools.Source: §A.2 · p. 12
  • LiveVQA retains image-based questions about identifiable attributes, while ToolVQA selects categories aligned with the agent’s available tools.LiveVQA focuses on attributes such as titles or authors; ToolVQA uses ImageDescription, GoogleSearch, OCR, and Calculator categories.Source: §A.2 · p. 12
  • SimpleDeepSearcher is converted to a Qwen-compatible format, AFM-WebAgent steps are linearized into single-agent trajectories, and LongAudio retains clips lasting 5 to 40 seconds.AFM-WebAgent linearizes planning, verification, and reflection; clips outside the LongAudio range are excluded for insufficient information or excessive computational overhead.Source: §A.2 · p. 12

A.3 Agent-RRM Construction Details

  • Agent-RRM training prompts target reliable, informative semantic and scalar feedback, with the structured-judgment prompt template shown in Figure 4.Source: §A.3 · p. 12, Figure 4 · p. 13
  • Agent-RRM assesses trajectories through <think> logical-consistency reasoning, <critique> of reasoning and tool-use flaws, and a holistic <score> used for RL advantage calculation.The three components make judgments more interpretable while converting trajectory quality into a standardized reward signal.Source: §A.3 · p. 12
  • Reagent-RRM-SFT-28K contains 28,000 trajectories for initial supervised fine-tuning, while Reagent-RRM-RL-90K contains 90,000 instances for large-scale RL training.Source: §A.3 · p. 12

B Training Details … C.2 Tool Configurations

The appendix specifies the training code and duration, the tools used in the agentic framework, and benchmark-specific evaluation subsets. It also states that evaluation uses the tool configurations described in Appendix B.2.

B.1 Training Codebase

B.2 Tools configuration

  • Training used 8 NVIDIA A800-80G GPUs; Search used the Bing Search API to provide top-k results with URLs, titles, and content snippets.Source: §B.2 · pp. 13–14
  • Browse retrieved website content with Jina Reader and condensed it using DeepSeek-Chat; Image2text used GPT-4.1, and Audio2text used Whisper-large-v3.Source: §B.2 · p. 14

C.1 Evaluation Benchmarks

  • GAIA evaluation covered its 103-task Text subset and, separately, its 165-task full set; HLE used a 500-task subset, and xbench results were for xbench-DeepSearch.Source: §C.1 · p. 14

C.2 Tool Configurations

  • Evaluation tool configurations were identical to those specified in Appendix B.2.Source: §C.2 · p. 14

C.3 Agent Evaluation Settings

Benchmark evaluations use fixed decoding and interaction limits, with an automated judge scoring predictions against ground-truth answers.

C.3 Agent Evaluation Settings

  • Evaluations use temperature 0.6, top_p 0.95, a 32,768-token response cap, 30 tool-use steps per query, and Qwen2.5-72B-Instruct for binary judging against ground truth (Dong et al., 2025).Source: §C.3 · p. 14

D Case Study

Two Reagent-C case studies show how Agent-RRM critiques flag reasoning or tool-use flaws and support subsequent refinement and error correction.

D Case Study

  • The two case studies show critiques highlighting logical inconsistencies or inappropriate tool use, enabling successful self-correction and reasoning refinement.The cases are presented in Figures 5 and 6.Source: D Case Study · p. 14
  • The agent inferred a yield of about 5 megatons from search results and concluded that Cannikin was the matching test, rather than confirming the source directly.The critique also describes the test selection as ambiguous because multiple tests were listed.Source: D Case Study · p. 15
  • In the GAIA example, the critique faults the agent for not opening the Phys.org or Britannica pages and for assuming the catastrophe and its yield.It also notes that the agent chose Cannikin without verifying which test matched Britannica’s yield.Source: D Case Study · p. 15

Related papers

Loading 2601.22154v2…