PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, Juanzi Li

arXiv:2502.19328v1cs.CLcs.AI

TL;DR

Existing reward models primarily model human preferences, leaving verifiable correctness signals underused despite their importance for reliable rewards. The paper proposes agentic reward modeling and implements REWARDAGENT, which combines preference rewards with factuality and instruction-following verification. REWARDAGENT significantly outperforms other reward models and LLM-based generative reward models, while its verification agents remain imperfect and limited to two correctness aspects.

  • Problem

    Existing reward models primarily focus on human preferences and may neglect verifiable correctness signals such as factuality, limiting reward reliability.

  • Method

    REWARDAGENT combines a human-preference reward model with Router-selected verification agents for factuality and instruction-following, plus a Judger.

  • Results

    REWARDAGENT significantly outperforms other reward models and LLMs used as generative reward models.

  • Takeaways & Limitations

    Combining human preferences with verifiable correctness signals provides a more reliable reward-system design within the evaluated scope.

  • Takeaways & Limitations

    Verification agents are imperfect, reaching an average reward-model benchmark score of 72.5%, and the implementation covers only factuality and instruction-following.

Abstract

from arXiv · show

Reward models (RMs) are crucial for the training and inference-time scaling up of large language models (LLMs). However, existing reward models primarily focus on human preferences, neglecting verifiable correctness signals which have shown strong potential in training LLMs. In this paper, we propose agentic reward modeling, a reward system that combines reward models with verifiable correctness signals from different aspects to provide reliable rewards. We empirically implement a reward agent, named RewardAgent, that combines human preference rewards with two verifiable signals: factuality and instruction following, to provide more reliable rewards. We conduct comprehensive experiments on existing reward model benchmarks and inference time best-of-n searches on real-world downstream tasks. RewardAgent significantly outperforms vanilla reward models, demonstrating its effectiveness. We further construct training preference pairs using RewardAgent and train an LLM with the DPO objective, achieving superior performance on various NLP benchmarks compared to conventional reward models. Our codes are publicly released to facilitate further research (https://github.com/THU-KEG/Agentic-Reward-Modeling).

1 Introduction

Existing reward models can overvalue human-preference cues while missing factuality and instruction adherence, motivating agentic reward modeling. REWARDAGENT combines preference rewards with verifiable signals and outperforms advanced reward models in benchmark evaluations.

  • Motivation: Reward models support LLM post-training and inference-time best-of-n search, making reliable scoring important.They are used in RL, DPO training, and inference-time scaling.
  • Motivation: Existing reward models may favor stylistically polished or longer responses while overlooking factual errors and instruction failures.This preference bias can affect reward reliability and the reliability of trained LLMs.
  • Proposed approach: Agentic reward modeling combines conventional reward models with verifiable correctness signals from different aspects.Verification agents can assess factual accuracy or adherence to instruction constraints alongside human preferences.
  • Proposed approach: REWARDAGENT integrates human-preference rewards with factuality and instruction-following signals through a Router, Verification Agents, and Judger.The factuality signal assesses claimed facts, while instruction-following evaluates hard constraints.
  • Experiments: REWARDAGENT significantly outperforms other advanced reward models on RM-Bench, JudgeBench, and IFBench.The experiments include factual-correctness and instruction-following evaluations, including IFBench’s 444 constrained instances.

2 Preliminaries

Reward models score instruction-response pairs but can inherit subjective preference biases and neglect verifiable correctness. Agentic reward modeling addresses this gap by combining a base reward with instruction-selected correctness signals.

  • Reward models: A reward model takes an instruction and response as input and outputs a reward score r_RM(x, y).Here, x denotes the instruction and y denotes the response.
  • Limitations: Human-preference reward models can favor longer, detailed outputs while neglecting verifiable signals such as factuality.The stated motivation links these limitations to subjectivity, complexity, and Bradley-Terry model capacity.
  • Agentic reward modeling: Agentic reward modeling combines a base reward model with verifiable correctness signals from multiple aspects.The formulation is introduced as a general reward-system concept.
  • Agentic reward modeling: The final reward weights the base model by λ and each selected verification agent a_i by w_i.A_x selects the verification agents appropriate for instruction x; weights may be fixed or instruction-adaptive.

3 REWARDAGENT

REWARDAGENT operationalizes agentic reward modeling with a Router, correctness-focused Verification Agents, and a Judger. Its agents assess factuality and hard instruction constraints, while the Judger combines those scores with human-preference rewards.

  • Architecture: REWARDAGENT integrates a base human-preference reward model with factuality and instruction-following verification agents.Factuality assesses claimed facts, while instruction-following checks hard constraints.
  • Router: The Router analyzes each instruction and selects the verification agents appropriate for its requirements.Dynamic selection corresponds to A_x and can reduce inference costs and cumulative errors.
  • Verification Agents: Verification agents evaluate response correctness using LLM backbones augmented with additional tools.The implementation includes dedicated agents for factuality and instruction-following.
  • Verification Agents: The factuality agent compares response differences, generates queries and evidence, and assigns each response a score from 0 to 1.Verifying only differences is intended to reduce search queries and inference-time costs.
  • Verification Agents: The instruction-following agent focuses on objectively evaluable hard constraints such as format and length.It generates and executes Python checks, refining code after execution errors; the final score averages binary constraint scores.
  • Judger: The Judger uses a weighted sum to combine verification-agent scores with base reward-model preferences.In the implementation, λ and w_i are set to 1.0, while instruction-adaptive weighting is left for future work.

4 Experiments

The experiments evaluate REWARDAGENT on reward-model benchmarks and analyze verification-agent design. REWARDAGENT combines human-preference scoring with verifiable correctness signals and is reported to outperform conventional reward models.

  • Experimental Setup: REWARDAGENT is evaluated on RM-Bench, JudgeBench, and IFBENCH, which test factual correctness and instruction-following constraints.IFBENCH contains instructions with hard constraints and chosen responses satisfying all constraints.
  • Experimental Results: REWARDAGENT significantly outperforms ArmoRM and corresponding LLM backbones on the evaluated reward-model benchmarks.The paper also reports that REWARDAGENTLLAMA outperforms larger-parameter reward models and advanced proprietary LLMs such as GPT-4o.
  • Experimental Results: Search-engine retrieval slightly reduces performance on RM-Bench and JudgeBench, possibly because retrieved information contains noise or irrelevant content.The authors leave detailed retrieval-augmented-agent analysis for future work.
  • Analysis: Removing the well-designed verification agent causes a significant performance decrease in the ablation study.The ablations replace the factuality or instruction-following agent with a single LLM scorer.
  • Analysis: The oracle setting performs significantly better for REWARDAGENTMINI and REWARDAGENTLLAMA, while the planner still has room for improvement.The oracle invokes factuality verification on RM-Bench and JudgeBench and instruction-following verification on IFBench.

5 Applications

The paper applies REWARDAGENT to inference-time best-of-n search and DPO training. It reports improved downstream search performance and generally stronger LLMs when preference pairs are constructed with REWARDAGENT.

  • 5.1 Best-of-N Search: REWARDAGENT is applied to best-of-n search for factuality question answering and constrained instruction-following tasks.The experiments use TriviaQA, IFEval, and CELLO, generating 32 responses per instruction with Llama3-8B Instruct and GPT-4o policy models.
  • 5.1 Best-of-N Search: REWARDAGENT significantly improves best-of-n performance over ArmoRM, and its oracle setting further improves results.The same trends are reported when GPT-4o is used as the policy model.
  • 5.2 DPO Training: Preference pairs are constructed by selecting the highest-scored response as chosen and the lowest-scored response as rejected.The datasets include UltraFeedback and on-policy data with 20,000 instructions and eight policy-model responses per instruction.
  • 5.2 DPO Training: LLMs trained with REWARDAGENT-constructed data generally outperform those trained with ArmoRM, especially on factuality and instruction-following datasets.The improvement is reported as more significant for on-policy data.
  • 5.2 DPO Training: REWARDAGENT-annotated training data consistently outperforms original GPT-4-annotated UltraFeedback data in the reported comparisons.REWARDAGENTLLAMA uses Llama3-8B Instruct and Qwen2.5-Coder 7B backbones at lower cost than GPT-4.

6 Related Work

Related work characterizes reward models as central to LLM training and inference-time scaling while identifying biases and missed correctness signals. It also motivates combining human preferences with verifiable rewards, whose generalization beyond specific domains remains difficult.

  • Reward Models: Reward models are used for inference-time scaling and for RL or DPO training of LLMs.They score responses and are trained using preference-based approaches, including Bradley-Terry modeling.
  • Limitations: Existing reward models may exhibit subjective bias, reward hacking, verbosity bias, and insufficient attention to factual correctness.These limitations are reported as affecting reward-model reliability and the performance of trained LLMs.
  • Verifiable Rewards: Rule-based and verifiable reward signals have achieved impressive results in specific domains including math, safety, instruction-following, medical, and finance.The cited work spans several domain-specific applications.
  • Verifiable Rewards: Generalizing verifiable reward signals to general domains remains non-trivial, motivating their combination with human-preference reward models.The paper presents this combination as a promising direction for more reliable reward systems.

7 Conclusion

The paper proposes REWARDAGENT, which combines human-preference rewards with verifiable correctness signals through a reward-agent architecture. It reports that REWARDAGENT significantly outperforms other reward models and generative-reward LLMs across evaluated settings.

  • 7 Conclusion: REWARDAGENT integrates conventional reward models with verifiable correctness signals through a router, verification agents, and a judger.The system is evaluated on reward-modeling benchmarks, best-of-n search, and DPO training.
  • 7 Conclusion: REWARDAGENT significantly outperforms other reward models and LLMs used as generative reward models.

Limitations

The main limitations concern the implementation of REWARDAGENT: its verification agents do not provide perfect rewards, and the system covers only factuality and instruction-following signals.

  • Limitations: The verification agents are far from providing perfect rewards, with an average reward-modeling benchmark score of 72.5%.The paper states that achieving perfect rewards remains challenging and requires further research.
  • Limitations: REWARDAGENT implements verification agents only for factuality and instruction-following.The authors encourage exploring additional verifiable correctness signals.

Ethical Considerations

The paper reports adherence to artifact licenses and describes intended-use risks associated with incorrect rewards and reward hacking. It also states that the data were anonymized and that ChatGPT helped refine some sentences.

  • Ethical Considerations: The authors state that they adhered to the licenses of all utilized artifacts and will release REWARDAGENT, its code, and IFBench under the MIT license.
  • Ethical Considerations: The authors state that the data were well anonymized, while warning that REWARDAGENT may provide incorrect rewards.Users are advised not to conduct reward hacking and to check important information carefully.
  • Ethical Considerations: The paper states that ChatGPT was used to refine some sentences.

Appendices

The appendices document REWARDAGENT’s implementation prompts and evaluation process, then describe IFBENCH as a 444-instance benchmark for multi-constraint instruction-following. IFBENCH uses generated instructions and responses with systematically varied constraint violations, summarized by Figure 4.

  • Appendices: The implementation uses Serper as an external search engine and gpt-4o-mini-2024-07-18 for the REWARDAGENTMINI version.Tables 4 to 6 present the prompts used for implementation.
  • Appendices: The evaluation process covers IFBENCH construction and distribution, evaluation-dataset settings, and additional experimental results.
  • Appendices: IFBENCH contains 444 instances with instructions containing 3 to 5 constraints, chosen responses satisfying all constraints, and rejected responses violating specific constraints.The benchmark evaluates reward models for multi-constraint instruction-following.
  • Appendices: Initial instructions are sampled from Open Assistant, expanded with 3 to 5 generated constraints, and paraphrased into one or two sentences.
  • Appendices: IFBENCH’s constraint categories include content, style, length, keyword, and format requirements.
  • Appendices: Eight GPT-4o candidate responses are generated per instruction, with chosen responses satisfying all constraints and rejected responses balanced by violation categories and counts.Figure 4 stratifies instances by unsatisfied-constraint difficulty and shows the distribution of unsatisfied-constraint categories.

B.2 Evaluation Details

The evaluation uses zero-shot best-of-n searches across TriviaQA, IFEval, and CELLO, with results comparing ArmoRM and REWARDAGENT. Additional evaluations cover DPO training and prompt-based verification components.

  • Best-of-N Evaluation: Best-of-n evaluation uses 500 TriviaQA validation instances, reporting exact-match accuracy for direct answers.TriviaQA uses the rc.nocontext validation split and zero-shot generation.
  • Best-of-N Evaluation: Best-of-n results report average accuracy across four IFEval settings and average CELLO scores using the official evaluation script.IFEval includes strict and loose prompt and instruction settings.
  • Best-of-N Evaluation: Figure 5 compares ArmoRM and REWARDAGENT on TriviaQA, IFEval, and CELLO, including an REWARDAGENT oracle setting.The figure reports best-of-n results as percentages.
  • DPO Training: DPO-training evaluations use FastChat for MT-Bench, the official CELLO script, and lm-evaluation-harness for the remaining tasks.MMLU and MMLU-Pro use 5-shot evaluation, while TriviaQA and TruthfulQA use zero-shot evaluation.
  • Best-of-N Evaluation: REWARDAGENT significantly improves best-of-n performance over ArmoRM with gpt-4o-2024-11-20 as the policy model.The policy model is more powerful than REWARDAGENT itself.
  • Verification Components: The study also provides prompts for routing, factuality assessment, and instruction-following assessment in its verification-agent components.These prompts are documented in Tables 4–6.
Loading 2502.19328v1…