Source-linked AI summary
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, Liang Wang
TL;DR
MRM research has emphasized data and architecture, leaving long-term reasoning in reward modeling underexplored. This paper reformulates reward modeling as rule-based RL and introduces StableReinforce to address instability, normalization issues, and reasoning–answer inconsistencies. R1-Reward improves multimodal reward-modeling benchmark performance, with further gains from increased inference compute.
Problem
Recent MRM improvements focus mainly on data and structure, with limited exploration of using RL to activate long-term reasoning for reward modeling.
Method
StableReinforce reformulates reward modeling as rule-based RL and refines clipping, advantage estimation, reward design, and training-data progression.
Results
R1-Reward outperforms SOTA models across three multimodal reward-modeling benchmarks, with performance improving further as inference samples increase.
Takeaways & Limitations
The results demonstrate that RL can be effectively applied to reward modeling and can improve both benchmark performance and data efficiency.
Takeaways & Limitations
Standard reward models do not fully use detailed human feedback, and scalar rewards provide limited transparency into how rewards are generated.
Abstract
from arXiv · showhide
Multimodal Reward Models (MRMs) play a crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs). While recent advancements have primarily focused on improving the model structure and training data of MRMs, there has been limited exploration into the effectiveness of long-term reasoning capabilities for reward modeling and how to activate these capabilities in MRMs. In this paper, we explore how Reinforcement Learning (RL) can be used to improve reward modeling. Specifically, we reformulate the reward modeling problem as a rule-based RL task. However, we observe that directly applying existing RL algorithms, such as Reinforce++, to reward modeling often leads to training instability or even collapse due to the inherent limitations of these algorithms. To address this issue, we propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of existing RL methods. These refinements result in more stable training dynamics and superior performance. To facilitate MRM training, we collect 200K preference data from diverse datasets. Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks. Compared to previous SOTA models, R1-Reward achieves a $8.4\%$ improvement on the VL Reward-Bench and a $14.3\%$ improvement on the Multimodal Reward Bench. Moreover, with more inference compute, R1-Reward's performance is further enhanced, highlighting the potential of RL algorithms in optimizing MRMs.
1 Introduction
MRMs support MLLM training, data filtering, and inference, but RL-based long-term reasoning remains underexplored for reward modeling. The paper introduces StableReinforce and R1-Reward to address instability and improve benchmark performance.
- MRMs provide reward signals for RL, filter noisy training data, and support inference-time selection strategies for MLLMs.
- RL offers long-term reasoning capabilities, but prior MRM advances mainly focus on data and model structure.
- Reward modeling can be reformulated as rule-based RL, where a policy selects the better answer and receives correctness-based rewards.
- PPO-style clipping can fail when advantages are negative and the policy differs substantially from the reference, causing divergence or crashes.
- Advantage normalization becomes unstable when batch rewards have very low variance, producing extremely large or small advantages.
- StableReinforce refines clipping and advantage normalization, adds an MLLM consistency referee, and uses progressive difficulty training with 200K preference data.
- 3.5%, 13.5%, and 14.6% improvements over SOTA are reported on MM-RLHF Reward Bench, VL Reward-Bench, and Multimodal Reward Bench, respectively.Performance continues improving as inference samples increase.
2 Related Work
Prior reward-modeling approaches include prompting multimodal models directly, while this paper trains a reward model through reinforcement learning. The RL formulation produces rankings while preserving interpretability without requiring an additional reward head.
- Reward models can directly prompt language or multimodal models to output scores or rankings, but this depends heavily on instruction following and comprehension.
- The paper trains a reward model through RL by generating an inference and ranking for each question-answer pair.
- This RL approach enforces the reward-modeling format without an additional reward head while maintaining interpretability.
3 Preliminary and Limitations
Reward models score preferred responses higher using pairwise comparisons, while PPO and Reinforce++ use clipped objectives and advantage normalization to stabilize reinforcement-learning updates. For reward-model training, however, loss behavior and highly imbalanced rewards can still produce numerical instability and poor performance.
- Standard Reward Models: Reward models replace an LLM head with a linear reward head that outputs scalar values and learn from preferred and less-preferred response pairs.Given query x, preferred response y_w, and less-preferred response y_l, training encourages the preferred response to receive the higher reward.
- Standard Reward Models: Standard reward models provide limited transparency because scalar rewards do not reveal how the reward was generated.They also do not fully use detailed human feedback such as textual explanations and nuanced reasoning.
- PPO and Reinforce++: PPO constrains policy updates by clipping the probability ratio between the current and old policies within [1 − ϵ, 1 + ϵ].The objective uses the minimum of an unclipped and clipped surrogate, with ϵ controlling how much the policy can change.
- PPO and Reinforce++: Reinforce++ adds a token-level KL penalty, reward normalization, clipping, scaling, and advantage normalization to improve stability and efficiency.Its advantage normalization subtracts the mean advantage and divides by the advantage standard deviation.
- Drawbacks of Traditional PPO/Reinforce++: PPO-style losses can become unstable when policy probabilities differ greatly, because exponentiation may overflow and negative advantages can produce excessively large losses.The paper gives an example with a loss value of 19930.4 and notes that such large losses can destabilize optimization.
- Drawbacks of Traditional PPO/Reinforce++: Advantage normalization can amplify rare errors when reward labels are highly imbalanced, producing extreme values such as -15.96 for one zero reward among 255 rewards of 1.The resulting large advantage can cause training instability.
4 R1-Reward
StableReinforce improves RL-based multimodal reward modeling by filtering unstable updates, controlling extreme advantages, and aligning reasoning with final answers.
- Algorithm: StableReinforce removes unstable gradients and extreme advantages while adding a consistency reward between reasoning and final answers.The method combines pre-CLIP, an advantage filter, and consistency-aware reward design.
- Algorithm: Pre-CLIP clips large probability ratios before exponentiation to reduce overflow and mitigate instability from large log-probability differences.The method uses 1e3 as a hyperparameter and is described as relatively insensitive to hyperparameter variations.
- Algorithm: The advantage filter retains standardized advantages within [−3, 3] and excludes outliers beyond three standard deviations.This filtering addresses extreme imbalance in the advantage distribution.
- Training Data: R1-Reward training uses a 200K preference dataset, cold-start SFT data, and harder samples selected by GPT-4o’s attempts to reach the ground-truth answer.The data include balanced answer choices and samples requiring at least two attempts or remaining incorrect after three attempts.
5 Experiments
R1-Reward is evaluated against multimodal reward-model baselines across three benchmarks, with additional analyses of inference-time scaling, data efficiency, and StableReinforce components. It achieves strong benchmark performance and benefits from increased sampling, while ablations show that its stability mechanisms are important.
- Main Results: R1-Reward achieves 71.92% average accuracy on VLReward Bench, roughly 9.3% above IXC-2.5-Reward despite using 200k rather than more than 1 million training samples.The comparison highlights data efficiency against the previous best open-source model.
- Main Results: R1-Reward improves over the previous state-of-the-art by 14.3% on Multimodal Reward Bench and performs best across all reported dimensions.The benchmark combines more than ten existing benchmarks with minimal overlap with the training data.
- Main Results: On MM-RLHF-Reward Bench, five-sample voting reaches 85.3% accuracy and fifteen-sample voting reaches 86.47%, outperforming existing models.The benchmark is described as particularly difficult for directly using language models as reward models.
- Test-Time Scaling: Increasing inference samples consistently improves performance: at k = 15, the “Any” strategy approaches 100% accuracy, while voting gains diminish as samples increase.The figure compares Majority Vote and Any Correct across MM-RLHF Reward Bench, VL Reward Bench, and Multimodal Reward Bench.
- Data Efficiency: With the same training data, reinforcement learning substantially improves reward-modeling capability over traditional reward-model baselines, while SFT also outperforms both baselines.The comparison uses Qwen2.5-VL-7B-Instruct backbones and 200K samples.
- Ablations and Analysis: Removing StableReinforce components reduces accuracy and stability; direct Reinforce++ training can produce NaN loss, maximum-length responses, and garbled output.The Consistency Reward Function addresses hallucination behavior, while Advantage Filter and Pre-Clip remove loss outliers and support convergence.
6 Conclusion
The paper introduces R1-Reward, a multimodal reward model trained with StableReinforce, and reports that reinforcement learning improves reward modeling while addressing key training challenges. R1-Reward outperforms state-of-the-art models on several multimodal reward benchmarks and supports test-time scaling, though more advanced scaling methods remain unexplored.
- Contributions: StableReinforce enables effective reinforcement learning for multimodal reward modeling while addressing training instability, advantage normalization limits, and reasoning–result inconsistencies.The approach incorporates pre-clipping, advantage filtering, consistency reward, and progressive difficulty training.
- Results: R1-Reward outperforms state-of-the-art models on several multimodal reward modeling benchmarks, improving accuracy and data efficiency.
- Limitations and future work: R1-Reward demonstrates test-time scaling capabilities, while the paper tests only a simple majority-voting strategy and leaves more advanced methods for future work.