Source-linked AI summary
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang
TL;DR
LVLMs lack broadly capable, publicly available multi-modal reward models despite the usefulness of reward signals for training and inference. The paper introduces IXC-2.5-Reward, trained on diverse text-image-video preferences, and reports strong multi-modal benchmark performance alongside applications in RL, test-time scaling, and data cleaning.
Problem
Publicly available multi-modal reward models for LVLMs are scarce, while existing models often use narrow-domain data or generative evaluation prompts.
Method
IXC-2.5-Reward combines diverse multi-modal preference data with an LVLM-based scalar reward model and supports PPO training, response selection, and data cleaning.
Results
70.0% on VL-RewardBench was the best reported performance, exceeding Gemini-1.5-Pro at 62.5% and GPT-4o at 62.4%.
Takeaways & Limitations
IXC-2.5-Reward provides a multi-modal reward signal for RL training, test-time selection, and filtering problematic image and video instruction-tuning samples.
Takeaways & Limitations
The training data is primarily English-centric, potentially limiting multilingual capabilities and reflecting cultural viewpoints and societal biases in English-speaking communities.
Abstract
from arXiv · showhide
Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward
1 Introduction
IXC-2.5-Reward addresses the scarcity and narrow coverage of multi-modal reward models by combining diverse preference data with an LVLM-based scoring framework. It supports reinforcement learning, test-time response selection, and data cleaning, while achieving strong benchmark results.
- Motivation: Multi-modal reward models remain underexplored because preference data is scarce, text-dominated, and concentrated in narrow domains.Existing models are often limited to hallucination mitigation or operate as generative evaluators.
- Approach: IXC-2.5-Reward augments InternLM-XComposer2.5 with a scoring head and is trained to evaluate image, video, and textual inputs across diverse domains.The preference pipeline uses prompts, generated responses, and GPT-4o or verifiers for preference judgments.
- Results: 70.0% on VL-RewardBench was the best reported performance, exceeding Gemini-1.5-Pro at 62.5% and GPT-4o at 62.4%.The model also averaged 88.6% on RewardBench and 68.8% on RM-Bench.
- Applications: PPO training with IXC-2.5-Reward improved IXC-2.5-Chat on multi-modal instruction following and in-the-wild chatting benchmarks.The result supports using IXC-2.5-Reward as a supervisory signal during reinforcement learning.
- Applications: Best-of-N sampling with IXC-2.5-Reward produced further gains over the RL-trained chat model for selecting responses.Low reward scores also correlated strongly with hallucinations and mismatched image/video question-answer content, enabling data cleaning.
2 Related Work
Prior reward-model research distinguishes discriminative, generative, implicit, outcome, and process formulations, while multi-modal reward models remain limited in scope and evaluation coverage. Existing resources include preference datasets and benchmarks developed primarily around language-model settings.
- Reward Models: Reward models support RLHF and test-time scaling by evaluating whether outputs align with human preferences.They can guide training by penalizing undesirable behavior and inference by selecting among candidate responses.
- Reward Models: Discriminative, generative, and implicit reward models differ in whether they classify outputs, generate textual feedback, or use preference-optimized log probabilities.Reward models are also categorized as outcome or process reward models; IXC-2.5-Reward is discriminative and outcome-based.
- Multi-Modal Reward Models: Existing LVLM reward models are limited to specific domains or weaker base models, leaving vision RLHF reliant on variants of off-policy DPO.Open-source LVLMs have also been used as generative reward models.
- Evaluation: Language-model reward benchmarks cover general abilities, multilingual evaluation, retrieval-augmented generation, and mathematical process rewards.Multimodal benchmark development has been slowed by the limited availability of multimodal reward models.
3 IXC2.5-Reward
IXC-2.5-Reward is trained on pairwise preferences spanning text, images, videos, and diverse domains, using an LVLM backbone with a scalar score head. Its training strategy freezes visual components while optimizing language-model and scoring components, with controls against length bias.
- Data Preparation: IXC-2.5-Reward combines open-source preference data with newly collected examples because public data contains limited image and scarce video coverage.The data uses chosen and rejected responses to represent pairwise preferences.
- Data Preparation: Newly collected data covers text-rich document understanding, mathematical reasoning, video understanding, and instruction following.Rejected responses are generated from supervised fine-tuning data and additional in-house instruction-following data.
- Model Architecture: The model reuses IXC-2.5-Chat components, including the visual encoder and MLP projector, to align image and video data with text.This lets preference training focus on reward prediction without additional modality-alignment pretraining.
- Model Architecture: A score head converts averaged hidden-state features for prompt-response pairs into a binary scalar reward r(x, y).The model replaces the final linear layer of IXC-2.5 with this scoring head.
- Training Strategy: The vision encoder and projector are frozen, while the language model and score head are trained on preference data.Removing pairs where the chosen response is significantly longer than the rejected response helps prevent length-quality associations.
4 The Applications of IXC-2.5-Reward
IXC-2.5-Reward is applied to reinforcement learning, test-time response selection, and instruction-tuning data cleaning. Together, these applications improve multimodal dialogue and instruction following, select higher-scoring outputs, and identify problematic training samples.
- 4.1 IXC-2.5-Reward for RL training: IXC-2.5-Reward supplies PPO rewards for training IXC-2.5-Chat while keeping the policy close to a reference model for stability.A critic initialized from IXC-2.5-Reward is trained alongside the policy.
- 4.1 IXC-2.5-Reward for RL training: IXC-2.5-Chat improves multi-modal chat quality and instruction following when trained with IXC-2.5-Reward and PPO.The authors use this result to demonstrate the reward model’s value as a reward-score provider.
- 4.2 IXC-2.5-Reward for Test-Time Scaling: Best-of-N sampling generates multiple responses, scores them with IXC-2.5-Reward, and selects the response with the highest reward.The paper states that this produces additional performance gains compared to the RL-trained IXC-2.5-Chat.
- 4.3 IXC-2.5-Reward for Data Cleaning: Problematic instruction-tuning samples negatively affect LVLM training, while classifier-based filtering has limitations for long-context, high-resolution, and video inputs.These limitations motivate using IXC-2.5-Reward to clean image and video instruction-tuning data.
- 4.3 IXC-2.5-Reward for Data Cleaning: Low IXC-2.5-Reward scores correlate strongly with hallucinations, empty answers, and irrelevant image/video-text pairings in instruction-tuning data.Figure 2 visualizes detected outlier and noisy examples from datasets including ALLaVA and LLaVA-Video-178K.
5 Experiments
The experiments evaluate IXC-2.5-Reward on multimodal and text-only reward-model benchmarks, then assess its use for training IXC-2.5-Chat. Results show strong reward-model performance and improved instruction following and multimodal chat without sacrificing other evaluated capabilities.
- 5.1.1 Results on VL-RewardBench: 70.0% Macro Accuracy on VLRewardBench is the highest among all evaluated models, while open-source IXC-2.5-Reward reaches 65.8% overall accuracy.The model is a 7B-parameter discriminative reward model that predicts a scalar reward.
- 5.1.1 Results on VL-RewardBench: 84.7% accuracy on general problems exceeds other generative reward models on VLRewardBench.The authors attribute this to more correct scalar-score classifications where previous LVLMs often produced tied judgments.
- 5.2 Evaluation Results of IXC-2.5-Chat: IXC-2.5-Chat outperforms previous state-of-the-art models across WildVision, MIA, and MM-MT instruction-following and chat benchmarks.These benchmarks cover real-world multimodal interactions and instruction-following ability, with evaluation using LLM-as-a-Judge for open-ended responses.
- 5.2 Evaluation Results of IXC-2.5-Chat: IXC-2.5-Chat performs comparably to the supervised-finetuned IXC-2.5 model on knowledge, reasoning, and text-rich categories.The reported result indicates improved instruction following and conversational ability without sacrificing performance in these areas.
6 Conclusion and Future Work
IXC-2.5-Reward supports multi-modal RL training, test-time scaling, and data cleaning, while IXC-2.5-Chat is trained to improve multi-modal user conversations. The authors identify on-policy reinforcement learning and future reward benchmarks as promising directions.
- IXC-2.5-Reward enables multi-modal RL training, test-time scaling, and data cleaning.
- IXC-2.5-Chat is trained with RLHF techniques to improve multi-modal chat experiences through detailed explanations and in-depth answers.
- The authors highlight on-policy reinforcement learning as a promising direction for future multi-modal reward-model research.
7 Limitations
The reward model is trained primarily on English-language corpora, which may limit multilingual capabilities and reflect cultural viewpoints and societal biases associated with English-speaking communities.
- English-centric training data potentially limits the reward model’s multilingual capabilities.
- English-language datasets may reflect cultural viewpoints and societal biases prevalent in English-speaking communities.
- Future work should incorporate multilingual datasets to improve generalizability and fairness.
A More Experimental Results
Additional experiments examine training settings, response-length constraints, best-of-N test-time scaling, and qualitative multi-modal dialogues. Length constraints reduce reward inflation from verbosity, while best-of-N sampling improves responses with only a small increase in average length.
- Implementation Details: IXC-2.5-Reward uses a 1e-5 learning rate and batch size 256, while IXC-2.5-Chat uses a 5e-5 learning rate and batch size 256.The PPO hyperparameters are γ = 0.99, β = 0.95, and ϵ = 0.2.
- The Impact of Length Constraints: Length constraints prevent overly long responses from artificially inflating rewards during IXC-2.5-Chat training.Without constraints, average response length increased from 274 to 361 tokens; the authors retained constraints because their goal is user experience rather than benchmark scores.
- Results on Test-Time Scaling: Best-of-N sampling further improves test-time results while increasing average response length only from 274 to 283 tokens.The authors attribute the improvement to higher-quality responses rather than exploiting length bias.
- Qualitative Results: Qualitative dialogues emphasize superior organization, more comprehensive answers, and more detailed explanations in multi-modal chat interactions.
- Qualitative Results: The qualitative examples include requests for structured stock information and professional redesign advice.