PaperlayerView exact revision on arXiv

Source-linked AI summary

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang

arXiv:2501.12368v2cs.CVcs.CL

TL;DR

Public multimodal reward models for large vision-language models are scarce. InternLM-XComposer2.5-Reward combines a multimodal preference corpus with a reward-scoring model, reaching 70.0% on VL-RewardBench, ahead of Gemini-1.5-Pro and GPT-4o.

  • Problem

    Public multimodal reward models for LVLMs remain scarce, despite their crucial role in training and inference and the need to cover image, video, and text.

    Source: §1 · p. 1

  • Method

    IXC-2.5-Reward augments InternLM-XComposer2.5 with a scoring head and trains on a preference corpus spanning text, image, video, and diverse domains.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    70.0% on VL-RewardBench puts IXC-2.5-Reward ahead of Gemini-1.5-Pro (62.5%) and GPT-4o (62.4%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model serves as an RL reward signal, a best-of-N response selector, and a filter for problematic instruction-tuning samples.

    Source: §1 · p. 3

    Training data is primarily English-language, potentially limiting multilingual capabilities and reflecting cultural viewpoints and biases prevalent in English-speaking communities.

    Source: §7 · p. 9

Abstract

from arXiv · show

Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward

1 Introduction

The introduction identifies scarce, domain-skewed preference data as a barrier to multimodal reward models for LVLMs and presents IXC-2.5-Reward, designed to score text, image, and video inputs across diverse domains.

  • Multimodal RMs for LVLMs remain underexplored because preference data is scarce and domain-skewed, constraining open-source LVLM instruction following and safety-refusal capabilities.Source: §1 · p. 1
  • IXC-2.5-Reward adds a scoring head to InternLM-XComposer2.5 and trains on preference data spanning text, image, and video across diverse domains.Its pipeline selects prompts, generates responses, and uses GPT-4o or a verifier for preference judgments.Source: §1 · p. 2
  • IXC-2.5-Reward scores 70.0% on VL-RewardBench, ahead of Gemini-1.5-Pro (62.5%) and GPT-4o (62.4%), plus 88.6% average on RewardBench and 68.8% on RM-Bench.Source: §1 · p. 2
  • PPO training IXC-2.5-Chat with IXC-2.5-Reward yields clear improvements on multimodal instruction-following and in-the-wild chatting benchmarks.Source: §1 · pp. 2–3
  • Best-of-N sampling with IXC-2.5-Reward adds performance gains over RL-trained IXC-2.5-Chat by selecting stronger candidate responses.Source: §1 · p. 3
  • Low IXC-2.5-Reward scores correlate with problematic samples, including hallucinations and image/video-question/answer mismatches, supporting data cleaning.The paper proposes using the model to clean LVLM pre-training and post-training data.Source: §1 · p. 3

2 Related Work

The related work frames reward models as tools for RLHF and test-time scaling, then identifies gaps in multimodal reward modeling and evaluation that motivate IXC-2.5-Reward.

Reward Model in Large Language Models

  • RMs support RLHF and test-time scaling, take discriminative, generative, or implicit forms, and may provide outcome or process supervision; IXC-2.5-Reward is a discriminative ORM.Discriminative RMs classify sequences, generative RMs produce textual feedback, and implicit RMs use predicted log probabilities as reward signals.Source: §2 · p. 3

Reward Model in Large Vision-Language Models

  • Prior LVLM reward models are domain-limited or use weaker base models, while scarce effective multimodal RMs have driven reliance on DPO variants, human annotation, and proprietary models.The paper positions IXC-2.5-Reward as enabling PPO training and DPO data selection at low cost.Source: §2 · p. 3

Reward Model Evaluations

  • LLM reward-model benchmarks cover general abilities, multilingual evaluation, RAG, and mathematical process reward, but existing multimodal evaluation focuses on generative RMs and omits process supervision.The paper anticipates significant future progress in multimodal reward-model benchmarking.Source: §2 · p. 3

3 IXC2.5-Reward

IXC-2.5-Reward is trained on open-source and newly collected multimodal pairwise preference data, using an IXC-2.5-based model to predict reward scores. Its training freezes the vision encoder and projector, and its data strategy addresses response-length bias.

Data Preparation

  • The preference corpus combines open-source data with newly collected examples spanning text-rich documents, math reasoning, video understanding, and instruction following.New examples start from SFT prompts and chosen responses; IXC-2.5 generates candidate outputs, with GPT-4o pairwise evaluation for general and text-rich data and verifier functions for math and instruction following. Some in-house instruction-following data is planned for future release.Source: §3 · p. 4

Model Architecture

  • IXC-2.5-Reward replaces IXC-2.5’s final linear layer with a score head that maps averaged token hidden states to a scalar reward.It reuses most pretrained IXC-2.5-Chat weights, including the visual encoder and MLP projector, so preference training does not require separate modality-alignment pretraining (Zhang et al., 2024d).Source: §3 · p. 4

Training Strategy

  • Training freezes the vision encoder and projector and updates only the LLM and score head, while leaving dynamic image partitioning unchanged (Zhang et al., 2024c).The vision encoder and projector are initialized from IXC-2.5 (Zhang et al., 2024c).Source: §3 · p. 4

Length Constraints

  • The method filters pairs where the chosen response is much longer than the rejected one, yet removing the reward model’s length constraint improved PPO policy performance.The authors note that length bias also affects LVLM evaluations of open-ended VQA, where overly long responses can inflate scores.Source: §3 · p. 4

4 The Applications of IXC-2.5-Reward

IXC-2.5-Reward is applied to PPO training, best-of-N test-time response selection, and cleaning image and video instruction-tuning data. The reported applications show improved chat and instruction following, reward-based selection, and identification of problematic samples.

4.1 IXC-2.5-Reward for RL training

  • PPO training with IXC-2.5-Reward yields IXC-2.5-Chat, which improves multi-modal chat quality and instruction following.Training maximizes reward while staying close to IXC-2.5 for stability; a critic initialized from the reward model reduces update variance, and prompts prioritize general chat and instruction following while retaining domain diversity.Source: §4.1 · p. 5

4.2 IXC-2.5-Reward for Test-Time Scaling

  • For test-time scaling, IXC-2.5-Reward scores N outputs generated by IXC-2.5-Chat and selects the highest-scoring response.The method uses Best-of-N sampling with different random seeds for a given prompt.Source: §4.2 · p. 5

4.3 IXC-2.5-Reward for Data Cleaning

  • Low IXC-2.5-Reward scores correlate strongly with hallucinations, empty answers, and irrelevant image/video-text pairings, enabling cleaning of LVLM pre- and post-training data.The approach addresses limitations of classifier-based filtering on long-context inputs, high-resolution images, and videos.Source: §4.3 · pp. 5–6, Figure 2 · p. 6

5 Experiments

Experiments find strong performance for IXC-2.5-Reward on multi-modal and language-only reward benchmarks, while IXC-2.5-Chat improves instruction following and in-the-wild chat without sacrificing performance in other evaluated categories.

5.1 Evaluation Results of IXC-2.5-Reward

  • Evaluation covers VLRewardBench's 1,250 multi-modal problems, Reward-Bench's 2,985 language-only problems, and RM-Bench's 1,237 language-only problems.Reward-Bench includes chat, chat hard, safety, and reasoning; RM-Bench spans chat, math, code, and safety across easy, normal, and hard tracks.Source: §5.1 · p. 6

5.1.1 Results on VL-RewardBench

  • 65.8% overall accuracy among open-source models and 70.0% Macro Accuracy among all models: IXC-2.5-Reward leads on VLRewardBench.The open-source model has 7B parameters.Source: §5.1.1 · pp. 6–7, Table 3 · p. 7

5.1.2 Results on Reward Bench and RM-Bench

  • IXC-2.5-Reward performs considerably and surpasses other multi-modal models on Reward Bench and RM-Bench, but remains sensitive to subtle content variations and style biases.The authors identify improving robustness to these sensitivities as an area for further research.Source: Table 4 · p. 7, §5.1.2 · p. 7, Table 5 · p. 8

5.1.1 Results on VL-RewardBench

  • 84.7% accuracy on VLRewardBench general problems exceeds other generative reward models, with scalar scores helping resolve judgments that previous LVLMs often tied.Source: §5.1.1 · p. 7

5.2 Evaluation Results of IXC-2.5-Chat

  • IXC-2.5-Chat outperforms previous state-of-the-art models on WildVision, MIA, and MM-MT, improving multi-modal understanding and in-the-wild chat responses.The evaluation uses open-ended questions scored with an LLM-as-a-Judge approach.Source: Table 6 · p. 8, §5.2 · p. 8
  • On knowledge, reasoning, and text-rich categories, IXC-2.5-Chat performs comparably to the IXC-2.5 SFT model, without sacrificing those areas.The results indicate that reward-model training improves instruction following and conversational ability while preserving performance in these categories.Source: §5.2 · p. 8

6 Conclusion and Future Work

IXC-2.5-Reward supports multi-modal RL training, test-time scaling, and data cleaning. It was also used to train IXC-2.5-Chat with RLHF, while future work may explore on-policy algorithms, reward benchmarks, and video alignment.

  • IXC-2.5-Reward supports multi-modal RL training, test-time scaling, and data cleaning.Source: §6 · p. 8
  • Using RLHF, the authors trained IXC-2.5-Chat to optimize multi-modal chat with detailed explanations and in-depth answers.Source: §6 · p. 8
  • Future research may explore on-policy reinforcement learning, including reward benchmarks and algorithms for video alignment.Source: §6 · p. 8

7 Limitations

The reward model’s training data is primarily English-sourced, which may limit its multilingual capabilities and reflect cultural viewpoints and societal biases prevalent in English-speaking communities.

7 Limitations

  • English-centric training data may constrain multilingual capabilities and carry English-speaking cultural viewpoints and societal biases; the authors suggest adding multilingual datasets to improve generalizability and fairness.Source: §7 · p. 9

Appendix · A More Experimental Results

The appendix reports that removing response-length constraints increases response length and improves open-ended benchmark results, while the authors retain constraints to prioritize user experience. Best-of-N sampling improves results with only a slight increase in average response length.

A More Experimental Results

  • IXC-2.5-Reward training used a learning rate of 1e-5 and batch size 256; IXC-2.5-Chat used 5e-5 and batch size 256.PPO hyperparameters were γ = 0.99, β = 0.95, and ϵ = 0.2.Source: A More Experimental Results · p. 15
  • Without length constraints, WildVision response length rises from 274 to 361 average tokens, while open-ended benchmark results reach state-of-the-art levels.The length-constraint ablation is reported in Table 7.Source: A More Experimental Results · p. 15, Table 7 · p. 15
  • The authors retain length constraints to avoid overly long responses and prioritize user experience, noting that evaluators may favor unnecessary detail when prompts do not penalize length.They suggest future multimodal chat and dialogue evaluations address potential length and style biases.Source: A More Experimental Results · p. 15
  • Best-of-N sampling further improves results while average response length increases only slightly, from 274 to 283 tokens.The authors attribute the gains to higher-quality responses rather than exploiting length bias; Table 8 reports the Best-of-N results.Source: A More Experimental Results · p. 15, Table 8 · p. 15
  • IXC-2.5-Chat’s multimodal dialogues show stronger organization, presentation, answer depth, and explanatory detail, enhancing its effectiveness in chat interactions.Figures 3 and 4 visualize instruction-following and open-ended-question dialogues, respectively.Source: A More Experimental Results · p. 15, Figure 3 · p. 16, Figure 4 · p. 17

Related papers

Loading 2501.12368v2…