Source-linked AI summary
Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
Xiaokun Wang, Peiyu Wang, Jiangbo Pei, Wei Shen, Yi Peng, Yunzhuo Hao, Weijie Qiu, Ai Jian, Tianyidan Xie, Xuchen Song, Yang Liu, Yahui Zhou
TL;DR
Existing multimodal reward models have limited generalizability across tasks and struggle to evaluate advanced VLM reasoners with complex inference. Skywork-VL Reward is built upon Qwen2.5-VL-7B-Instruct [13], a 7-billion-parameter VLM with a ViT visual encoder, vision-language adapter/projector, and Qwen language-model decoder. Skywork-VL Reward achieves state-of-the-art results on VLRewardBench, competitive performance in text-only scenarios, and its preference data proves effective for MPO training.
Problem
Existing multimodal reward models have limited generalizability across tasks and struggle to evaluate advanced VLM reasoners with complex inference.
Source: §1 · p. 1Method
Skywork-VL Reward is built upon Qwen2.5-VL-7B-Instruct [13], a 7-billion-parameter VLM with a ViT visual encoder, vision-language adapter/projector, and Qwen language-model decoder.
Source: §3.3 · p. 5Results
Skywork-VL Reward achieves state-of-the-art results on VLRewardBench, competitive performance in text-only scenarios, and its preference data proves effective for MPO training.
Source: §1 · p. 2Takeaways & Limitations
The authors present Skywork-VL Reward as a multimodal evaluator for standard VLMs and advanced VLM reasoners across diverse domains and tasks.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility.
1 Introduction
The introduction frames multimodal reward modeling as an early-stage field whose models struggle to generalize across tasks and assess advanced VLM reasoners. Skywork-VL Reward combines broad preference data with a two-stage reward-model training approach, achieving strong benchmark results and supporting MPO training.
1 Introduction
- Multimodal reward models remain early-stage, with limited generalizability across tasks and difficulty evaluating advanced VLM reasoners with complex inference.The introduction identifies a need for models that assess both standard VLMs and advanced reasoners across diverse domains and tasks.Source: §1 · p. 1
- The preference dataset combines public data and in-house annotations, with pairs covering basic image descriptions through intricate reasoning and responses from standard VLMs and advanced reasoners.Pairs include an image when applicable, a textual prompt, and candidate responses.Source: §1 · p. 2
- The model uses Qwen2.5-VL-7B-Instruct with a reward head and a two-stage training paradigm combining pure-text and multimodal data.The scalar reward scores are designed to align with human preferences.Source: §1 · p. 2
- Skywork-VL Reward achieves state-of-the-art results on VLRewardBench, remains competitive in text-only scenarios, and its preference data proves effective for MPO training.The authors describe the model as evaluating both standard VLMs and advanced VLM reasoners across diverse domains and tasks.Source: §1 · p. 2
2 Related Work
Prior work spans text-only reward-model designs and a developing multimodal field, where generative approaches, discriminative models, and large-scale preference-data synthesis have been explored. Most multimodal approaches score final outputs, while VisualPRM provides process-level signals.
Reward Models for Text-only Large Language Models
- Text-only reward models differ by model form—discriminative, generative, or implicit—and by whether feedback evaluates overall outcomes or intermediate reasoning steps.Discriminative models output a score or preference probability; generative models produce an evaluation or verdict, while implicit models reparameterize preference learning through DPO.Source: §2 · p. 2
Reward Models for Vision-Language Models
- Multimodal reward-model work includes generative VLMs, discriminative models such as LLaVA-RLHF, and large-scale preference datasets labeled using advanced models as proxies for human evaluation.IXC-2.5-Reward uses GPT-4o and verifier functions to generate preference labels.Source: §2 · p. 2
- Most current multimodal approaches provide outcome rewards for final VLM responses, whereas VisualPRM supplies fine-grained process rewards and serves as a critic for test-time scaling.The cited work reports that VisualPRM enhances existing VLM reasoning capabilities.Source: §2 · p. 2
3 Method
Skywork-VL Reward evaluates multimodal responses with scalar preference scores, using a curated comparison dataset and a fine-tuned vision-language model. Its method combines staged data curation, a reward-specific architecture, and pairwise preference training.
3.1 Dataset Construction
- The model is trained on preference comparisons drawn from open-source datasets and in-house annotations spanning multimodal understanding and complex reasoning tasks.Sources include LLaVA-Critic-113k, Skywork-Reward-Preference-80K-v0.2, RLAIF-V-Dataset, and approximately 50,000 human-annotated reasoning comparisons.Source: §3.1 · pp. 3–4
3.2 Data Curation Procedure
- Curation yielded approximately 200,000 distinct, high-confidence preference pairs; these pairs were used to train a surrogate RM, which then scored the entire dataset.Chosen responses with low surrogate-RM scores were regenerated using GPT-4o. Chosen responses were also regenerated when the score difference between chosen and rejected responses was minimal.Source: §3.2 · p. 4
- Reasoning-style responses are generated either directly by Skywork R1V or through image descriptions passed to an InternVL and Deepseek R1 pipeline.The two-step route uses a textual image description in place of the original visual input.Source: §3.2 · pp. 4–5
- The training set contains approximately 190k preference comparisons, about 70% with images, after staged curation and reasoning-data augmentation.The process retained 150,000 samples after refinement; the final set combines open-source and in-house data.Source: §3.2 · pp. 4–5
3.3 Model Architecture
- The model uses Qwen2.5-VL-7B-Instruct as its base, with a reward head that outputs a scalar score instead of a text sequence.The score is produced from the final hidden state following the answer’s last token.Source: §3.3 · p. 5
3.4 Reward Model Loss Function
- Pairwise preference training maximizes the score difference between preferred and rejected responses, learning relative rankings rather than calibrated absolute rewards.Equal or near-equal preference examples are excluded because they can introduce ambiguity.Source: §3.4 · p. 5
4 Experiment
Skywork-VL Reward is evaluated on multimodal and text-only reward benchmarks, with case studies and MPO training assessing its behavior and downstream use. It leads on VL-RewardBench overall and performs competitively on RewardBench.
4.1 Training Details
- Training uses two stages: multimodal preference data first, followed by pure-text preference data to improve text-only generalization and reasoning.The visual encoder is frozen; the projector, language backbone, and reward head remain trainable.Source: §4.1 · pp. 5–6
4.4 VL-RewardBench Evaluation
- 73.1% overall accuracy and 69.0% Macro Average exceed Gemini-2.0-flash-exp and IXC-2.5-Reward-7B on VL-RewardBench.The comparisons are 73.1% vs. 68.8% and 66.3% overall accuracy, and 69.0% vs. 64.5% and 68.6% Macro Average, respectively.Source: Table 3 · p. 6, §4.4 · p. 7
4.2 Evaluation Benchmarks
- VL-RewardBench covers general multimodal queries, visual hallucination detection, and image-based complex reasoning; RewardBench tests text-only chat, safety, and reasoning.The evaluation reports key dimensions and aggregated overall accuracy on both benchmarks.Source: §4.2 · p. 6
4.3 Baselines
- Comparisons include proprietary and open-source multimodal reward models on VL-RewardBench, and language-only and similarly sized multimodal models on RewardBench.Some evaluated results were reproduced by the authors, while others came from official reports.Source: §4.3 · p. 6
4.5 RewardBench Evaluation
- 90.1% average score on RewardBench outperforms the second-best comparable-scale multimodal RM, IXC-2.5-Reward-7B, by 2.0%.Skywork-VL Reward leads comparable-scale multimodal RMs in Chat Hard (87.5%), Safety (91.1%), and Reasoning (91.8%).Source: Table 4 · p. 7, §4.5 · p. 7
4.6 Case Study
- In two examples, the reward model favors concise, well-structured reasoning over rambling or redundant responses across geometry and chart tasks.For the geometry problem, both answers reach the same numerical conclusion, but differ in their derivations.Source: §4.6 · pp. 7, 10, Figure 2 · p. 8, Figure 3 · p. 9
4.7 Skywork-VL Reward for Mixed Preference Optimization
- 69.2% to 73.5%: adding Skywork-VL Reward as an MPO reward raises the VLM reasoner’s MathVista score.The preference data generated with the model is used to fine-tune a VLM reasoner via MPO.Source: Table 5 · p. 10, §4.7 · p. 10
5 Conclusion
Skywork-VL Reward combines a curated multimodal preference dataset with two-stage training to evaluate responses from standard VLMs and VLM reasoners. It achieves state-of-the-art performance on VL-RewardBench, performs competitively on text-only RewardBench, and its supervised signals for MPO significantly enhance VLM multimodal reasoning.
5 Conclusion
- Skywork-VL Reward achieves state-of-the-art performance on VL-RewardBench and competitive performance on text-only RewardBench.Source: §5 · p. 10
- The model uses a large-scale, curated preference dataset and a two-stage training paradigm to assess outputs from standard VLMs and VLM reasoners.Source: §5 · p. 10
- Using Skywork-VL Reward to provide supervised signals for MPO significantly enhances VLM multimodal reasoning abilities.Source: §5 · p. 10