BaseReward: A Strong Baseline for Multimodal Reward Model

1 Introduction

• The experiments cover reward-model paradigms, reward-head design, training strategies, more than ten preference datasets, backbone and scale, and ensemble methods.The data analysis includes multimodal and text-only preference data.Source: §1 · p. 2
• Approximately 11% on MM-RLHF-Reward Bench and 18% on VL-Reward Bench; BaseReward sets new SOTA and improves RL-tuned MLLM performance across perception, reasoning, and conversational tasks.The benchmark improvements are reported against previous models; the RL gains are reported across a diverse range of tasks.Source: §1 · p. 3
• BaseReward combines a Qwen2.5-VL backbone, an optimized two-layer reward head, and curated multimodal and text-only preference data.The paper presents it as a powerful and efficient baseline for multimodal reward modeling.Source: §1 · pp. 2–3

2 Related Work

• MLLM research addresses multimodal understanding and generation alongside challenges such as long-form context, efficiency, hallucinations, and conversation.The passage connects alignment with human preferences to RLHF and reward models as its learning signal.Source: §2 · p. 3
• Reward modeling includes prompted scoring or ranking, direct scalar scoring with a reward head [Liu et al., 2024a; Zang et al., 2025; Minghao Yang, 2024; Lou et al., 2024; Wang et al., 2024a], and critic-style evaluation with an additional reward head.The direct scalar approach is computationally efficient but offers limited interpretability, while prompted scoring depends on instruction-following and comprehension.Source: §2 · p. 3
• Prior work lacks a unified, fair comparison of reward-model paradigms and deeper exploration of reward-head structure, training strategies, and data sources.The paper addresses these gaps through experimental analysis aimed at a recipe for building high-performance MRMs.Source: §2 · p. 3

3.1 Preliminary

• A reward model replaces a pretrained language-model head with a reward head that outputs a scalar, trained on human pairwise comparisons to score preferred responses higher.The passage defines r(y|x) as the scalar reward and σ as the sigmoid function in the ranking loss.Source: §3.1 · p. 3

3.2 Evaluation Benchmarks and Metrics

• Multimodal evaluation uses VL-Reward Bench, Multimodal RewardBench, and MM-RLHF-Reward Bench, which measure preference accuracy, six capability areas, and response-pair ranking, respectively.VL-Reward Bench reports Overall Accuracy and Macro Average Accuracy; MM-RLHF-Reward Bench reports Acc and the stricter Acc+ metric.Source: §3.2 · p. 4
• Text-only generalization is tested with RMBench’s Easy, Normal, and Hard Accuracy across four domains and Reward Bench’s five capability subtasks.The subtasks include Chat, Chat Hard, Safety, Reasoning, and Prior Sets.Source: §3.2 · p. 4
• Default experiments use Qwen2.5-VL-7B and about 200,000 preference pairs; data ablations cover all benchmarks, while architecture ablations use a sufficient subset.The pairs come from the R1-Reward SFT dataset, aggregated from MM-RLHF, VL-Feedback, and RLHF-V.Source: §3.2 · p. 4

4.1 Reward Modeling Approaches

• Naive-RM is comparable or better in VQA, general, and hallucination tasks, while Long-CoT-GRM excels in coding and safety/bias; Naive-RM is cheaper to apply during reinforcement learning.The paradigm comparison used a standardized training protocol and curated default dataset.Source: §4.1 · p. 5

4.2 Reward Model Design

• A two-layer SiLU reward head performs best; a one-layer linear head performs worst, and other activations or deeper heads add no significant gains.The experiments find that using an MLP reward head significantly improves capability over the simple linear-layer design.Source: §4.2 · pp. 5–6, Table 2 · p. 5

4.3 Training Regularization Strategies

• Increasing zero-coefficient weight λ from 0 to 0.1 degrades performance, and length normalization alone does not improve on the unregularized baseline, so default training omits regularization loss.The figure shows the performance trend as λ changes and marks the length-normalization-only baseline.Source: §4.3 · p. 6, Figure 1 · p. 7

4.4 Common Training Datasets

• MMIF and SHP offer limited training benefit, and no single dataset substantially advances coding capability, indicating specialized downstream tasks require dedicated additional data.The authors attribute limited dataset benefit likely to insufficient diversity or data-quality issues.Source: §4.4 · pp. 6–7
• MMPR and RLAIF-V push VL-Reward Bench hallucination accuracy beyond 90%, while text-only datasets match multimodal average performance and improve Multi-Modal Reward Bench results.Text-only preference data also helps preserve performance on text-based reward benchmarks.Source: §4.4 · pp. 6–7, Table 5 · p. 8

4.5 Optimizing Multimodal RMs for Pure-Text Tasks

• On two pure-text benchmarks, mixed-data Qwen 2.5 VL-7B brings no gain over text-only training, while LLMs trained on identical text data perform better; the authors recommend separate RMs.They propose dynamically selecting a dedicated pure-text or multimodal RM by input type during reinforcement learning.Source: §4.5 · p. 8, Figure 2 · p. 9

4.6 Impact of Base Model Selection and Scale

• Qwen-VL leads multimodal benchmarks (93.5 vs. 83.7 on MM-RLHF-Reward), whereas InternVL leads RewardBench (84.0 vs. 75.8); scaling gains are marginal, favoring sub-10B options under compute constraints.The reported accuracy values are for Qwen2.5-VL-7B and Intern-VL3-8B, respectively.Source: §4.6 · pp. 8–9, Table 6 · p. 9

4.7 Ensemble Strategies for Reward Models

• Simple averaging raises the three multimodal benchmark average from below 81.0 to 82.6; adding a text-only Qwen 3 raises Pure Text Avg from 80.7 to 82.7, while validation weights add complexity without clear advantage.The Qwen 3 was trained exclusively on the text-only data.Source: §4.7 · p. 10, Table 7 · p. 10

5.1 Structure and Training Strategy

• BaseReward uses Qwen2.5-VL-7B with a two-layer SiLU MLP reward head, no auxiliary losses, and seven datasets totaling 2.8M preference pairs.The final learning rate is 3e−6, selected from a grid search; training uses batch size 128 on 64 Nvidia H100 GPUs.Source: §5.1 · p. 10

5.2 Baseline Algorithms

• The baseline comparison includes prominent multimodal models and reward models, contrasting critic-based MM-RLHF-Reward-7B with IXC-2.5-Reward’s direct reward-head scoring.The comparison covers models including GPT-4o, Claude, Gemini, and other multimodal systems.Source: §5.2 · p. 10

5.3 Evaluation Results on MRM Benchmark

• BaseReward beats prior SOTA by 11.9% in MM-RLHF-Reward Bench accuracy, 23.32% on MM-RLHF Acc+, and 14.2% in VL-Reward Bench Overall Accuracy, but ranks second on Multimodal Reward Bench.The authors primarily attribute the latter result to missing coding and related preference data in training.Source: §5.3 · pp. 10–11, Table 8 · p. 11, Table 9 · p. 11

5.4 Reinforcement Learning with BaseReward

• The RL evaluation uses one BaseReward model derived from Qwen2.5-VL, without voting or an ensemble strategy because of computational constraints.The RL stage tests reward-model signals in a genuine reinforcement-learning process.Source: §5.4 · p. 12

5.4.1 Experimental Setup

• The RL setup trains Qwen-2.5-VL 3B with GRPO for one epoch, using eight rollouts per prompt and batch size 256, on prompts from perception, chart-recognition, and reasoning datasets.Evaluation spans general-purpose, perception, mathematical-reasoning, and conversational benchmarks.Source: §5.4.1 · pp. 12–13
• The tested reward schemes are binary exact-match rewards, direct BaseReward scores, and a hybrid that gives exact matches 1 and otherwise sigmoid-normalizes BaseReward scores.For exact-match problems, the binary reward is 1 for a match and 0 otherwise.Source: §5.4.1 · p. 12
• For R1-Reward, each of eight responses is scored against the other candidates, and the pairwise preference scores are aggregated into its final reward.The setup forms 56 ordered response pairs and denotes each relative preference score S(yi, yj).Source: §5.4.1 · pp. 12–13

5.4.2 Results and Analysis

• The hybrid rule-based and BaseReward approach performs best, with consistent gains across logical reasoning, perception, and conversational tasks.The authors describe it as combining rule-based precision for objective tasks with BaseReward’s nuanced semantic understanding for complex, subjective evaluations.Source: §5.4.2 · p. 13, Table 11 · p. 13

• An optimized Naive-RM with a two-layer SiLU MLP and no auxiliary regularization is efficient and effective; curated multimodal and text-only data is essential, with text-only gains in safety and mathematics.Source: §6 · p. 13
• BaseReward establishes new SOTA on major MRM benchmarks and consistently improves an MLLM across perception, reasoning, and conversational tasks when used as an RL reward signal.Source: §6 · p. 13
• The study excludes backbones at 72B parameters or larger, leaving scaling gains open; LLM-based models outperform MLLM-based counterparts on pure-text reward tasks.Source: §6 · p. 14