Source-linked AI summary
VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Lingpeng Kong, Qi Liu
TL;DR
VL-GenRMs lack a comprehensive benchmark for assessing their reliability, despite their role in multimodal evaluation and alignment. VL-RewardBench uses curated challenging samples and human-verified preference labels across diverse task domains. Evaluations find modest model accuracy and a strong association with downstream Best-of-N performance, while identifying perception errors and model-dependent scaling effects.
Problem
VL-GenRM evaluation lacks a comprehensive benchmark, while existing approaches can use biased AI preferences or traditional tasks that are insufficiently complex.
Source: §1 · p. 2Method
VL-RewardBench combines challenging-sample curation and AI-assisted preference labeling with human verification across general, hallucination, and multimodal reasoning tasks.
Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 3Results
GPT-4o achieves 62.4% macro average accuracy, and VL-RewardBench scores strongly correlate with downstream MMMU-Pro Best-of-N results.
Source: §1 · p. 2, §3.3 · p. 5Takeaways & Limitations
The benchmark and evaluation identify visual perception as a key bottleneck, model-capacity-dependent test-time scaling effects, and gains from training VL-GenRMs to judge.
Source: §1 · p. 2, §6 · p. 8Pointwise-score comparisons use only the subset for which pointwise scores were successfully extracted.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Vision-language generative reward models (VL-GenRMs) play a crucial role in aligning and evaluating multimodal AI systems, yet their own evaluation remains under-explored. Current assessment methods primarily rely on AI-annotated preference labels from traditional VL tasks, which can introduce biases and often fail to effectively challenge state-of-the-art models. To address these limitations, we introduce VL-RewardBench, a comprehensive benchmark spanning general multimodal queries, visual hallucination detection, and complex reasoning tasks. Through our AI-assisted annotation pipeline that combines sample selection with human verification, we curate 1,250 high-quality examples specifically designed to probe VL-GenRMs limitations. Comprehensive evaluation across 16 leading large vision-language models demonstrates VL-RewardBench's effectiveness as a challenging testbed, where even GPT-4o achieves only 65.4% accuracy, and state-of-the-art open-source models such as Qwen2-VL-72B, struggle to surpass random-guessing. Importantly, performance on VL-RewardBench strongly correlates (Pearson's r $>$ 0.9) with MMMU-Pro accuracy using Best-of-N sampling with VL-GenRMs. Analysis experiments uncover three critical insights for improving VL-GenRMs: (i) models predominantly fail at basic visual perception tasks rather than reasoning tasks; (ii) inference-time scaling benefits vary dramatically by model capacity; and (iii) training VL-GenRMs to learn to judge substantially boosts judgment capability (+14.7% accuracy for a 7B VL-GenRM). We believe VL-RewardBench along with the experimental insights will become a valuable resource for advancing VL-GenRMs.
1. Introduction
Reliable VL-GenRM evaluation is needed for multimodal assessment, sample selection, and reward modeling, but existing approaches lack a comprehensive benchmark. VL-RewardBench addresses this gap with curated, human-verified examples; evaluations show modest model performance and a strong association with downstream Best-of-N results.
1. Introduction
- Reliable VL-GenRM evaluation supports scalable performance assessment, instructive-example selection for synthetic training, and objective evaluation for LVLM reward modeling.Existing assessments rely on AI-annotated preferences or traditional benchmarks, which can exhibit systematic or stylistic bias and may use overly simple queries.Source: §1 · pp. 1–2
- VL-RewardBench covers general multimodal queries, visual hallucination detection, and multimodal knowledge and mathematical reasoning in 1,250 human-verified samples.Its curation targets challenging cases and objective labels, using small-model filtering for preference-pair datasets and commercial-model responses with GPT-4o assessment for reasoning tasks.Source: §1 · p. 2
- 62.4% accuracy for GPT-4o and 62.5% for Gemini-1.5-Pro lead a 16-model evaluation, while Qwen2-VL-72B reaches 43.0% and LLaMA-3.2-90B 53.9%.VL-RewardBench performance strongly correlates with downstream MMMU-Pro results using VL-GenRMs for Best-of-N sampling (Pearson’s r>0.9).Source: §1 · p. 2
- The main bottleneck is visual perception: existence and recognition errors exceed 67%, compared with 41.8% for reasoning tasks.Test-time scaling benefits larger models but may degrade smaller ones; training LLaVA-OneVision-7B-ov to judge improves accuracy by 14.7%.Source: §1 · p. 2
2. VL-RewardBench
VL-RewardBench evaluates preference judgments on single-image, single-turn multimodal queries, drawing from general, hallucination-oriented, and reasoning datasets. Its construction combines challenging-sample filtering, AI-assisted labeling, and human verification, yielding a diverse benchmark with substantial task and error-type imbalance.
2. VL-RewardBench
- VL-RewardBench represents each sample as a multimodal query with preferred and rejected responses, focusing on single-image, single-turn interactions.The framework can be extended to multi-turn dialogues and multiple images.Source: §2 · p. 2
2.1. Dataset Source
- Source data span general multimodal instructions, hallucination-oriented queries, and reasoning tasks, including VLFeedback, WildVision, POVID, RLAIF-V, RLHF-V, MMMU-Pro, and MathVerse.The reasoning sources use 1,568 MMMU-Pro samples and 1,546 selected MathVerse samples; the latter are Vision Dominant and Vision Intensive subsets.Source: §2.1 · pp. 2–3
2.2. Preference Annotation
- The annotation framework filters challenging general and hallucination examples with small models and creates preference labels for reasoning examples, with graduate-student annotators refining guidelines through discussion.The two dataset-specific strategies are summarized in Figure 2, and all labels receive human verification.Source: §2.2 · p. 3, Figure 2 · p. 3
2.2.1. Ensemble Filtering with Small Models
- Small-model ensemble filtering retains pairs that all models consistently misjudge, then human checks remove incorrect, ambiguous, or unsuitable cases, yielding 932 high-quality pairs.The ensemble uses four 7B models; three randomized-position judgments per model help mitigate position bias, and remaining pairs receive error-type labels.Source: §2.2.1 · pp. 3–4
2.2.2. AI-aided Preference Labeling for Reasoning Tasks
- For reasoning tasks, commercial models generate candidate responses, GPT-4o drafts preference labels with rationales, and human review validates them, producing 318 challenging samples.Responses are paired under a verbosity constraint; reviewers retain pairs with at least one partly correct answer and prefer more complete, logical reasoning when both are correct.Source: §2.2.2 · pp. 4–5
2.3. Dataset Statistics
- The benchmark contains 1,250 pairs: 59.9% hallucination-related, 25.4% reasoning, and 14.7% general instructions, with existence errors making up 59.3% of tagged cases.The authors recommend macro-average metrics; annotator agreement averaged 0.70 Cohen’s kappa, and preferred-versus-rejected length differences were zero-centered and bell-shaped.Source: §2.3 · p. 5
3. Experiments
The experiments evaluate 16 commercial and open-source vision-language models using a standardized preference-judgment protocol. Models achieve modest accuracy on VL-RewardBench, and benchmark performance correlates with downstream Best-of-N gains.
3.1. Evaluated Models
- The evaluation covers 16 state-of-the-art models, including open-source systems from 4B to 90B parameters and commercial GPT, Gemini, and Claude models.LLaVA-Critic models are also included to examine the impact of learning to judge.Source: §3.1 · p. 5
3.2. Evaluation Settings
- The evaluation presents each model with a query and preferred/rejected responses, randomizes response order across K = 5 runs, and uses majority voting.It reports overall accuracy and macro average accuracy; decoding uses temperature=0.2 and top-p=0.2.Source: §3.2 · p. 5
3.3. Evaluation Results
- Scaling improves accuracy across families: Llama-3.2 rises from 42.8% to 53.9%, Qwen2-VL from 33.9% to 43.0%, and Molmo from 39.7% to 43.7%.GPT-4o (62.4%) also substantially outperforms GPT-4o-mini (44.8%).Source: §3.3 · pp. 5–6
- 62.5% macro average accuracy for Gemini-1.5-Pro and 62.4% for GPT-4o lead; reasoning reaches 51.1%–70.5%, while general instructions are hardest at 28.0%–50.8%.The table reports results on VL-RewardBench; hallucination judgments remain difficult even for top models.Source: §3.3 · p. 5, Table 2 · p. 6
- Randomly sampled pairs are easier: Gemini models exceed 95% accuracy, supporting the curation strategy’s selection of challenging examples.The random examples follow the same task distribution as VL-RewardBench.Source: §3.3 · p. 6
- Pearson r = 0.946 for Qwen2-VL-7B and r = 0.914 for LLaVA-OneVision-7B-ov link benchmark accuracy with Best-of-N gains on MMMU-Pro.With GPT-4o as selector, LLaVA-OneVision-7B-ov accuracy rises from 35.7% to 52.5%.Source: §3.3 · p. 6, Figure 4 · p. 6
4. Analysis
The analysis probes error patterns across model scales, test-time voting, and critic training. It finds that perception errors remain prominent, scaling effects vary by model, and critic training improves judgment.
4. Analysis
- The three investigations examine error patterns across model scales, inference-time scaling effects, and potential improvements through critic training and scoring methods.The section introduces these analyses as investigations of VL-GenRM capabilities and limitations.Source: §4 · pp. 6–7
4.1. Error Analysis
- 67.9% error on Existence for GPT-4o-mini and 80.9% on Recognition for Qwen2-VL-7B contrast with 41.8% average error on Reasoning tasks.Scaling Qwen2-VL from 7B to 72B reduces Counting errors by 18.2 percentage points (74.1% to 55.9%); reasoning error falls by 6.0% on average, and the authors suggest architectural innovations may be needed for complex reasoning.Source: Figure 5 · p. 7, §4.1 · p. 7
4.2. Does Inference-time Scaling Help?
- 60.3% to 62.7% macro accuracy for GPT-4o as K rises from 1 to 7 contrasts with steady GPT-4o-mini results and drops of 1.7 and 2.6 points for Qwen2-VL-72B and Molmo-72B by K=5.The authors conclude that text-only inference-time scaling strategies may not transfer directly to vision-language judgment.Source: Figure 6 · p. 7, §4.2 · p. 7
4.3. Critic Training Improves VL-GenRMs
- +14.7% and +9.2% judgment-accuracy gains over the base model come from pointwise and pairwise critic training, respectively.The pointwise critic scores 52.9% overall versus 47.4% for the pairwise critic; their relative strengths differ across hallucination and reasoning subsets.Source: Figure 7 · p. 8, §4.3 · p. 8
4.4. Takeaways
- The authors prioritize stronger visual perception and advanced scaling strategies, and propose a co-evolution cycle in which stronger LVLMs improve judges that curate training data.Suggested strategies include complex planning, process-level supervision, and critic training.Source: §4.4 · p. 8
5. Related Work
Prior VL-GenRM work examined agreement on academic tasks and critic-based improvement, while this study evaluates state-of-the-art LVLMs and presents a broader benchmark targeting difficult real-world queries and reasoning tasks.
Prior VL-GenRM evaluation
Prior VL-GenRM improvement
VL-RewardBench
- VL-RewardBench broadens coverage to real-world queries and reasoning tasks, using a difficulty-elevation pipeline aimed at cases challenging even state-of-the-art models.The study also evaluates state-of-the-art LVLMs and analyzes directions for future VL-GenRM improvement.Source: §5 · p. 8
6. Conclusions
The paper presents VL-RewardBench as a challenging benchmark for VL-GenRMs and analyzes 16 state-of-the-art LVLMs. Its conclusions highlight model-scale benefits, variable test-time scaling effects, and the potential of critic training.
6. Conclusions
- VL-RewardBench raises the evaluation bar through systematically curated challenging cases and complex multimodal reasoning tasks.Source: §6 · p. 8
- Analysis of 16 state-of-the-art LVLMs identifies benefits from increased model scale, variable effectiveness of test-time scaling, and potential improvements from specialized critic training.Source: §6 · p. 8
- The benchmark and analysis provide concrete pathways for developing more capable and reliable VL-GenRMs.Source: §6 · p. 8