Source-linked AI summary
VisualPRM: An Effective Process Reward Model for Multimodal Reasoning
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, Lewei Lu, Haodong Duan, Yu Qiao, Jifeng Dai, Wenhai Wang
TL;DR
MLLM reasoning still trails proprietary systems, while Best-of-N scaling lacks effective multimodal critics and suitable evaluation benchmarks. The paper introduces VisualPRM, trained with VisualPRM400K and evaluated with VisualProcessBench, and reports gains across model families and scales, including 5.9 points for InternVL2.5-78B across seven benchmarks.
Problem
Best-of-N scaling for MLLMs lacks effective critic models and benchmarks that isolate multimodal step-wise criticism.
Method
The paper builds VisualPRM400K with automated step-supervision, trains an 8B-parameter VisualPRM to predict step correctness, and introduces human-annotated VisualProcessBench.
Results
VisualPRM improves four MLLMs across seven multimodal reasoning benchmarks, including a 5.9-point gain for InternVL2.5-78B, and outperforms ORMs and Self-Consistency in BoN evaluation.
Takeaways & Limitations
Process Reward Models can serve as effective critic models for multimodal test-time scaling across model families and scales.
Abstract
from arXiv · showhide
We introduce VisualPRM, an advanced multimodal Process Reward Model (PRM) with 8B parameters, which improves the reasoning abilities of existing Multimodal Large Language Models (MLLMs) across different model scales and families with Best-of-N (BoN) evaluation strategies. Specifically, our model improves the reasoning performance of three types of MLLMs and four different model scales. Even when applied to the highly capable InternVL2.5-78B, it achieves a 5.9-point improvement across seven multimodal reasoning benchmarks. Experimental results show that our model exhibits superior performance compared to Outcome Reward Models and Self-Consistency during BoN evaluation. To facilitate the training of multimodal PRMs, we construct a multimodal process supervision dataset VisualPRM400K using an automated data pipeline. For the evaluation of multimodal PRMs, we propose VisualProcessBench, a benchmark with human-annotated step-wise correctness labels, to measure the abilities of PRMs to detect erroneous steps in multimodal reasoning tasks. We hope that our work can inspire more future research and contribute to the development of MLLMs. Our model, data, and benchmark are released in https://internvl.github.io/blog/2025-03-13-VisualPRM/.
1. Introduction
Multimodal large language models still lag proprietary systems in reasoning, and Best-of-N scaling is limited by weak critic models and costly, policy-dependent evaluation. The paper addresses these gaps with VisualPRM400K, VisualProcessBench, and VisualPRM for step-wise criticism.
- MLLMs show strong perception and recognition but retain a substantial reasoning gap relative to proprietary models.
- Best-of-N evaluation requires critic models to estimate candidate quality, but existing open-source MLLMs provide only marginal gains over no test-time scaling.The limitation is attributed to insufficient critic data in their training corpora.
- BoN critic evaluation is expensive and difficult to compare because policy-model generation dominates cost and affects performance.
- VisualPRM400K contains approximately 400K multimodal process-supervision examples with step-level correctness generated by averaging continuation accuracies.
- VisualProcessBench contains 2,866 samples and 26,950 human-annotated step-wise labels for detecting erroneous reasoning steps.Unlike benchmarks identifying only the first error, it evaluates detection of all errors in a solution.
- VisualPRM is an 8B-parameter multimodal PRM that predicts step correctness and improves four MLLMs by 8.0, 3.7, 8.4, and 5.9 points across seven benchmarks.The improvements correspond respectively to MiniCPM-V2.6, QwenVL2.5-7B, InternVL2.5-8B, and InternVL2.5-78B.
2. Related Work
Prior work advances MLLMs through architecture, data, and training, while reward-model research distinguishes outcome from process scoring. Multimodal process supervision and benchmarking remain underdeveloped areas addressed by this paper.
- Multimodal Large Language Models: MLLM development has pursued architectural connectors aligning visual foundation-model embeddings with language-model latent spaces.
- Process Reward Models: Outcome Reward Models assign an overall response score, whereas Process Reward Models estimate the quality of individual steps before aggregation.
- Process Reward Models: Prior process-supervision datasets use human annotation or Monte Carlo pipelines to reduce the cost of labeling step quality.
- Process Reward Models: Multimodal process reward models remain largely underexplored, motivating VisualPRM400K as a multimodal process-supervision dataset.
- Benchmarks for Reward Models: Existing reward-model benchmarks evaluate binary preferences, pairwise or Best-of-N choices, and step-wise judgments, but lack a multimodal process benchmark.
3. Method
The method constructs multimodal process supervision data, trains a PRM to score solution steps, and introduces a benchmark for detecting erroneous reasoning steps. It evaluates step correctness through expected accuracy or step-to-step improvement and aggregates step scores into response scores.
- VisualPRM400K: VisualPRM400K contains approximately 400K multimodal process-supervision examples generated from images, questions, step-by-step solutions, and step correctness annotations.The construction samples continuations from each partial solution to estimate expected step accuracy.
- VisualProcessBench: VisualProcessBench evaluates whether PRMs and MLLMs identify erroneous steps using multimodal reasoning solutions with step-wise correctness labels.Its evaluation labels each step as positive, negative, or neutral and uses human annotation procedures.
- VisualPRM400K: The data pipeline estimates each step’s expected accuracy by sampling continuations after the step and measuring their completion accuracy.Each image-question pair uses sampled solutions and multiple continuations, with steps merged when solutions exceed the maximum step limit.
- VisualPRM: VisualPRM formulates process supervision as a multi-turn chat, presenting the image, question, and successive solution steps while predicting each step’s quality.Unlike approaches supervising only through the first incorrect step, this method supervises all steps.
- PRM modeling: Value-based PRMs classify steps using expected accuracy above zero, whereas advantage-based PRMs classify the change in expected accuracy between consecutive steps.During inference, discretized-token probabilities produce step scores that are aggregated into a response score, typically by averaging.
4. Experiments
Experiments evaluate VisualPRM as a critic for Best-of-N reasoning and step correctness across multimodal models, benchmarks, and critic designs. VisualPRM improves reasoning across model families and scales, outperforms Self-Consistency and ORM in BoN, and achieves strong step-judgment performance.
- Best-of-N evaluation: VisualPRM improves existing open-source MLLMs on seven multimodal reasoning benchmarks under Best-of-8 evaluation.The overall score averages MMMU, MathVista, MathVision, MathVerse, DynaMath, WeMath, and LogicVista.
- Best-of-N evaluation: 8.4, 8.0, and 3.7 points improve InternVL2.5-8B, MiniCPM-V-8B, and Qwen2.5-VL-7B, respectively, with VisualPRM.InternVL2.5-26B, InternVL2.5-38B, and InternVL2.5-78B also achieve substantial gains over counterparts without TTS.
- VisualProcessBench: VisualPRM achieves an overall F1 score of 62.0 on VisualProcessBench, outperforming GPT-4o and GPT-4o-Mini and matching Gemini-2.0-Flash.Most open-source MLLMs perform close to the random-guessing baseline of 50.0 and tend to label most steps correct.
- Best-of-N evaluation: As N increases, PRM produces the largest reasoning gains among Self-Consistency, ORM, and PRM for InternVL2.5-8B and MiniCPM-V2.6-8B.For InternVL2.5-8B, PRM exceeds SC and ORM by 2.4 and 1.5 points at Best-of-8, widening to 3.1 and 4.3 points at Best-of-128.
- Ablation studies: Value-based PRMs outperform advantage-based PRMs in BoN and VisualProcessBench, while averaging step scores outperforms selecting the maximum.The maximum can rely on a single high-scored step, whereas averaging uses scores across the solution.
- Ablation studies: Existing MLLMs provide marginal BoN gains as critics because they assign similar, predominantly positive scores to most solution steps.Their autoregressive per-step judgments are also slower than VisualPRM’s single-forward computation for all steps.
5. Conclusion
The paper presents VisualPRM400K, VisualPRM, and VisualProcessBench as resources and a model for multimodal process supervision and test-time scaling. Results show improved MLLM reasoning, stronger BoN performance than ORM and Self-Consistency, and difficulty for existing open-source MLLMs in judging step correctness.
- VisualPRM400K contains about 400K multimodal process-supervision examples for training multimodal PRMs.
- VisualPRM estimates each reasoning step’s value and improves MLLM reasoning across model scales and families with Best-of-N evaluation.
- VisualPRM outperforms Outcome Reward Models and Self-Consistency during Best-of-N evaluation.
- VisualProcessBench measures PRM and MLLM detection of incorrect multimodal reasoning steps using step-wise correctness labels.Existing open-source MLLMs struggle to judge each step effectively.
6. Training Hyper-parameters
VisualPRM training uses data packing, AdamW optimization, a short linear warmup, cosine decay, and one training epoch.
- VisualPRM training uses AdamW with β1=0.9, β2=0.999, and weight decay 0.05.
- The learning rate starts at 1e-5, warms up linearly for the first 5% of training steps, then follows cosine decay to zero.
- Training runs for one epoch with data packing enabled.
7. More Ablation Studies
Ablations examine correctness thresholds, generation temperature, Best-of-N scaling, and VisualProcessBench step distributions. Performance favors a zero correctness threshold, an intermediate temperature, and increasing candidate counts.
- Training thresholds: Increasing the expected-accuracy threshold reduces both Best-of-8 performance and VisualProcessBench scores.The training definition therefore treats a step as correct when its expected accuracy exceeds 0.
- Generation temperature: As generation temperature rises from 0.3 to 1.3, InternVL2.5-8B performance first improves and then declines.The paper attributes this pattern to a trade-off between response diversity and accuracy.
- Best-of-N scaling: As N increases from 8 to 128, InternVL2.5-8B improves from 41.2 to 44.0 and MiniCPM-V2.6 improves from 37.5 to 40.4.These results demonstrate scalable test-time scaling for MLLMs.
- VisualProcessBench analysis: VisualProcessBench reports step-position distributions on the x-axis by step index, with y-axes showing position proportion and error rate.
8. More Statistics for VisualProcessBench
VisualProcessBench solutions usually contain fewer than 15 steps, while step errors generally become more frequent later in the reasoning process.
- Most VisualProcessBench solutions contain fewer than 15 steps, with about 7 or 13 steps especially common.
- Step error rates are lower during the first three steps and increase as the step index grows.The passage attributes this pattern to longer, more challenging reasoning problems.
- From step 15 onward, the error rate drops sharply because few samples create substantial statistical fluctuations.
9. More Data Examples in VisualPRM400K
VisualPRM400K includes examples spanning diverse multimodal domains, including visual question answering, science, charts, mathematics, OCR, and documents.
- VisualPRM400K examples cover general VQA, science, chart, mathematics, OCR, and document data.
10. More Data Examples in VisualProcessBench
The VisualProcessBench examples illustrate multimodal questions paired with step-by-step reasoning, correctness labels, and both correct and incorrect model behavior across several domains.
- VisualProcessBench includes randomly selected examples and an example where a model corrects an initially incorrect answer.
- The examples include visual art, melody-key identification, receipts, economics, arithmetic, quadratic graphs, and spatial fish-alignment questions.
- One visual-art example labels the reasoning steps correct and selects B, Shape, for an enclosed area formed by lines.
- One melody example reasons from a one-sharp key signature, a D# accidental, and E as tonic to select D, E minor.
- The arithmetic example demonstrates both correct and incorrect labels, with 54 + 32 = 86 marked as an incorrect final answer.
- The quadratic-graph example identifies C as the non-parabolic piece and therefore the one that does not belong.
- The fish-alignment example shows a question whose reasoning requires identifying heads pointing toward the ring before mentally straightening the line.