PaperlayerView exact revision on arXiv

Source-linked AI summary

GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning

Jianghangfan Zhang, Yibo Yan, Kening Zheng, Xin Zou, Song Dai, Xuming Hu

arXiv:2508.04088v2cs.CL

TL;DR

Without a mechanism to correct errors, the reasoning process remains fundamentally broken, limiting utility in Best-of-N sampling. GM-PRM analyzes step intent, image alignment, and logical soundness, generates corrections for first erroneous steps, and uses them in Refined-BoN. GM-PRM with Refined-BoN consistently improves performance across five benchmark datasets for six MLLMs, with average gains up to +5.9.

  • Problem

    Without a mechanism to correct errors, the reasoning process remains fundamentally broken, limiting utility in Best-of-N sampling.

    Source: §1 · p. 2

  • Method

    GM-PRM analyzes step intent, image alignment, and logical soundness, generates corrections for first erroneous steps, and uses them in Refined-BoN.

    Source: §1 · p. 2

  • Results

    GM-PRM with Refined-BoN consistently improves performance across five benchmark datasets for six MLLMs, with average gains up to +5.9.

    Source: §4.2 · p. 5, §5 · p. 7

  • Takeaways & Limitations

    The corrective reward-model paradigm improves multimodal reasoning trajectories and achieves state-of-the-art benchmark results with a 20K-sample training dataset.

    Source: §1 · p. 2, §5 · p. 7

    Existing Best-of-N mechanisms remain passive selection processes over a static set of potentially flawed solutions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities but often struggle with complex, multi-step mathematical reasoning, where minor errors in visual perception or logical deduction can lead to complete failure. While Process Reward Models (PRMs) offer step-by-step supervision, existing multimodal PRMs are limited to being binary verifiers that can identify but not correct errors, offering little explanatory power. To address these deficiencies, we introduce the Generative Multimodal Process Reward Model (GM-PRM), a novel paradigm that transforms the PRM from a passive judge into an active reasoning collaborator. Instead of a simple scalar score, GM-PRM provides a fine-grained, interpretable analysis of each reasoning step, evaluating its step intent, visual alignment, and logical soundness. More critically, GM-PRM is trained to generate a corrected version of the first erroneous step it identifies. This unique corrective capability enables our new test-time inference strategy, Refined Best-of-N (Refined-BoN). This framework actively enhances solution quality by using the PRM's generated correction to guide the policy model toward a more promising reasoning trajectory, thereby improving the diversity and correctness of the solution pool. We demonstrate that GM-PRM achieves state-of-the-art results on multiple multimodal math benchmarks, significantly boosting policy model performance with remarkable data efficiency, requiring only a 20K-sample training dataset. Our code will be released upon acceptance.

1 Introduction

MLLMs often fail on complex multimodal mathematical reasoning because small visual or logical errors can invalidate final answers. GM-PRM addresses limited multimodal PRM interpretability and correction by analyzing reasoning steps and generating corrections that drive Refined-BoN.

1 Introduction

  • Minor visual-perception or logical-inference errors can cascade into entirely incorrect final answers in complex multimodal mathematical reasoning.Source: §1 · p. 1
  • Existing multimodal PRMs provide limited explanatory power and generally identify errors without correcting them, leaving Best-of-N selection passive over flawed solutions.Source: §1 · pp. 1–2
  • GM-PRM transforms multimodal PRMs from binary verifiers into generative collaborators that analyze step intent, image alignment, and logical soundness.It generates a refined correction for the first erroneous step it identifies.Source: §1 · p. 2
  • Refined-BoN uses GM-PRM corrections to guide the policy model toward new reasoning trajectories, improving solution-pool diversity and correctness through active refinement.Source: §1 · p. 2
  • GM-PRM and Refined-BoN achieve state-of-the-art results on multiple multimodal math benchmarks using a 20K-sample training dataset.Source: §1 · p. 2

2 Related Work

Prior work developed fine-grained process supervision for language and multimodal reasoning, but multimodal PRMs still face interpretability and error-correction challenges. GM-PRM is introduced to address these limitations.

Process Reward Models (PRMs)

  • Process Reward Models evaluate the correctness of intermediate reasoning steps, with later approaches reducing annotation costs or using model-generated evaluations.Source: Process Reward Models (PRMs) · p. 2

Multimodal PRMs

  • Multimodal PRM research includes error-insertion datasets, Monte Carlo estimation, human-annotated benchmarks, consistency-based annotation, and other capability improvements.Source: Multimodal PRMs · p. 2
  • Existing multimodal PRMs remain limited by insufficient interpretability of step labels and inability to correct identified erroneous steps.Source: Multimodal PRMs · p. 2

3 Methodology

The methodology develops GM-PRM as a multimodal generative process evaluator and pairs it with Refined-BoN to detect, explain, and correct flawed reasoning during inference. It constructs training data through targeted selection, GPT-4o analysis, and agreement-based filtering, then uses corrected steps to diversify and improve candidate solutions.

Problem and Reasoning Steps Generation

  • In conventional process reward modeling, a policy model generates reasoning steps and a trained PRM assigns each step a confidence score based on binary correctness labels.Generative PRMs replace binary labels with textual analyses and judgments, motivating the multimodal GM-PRM formulation.Source: Problem and Reasoning Steps Generation · p. 3

GM-PRM

  • GM-PRM extends process reward modeling from scalar or binary step judgments to multimodal critiques covering step intent, image alignment, and reasoning logic.It accepts textual and visual problem inputs and produces textual critiques and judgments for each reasoning step.Source: GM-PRM · p. 3, Problem and Reasoning Steps Generation · p. 3

Data Selection

  • The training-data pipeline selects plane-geometry and function problems from VisualPRM400K, supplements them with textual critiques, and targets multimodal mathematical reasoning.VisualPRM400K contains approximately 400,000 multimodal process-supervision samples.Source: §3.2 · p. 3, Data Selection · p. 4

Generation of Analysis and Judgment

  • GPT-4o generates step-level analyses for intent, image alignment, reasoning logic, and refinement, including a corrected version of the first detected erroneous step.The resulting corrections are intended to be logically coherent, visually accurate, and semantically aligned with the original step intent.Source: Generation of Analysis and Judgment · p. 4

Data Filtering

  • Data filtering combines GPT-4o judgments with Monte Carlo hard estimation and retains samples whose evaluations agree.Hard estimation labels a step correct when at least one subsequent completion reaches the correct final answer; otherwise it labels the step incorrect.Source: Data Filtering · p. 4

Refined-BoN Method

  • Refined-BoN uses GM-PRM to refine the first erroneous step, feed it with validated preceding steps back to the policy model, and generate additional solution trajectories.The method generates N/2 initial solutions, conditions subsequent generations on step-level evaluations, and regenerates unchanged solutions when all steps are judged correct.Source: Figure 2 · p. 3, Refined-BoN Method · p. 5, §3.3 · p. 5

Solution Selection

  • Solutions with all steps judged correct are ranked by their average GM-PRM step scores, while otherwise the highest-scoring solution is selected among candidates containing errors.The score for each step is based on GM-PRM’s probabilities for “Correct” and “Incorrect” tokens.Source: Solution Selection · p. 5

4 Experiments

Across five multimodal mathematics benchmarks, GM-PRM with Refined-BoN consistently improves six policy models, with stronger gains as sampling increases and when averaging step scores. The experiments also show larger benefits on challenging or weaker-baseline settings and improved solution diversity over standard BoN.

Benchmarks.

  • The experiments evaluate GM-PRM with default N=8 across MathVista, MathVision, MathVerse, DynaMath, and WeMath using six policy models.MathVerse uses its Vision-Only subset and DynaMath uses its Plane-Geometry subset.Source: Benchmarks · p. 5, Settings · p. 5, §4 · p. 5

4.2 Main Results

  • GM-PRM with Refined-BoN consistently improves accuracy across five benchmark datasets and six MLLMs, with average gains of +5.9 for MiniCPM-V2.6-8B, +4.5 for Llama-3.2-11B-Vision, +4.5 for Qwen2.5-VL-7B, and +5.6 for InternVL3-8B.Table 1 reports the baseline, final, and improvement values across the five datasets.Source: §4.2 · p. 5, Table 1 · p. 6
  • Dataset-level gains are largest on WeMath, where MiniCPM-V2.6-8B improves by +12.4 points, while MathVerse and DynaMath gain +4.5–6.7 points and MathVision gains +1.9–3.9.The reported pattern suggests stronger benefits on complex symbolic and multi-step reasoning datasets than on MathVision.Source: §4.2 · p. 5
  • GM-PRM generalizes beyond its plane-geometry and function training problems, with Figure 3 showing especially large improvements for plane geometry across question categories.Plane-geometry gains exceed those for the overall dataset, function problems, and other categories.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
  • Lower-baseline models benefit more from refinement: MiniCPM-V-2.6-8B gains +5.9 points (+21.0%), compared with InternVL3-38B’s +2.9 points (+5.9%).Qwen2.5-VL-7B also gains +4.5 points (+10.4%) from a 43.3% baseline.Source: §4.2 · p. 6

Number of solution samples N in Refined BoN.

  • As N increases from 1 to 8, GM-PRM’s accuracy grows faster than Self-Consistency across MiniCPM-V-2.6-8B, Qwen2.5-VL-7B, and InternVL3-78B.At Best-of-8, GM-PRM exceeds Self-Consistency by 4.9, 3.5, and 4.1 points for the three policies, respectively.Source: Number of solution samples N in… · p. 6, Figure 4 · p. 7

Methods for aggregating step scores.

  • Averaging step scores and selecting the minimum outperform selecting the maximum across policy models and datasets, with averaging performing slightly better than the minimum.The comparison uses solutions whose steps are all judged correct.Source: Methods for aggregating step scores · p. 7, Figure 5 · p. 7

Refined-BoN vs. BoN.

  • Refined-BoN improves average Pass@8 over standard BoN by 0.9 points for MiniCPM-V-2.6-8B, 1.3 for Llama-3.2-11B-Vision, and 0.9 for InternVL3-8B across five datasets.The result supports improved diversity and correctness of generated solution sets.Source: Refined-BoN vs. BoN · p. 7, Table 2 · p. 7

5 Conclusion

GM-PRM reframes multimodal reward modeling as active collaboration by analyzing reasoning steps and correcting errors through Refined-BoN. Experiments report state-of-the-art results, consistent gains across models and problem types, and data efficiency.

5 Conclusion

  • GM-PRM achieves state-of-the-art results on multiple multimodal benchmarks while significantly boosting policy-model performance with remarkable data efficiency.The reported gains are consistent across diverse models and problem types.Source: §5 · p. 7
  • GM-PRM provides fine-grained, interpretable step analysis and generates corrections for erroneous steps, moving beyond binary verification.Its corrective capability powers the Refined Best-of-N framework for multimodal mathematics.Source: §5 · p. 7
  • Refined-BoN actively improves flawed reasoning trajectories at test time, shifting reward models from passive error detection toward collaborative correction.This framework uses GM-PRM’s corrective capability to refine candidate solutions.Source: §5 · p. 7

Multimodal Large Language Models (MLLMs)

MLLMs integrate visual and textual information but leave substantial room for improving test-time scaling. This work explores incorporating Process Reward Models into test-time scaling to enhance MLLM performance.

Multimodal Large Language Models (MLLMs)

  • MLLMs integrate visual and textual information to perceive and reason across modalities, extending language-centric model capabilities.They differ from text-only LLMs through multimodal information fusion and interaction.Source: Multimodal Large Language Models (MLLMs) · p. 10
  • MLLM research has advanced through vision-language connectors, added visual-feature layers, and increasingly curated multimodal pretraining and instruction-tuning datasets.Connector-based methods align vision embeddings with LLM latent spaces, while added layers reduce visual tokens but incur training costs.Source: Multimodal Large Language Models (MLLMs) · p. 10
  • Most prior efforts focus on MLLM training processes, leaving significant room for exploration in test-time scaling techniques.The paper investigates incorporating Process Reward Models into test-time scaling.Source: Multimodal Large Language Models (MLLMs) · p. 10

A.2 Benchmark

Appendix A.2 points to Table 3 for additional details about the Refined-BoN test benchmarks.

A.2 Benchmark

  • Table 3 provides additional details about the benchmarks used to evaluate Refined-BoN.The supplied passages identify the table’s scope but do not report its benchmark entries or comparisons.Source: Table 3 · p. 10, §A.2 · p. 10
  • The Refined-BoN benchmark details are presented in Table 3.The caption and accompanying text identify the table as the relevant benchmark reference.Source: Table 3 · p. 10, §A.2 · p. 10
  • The supplied benchmark passages do not specify the table’s rows, columns, datasets, or results.No further encoding or outcome is stated in the provided text.Source: Table 3 · p. 10, §A.2 · p. 10

A.3 Dataset

The GM-PRM training dataset contains 19,614 samples, balancing incorrect-step solutions with solutions judged fully correct.

A.3 Dataset

  • 19,614 samples comprise the final dataset used to train GM-PRM.The dataset is constructed to ensure a balanced distribution of process labels.Source: §A.3 · p. 10
  • 9,061 solutions contain incorrect steps identified jointly by GPT-4o and Monte Carlo estimation.These samples form the incorrect-step portion of the training data.Source: §A.3 · p. 11
  • 10,553 solutions are judged to have all steps correct.Together with the incorrect-step solutions, they constitute the final training dataset.Source: §A.3 · p. 11

A.4 Prompt

This appendix specifies prompts for generating reasoning and answers, constructing the training dataset, and illustrating additional GM-PRM and Refined-BoN cases.

A.4 Prompt

A.5 More Cases

  • The appendix reports two additional cases illustrating GM-PRM and the Refined-BoN process.Source: §A.5 · p. 11

A.4 Prompt

A.5 More Cases

Loading 2508.04088v2…