Unlocking Multimodal Mathematical Reasoning via Process Reward Model

• URSA targets the largely unexplored use of Process Reward Models in multimodal mathematical reasoning.Source: §1 · pp. 1–2
• The framework addresses scarce high-quality reasoning data, inadequate multimodal process labeling, and reward hacking or length bias in process-reward online RL.Source: §1 · p. 2
• URSA constructs MMathCoT-1M from 1.43 million open-source examples to strengthen the foundation model, then synthesizes DualMath-1.1M for process reward modeling.Source: §1 · p. 2
• PS-GRPO incorporates multimodal PRMs by comparing rollout quality rather than relying on scalar rewards, implicitly penalizing process-level inconsistencies.Source: §1 · p. 2
• Results across 6 multimodal reasoning benchmarks show that the PRM improves Best-of-N verification over self-consistency and outcome-based baselines.Source: §1 · p. 2
• With PS-GRPO, URSA-8B-PS-GRPO outperforms Gemma3-12B by 8.4% and GPT-4o by 2.7% on average across 6 benchmarks.Source: §1 · p. 2, Figure 1 · p. 2

2.1 Collection of Vision-Language Alignment Data

• URSA-Alignment-860K collects math-intensive vision-language data from Multimath, MAVIS, and Geo170K, then trains only the MLP projector during alignment.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 CoT Reasoning Data Synthesis

• The CoT synthesis stage collects 1.43 million samples from five existing math reasoning datasets and categorizes them as answer-only, analysis-formatted, or CoT-formatted.Source: §2.2 · p. 3
• Answer-only samples are expanded into reasoning trajectories conditioned on their questions and ground-truth answers.Source: §2.2 · p. 3
• Analysis-formatted samples are rewritten to improve step-by-step logic and linguistic diversity because their original solutions can contain reasoning jumps or brief rationales.Source: §2.2 · p. 3
• CoT-formatted solutions from Multimath-EN-300K are natural-language stylized to unify their mathematical and symbolic presentation.Source: §2.2 · p. 3
• Generated solutions are filtered when they alter the original answer or introduce consistency problems such as questioning the answer or forcing it through new assumptions.Source: §2.2 · p. 3
• The curated solution sets are unioned into the SFT dataset and used for full-parameter instruction fine-tuning of URSA-8B.Source: §2.2 · p. 4
• URSA-8B is constructed as a stronger reasoning foundation expected to raise inference-time performance and support more diverse process supervision data.Source: §2.2 · p. 4

3.1 Binary Error Locating Engine

• 773K process annotations are produced with Binary Error Locating, which uses rollout correctness potential to identify the first erroneous step and adds correct solutions as positive examples.A step is potentially correct when its Monte Carlo estimate is positive; binary search localizes the error.Source: §3.1 · p. 4

3.2 Misinterpretation Insertion Engine

• 302K multimodal supervision samples are synthesized by inserting perception-related hallucinations into correct solutions and labeling subsequent steps as incorrect.The insertion process targets potentially confusable image conditions, then continues reasoning from the altered step.Source: §3.2 · p. 5

3.3 PRM Training

• DualMath-1.1M merges binary error-location and misinterpretation-insertion data to supervise both logical correctness and perceptual consistency.The merged dataset contains approximately 1.1 million process-supervision examples.Source: §3.3 · p. 5
• The PRM is trained as a binary classifier that appends a correctness token after each reasoning step.Each step ej receives a binary label yj indicating predicted correctness.Source: §3.3 · p. 5

Scalar process rewards

• Scalar process-reward variants reduce test accuracy below vanilla GRPO, while training also produces shorter responses and fewer reasoning steps.The paper attributes these patterns to reward hacking and conservative later-step rewards caused by process-label bias.Source: §4 · pp. 5–6, Figure 5 · p. 6

PS-GRPO

• PRM-based Best-of-N selection and drop-moment error identification remain stable during online RL, indicating that relative solution quality is more reliable than scalar rewards.A drop-moment is a significant decrease between consecutive process rewards.Source: §4 · p. 6
• PS-GRPO uses drop-moment detection and a penalty for correct-result rollouts to focus learning on solutions with accurate results and rigorous processes.Relative reward changes are used to avoid the PRM’s length bias in rewarding.Source: §4 · p. 6

Benchmark evaluation

• Table 1 evaluates MLLMs across six benchmarks using task-specific accuracy, loose score, or average-case accuracy metrics.The table distinguishes closed-source best results from open-source best and runner-up results.Source: Table 1 · p. 7

Test-time scaling

• Table 2 compares Best-of-N test-time scaling on URSA-8B and AtomThink-EMOVA.The comparison evaluates verifier-assisted rollout selection.Source: Table 2 · p. 7

5.1 Experimental Setup

• The evaluation uses zero-shot inference on MathVerse, DYNAMATH, MathVista, WE-MATH, GeoQA, and MathVision.Baselines include proprietary models, comparable-size open-source MLLMs, and self-consistency or ORM alternatives for PRM test-time scaling.Source: §5.1 · p. 7
• URSA uses SAM-B+SigLIP-L with Qwen2.5-Math-Instruct and selects 15K MMathCoT-1M examples for PS-GRPO.The implementation uses a two-layer MLP for vision-language alignment; γ and ρ are set to 0.5 and 0.3.Source: §5.1 · pp. 7–8

5.2 Main Results

• 8.5% and 8.2% average-performance advantages over AtomThink-EMOVA-8B and Gemma3-12B, respectively, are reported for URSA-8B-PS-GRPO.It also outperforms GPT-4o across six benchmarks on average, including MathVista-GPS, GeoQA, and MathVision comparisons.Source: §5.2 · p. 8
• 16.6% and 10.1% relative improvements on MathVerse and MathVision are achieved with URSA-8B-RM using only four samplings.In Best-of-32 evaluation, URSA-8B reaches 35.1 on MathVision and 55.0 on MathVerse.Source: §5.2 · p. 8
• 6.8% versus 3.1% average improvement shows that PS-GRPO outperforms vanilla GRPO under matched data, hyperparameters, and rollout counts.The gain is nearly doubled on WE-MATH and MathVision, while improvements on MathVista-GPS and GeoQA are relatively small.Source: §5.2 · p. 8

6.1 How Each Stage Contributes the Performance

• MMathCoT-1M provides the highest absolute performance gain among the training stages, while all stages contribute to performance.Alignment-860K is more evident on MathVerse and MathVision, whereas PS-GRPO is especially prominent on WE-MATH and MathVerse.Source: §6.1 · pp. 8–9
• 13.2% and 11.4% relative improvements on WE-MATH and MathVerse are reported for PS-GRPO over URSA-8B.The analysis also reports generalization validation on InternVL2.5-8B and Multimath in Appendix C.4.Source: §6.1 · p. 9

6.2 Ablation Studies on Automatic Process Labeling

• Both BEL and MIE contribute positively to URSA-8B-RM, showing that logical correctness and perceptual consistency each matter in process supervision.BEL has the larger impact, indicating that synthesized-data quality remains improvable; the ablation is reported for DualMath-1.1M Best-of-N evaluation.Source: §6.2 · p. 9, Table 3 · p. 9

6.3 Sensitivity Analysis on Reward Penalty and Drop-moment

• γ values that are too high can make group rewards vanish and destabilize training, while an appropriate tested range from 0.3 to 0.7 is generally beneficial when ρ=0.3.Excessively large ρ reduces reward differentiation toward vanilla GRPO, whereas excessively small ρ over-penalizes process-reward changes.Source: §6.3 · p. 9, Table 4 · p. 9

7 Conclusion

• URSA introduces MMathCoT-1M, DualMath-1.1M, and PS-GRPO to address multimodal reasoning-data scarcity, process labeling, and reward-modeling challenges.The pipeline builds URSA-8B, trains a multimodal PRM, and integrates relative process quality into online reinforcement learning.Source: §7 · pp. 9–10
• 8.4% and 2.7% superior average performance over Gemma3-12B and GPT-4o, respectively, is reported for URSA-8B-PS-GRPO.The comparison spans six benchmarks and is presented as the resulting model’s average-performance outcome.Source: §7 · p. 10
• DualMath-1.1M provides large-scale automated process supervision covering logical validity and perceptual consistency in multimodal reasoning.PS-GRPO is described as addressing reward hacking and rewarding length bias through process-as-outcome modeling.Source: §7 · p. 10

A Related Work

• Multimodal mathematical reasoning requires interpreting visual information and performing cross-modal reasoning between images and text.The related work highlights geometric problems and graph analysis as particularly challenging tasks.Source: A Related Work · p. 18
• Prior PRM research uses intermediate-step evaluation, verifiers, and search or scaling methods, but multimodal reasoning still lacks robust CoT models and diverse reward-modeling data.Concurrent work has begun exploring PRM-assisted visual reasoning through construction and benchmarking.Source: A Related Work · p. 18

B.1 Group Relative Policy Optimization

• Vanilla GRPO samples response groups, normalizes in-group rewards to estimate advantages, and applies PPO clipping with a KL penalty.The formulation removes PPO’s value function and computes relative advantages within each online rollout group.Source: §B.1 · p. 18
• The appendix describes PRM-integrated GRPO variants using process-reward sequences either with outcome rewards or through step-level multiple relative advantages.These variants are introduced as alternatives to vanilla GRPO’s outcome-reward formulation.Source: §B.1 · p. 18

B.2 Test-Time Scaling by Best-of-N evaluation

• Best-of-N evaluation samples N responses and uses mean process rewards to select the best single response for test-time scaling.The appendix chooses this direct selection procedure rather than combining PRM scores with self-consistency voting.Source: §B.2 · p. 19

C.1 Fine-grained Comparison on Used benchmarks

• URSA models outperform open-source competitors in DYNAMATH plane geometry and algebra, with especially strong undergraduate-level performance.Source: §C.1 · p. 19, Table 6 · p. 20
• URSA-8B-PS-GRPO marginally surpasses GPT-4o on MathVerse and exceeds AtomThink-EMOVA-8B and InternVL2.5-8B by 8.4 and 11.4 points.Source: §C.1 · p. 19, Table 7 · p. 20

C.2 Scaling Law of MMathCoT-1M

• MMathCoT-1M exhibits a training-time scaling law, with larger sampled-data ratios improving URSA-8B performance.Source: §C.2 · p. 20, Table 9 · p. 21

C.3 Higher Upper Bound Taken from Stage I

• URSA-8B consistently outperforms leading general and math MLLMs on Pass@N across MathVerse, MathVision, and MathVista-GPS, raising the upper bound available to later PRM and RL stages.Source: §C.3 · p. 21, Figure 7 · p. 22

C.4 Generalization Validation

• On InternVL2.5-8B, MMathCoT-1M raises the average score from 45.2 to 51.7 and PS-GRPO to 54.7; on MultiMath, scores rise from 43.1 to 48.7 and 51.2.Source: §C.4 · p. 22, Figure 8 · p. 22

C.5 Implementary Results on Other Benchmarks

• URSA-8B-RM improves Best-of-8 performance over self-consistency by 4.6%, 4.5%, and 2.7% on WE-MATH, DYNAMATH, and GeoQA.Source: §C.5 · p. 22, Table 10 · p. 22

D Module Selection Criteria

• The hybrid SigLIP-L+SAM-B vision tower and Qwen2.5-Math-Instruct backbone were selected after qualitative captioning and small-scale comparisons.Source: D Module Selection Criteria · pp. 22–23

E.1 Effectiveness of Different Data Category

• Complete synthesized data achieves the best results across datasets, with answer-only CoT expansion strongest on MathVerse and MathVision.Source: §E.1 · p. 23, Figure 9 · p. 23
• Replacing analysis rewriting most harms WE-MATH, while rewriting and natural-language formulation improve DYNAMATH robustness under image transformations.Source: §E.1 · p. 23

E.2 Selection of External Closed-source MLLM

• Gemini-1.5-Flash-002 offers strong unimodal and multimodal math performance, while GPT-4o provides no clear SFT advantage in the 50K-example comparison.Source: §E.2 · p. 24, Table 11 · p. 24
• Gemini-1.5-Flash-002 costs one-thirty-third as much in output tokens as GPT-4o, making it the cost-effective synthesis choice for approximately 2.7 million API calls.Source: §E.2 · p. 24
• The authors expect stronger results if researchers can afford more powerful closed-source models.Source: §E.2 · p. 24

F.1 RL Data Curation

• Approximately 15K filtered examples remain after eight-sample screening removes items whose rollouts are all correct or all incorrect.Source: §F.1 · p. 25, Figure 10 · p. 25

F.2 Parameters and Time Cost

• Experiments use one-time difficulty-based data selection without additional GRPO optimization tricks, supporting a focused comparison of vanilla GRPO and PS-GRPO.Source: §F.2 · p. 25, Table 13 · p. 25

F.3 Benchmarks

• Evaluation combines rule-based matching for WE-MATH and GeoQA with LLM-as-a-Judge scoring for MathVista, MathVision, MathVerse, and DYNAMATH.Source: §F.3 · pp. 25–26

F.4 Algorithm

• Binary Error Locating uses midpoint Monte Carlo checks to identify the first erroneous step in O(log N) searches before collecting sequential labels.Source: §F.4 · pp. 26–28
• PS-GRPO merges outcome and process-as-outcome rewards before calculating relative advantages for policy updates.Source: §F.4 · pp. 27–28

G.1 Prompt Utilized in MMathCoT-1M Synthesis

• MMathCoT-1M synthesizes answer-only expansions, analysis rewrites, and unified reasoning formats, then filters altered answers, unsupported assumptions, and unreasonable trajectories.Source: §G.1 · pp. 28–29

G.2 Prompt Utilized in DualMath-1.1M Synthesis

• DualMath-1.1M synthesis inserts geometry and chart/function misinterpretations to create incorrect reasoning paths for process supervision.Source: §G.2 · pp. 29–30, Figure 13 · p. 30

H.1 Showcase on Best-of-N Evaluation

• URSA-8B-RM detects false theorem applications and angle-number misunderstandings in a MathVista-GPS Best-of-N case, supporting selection of valuable learning samples.Source: §H.1 · p. 30, Figure 18 · p. 35

H.2 Process of Misinterpretation Insertion Engine

• MIE alters extracted visual information at a selected step and continues reasoning from the modified conditions to construct incorrect multimodal paths.Source: §H.2 · p. 30, Figure 19 · p. 36

H.3 Failure Pattern in Process During GRPO

• False-positive rollouts either misalign visual conditions or exploit shortcut patterns, so PS-GRPO penalizes these process errors while retaining outcome rewards.Source: §H.3 · pp. 30–31

H.4 Cases on How Naive Process Reward Modeling Fails

• Naive process-reward RL can favor isolated-looking correct steps over final accuracy, producing conservative or misleading reasoning patterns.Source: §H.4 · pp. 31–32, Figure 22 · p. 39