PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Data Efficiency in Multimodal Process Reward Models

Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, Haolin Liu, Wenxuan Zhang, Jiaxin Huang

arXiv:2602.04145v2cs.LGcs.CLcs.MM

TL;DR

MPRM training uses costly MC-annotated rollouts, but the data may be redundant and its informative content depends on label composition and reliability. The paper formalizes these factors and introduces BIS, which selects mixed, reliable rollouts using existing MC signals. Across two backbones on VisualProcessBench, BIS-selected subsets match or surpass full-data performance, including at 10% of the rollouts.

  • Problem

    MPRM training typically requires large MC-annotated corpora, motivating study of how performance scales with rollout budget and how informative subsets can preserve full-data performance.

  • Method

    The paper combines a teacher–student label-noise analysis with BIS, a model-agnostic rollout-level score balancing positive/negative mixture and positive-step reliability from stored MC signals.

  • Results

    BIS-selected subsets match or surpass full-data performance across InternVL2.5-8B and Qwen2.5-VL-7B on VisualProcessBench, with InternVL2.5-8B reaching 65.46% overall micro-F1 at 10%.

  • Takeaways & Limitations

    MPRM training data can be curated by prioritizing uncertain yet reliably labeled steps, reducing rollout requirements without sacrificing reported performance.

  • Takeaways & Limitations

    The theoretical analysis uses simplifying assumptions including symmetric, constant label noise and a bounded-norm assumption.

Abstract

from arXiv · show

Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.

1. Introduction

MPRM training depends on costly MC-annotated rollouts, yet these corpora contain substantial redundancy. The paper analyzes rollout informativeness and proposes BIS to select mixed, reliable rollouts without extra computation.

  • MPRMs provide step-level supervision for visual reasoning by scoring intermediate steps rather than only final answers.
  • Large-scale MC-annotated rollouts make MPRM training computationally expensive and motivate studying performance scaling with rollout budgets.
  • Random subsampling causes performance to saturate quickly, revealing substantial redundancy in existing MC-annotated training data.
  • Informative rollouts combine positive and negative steps with reliable positive labels, whereas very low average MC scores introduce noisy pseudo-positive labels.
  • BIS selects rollouts using label mixture and reliability from stored MC signals, and experiments across two backbones show strong performance at small subset ratios.
  • 10% of the BIS-selected rollouts reaches and can surpass full-data performance on InternVL2.5-8B while saving 95.5% computational cost.

2. Preliminary Study

The preliminary study finds that random subsampling quickly reaches diminishing returns, indicating redundancy in MC-annotated rollouts. Controlled subset comparisons show that mixed rollouts with reliable positive labels provide more informative supervision than random or low-MC selections.

  • Background and General Setup: VisualProcessBench evaluates MPRMs with per-source macro-F1 and micro-averaged F1 across five human-annotated sources.
  • Random Sub-Sampling: Evidence of Redundancy: Random-ρ preserves each source’s relative composition while retaining fraction ρ of rollouts for single-pass fine-tuning.
  • Random Sub-Sampling: Evidence of Redundancy: Performance quickly plateaus as ρ increases, so discarding many rollouts only modestly degrades micro-F1.
  • Random Sub-Sampling: Evidence of Redundancy: Under matched updates, Full-Data eventually performs better than Random-25%, but their gap remains moderate and Random-25% slightly overfits late.
  • Characterizing Informative Rollouts: Mixed-25%, Low-MC-25%, and Random-25% test whether increased exposure to negative steps improves supervision under a fixed data budget.
  • Characterizing Informative Rollouts: Mixed-25% has an 11.02% incorrect-step ratio and 0.716 average MC score, while Low-MC-25% has 12.57% and 0.601, respectively.
  • Characterizing Informative Rollouts: At best checkpoints, Mixed-25% > Low-MC-25% > Random-25%, showing that mixed rollouts with reliable positives outperform selections emphasizing low MC scores alone.

3. Theoretical Analysis

The theoretical analysis models MPRM training as learning from noisy MC-annotated labels and argues that gradient noise, rather than data scarcity, limits performance. It shows that informative rollouts combine teacher uncertainty through label mixture with reliable positive labels reflected by MC scores.

  • Teacher–Student Abstraction: The teacher–student framework treats the teacher as an ideal correctness model and the student as learning noisy MC-annotated step labels with logistic loss.The analysis uses a linear representation and logistic-regression abstraction for step-level prediction.
  • Data Efficiency: MPRM training mostly suffers from noisy gradients instead of insufficient training data.The analysis links MC label noise to stochastic-gradient noise and optimization error.
  • Data Efficiency: Random subsampling has only a modest effect when the data-complexity term is small and optimization error dominates.This explains why Random-γ can closely track Full-Data under matched-update or single-pass settings.
  • Label Reliability: Low-MC positive steps have high effective noise, reducing useful gradient signal and increasing the noise contribution.Averaging MC scores over positive steps therefore provides a rollout-level reliability estimate associated with less label-noise interference.
  • Rollout Uncertainty: Rollout label mixture estimates teacher uncertainty with O(1/n) bias and is largest for rollouts containing balanced positive and negative steps.Nearly all-positive or all-negative rollouts have low mixture, whereas mixed rollouts can provide stronger gradient signal.
  • Combined Criterion: Rollouts are most informative when both label mixture and reliability are high.This theoretical finding motivates selecting mixed rollouts with trustworthy positive labels.

4. Balanced-Information Score

The Balanced-Information Score selects MC-annotated rollouts using both label mixture and positive-label reliability. It ranks rollouts within each source using existing MC signals, enabling fixed-ratio subsets without additional supervision or model calls.

  • Score Definition: BIS combines the positive-step mixture ppos(1 − ppos) with reliability R(x) to score rollout information.The mixture term favors rollouts containing correct and incorrect steps, while reliability favors trustworthy positive steps.
  • Score Definition: BIS is highest for rollouts that provide both clear negative signals and trustworthy positive anchors.A smoothing constant α > 0 gives non-zero weight to low-mixture rollouts.
  • Subset Construction: BIS ranks rollouts within each source and retains the top ρ fraction to form the selected training subset.The same global keep ratio is applied across sources before concatenating the selected rollouts.
  • Evaluation: Table 2 reports overall micro-F1 and per-source macro-F1 for full-data and subsampled training under different keep ratios and labeling schemes.The table distinguishes soft targets, thresholded hard labels, and the untrained backbone baseline.

5. Experiments

Experiments show that BIS efficiently selects informative rollouts, matching or exceeding full-data performance at small budgets and outperforming random or single-component heuristics. Its gains persist across backbones and reranking benchmarks, while both mixture and reliability are necessary.

  • Main results: BIS reaches full-data performance at small keep ratios and consistently outperforms random subsampling across both backbones.It is evaluated against matched rollout budgets on VisualProcessBench.
  • Main results: BIS improves over random subsampling by +10.9 points at ρ=5% and +5.5 points at ρ=15% for Qwen2.5-VL-7B, reaching the full-data reference at ρ=25%.Performance rises rapidly at small budgets, peaks at a moderate ratio, and can slightly decline afterward.
  • Best-of-N reranking: BIS-selected subsets achieve the strongest best-of-N reranking performance across MM-K12, OlympiadBench, MathVerse, and MathVista.The evaluation uses N=16 candidates per problem and surpasses both Random-25% and the full-data MPRM.
  • Selection behavior: Figure 3 shows BIS suppressing low-reliability rollouts and concentrating coverage around moderate R(x), rather than simply maximizing reliability.The BIS-25% subset is compared with all rollouts using reliability, mixture, and coverage distributions.
  • Ablations: Ablations show that mixture-only and reliability-only selection are weaker than BIS, demonstrating that both components contribute under the 25% budget.Reliable-25% lags on nearly all benchmarks, while Mixed-25% remains consistently weaker than BIS.
  • Sensitivity: Performance is broadly stable across α values, with α=0.05 consistently best because it balances low-mixture coverage against reliability emphasis.Very small α can reduce supervision diversity, while very large α weakens the mixture term.

6. Conclusion

The paper finds substantial redundancy in MC-annotated MPRM training data and explains why informative updates favor uncertain yet reliably labeled steps. BIS uses these signals to match or surpass full-data performance with as little as 10% of rollouts.

  • Findings: Randomly discarding most rollouts only mildly degrades performance, indicating substantial redundancy in current MPRM training sets.The conclusion reports this finding as the basis for studying rollout selection.
  • Theory: Informative gradient updates concentrate on uncertain yet reliably labeled steps, while low-MC pseudo-positives mainly add variance.This summarizes the paper’s theoretical explanation of data efficiency.
  • Method: BIS ranks rollouts by label mixture and reliability using only MC signals already stored in the dataset.The method is presented as a data-centric principle for curating future MPRM corpora.
  • Results: BIS-selected subsets match or surpass full-data MPRM performance with as little as 10% of rollouts.The conclusion identifies this as the principal empirical outcome.

Impact Statement

The paper frames MPRM training as a data- and compute-efficiency problem, with potential societal benefits from reducing training energy costs. The supplied passages also include supporting appendices and related background.

  • Impact: The work aims to improve data and compute efficiency when training multimodal process reward models.The stated societal benefit is reduced energy cost if the method adapts successfully to practical training.
  • Impact: Successful adaptation to practical model training is described as potentially reducing energy cost.This is presented as a prospective benefit rather than an experimentally established outcome.

A.2. Data-efficient Process Supervision

The paper situates its approach within data-efficient process supervision and evaluates it on VisualProcessBench, a diverse step-level benchmark with multiple source datasets and annotated trajectories.

  • Related work: Data-efficient process-supervision research includes annotation-pipeline optimization, learning from noisy feedback, and data selection or supervision allocation.The paper places its work in the third category.
  • Related work: The paper’s approach focuses on data selection and supervision allocation rather than improving annotation pipelines or training frameworks.DreamPRM and DreamPRM-1.5 are cited as related data-selection methods.
  • Benchmark: VisualProcessBench contains 2,866 solution trajectories and 26,950 annotated steps across five source datasets.The benchmark provides a diverse testbed for step-level evaluation.
  • Benchmark: Table 6 summarizes source-level problem counts, base-model solution distributions, step-label composition, and trajectory-length statistics.These statistics characterize the benchmark’s composition.

C. Experimental Setup and Implementation Details

The experiments use two multimodal backbones, standardized training and evaluation procedures, and shared implementation settings across data-selection conditions.

  • InternVL2.5-8B is the default backbone, with Qwen2.5-VL-7B included as a second-backbone experiment.
  • MPRM supervision is applied at <prm> positions using binary Yes/No reward-token classification with two-way cross-entropy.
  • Under the single-pass protocol, each data regime recomputes optimization steps from its selected rollout count and global batch size.
  • VisualProcessBench evaluation produces per-step scores, thresholds them into positive or negative predictions, and reports benchmark metrics using a global threshold.
  • Best-of-N reranking uses N = 16 candidate rollouts and selects the candidate with the highest trajectory-level score.

D. Extended Results for Random Sub-sampling (Section 2.2)

Random subsampling shows rapid performance gains at small fractions followed by saturation, while retaining 25% of the rollouts leaves only a moderate gap to full-data training.

  • Random-subset performance rises sharply from 0 to a small fraction and then quickly saturates across VisualProcessBench source datasets.
  • At ρ = 25%, Full-Data has a systematic but moderate edge over Random-25% under a matched update budget.
  • The modest 25%-versus-full-data gap after discarding 75% of rollouts is consistent with diminishing returns from additional rollouts.

E. Extended Results for 25% Subsets (Sections 2.3 and 5.2)

The extended analysis compares 25% subset training dynamics and develops theoretical support for mixture and reliability as proxies for informative rollout supervision.

  • Extended 25% results: BIS-25% produces the highest or near-highest performance curve at almost all training steps across sources.
  • Extended 25% results: The 25% subset curves compare BIS with Random, Low-MC, Mixed, and Reliable selections under the same rollout and update budget.
  • Theoretical analysis: The theory decomposes excess population loss into an O(N_eff^-1/2) data term and an O(T^-1/2) optimization term.
  • Theoretical analysis: Higher Monte Carlo noise can increase the gradient second-moment bound and make the optimization term dominant in a noise-limited regime.
  • Mixture and reliability: Rollout-level label mixture is an approximately unbiased proxy for teacher mixture, with bias bounded by O(1/n).
  • Mixture and reliability: Observed-label mixture remains within O(η) + O(1/n) of the noise-free target in expectation under small η and sufficiently large n.
  • Mixture and reliability: Under bounded representation norms, A(x) and A_full(x) measure the same step-wise information up to global multiplicative constants.
  • MC-induced pseudo-positive probability: The pseudo-positive probability Pr(r < τ | K = k) decreases strictly with k, while Pr(r ≥ τ | K = k) increases strictly.

G. Training Dynamics

Training-dynamics results compare BIS and random subsets across keep ratios and backbones, showing faster learning and stronger performance for BIS, especially at low budgets.

  • Across ρ ∈ {5%, 10%, 15%, 25%, 35%, 50%} and both backbones, BIS achieves stronger final performance and improves faster than Random-ρ.
  • BIS reaches high accuracy in substantially fewer training steps and maintains advantages over random subsampling throughout training.

H. Per-source BIS Histograms

Per-source BIS distributions reveal why rollout quality varies across datasets: low scores can reflect either insufficient label mixture or unreliable positive supervision. BIS shifts selection toward rollouts that combine informative mixtures with reliable positive steps, while case studies show why reliability is necessary beyond mixture alone.

  • Distributional interpretation: BIS scores low for nearly pure rollouts because their mixture term ppos(1 − ppos) is small, regardless of source.Per-source interpretation therefore requires separating mixture scarcity from reliability scarcity.
  • Source difficulty: High-¯s sources such as ScienceQA (¯s = 0.9723) and NLVR2 (¯s = 0.9672) have thinner high-BIS tails because naturally mixed trajectories are rarer.DVQA is another high-¯s example with ¯s = 0.9453.
  • Source difficulty: Low-¯s sources such as GeoQA+ (Open) (¯s = 0.5720) and GeomVerse (0.6819) lack reliable positive anchors, limiting mixed-but-reliable rollouts.Many steps rarely lead to successful continuations under MC sampling, even when errors are plentiful.
  • Source difficulty: Medium-¯s sources provide the richest BIS substrate because they retain both non-trivial label mixtures and reasonably reliable positives.FigureQA (¯s = 0.7615) and VQAv2 (¯s = 0.8253) show heavier moderate-to-high BIS mass.
  • Selection behavior: Under the same per-source top-25% budget, BIS shifts selected-rollout mass toward higher scores and favors mixed, reliable trajectories.The component histograms show suppression of low R(x) and increased coverage at moderate-to-high ppos(1 − ppos).
  • Case studies: A high-quality rollout receives a large BIS score when positive steps have consistently high MC scores alongside meaningful step-level variation.This case matches the intended mixed-but-reliable criterion.
  • Case studies: A mixed rollout receives only a moderate BIS score when some positive steps have low MC scores, showing that mixture alone is insufficient.The borderline case later collapses to an incorrect final choice.
  • Case studies: Low-but-nonzero MC scores can create unreliable pseudo-positive labels under hard thresholding, while BIS down-weights these low-confidence rollouts.The diagnostic failure case includes scores often at the 1/16 level despite clear geometry or number mistakes.
Loading 2602.04145v2…