MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources

Sicong Leng; Jing Wang; Jiaxi Li; Hao Zhang; Zhiqiang Hu; Boqiang Zhang; Yuming Jiang; Hang Zhang; Xin Li; Lidong Bing; Deli Zhao; Wei Lu; Yu Rong; Aixin Sun; Shijian Lu

Source-linked AI summary

MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources

Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Yuming Jiang, Hang Zhang, Xin Li, Lidong Bing, Deli Zhao, Wei Lu, Yu Rong, Aixin Sun, Shijian Lu

arXiv:2509.21268v1Sep 25, 2025cs.CV

TL;DR

Multimodal reasoning is limited by scarce open long-CoT data and unstable GRPO optimization when reward variance is low. The paper introduces VAS, which selects prompts using outcome variance and trajectory diversity, and releases curated resources and models. Across multimodal reasoning benchmarks, the approach improves stability and performance, while not fully resolving RL instability and adding VPS computation overhead.

Problem
Open, large-scale, high-quality multimodal long-CoT data are limited, while low reward variance makes GRPO optimization unstable through gradient vanishing.
Method
VAS uses a Variance Promotion Score combining outcome variance and trajectory diversity to select training prompts, alongside curated datasets, reproducible code, and open models.
Results
VAS improves convergence, stability, and downstream performance across multimodal reasoning benchmarks, while the 7B model reaches a 58.4 average score and the 3B variant reaches 52.7.
Takeaways & Limitations
Reward-variance-aware sampling and released resources provide a theoretically grounded and empirically supported basis for multimodal reasoning training and benchmarking.
Takeaways & Limitations
VAS does not fully resolve multimodal RL training instability, incurs VPS computation overhead, and leaves systematic integration with algorithmic RL advances for future work.

Abstract

from arXiv · show

Large multimodal reasoning models have achieved rapid progress, but their advancement is constrained by two major limitations: the absence of open, large-scale, high-quality long chain-of-thought (CoT) data, and the instability of reinforcement learning (RL) algorithms in post-training. Group Relative Policy Optimization (GRPO), the standard framework for RL fine-tuning, is prone to gradient vanishing when reward variance is low, which weakens optimization signals and impairs convergence. This work makes three contributions: (1) We propose Variance-Aware Sampling (VAS), a data selection strategy guided by Variance Promotion Score (VPS) that combines outcome variance and trajectory diversity to promote reward variance and stabilize policy optimization. (2) We release large-scale, carefully curated resources containing ~1.6M long CoT cold-start data and ~15k RL QA pairs, designed to ensure quality, difficulty, and diversity, along with a fully reproducible end-to-end training codebase. (3) We open-source a family of multimodal reasoning models in multiple scales, establishing standardized baselines for the community. Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS. Comprehensive ablation studies and analyses provide further insight into the contributions of each component. In addition, we theoretically establish that reward variance lower-bounds the expected policy gradient magnitude, with VAS serving as a practical mechanism to realize this guarantee. Our code, data, and checkpoints are available at https://github.com/LengSicong/MMR1.

1 INTRODUCTION

The paper addresses limited open multimodal long-CoT data and unstable GRPO optimization by introducing Variance-Aware Sampling and releasing curated resources and models.

1 INTRODUCTION: The work targets two bottlenecks: scarce open, high-quality multimodal long-CoT data and unstable optimization when GRPO rewards have low variance.These limitations constrain reproducibility and weaken optimization signals.
1 INTRODUCTION: VAS uses a Variance Promotion Score to select prompts that increase reward variance and mitigate gradient vanishing in GRPO training.The score combines outcome variance with trajectory diversity.
1 INTRODUCTION: Experiments show improved convergence, stability, and downstream performance across multimodal mathematical and logical reasoning benchmarks.Ablations find complementary benefits from balancing outcomes and increasing trajectory diversity.

2 RELATED WORK

Prior multimodal RL work modifies rewards, sampling, hyperparameters, or algorithms, but gradient vanishing and limited curated data remain insufficiently addressed.

2 RELATED WORK: Existing multimodal RL approaches refine reward design, sample diversification, hyperparameters, and training strategies but often overlook GRPO gradient vanishing.The resulting optimization can remain unstable and converge slowly.
2 RELATED WORK: Prior remedies mainly adjust RL algorithms or reward mechanisms, whereas this work addresses gradient vanishing through data sampling during training.The sampling perspective is presented as complementary to existing GRPO variants.

3 VARIANCE-AWARE SAMPLING FRAMEWORK

VAS dynamically prioritizes prompts using a score that combines outcome variance and trajectory diversity, while retaining uniform sampling and periodically refreshing scores.

3 VARIANCE-AWARE SAMPLING FRAMEWORK: OVS is maximized when a prompt produces balanced correct and incorrect responses, while TDS measures diversity among sampled reasoning sequences.The two scores operationalize complementary sources of reward variance.
3 VARIANCE-AWARE SAMPLING FRAMEWORK: VPS combines OVS and TDS to prioritize prompts expected to produce higher reward variance and stronger, more consistent GRPO gradient signals.OVS targets outcome variation, while TDS encourages diversity among reasoning trajectories.
3 VARIANCE-AWARE SAMPLING FRAMEWORK: Each batch mixes VPS-weighted sampling with uniform random sampling, with λ controlling the weighted-sampler fraction.This preserves broader data coverage while emphasizing informative prompts.
3 VARIANCE-AWARE SAMPLING FRAMEWORK: VPS values are periodically recomputed from fresh rollouts so prompt selection reflects changes in the evolving policy.The update interval trades off score freshness against computation.

4 THEORY

The theory links reward variance to policy-gradient strength and expected progress, then decomposes variance into outcome and trajectory components that motivate OVS and TDS.

4 THEORY: The gradient variance factorizes into reward variance and a policy-dependent Fisher-information term, making reward variance the prompt-dependent control factor.The Fisher-information term does not depend on rewards and is bounded under the stated conditions.
4 THEORY: Higher reward variance yields stronger gradient signals and larger provable minimum improvements per update, while similar rewards can make gradients vanish.The result is established for REINFORCE and extended to GRPO, including clipped and unclipped settings.
4 THEORY: For binary rewards, total variance separates correctness variability conditioned on a reasoning path from variation across reasoning paths.The first component motivates OVS, while the second motivates TDS through trajectory-diversity measures.
4 THEORY: GRPO normalizes rewards within rollout groups, and clipping introduces bias of order O(ε) while preserving a lower bound that favors higher reward variance.The theoretical guarantee therefore remains qualitatively applicable under both estimator settings.

5 DATA CURATION

The training pipeline combines supervised cold-start fine-tuning on diverse long-CoT data with GRPO using a curated, challenging 15k RL dataset.

5 DATA CURATION: The pipeline uses supervised cold-start fine-tuning followed by GRPO-based reinforcement learning, with each stage supported by curated data.This two-stage design separates long-CoT initialization from reward-driven optimization.
5 DATA CURATION: The cold-start dataset combines instruction-tuning corpora with biology, chemistry, geography, and physics practice problems to broaden multimodal domain coverage.The final collection spans five domain categories, including Math and General Science.
5 DATA CURATION: The RL prompts use concise, verifiable short answers extracted and rephrased from original chain-of-thought annotations for reward modeling.GPT-4o performs the answer extraction and rephrasing.
5 DATA CURATION: The RL dataset contains 8k hard math problems and 7k logical reasoning problems, emphasizing difficulty, diversity, and balanced coverage.The math items are selected for low pass rates, while logical problems come from Raven, MM-IQ, and EasyArc.

6 EXPERIMENTS

MMR1 achieves strong multimodal reasoning performance while VAS improves optimization stability, convergence, and benchmark outcomes across model scales and training analyses.

6.2 MAIN RESULTS: 58.4 average score makes the 7B MMR1 model the highest-performing reasoning-oriented MLLM, ranking first on most evaluated benchmarks.It scores 55.4 on MathVerse, 31.8 on MathVision, 48.9 on LogicVista, and 83.7 on ChartQA.
6.2 MAIN RESULTS: 52.7 average score shows that the 3B variant matches or surpasses several 7B models, including R1-VL at 47.7 and R1-OneVision at 47.8.The result supports competitiveness under smaller model scale.
6.3 EFFECT OF COLD-START AND VAS: Cold-start supervision, GRPO, and VAS provide complementary contributions, with VAS producing the highest overall scores in the component ablation.Cold-start initializes reasoning trajectories, RL encourages exploration, and VAS supports stable variance-aware training.
6.5 EFFECT OF VAS HYPER-PARAMETERS: Moderate update intervals, balanced VPS weighting, and non-extreme mixture ratios provide the most consistent performance, while stale scores or excessive VAS weighting degrade results.Increasing rollouts from 8 to 16 gives marginal gains, whereas 32 adds cost with limited benefit.
6.4 EFFECT OF VAS HYPER-PARAMETERS: VAS configurations yield higher gradient norms and more stable clip fractions than the shuffle baseline, indicating stronger update signals and more effective policy-space exploration.These observations are reported for actor gradients and policy-gradient clipping in Figure 2.
6.4 EFFECT OF VAS HYPER-PARAMETERS: VAS improves validation convergence and final accuracy over uniform sampling, with full VAS sampling converging fastest and mixed sampling also outperforming the baseline.The difference between λ = 1.0 and λ = 0.5 is not pronounced on the mathematics-focused validation set.
6.6 DYNAMICS OF VARIANCE PROMOTION SCORE: VPS rankings stabilize over time while bidirectional transitions remain, allowing the sampler to track prompts whose informativeness changes during training.The distribution shifts toward mid-VPS values as prompts become easier, polarized, or less trajectory-diverse.
6.7 QUALITATIVE DEMONSTRATION: The MathVerse example follows analyze–plan–execute reasoning with verification and an alternative solution, reaching the correct 140° answer.The example illustrates systematic decomposition, reflective checking, and multiple-solution reasoning.

7 CONCLUSION AND LIMITATION

The work introduces VAS for more stable multimodal reinforcement learning and releases curated datasets and models. It acknowledges incomplete resolution of training instability, sampling overhead, and limited integration with algorithmic RL advances.

7 CONCLUSION AND LIMITATION: VAS enhances training stability and RL effectiveness by prioritizing prompts using outcome variance and trajectory diversity.The approach is supported by theoretical analysis and empirical evaluation.
7 CONCLUSION AND LIMITATION: The released cold-start datasets and well-tuned models provide resources for benchmarking and advancing multimodal reasoning research.
7 CONCLUSION AND LIMITATION: VAS does not fully resolve all training instabilities inherent to multimodal reinforcement learning.
7 CONCLUSION AND LIMITATION: Variance-based prompt scoring adds computation, mitigated by longer update intervals or selectively updating subsets of samples.
7 CONCLUSION AND LIMITATION: The method focuses on data sampling, leaving systematic integration with algorithmic RL advances for future work.

A VARIANCE–PROGRESS THEORY

The variance–progress theory explains why reward variance supports informative policy-gradient updates. It formalizes prompt-level rewards, assumptions, and baselines underlying the analysis.

A VARIANCE–PROGRESS THEORY: Reward variance lower-bounds expected one-step improvement, so mixed outcomes and diverse reasoning paths strengthen learning signals.The intuition links outcome variance to informative updates and trajectory diversity to stronger learning.
A VARIANCE–PROGRESS THEORY: The analysis assumes bounded score gradients, positive-definite Fisher terms, smooth objectives, and a gradient lower bound tied to reward variance.
A VARIANCE–PROGRESS THEORY: For each prompt, responses are sampled from the policy, and residual reward subtracts the prompt’s expected reward.
A VARIANCE–PROGRESS THEORY: The expected reward is the optimal action-independent baseline because it minimizes total REINFORCE gradient-estimator variance.The baseline depends on the prompt but not on the sampled response.
A VARIANCE–PROGRESS THEORY: An action-dependent baseline can further reduce scalar variance but requires inner Monte-Carlo estimates.

A.1.2 GRADIENT-VARIANCE BOUNDS

The gradient-variance analysis bounds estimator variance using score-function assumptions and identifies reward variance as the prompt-dependent quantity governing the bounds.

A.1.2 GRADIENT-VARIANCE BOUNDS: Dropping a nonnegative outer-product term yields a valid but looser variance inequality.
A.1.2 GRADIENT-VARIANCE BOUNDS: The variance sandwich bound requires bounded score-function gradients and a uniformly positive smallest Fisher eigenvalue.
A.1.2 GRADIENT-VARIANCE BOUNDS: Reward variance is the only prompt-dependent factor, while Fisher terms contribute bounded model-dependent constants.
A.1.2 GRADIENT-VARIANCE BOUNDS: The upper Fisher bound follows from the score-gradient bound, while the lower bound uses a standard non-degeneracy assumption.The stated setting is a full-dimensional exponential family with parameters in a compact set.

A.2 PROOF OF THE VARIANCE–PROGRESS THEOREM

The variance–progress proof uses a second-order update analysis to lower-bound expected objective improvement by reward variance. Under a sufficiently small step size, the bound depends on model-family constants while prompt dependence enters through reward variance.

A.2 PROOF OF THE VARIANCE–PROGRESS THEOREM: Expected improvement is lower-bounded by a term proportional to reward variance when the update step is sufficiently small.The derivation uses a second-order Taylor expansion and bounds the quadratic remainder.
A.2 PROOF OF THE VARIANCE–PROGRESS THEOREM: The bound depends on constants cmin and L tied to the model family, with all prompt dependence entering through Var[R].

A.3 TWO-LEVEL DECOMPOSITION OF REWARD VARIANCE

Reward variance decomposes into within-trajectory Bernoulli variance and between-trajectory variation in success probabilities. OVS and TDS consistently estimate these components, while VPS combines them into a monotone surrogate that supports GRPO optimization.

Two-level decomposition: Reward variance equals expected within-trajectory Bernoulli variance plus between-trajectory variation in success probabilities.The first component captures outcome randomness within a reasoning trajectory; the second captures variation across trajectories.
Two-level decomposition: Under the stated Lipschitz condition, trajectory diversity supplies an Efron–Stein lower bound for the between-trajectory variance component.The bound depends on squared distances between chain-of-thought trajectories.
Estimator consistency: OVS and TDS estimators are strongly consistent for their respective population variance terms.This consistency follows from the strong law and U-statistic convergence.
Variance Promotion Score: VPS = α OVS + β TDS, with α, β > 0, converges almost surely to a positive affine transform of a lower bound on reward variance.Therefore, VPS is a strongly consistent monotone surrogate rather than an exact equality to reward variance.
Extension to GRPO: GRPO’s centered, variance-normalized estimator retains a positive policy-improvement lower bound whenever reward variance is positive.Importance ratios preserve the dependence after rescaling constants, while clipping introduces an O(ε) bias and reduces the bound by the same order.