Source-linked AI summary
R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, Jiaxing Huang
TL;DR
MLLM reinforcement learning suffers from sparse rewards and vanishing advantages, especially when reasoning paths are limited or homogeneous. Share-GRPO expands questions, shares diverse trajectories and reward information hierarchically, achieving superior results across six reasoning benchmarks, including 75.4% on MathVista with six responses per question.
Problem
Directly applying GRPO to MLLMs suffers from sparse rewards and advantage vanishing, causing inefficient exploration and unstable training.
Method
Share-GRPO expands each question into semantically consistent variants, shares diverse reasoning trajectories, and computes hierarchical advantages using shared reward information.
Results
Share-GRPO demonstrates superior performance across six reasoning benchmarks, achieving 75.4% on MathVista with six generated responses per question.
Takeaways & Limitations
Diverse reasoning paths and information sharing provide richer reward signals and more stable policy learning than simply increasing sampling.
Abstract
from arXiv · showhide
In this work, we aim to incentivize the reasoning ability of Multimodal Large Language Models (MLLMs) via reinforcement learning (RL) and develop an effective approach that mitigates the sparse reward and advantage vanishing issues during RL. To this end, we propose Share-GRPO, a novel RL approach that tackle these issues by exploring and sharing diverse reasoning trajectories over expanded question space. Specifically, Share-GRPO first expands the question space for a given question via data transformation techniques, and then encourages MLLM to effectively explore diverse reasoning trajectories over the expanded question space and shares the discovered reasoning trajectories across the expanded questions during RL. In addition, Share-GRPO also shares reward information during advantage computation, which estimates solution advantages hierarchically across and within question variants, allowing more accurate estimation of relative advantages and improving the stability of policy training. Extensive evaluations over six widely-used reasoning benchmarks showcase the superior performance of our method. Code will be available at https://github.com/HJYao00/R1-ShareVL.
1 Introduction
The paper identifies sparse rewards and advantage vanishing as obstacles when applying GRPO-like reinforcement learning to MLLMs. Share-GRPO addresses them by expanding question space, sharing reasoning trajectories and rewards, and estimating advantages hierarchically.
- Motivation: Directly applying GRPO to MLLMs suffers from sparse rewards and advantage vanishing, degrading reasoning-oriented policy training.Only a few paths receive positive rewards, while homogeneous responses can make relative advantages approach or collapse to zero.
- Share-GRPO: Share-GRPO generates semantically consistent question variants, explores diverse reasoning trajectories, and shares discovered trajectories across variants.Each expanded question can contribute to and benefit from a shared solution space.
- Share-GRPO: Share-GRPO shares reward information through hierarchical advantage estimation across and within question variants.Global estimation aggregates responses across variants, while local estimation captures responses within each variant.
- Evidence: The authors report extensive evaluations on six MLLM reasoning benchmarks demonstrating the superiority of Share-GRPO.Figure 1 also summarizes denser rewards, higher valid advantage ratios, and performance gains over baseline and other SOTA RL-based reasoning MLLMs.
2 Related Work
Related work covers multimodal language models, reinforcement learning for MLLMs, and information sharing across learning systems. The paper positions Share-GRPO as introducing information sharing into MLLM reasoning reinforcement learning.
- Multimodal Language Models: MLLM research progressed from text-image alignment and multimodal integration toward broader cross-modal understanding through multimodal instruction fine-tuning.The passage cites GPT-4V as an example of this progression.
- Reinforcement Learning for MLLMs: Recent MLLM reinforcement learning uses human feedback or rule-based rewards to improve model capabilities.DeepSeek-R1 is presented as using a simple rule-based reward function during reinforcement learning.
- Information Sharing: Information sharing supports learning through exchanged signals across modalities, tasks, or hierarchical model components.Examples include shared representations in contrastive learning and shared structures or value functions in reinforcement learning.
- Research Positioning: Unlike the cited prior work, this paper introduces information sharing into MLLM reasoning reinforcement learning to address sparse rewards and advantage vanishing.The stated target is more effective reasoning learning in MLLMs.
3 Method
Share-GRPO expands each question into semantically equivalent variants, explores and shares reasoning responses across them, and computes shared advantages at global and local levels. The method uses rule-based accuracy and format rewards within a GRPO-style optimization framework.
- GRPO Preliminary: GRPO samples response groups from a reference policy, computes rewards, and estimates each response’s relative advantage for policy optimization.Its clipped objective includes a KL penalty term.
- Share-GRPO: Share-GRPO applies semantically consistent transformations to generate varied but equivalent questions and expands the reasoning solution space.The framework then explores diverse reasoning paths and shares trajectories and rewards across question variants.
- Question Space Expansion: Offline textual transformation rewrites prompts while preserving their original intent and correct answer.The generated variants differ in syntax and lexical expression.
- Question Space Expansion: Online multimodal transformation alters visual content using transformations selected to preserve critical reasoning cues, with textual guidance aligned to the visual change.The method mentions rotation and noise injection as suitable transformations and avoids cropping or color distortion.
- Solution Space Expansion: For each question variant, the policy generates multiple candidate reasoning responses, creating an expanded response set for shared exploration.This enlarged solution space supports trajectory sharing across variants.
- Shared Advantage Estimation: Share-GRPO computes rule-based outcome accuracy and format rewards, then estimates advantages globally across variants and locally within each variant.The hierarchical estimate combines cross-variant diversity with within-variant response structure.
4 Experiments
Experiments evaluate Share-GRPO across six reasoning benchmarks, ablations, hyperparameter studies, and qualitative mathematical examples. Results show broad gains, improved reward utilization, and benefits from diverse question variants and shared trajectories.
- 4.2 Main Results: Share-GRPO improves R1-ShareVL-7B over its base model by +7.2% on MathVista and +3.6% on MathVerse.
- 4.2 Main Results: R1-ShareVL-7B outperforms MM-Eureka-7B by +2.1% on average and exceeds ThinkLite-7B by +5.0% on MMMU and +3.3% on MMStar.
- 4.3 Ablation Study: Ablations show that adding information sharing among offline variants with global shared advantage estimation improves performance over GRPO by +1.1%.
- 4.4 Discussion: Share-GRPO reaches 66.8% average performance versus 65.1% for GRPO with dynamic sampling, while their combination reaches 67.2%.
- 4.4 Discussion: With m = 2, Share-GRPO achieves 75.4% accuracy on MathVista, and increasing the number of question variants to 4 further improves performance.
- 4.4 Discussion: GRPO improves from 72.3% at N = 6 to 72.8% at N = 12 on MathVista, whereas Share-GRPO reaches 75.4% with six responses per question.
- 4.4 Discussion: The qualitative example shows R1-ShareVL correctly interpreting a mathematical question and producing a coherent symbolic reasoning solution.
5 Conclusion
The paper proposes Share-GRPO, a reinforcement learning framework for MLLMs that uses information sharing to address sparse rewards and advantage vanishing. It expands questions with semantically consistent variants, shares responses, and computes hierarchical advantages across global and local levels.
- Share-GRPO introduces information sharing to mitigate sparse rewards and advantage vanishing in MLLM reinforcement learning.
- The framework generates semantically consistent question variants and encourages MLLMs to explore and share responses across a more diverse solution space.
- Hierarchical advantage estimation operates within and across question variants at global and local levels to guide optimization.
- Experiments, ablations, and discussion report superior performance across various reasoning benchmarks.
A Benchmarks
The evaluation covers six multimodal reasoning benchmarks spanning mathematics, general multimodal understanding, scientific charts, and visual question answering. Together, these datasets test diverse reasoning and visual-understanding capabilities.
- MathVista contains 6141 questions covering arithmetic, geometry, algebra, and statistics for mathematical problem solving.
- MMStar provides 1500 visual key samples, while MMMU contains 11.5K multimodal questions from university exams, quizzes, and textbooks.
- MathVerse, MathVision, and AI2D cover multimodal mathematics, competition mathematics with visual contexts, and scientific-chart understanding.