Source-linked AI summary
OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks
Wenbo Hu, Xin Chen, Yan Gao-Tian, Yihe Deng, Nanyun Peng, Kai-Wei Chang
TL;DR
Open-source multimodal generalist models must jointly handle heterogeneous reward topologies while balancing fine-grained perception and multi-step reasoning. The paper introduces G2RPO with task-level response-length and entropy shaping, and reports robust performance across 18 benchmarks, including results surpassing GPT-4o on MMMU and MathVista.
Problem
Heterogeneous reward topologies across visual tasks and the difficulty of balancing fine-grained perception with multi-step reasoning constrain open-source multimodal generalist models.
Method
G2RPO maps each task’s advantage distribution toward N(0, 1) using nonlinear distributional matching, combined with response-length and entropy shaping mechanisms.
Results
Across 18 benchmarks, OpenVLThinkerV2 establishes new SOTA results among open-source models, achieving 71.6% on MMMU and 79.5% on MathVista while surpassing GPT-4o.
Takeaways & Limitations
The combined methodology produces a robust, general-purpose multimodal model that balances fine-grained perception with complex multi-step reasoning.
Takeaways & Limitations
The shaping mechanisms use hyperparameters selected from simple empirical observations, leaving systematic exploration and automated search for future work.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) has emerged as the de facto Reinforcement Learning (RL) objective driving recent advancements in Multimodal Large Language Models. However, extending this success to open-source multimodal generalist models remains heavily constrained by two primary challenges: the extreme variance in reward topologies across diverse visual tasks, and the inherent difficulty of balancing fine-grained perception with multi-step reasoning capabilities. To address these issues, we introduce Gaussian GRPO (G$^2$RPO), a novel RL training objective that replaces standard linear scaling with non-linear distributional matching. By mathematically forcing the advantage distribution of any given task to strictly converge to a standard normal distribution, $\mathcal{N}(0,1)$, G$^2$RPO theoretically ensures inter-task gradient equity, mitigates vulnerabilities to heavy-tail outliers, and offers symmetric update for positive and negative rewards. Leveraging the enhanced training stability provided by G$^2$RPO, we introduce two task-level shaping mechanisms to seamlessly balance perception and reasoning. First, response length shaping dynamically elicits extended reasoning chains for complex queries while enforce direct outputs to bolster visual grounding. Second, entropy shaping tightly bounds the model's exploration zone, effectively preventing both entropy collapse and entropy explosion. Integrating these methodologies, we present OpenVLThinkerV2, a highly robust, general-purpose multimodal model. Extensive evaluations across 18 diverse benchmarks demonstrate its superior performance over strong open-source and leading proprietary frontier models.
1 Introduction
Open-source multimodal RL must reconcile heterogeneous reward distributions with the competing demands of fine-grained perception and multi-step reasoning. The paper proposes G2RPO and task-level shaping mechanisms, then evaluates OpenVLThinkerV2 across 18 benchmarks.
- Visual tasks expose highly varied reward topologies, from sparse binary signals in math VQA to dense continuous IoU scores in grounding.
- Standard GRPO can favor low-variance rollouts, whereas removing normalization can let high-variance tasks dominate updates.
- G2RPO nonlinearly maps each task’s empirical reward distribution to N(0, 1) using efficient one-dimensional optimal transport.
- Multimodal RL must preserve both fine-grained perception and multi-step reasoning, while prior perception methods require costly annotations or extra modules.
- Task-level response length and entropy shaping balance vision-centric and reasoning-centric tasks by guiding response behavior and constraining exploration.
- Across 18 benchmarks, OpenVLThinkerV2 reports 71.6% on MMMU and 79.5% on MathVista, surpassing GPT-4o by a significant margin.
- The paper’s contributions combine G2RPO, task-level shaping, and a robust general-purpose multimodal model with reported SOTA performance.
2 Method
The method replaces linear reward normalization with Gaussian distributional matching and adds task-level response-length and entropy shaping. Together, these mechanisms aim to stabilize multi-task optimization while balancing visual perception and multi-step reasoning.
- Multi-Task Normalization: Standard GRPO can create intra-task imbalance, while removing its normalization can cause high-variance tasks to dominate inter-task updates.EMA-GRPO uses task-wise historical variance but retains linear transformations, leaving distributional pathologies unresolved.
- Task-Level Response Length Shaping: Response-length shaping encourages longer reasoning chains for reasoning-centric tasks while reducing unnecessary generation for vision-centric tasks.The observed trajectories converge toward an optimal range, supporting a balance between reasoning and perceptual grounding.
- Gaussian GRPO: By enforcing Gaussian topology, G2RPO caps outliers, symmetrizes bimodal reward distributions, and theoretically equalizes advantage distributions across tasks.This avoids rigid momentum hyperparameters while reducing sensitivity to anomalous reward spikes.
- Task-Level Entropy Shaping: Entropy shaping constrains task-specific exploration between minimum and maximum thresholds to prevent entropy collapse and entropy explosion.The regularization applies a margin-based linear correction only outside the target interval and is added to the optimization objective.
- Discussion: The shaping boundaries are selected from simple empirical observations, while systematic hyperparameter exploration and automated search remain future work.The authors report that coarse trend-based shaping already produces substantial performance improvements.
3 Experiments
Experiments evaluate OpenVLThinkerV2 across diverse visual reasoning, document, spatial, and grounding benchmarks, alongside an ablation of its training components. The model shows broad performance gains, while combining G2RPO with response-length and entropy shaping yields the strongest overall results.
- Main Results: OpenVLThinkerV2 is evaluated across general science, mathematics, chart understanding, document understanding, spatial reasoning, and visual grounding tasks.The evaluation spans six main domains and compares general multimodal models with task-specific experts.
- Visual Reasoning Tasks: 71.6% on MMMU, 88.2% on MMBench, 73.8% on MMStar, and 87.4% on ChartQA are reported for OpenVLThinkerV2.The model surpasses GPT-4o on the cited reasoning benchmarks and exceeds Gemini 2.5 Pro on ChartQA.
- Document Understanding Tasks: 911 on OCRBench surpasses DeepEyesV2, GPT-5, and Gemini 2.5 Pro in document understanding.DeepEyesV2 is described as a specialized model using dynamic zoom-in tools for document parsing.
- Grounding Tasks: OpenVLThinkerV2 achieves 93.4%/88.2%/90.4% on RefCOCO, RefCOCO+, and RefCOCOg, respectively, outperforming Grounding DINO.These results are reported on validation splits for the grounding task.
- Overall Performance: Overall evaluations report balanced performance across vision-centric and reasoning-centric tasks, with G2RPO and both shaping mechanisms associated with stabilized training and accelerated convergence.The authors describe the resulting model as a capable multimodal generalist.
- Ablation Study: G2RPO provides the largest initial ablation improvement, length shaping gives broader gains than entropy shaping alone, and combining both achieves the highest overall gains.Entropy shaping particularly improves reasoning-centric tasks, while its gains are more modest in saturated or out-of-distribution domains.
4 Related Work
Related work positions GRPO as a widely used reinforcement-learning objective for multimodal reasoning and highlights the challenge of preserving both perception and reasoning. It also distinguishes prior optimal-transport uses from G2RPO’s distribution-matching objective.
- Group Relative Policy Optimization: GRPO has become a de-facto reinforcement-learning objective for enhancing reasoning in language and multimodal language models.The related literature traces its use across recent LLM and MLLM reasoning studies.
- Multimodal Reasoning: Multimodal reasoning studies identify preserving fine-grained perception alongside multi-step reasoning as a primary challenge.The cited literature integrates reinforcement learning into MLLMs for complex reasoning across diverse visual tasks.
- Optimal Transport in LLM: Prior LLM applications of optimal transport typically use it as a distance or ordering mechanism, whereas G2RPO uses it to match reward distributions.Examples include semantic distances between token distributions, reward-distribution dominance, and latent safety-representation mapping.
5 Conclusion
The conclusion presents OpenVLThinkerV2 as a general-purpose multimodal model trained with multi-task reinforcement learning. It combines G2RPO with task-level shaping and identifies broader applications as future work.
- Conclusion: OpenVLThinkerV2 combines multi-task reinforcement-learning post-training with G2RPO and task-level response-length and entropy shaping.The stated goal is to balance fine-grained perception and complex multi-step reasoning.
- Conclusion: G2RPO is presented as suitable for broader tasks with highly divergent reward topologies, including SWE coding and GUI tasks.Large-scale expansion beyond multimodal tasks is left for future work.
A More Results
G2RPO shows stable training behavior across accuracy, length, format, and structure rewards, with early or sustained advantages over comparison baselines. These reward improvements support the model’s broader training stability across task requirements.
- G2RPO demonstrates training stability and efficiency across accuracy, length, format, and structure rewards.
- Accuracy Reward: G2RPO begins improving accuracy reward early, while GRPO oscillates and GDPO later declines to lower accuracy reward.G2RPO converges early within the 100-step training period.
- Length Reward: G2RPO achieves significantly higher length reward than GRPO and GDPO throughout training.
- Format Reward: G2RPO maintains the best format reward at the end of training after GDPO’s initially higher reward converges lower.
- Structure Reward: G2RPO maintains the best structure reward at the end of training after GDPO’s initially higher reward converges lower.
B Policy Gradient Derivation of G2RPO
The G2RPO policy-gradient derivation replaces standard advantage normalization with a Gaussian optimal-transport mapping while retaining group sampling, importance weighting, and PPO-style clipping. The resulting advantage has approximately zero mean and unit variance, supporting stable updates and reducing outlier influence.
- G2RPO derives its policy gradient by replacing the linear standard advantage with a nonlinear optimal-transport advantage mapping.
- Group Sampling and Importance Sampling: For each query, GRPO samples a response group from the behavior policy and uses token-level importance sampling to optimize the new policy.
- Injecting the Gaussian Advantage: G2RPO ranks sampled rewards and maps their empirical probabilities to N(0, 1) through the inverse CDF, with tie-breaking for equal rewards.
- Injecting the Gaussian Advantage: The mapped advantages approximately satisfy zero mean and unit variance, systematically reducing the influence of reward outliers.
- The Clipped Surrogate Objective: The surrogate objective uses PPO-style clipping to prevent destructively large policy updates before computing the final parameter gradient.