Source-linked AI summary
All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Peter Tu, Jing Zhang
TL;DR
The paper asks why RL models can improve VLM reasoning yet remain limited relative to their base models. It identifies GRPO-induced diversity collapse and introduces MUPO, which uses grouped optimization and diversity rewards to encourage multiple reasoning strategies. MUPO-Thinker achieves reported gains over strong baselines and establishes new state-of-the-art results on the evaluated benchmarks.
Problem
The mechanisms and limitations underlying RL models' reasoning behavior remain underexplored, including why RL models can underperform their base counterparts in some VLM settings.
Method
MUPO partitions responses into multiple groups, performs localized advantage estimation, and introduces diversity rewards to promote separation among reasoning strategies.
Results
MUPO-Thinker achieves average gains of 2–7% over strong baselines on established benchmarks and sets a new state of the art.
Takeaways & Limitations
Encouraging divergent reasoning enables models to explore diverse paths and discover globally better solutions across problem types.
Abstract
from arXiv · showhide
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
1. Introduction
The paper finds that RL models reason more deeply but narrowly, whereas base models explore broader alternatives that become advantageous across multiple attempts. It attributes this divergence to GRPO's early collapse of reasoning diversity and proposes MUPO to preserve diverse strategies during RL.
- Behavioral comparison: RL models generally achieve higher accuracy on single attempts, but base models solve more problems when multiple samplings are allowed.Base models often succeed through alternative pathways absent from RL models, including verification-based geometry strategies and efficient elimination for large object counts.
- Behavioral comparison: RL models pursue deeper deliberation through dominant strategies, while base models explore broader and more diverse reasoning patterns.The paper characterizes base-model reasoning as less refined along individual paths but more divergent across attempts.
- Divergent thinking: Greater reasoning diversity significantly increases the probability of reaching correct answers, especially when common strategies fail or are error-prone.The paper connects varied perspectives across multiple attempts with improved chances of success on challenging tasks.
- Diversity collapse: GRPO rapidly collapses reasoning diversity during early training, concentrating optimization on a narrow subset of strategies and discarding alternatives.This dynamic prioritizes exploitation over exploration, leading to local optima and constrained test-time scaling.
- Proposed method: MUPO partitions responses into multiple groups, applies localized advantage estimation, and adds diversity rewards to promote separation among reasoning strategies.Its objective combines multiple group-specific GRPO objectives so each group can represent and refine a distinct strategy.
2. Related Work
Prior work advances VLM reasoning through prompting, supervised trajectories, reinforcement learning, reward design, and test-time computation. This paper positions MUPO as a way to integrate parallel, divergent thinking into RL.
- Reasoning in VLMs: VLM reasoning research has progressed from prompt-based chain-of-thought elicitation to supervised fine-tuning on human-designed trajectories and RL-based self-improvement.GRPO shifts training toward autonomous discovery and refinement of strategies.
- RL for reasoning: Recent RL research explores sampling strategies, visual-feature rewards, and entropy mechanisms to improve reasoning training.These approaches target informative trajectories, perceptually aware reasoning, or uncertainty-aware optimization.
- Test-time scaling: Test-time scaling uses additional computation through sequential deeper reasoning or parallel divergent reasoning.In VLMs, extra thinking budgets provide a cost-effective alternative for improving performance relative to additional training.
- Test-time scaling: MUPO bridges sequential and parallel test-time reasoning by integrating parallel thinking into RL, yielding gains in accuracy and scalability.The paper presents this integration as the role of its proposed method within the test-time-scaling landscape.
3. Exploring Divergent Thinking in VLMs
Experiments show that reasoning diversity benefits VLM performance, while GRPO training narrows the explored reasoning space. The paper uses these findings to motivate divergent, multi-strategy optimization.
- Experimental setup: The study compares 3B and 7B RL and base models on MathVerse, LogicVista, WeMath, and HallusionBench using temperature 1.0 sampling.The setup evaluates how multiple generated responses affect reasoning-centric benchmark performance.
- Divergent versus convergent reasoning: As the number of samples increases, base models solve substantially more problems while RL-model gains remain marginal.This pattern indicates greater capacity for effective alternative solutions in base models on challenging cases.
- Divergent versus convergent reasoning: RL reasoning embeddings are densely clustered, whereas base-model embeddings are more widely distributed across the reasoning space.The concentrated RL region supports higher pass rates in successful cases but may exclude correct trajectories in failures; broader base-model regions can retrieve alternatives.
- Diversity collapse: GRPO selects one reasoning mode early and progressively sharpens its policy distribution toward that mode.The conceptual comparison contrasts this convergent dynamic with divergent training that explores and refines multiple modes.
4. The Proposed Method
MUPO replaces GRPO’s global optimization with multiple locally refined reasoning modes and an explicit incentive for separation between them. Its annealed diversity reward encourages broad exploration early, then shifts optimization toward globally optimal solutions.
- Multi-Group Objective: MUPO partitions sampled responses into groups representing distinct reasoning modes and estimates advantages locally within each group.This separates the refinement of different strategies instead of comparing all responses globally.
- Multi-Group Objective: MUPO combines multiple GRPO objectives so diverse modes can search for optima while load balancing prevents larger groups from dominating optimization.The group contribution is controlled by a sensitivity exponent β.
- Diversity Reward: The diversity reward increases separation between groups using distances between reasoning embeddings from different groups.Responses farther from other groups receive a higher advantage.
- Reward Design: MUPO conditions the diversity reward on perfect accuracy to prevent models from pursuing diverse outputs at the expense of correctness.The total reward includes accuracy, format, and conditionally applied diversity components.
- Reward Design: The diversity-reward weight follows a smooth monotonic decay, encouraging broad exploration early and globally optimal solutions later.The schedule uses initial and final weights λmax and λmin over training steps.
5. Experiment and Results
MUPO improves benchmark performance by preserving diverse reasoning strategies while retaining effective exploitation during training. Experiments show gains across mathematical and general-purpose benchmarks, stronger test-time scaling, and broader reasoning behavior than GRPO.
- Main Results: MUPO-Thinker-7B improves average acc@1 by 2.5% on mathematical benchmarks and 2.3% on general-purpose benchmarks over previous best results.The reported averages increase from 49.1% to 51.6% and from 63.3% to 65.6%, respectively.
- Main Results: MUPO-Thinker-3B improves average benchmark performance by 2.0% and 2.4% over same-scale strong baselines, while its acc@4 gain reaches 5.9%.The acc@1 gains are reported as 41.5% →43.5% and 55.4% →57.8%; acc@4 improves from 50.1% to 56.0%.
- Main Results: MUPO-Thinker-7B improves acc@4 by 6.0% on mathematical benchmarks and 6.2% on general-purpose benchmarks over strong RL models.The reported scores increase from 52.8% to 58.8% and from 66.2% to 72.4%, respectively.
- Qualitative Analysis: MUPO produces a broad, multimodal reasoning-embedding structure, unlike GRPO’s narrow sampling region, with modes corresponding to distinct solution strategies.In a cited failure case, MUPO finds correct reasoning through alternative modes while GRPO fails.
- Training Dynamics: MUPO’s diversity reward rises during exploration, declines as its weight anneals during exploitation, and then plateaus as training stabilizes around effective solutions.Validation pairwise diversity declines more gradually than the sharp collapse observed in GRPO.
- Ablation Study: Accuracy peaks at K = 3 groups; larger K benefits mathematical benchmarks, whereas smaller K suits general-purpose problems.When K = 1, MUPO reduces to GRPO.
6. Conclusion
The paper identifies diversity collapse as a limitation of GRPO and proposes MUPO to preserve divergent reasoning across multiple solutions. MUPO combines localized group-wise optimization with diversity rewards and improves established benchmark performance.
- Conclusion: RL models show deeper but narrower reasoning, whereas base models exhibit broader and more diverse reasoning strategies.The paper presents this as a fundamental behavioral distinction between RL and base models.
- Conclusion: GRPO’s diversity collapse makes models converge to limited strategies, neglect alternatives, and reach local optima with poor scalability.The paper attributes this pattern to training dynamics.
- Conclusion: MUPO is a policy algorithm designed to incentivize divergent thinking across multiple solutions.It is presented as a simple yet effective approach evaluated on established benchmarks.