Source-linked AI summary
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, Dacheng Tao
TL;DR
MLLM reasoning work often relies on supervised imitation of successful chains, leaving incorrect reasoning paths underused and outcome-level online RL feedback sparse. The paper introduces StepGRPO with dense, rule-based step-wise rewards, and reports that R1-VL achieves superior reasoning performance across eight benchmarks.
Problem
SFT-based MLLM reasoning methods largely neglect incorrect reasoning paths, while outcome-level rewards can provide positive feedback for only a few paths, limiting exploration and destabilizing learning.
Method
StepGRPO is an online reinforcement learning framework that combines outcome-level rewards with rule-based StepRAR and StepRVR for dense step-wise reasoning supervision without process reward models.
Results
R1-VL achieves superior performance across multiple benchmarks, including 4.6% and 3.8% improvements over Qwen2-VL-2B and Qwen2-VL-7B, respectively.
Takeaways & Limitations
StepGRPO supports MLLM reasoning self-improvement through step-wise online reinforcement learning rather than merely imitating positive reasoning paths.
Abstract
from arXiv · showhide
Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs' reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive experiments over 8 benchmarks demonstrate the superiority of our methods.
1. Introduction
Existing MLLM reasoning methods often rely on supervised imitation of successful chains, while outcome-level online reinforcement learning provides sparse feedback. StepGRPO addresses these limitations with dense, rule-based step-wise rewards and yields R1-VL models with stronger reasoning performance.
- SFT methods use high-quality chain-of-thought data but largely neglect incorrect reasoning paths, encouraging imitation of successful solutions without learning from flawed ones.
- Outcome-level rewards give positive feedback to only a few MLLM-generated paths, reducing exploration efficiency and destabilizing learning.
- StepGRPO adds dense step-wise reasoning rewards to outcome-level rewards, enabling MLLMs to self-improve without additional process reward models.
- StepRAR rewards necessary intermediate steps through soft key-step matching, while StepRVR evaluates reasoning completeness and logical consistency.
- Rule-based step-wise rewards provide fine-grained supervision across reasoning trajectories while reducing the computational overhead associated with process reward models.
2. Related Work
Related work covers MLLM vision-language understanding, supervised chain-of-thought reasoning, online reinforcement learning, and general RL foundations. StepGRPO is positioned as an online RL approach distinct from prior MLLM reasoning methods.
- MLLMs have advanced across vision-language understanding tasks involving visual content and multiple modalities such as images and videos.
- Many MLLM reasoning methods generate high-quality chain-of-thought data with powerful models and use supervised fine-tuning on the collected data.
- Some recent studies explore online reinforcement learning to improve MLLM reasoning, while StepGRPO is presented as a distinct self-improvement framework.
- Reinforcement learning trains an agent through interactions, rewards, and policy updates that maximize long-term return.
- LLM reasoning RL methods rely on reward functions or models to reinforce high-quality reasoning paths and penalize low-quality trajectories.
3. Method
StepGRPO combines policy warm-up with online optimization that rewards multimodal reasoning trajectories at each step. Its rule-based rewards assess necessary intermediate steps and the completeness and logical consistency of generated reasoning.
- Task Formulation: The task models each reasoning step as an action selected from the previous state, optimizing the policy to improve reasoning quality.A reasoning step consists of one or more sentences containing multiple word tokens.
- Step-wise Group Relative Policy Optimization: StepGRPO first warm-starts the policy with multimodal chain-of-thought data, then performs step-wise online policy optimization.The warm-up is intended to establish basic step-by-step reasoning before reinforcement learning.
- Step-wise Reasoning Accuracy Reward: StepRAR uses soft matching against pre-extracted and augmented key steps to reward paths containing intermediate reasoning steps that contribute to the final solution.The match score is the ratio of matched key steps to the total number of extracted key steps.
- Step-wise Reasoning Validity Reward: StepRVR rewards trajectories only when they satisfy both reasoning completeness and logical coherence.Completeness requires background analysis, step-by-step reasoning, and a final answer.
- Optimization with the Step-wise Rewards: The method computes group-relative advantages from rewards across generated trajectories and optimizes the policy using those advantages.The group mean reward serves as the baseline for measuring whether each trajectory performs better or worse than its peers.
4. Experiment
Experiments evaluate StepGRPO across eight multimodal benchmarks using Qwen2-VL baselines and compare its rewards, optimization behavior, and qualitative reasoning against existing methods. R1-VL improves baseline and competing-model performance, with gains linked to dense step-wise supervision and self-exploration.
- Experimental Setup: Experiments evaluate StepGRPO on eight multimodal benchmarks using Qwen2-VL-2B and Qwen2-VL-7B.The benchmarks span mathematical reasoning, chart understanding, visual hallucination, and general visual understanding.
- Main Results: R1-VL improves over Qwen2-VL-2B by 4.6% and Qwen2-VL-7B by 3.8%.The authors attribute these gains to step-wise reasoning accuracy and validity rewards that provide supervision at each reasoning step.
- Main Results: R1-VL-7B surpasses Mulberry-7B and LlamaV-o1-11B by 0.6% and 9.3% on MathVista, respectively.R1-VL-2B also exceeds LLaVA-Reasoner-8B and LLaVA-CoT-11B by 13.1% and 9.3% on MathVista.
- Main Results: R1-VL-7B reaches 63.7 accuracy on MathVista, closely matching GPT-4o at 63.8.The model outperforms most open-source MLLMs and achieves competitive results against closed-source models.
- Ablation Study: Combining StepRAR and StepRVR achieves the best ablation performance of 63.7% on Qwen2-VL-7B over MathVista.Warm-up alone reaches 61.2%, while either step-wise reward improves on the warm-up model.
- Reward and Optimization Analysis: Step-wise rewards outperform outcome-level rewards because they provide finer-grained supervision and mitigate sparse rewards.Under the same training steps on Qwen2-VL-7B over MathVista, StepGRPO consistently outperforms SFT.
5. Conclusion
StepGRPO enables MLLMs to self-improve through dense step-wise rewards without process reward models, while R1-VL achieves superior reasoning capability across eight benchmarks.
- StepGRPO is an online reinforcement learning framework using dense step-wise rewards to improve MLLM reasoning without process reward models.It introduces rule-based rewards for intermediate reasoning steps, reasoning structure, and logical consistency.
- StepRAR rewards intermediate reasoning steps through soft key-step matching.
- StepRVR rewards reasoning structure and logical consistency through reasoning completeness and logic evaluation.
- R1-VL, developed with StepGRPO, demonstrates superior reasoning capability compared with state-of-the-art MLLMs across eight benchmarks.