Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

Wenxuan Huang; Bohan Jia; Zijie Zhai; Shaosheng Cao; Zheyu Ye; Fei Zhao; Zhe Xu; Xu Tang; Yao Hu; Shaohui Lin

Source-linked AI summary

Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Xu Tang, Yao Hu, Shaohui Lin

arXiv:2503.06749v4Mar 9, 2025cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLMs lack substantial high-quality multimodal reasoning data, making direct RL ineffective at eliciting complex processes such as questioning and reflection. Vision-R1 combines an annotation-free 200K multimodal CoT cold start with PTST and GRPO, achieving strong multimodal math reasoning performance, including 73.5% on MathVista for its 7B model.

Problem
Direct RL struggles to activate complex MLLM reasoning because substantial high-quality multimodal reasoning data is absent.
Method
Vision-R1 uses modality bridging and filtering to build a 200K multimodal CoT cold-start dataset, then applies PTST with GRPO and hard-formatting result rewards.
Results
Vision-R1 shows strong math reasoning across multimodal benchmarks, with Vision-R1-7B scoring 73.5% on MathVista, 0.4% below OpenAI O1.
Takeaways & Limitations
Combining cold-start initialization with RL provides a promising route for incentivizing reasoning capability in MLLMs.
Takeaways & Limitations
Direct RL remains challenging under constraints of data quality and quantity and computation resources.

Abstract

from arXiv · show

DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .

1 INTRODUCTION

Vision-R1 addresses the difficulty of eliciting complex reasoning in MLLMs by combining a large multimodal CoT cold start with RL and PTST. The approach constructs annotation-free reasoning data and progressively refines reasoning under optimization challenges caused by overthinking.

Direct RL struggles to elicit complex multimodal reasoning because high-quality multimodal reasoning data is limited.
A 200K multimodal CoT dataset is constructed without human annotations for cold-start initialization.
Vision-R1 combines cold-start initialization with RL to incentivize reasoning capability in MLLMs.
PTST with GRPO and hard-formatting rewards addresses overthinking by progressively developing more complex reasoning processes.
Vision-R1-7B performs comparably to much larger state-of-the-art MLLMs on math reasoning tasks, while larger variants improve further over the base model.Vision-R1-32B and Vision-R1-72B achieve around 10% average accuracy improvement over the base model across multiple multimodal math benchmarks.

2 RELATED WORK

Prior multimodal reasoning methods use prompting or manually constructed step-level datasets, but their CoT often lacks human-like cognitive processes. Vision-R1 builds on RL reasoning advances while addressing the open challenge of applying them to MLLMs.

DeepSeek-R1 showed that large-scale RL with formatting and result-only rewards can induce human-like complex CoT reasoning in LLMs.
Applying this RL approach to MLLMs remains an open area of inquiry.
MLLMs map non-text inputs into text before language-model processing, supporting vision understanding across tasks.
CoT prompting and manually constructed SFT datasets have been used to enhance MLLM reasoning.
These methods often produce CoT lacking questioning, reflection, and inspecting, limiting effectiveness on complex reasoning tasks.

3 METHOD

Vision-R1 combines annotation-free multimodal CoT cold-start initialization with RL to improve MLLM reasoning. Its method uses modality bridging, filtered 200K CoT data, and progressive training to address direct-RL and overthinking challenges.

Can only RL incentivize reasoning capability in MLLMs?: Direct RL training struggled to stimulate complex MLLM reasoning and produce lengthy, complex CoT under data-quality, data-quantity, and computation constraints.The RL-only Vision-R1-Zero model gradually generated longer reasoning without significant performance improvement.
Progressive Thinking Suppression Training: GRPO with PTST progressively loosens reasoning-length restrictions across stages, using 4K, 8K, and 16K limits with group numbers 16, 8, and 4.The final Vision-R1 version did not undergo the third stage; its reward requires both correct final answers and correct formatting.
Modality Bridging to obtain high-quality multimodal CoT data: Modality Bridging uses existing MLLMs and DeepSeek-R1 to convert multimodal information into textual information that captures complex cognitive reasoning processes.An MLLM first generates Pseudo-CoT with image descriptions and reasoning, which is then used to produce detailed image descriptions before DeepSeek-R1 generates CoT.
Experiments: Table 1 compares closed-source and open-source MLLMs across diverse multimodal math benchmarks, reporting both overall averages and MathVista subtask scores.The table covers geometry reasoning, algebraic reasoning, geometry problem solving, and math word problems.
Modality Bridging to obtain high-quality multimodal CoT data: The filtered Vision-R1-cold dataset contains 200K multimodal human-like complex CoT samples for cold-start initialization.Samples are retained when their final answers match ground truth, with rule-based filtering removing logically inconsistent examples and improving semantic coherence.
Overthinking Optimization Problem: Cold-start initialization teaches complex reasoning but creates an overthinking optimization problem, with prolonged incorrect reasoning complicating subsequent RL training.Directly allowing 16K-token thoughts caused longer answers without performance improvement.
Experiments: Table 2 evaluates generalization across comprehensive multimodal benchmarks using Llama-3.2-11B-V-Instruct as the base MLLM.Its caption states that the proposed data improves performance across all listed benchmarks.

4 EXPERIMENTS

Vision-R1 combines cold-start multimodal CoT initialization with RL and PTST, achieving competitive multimodal math reasoning while addressing overthinking and data-quality challenges.

73.5% MathVista accuracy places Vision-R1-7B only 0.4% below OpenAI O1.
80.3%, 79.0%, and 83.2% on GEO, ALG, and GPS exceed Qwen-2.5-VL-7B by over 10% average accuracy.
Cold-start dataset: Vision-R1-cold contains 200K multimodal human-like complex CoT samples generated without human annotations for cold-start initialization.
PTST ablation: 55.4% average accuracy is achieved by the two-stage PTST schedule 4K×16→8K×8, outperforming fixed short-length training by +1.1 Avg.
PTST ablation: Fixed 16K reasoning severely underperforms at 47.7%/47.9%, while additional sampling or stages provide no meaningful improvement.
Cold-start ablation: Cold start followed by PTST reaches 55.4% average accuracy, whereas PTST alone offers limited benefit and SFT without CoT is detrimental.
Visualization: Vision-R1-7B generates complex reasoning with questioning and reflection, described as an emergent “Aha moment” associated with improved inference performance.

5 CONCLUSION

The paper explores RL for MLLM reasoning and proposes Vision-R1, combining cold-start initialization with RL to achieve strong mathematical reasoning comparable to state-of-the-art MLLMs.

Vision-R1 combines cold-start initialization with RL training to incentivize reasoning capability in MLLMs.
The model achieves strong math reasoning ability with performance comparable to state-of-the-art MLLMs.

A EXPERIMENT SETTINGS

Vision-R1 is evaluated across multimodal math and general benchmarks using cold-start initialization, RL variants, and the two-stage PTST strategy. Stage 2 is selected as the final model checkpoint because it balances reasoning length and overall performance, while further training adds complexity without significant gains.

Datasets and Benchmarks: Vision-R1-cold is constructed from multimodal VQA datasets, while RL training mixes five multimodal math datasets.The cold-start dataset contains 200K examples; the RL data combines We-Math, MathVision, Polymath, SceMQA, and Geometry3K.
Datasets and Benchmarks: The evaluation uses three multimodal math benchmarks and four general multimodal benchmarks to assess reasoning and general ability.The math benchmarks are MM-Math, MathVista, and MathVerse; the general benchmarks include MMStar, ChartQA, MME, and another benchmark listed in the passage.
Implementation Details: Vision-R1-cold preparation uses Qwen2.5-VL-72B and DeepSeek-R1 to process the VQA datasets.The two models are deployed during Vision-R1-cold dataset preparation.
Training Variants: Vision-R1 uses Qwen2.5-VL cold-started by SFT, followed by GRPO under the two-stage PTST strategy.The cold-start model is trained for two epochs before subsequent RL training.
Training Variants: Vision-R1-Zero applies RL directly to the base MLLM without cold-start initialization, whereas Vision-R1-CI uses the Vision-R1-cold dataset first.Vision-R1-Zero uses a 4K-token limit and 16 samples, while Vision-R1-CI denotes the cold-started model.
Training Variants: Vision-R1-Long uses 16K-token generation and 300 training steps, while Vision-R1 applies PTST through two RL stages.Stage 1 trains Vision-R1-CI for 100 steps with a 4K-token limit; Stage 2 uses an 8K-token maximum and eight samples per input.
Training Variants: The Stage 2 checkpoint is the final Vision-R1 model because it achieves an optimal balance between reasoning length and overall performance.Vision-R1-32B and Vision-R1-72B continue RL with additional data under the same Stage 2 settings.
Training Variants: Further Stage 3 training does not yield significant performance improvements but generates more complex reasoning processes.The extension follows Vision-R1-Long settings for an additional 100 training steps.

B COT COMPARISON FOR DATA CONSTRUCTION STRATEGY

The comparison shows that naive image descriptions provide insufficient information for DeepSeek-R1, whereas Pseudo-CoT-enhanced descriptions support more accurate reasoning.

CoT Comparison: Naive image descriptions omit key visual information, causing DeepSeek-R1 to generate unusual or incorrect reasoning trajectories.The MLLM does not identify which image information matters for reasoning when producing the naive description.
CoT Comparison: Adding Pseudo-CoT incorporates converted image information into descriptions and enables more accurate reasoning trajectories.The comparison attributes the improvement to richer descriptions that expose information relevant to reasoning.

C ADDITIONAL DATASET ILLUSTRATIONS

The paper provides additional examples of Vision-R1-cold data and Vision-R1-7B outputs, illustrating multimodal reasoning processes with corresponding answers.

Dataset Examples: Figure 6 provides additional examples of the Vision-R1-cold dataset.The examples contain abundant information obtained through the Modality Bridging method.

D DATA SOURCES

Vision-R1-cold combines multimodal data from science, figure understanding, mathematics, and general question answering, with examples spanning several task types.

Mathematical Data: The mathematical sources include GLLaVA, GEOS, UniGeo, GeoQA Plus, Geo3K, MathVision, GeoMverse, MathV360K, IconQA, TabMWP, CLEVR, CLEVR-Math, and Super-CLEVR.The collection spans geometry, visual mathematics, tabular mathematics, and synthetic reasoning tasks.
Example Data: The examples include cone surface-area calculation and angle reasoning, each pairing a visual question with step-by-step reasoning and a selected answer.The cone example derives 3π, while the angle example derives 125° from a linear-pair relation.
Example Data: The dataset illustrations also include a triangle-perimeter question and examples showing abundant information produced through Modality Bridging.The cited passages present these as additional examples of the constructed data.
Example Outputs: Additional Vision-R1-7B examples show comprehensive reasoning processes and corresponding final answers on MathVerse.These outputs are presented as further model-response illustrations.
General QA Data: General QA sources include ShareGPT4V, PISC, VQA-AS, A-OKVQA, TextVQA, VizWiz, and VQA2.0.These datasets broaden the multimodal question-answering coverage beyond specialized science, figure, and mathematics tasks.
Science and Medical Data: Vision-R1-cold contains science and medical data from datasets including GeoQA+, CLEVR-Math, TQA, AI2D, ScienceQA, VQA-RAD, and PMC-VQA.These sources cover science, medical, visual-question-answering, and mathematical content.
Figure Understanding Data: Its figure-understanding data comes from chart, document, plot, and hierarchical reasoning datasets such as DVQA, DocVQA, FigureQA, PlotQA, ChartQA, InfoVQA, MultiHiertt, and LRV-Chart.The listed sources provide varied figure and document understanding examples.