PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Xu Tang, Yao Hu, Shaohui Lin

arXiv:2503.06749v4cs.CVcs.AIcs.CLcs.LG

TL;DR

Manually formatted reasoning datasets can yield pseudo-CoT that lacks questioning and reflection, limiting complex vision reasoning. Vision-R1 combines a multimodal CoT cold start with RL and achieves math-reasoning performance comparable to state-of-the-art MLLMs.

  • Problem

    Manually formatted reasoning datasets can yield pseudo-CoT lacking questioning, reflection, and inspection, limiting their application to complex vision reasoning tasks.

    Source: §1 · p. 2

  • Method

    Vision-R1 combines a 200K human-like multimodal CoT cold start with PTST-guided GRPO using a hard-formatting result reward.

    Source: §1 · p. 3

  • Results

    Around 10% average accuracy improvement over the base model across multiple multimodal math benchmarks is reported for Vision-R1-32B and Vision-R1-72B, while Vision-R1-7B performs comparably to SoTA MLLMs with over 70B parameters.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that Vision-R1 achieves strong math-reasoning ability comparable to state-of-the-art MLLMs.

    Source: §5 · p. 10

    Direct RL alone remains challenging for eliciting complex reasoning in MLLMs, particularly under constraints on data quality, quantity, and computational resources.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .

1 INTRODUCTION

Vision-R1 combines multimodal CoT cold-start initialization with reinforcement learning to improve MLLM reasoning. Its approach uses a 200K-sample dataset and Progressive Thinking Suppression Training to address challenges in direct RL and overthinking.

1 INTRODUCTION

  • Vision-R1-7B achieves performance comparable to SoTA MLLMs over 70B parameters; Vision-R1-32B and Vision-R1-72B improve average accuracy around 10% over the base model on multiple multimodal math benchmarks.The reported 7B comparison is on math reasoning tasks.Source: Figure 1 · p. 2, §1 · p. 3
  • Manually designed step-level reasoning data and SFT can produce “Pseudo-CoT” lacking questioning, reflection, and inspection, limiting performance on complex vision reasoning tasks.The introduction motivates generating human-like, high-quality complex CoT data for MLLM training.Source: §1 · p. 2
  • Direct RL struggles to elicit complex CoT reasoning in MLLMs without large-scale, high-quality multimodal data and prolonged training.The paper frames this as a limitation of directly applying the DeepSeek-R1-Zero paradigm to MLLMs (DeepSeek-AI, 2025).Source: §1 · p. 2
  • Vision-R1’s 200K multimodal human-like complex CoT dataset is built without human annotations and serves as cold-start initialization data.An existing MLLM generates vision descriptions and step-level reasoning; modality bridging converts image information to text before DeepSeek-R1 generates CoT, followed by rule-based filtering.Source: §1 · pp. 2–3
  • Progressive Thinking Suppression Training (PTST) with GRPO and a hard-formatting result reward addresses overthinking by compressing CoT early, then progressively extending reasoning.The cold-start model’s correct reasoning tends to appear in shorter CoT sequences, complicating subsequent RL optimization.Source: §1 · p. 3

2 RELATED WORK

Prior work explores stepwise reasoning in LLMs through designed methods and reinforcement learning, while multimodal reasoning studies prompting and step-level training. Vision-R1 is distinguished by combining cold-start initialization with RL to develop more complex multimodal reasoning.

2.1 LARGE LANGUAGE MODEL REASONING

  • LLM reasoning research has explored human-designed CoT, tree/graph planning, process rewards, search, and SFT datasets (Wei et al., 2022; Yao et al., 2023; Besta et al., 2024).These approaches typically format outputs to follow specific steps.Source: §2.1 · p. 3
  • DeepSeek-R1 showed that large-scale RL with formatting and result-only rewards can elicit complex CoT in LLMs, while its application to MLLMs remained open (DeepSeek-AI, 2025).The approach produced self-emerging thought processes and advantages on complex reasoning tasks.Source: §2.1 · p. 3

2.2 MULTIMODAL LARGE LANGUAGE MODEL REASONING

3 METHOD

Vision-R1 combines multimodal-CoT cold-start initialization with reinforcement learning because direct RL alone struggled to elicit complex MLLM reasoning. Its method uses modality bridging to construct reasoning data and PTST to guide correct reasoning while gradually allowing longer chains of thought.

3.1 CAN ONLY RL INCENTIVIZE REASONING CAPABILITY IN MLLMS?

  • Direct GRPO training on 10K open-source math problems struggled to elicit complex reasoning, and longer reasoning during extended training did not significantly improve performance.Vision-R1-Zero used equal formatting and result rewards.Source: §3.1 · p. 4

3.2 OVERVIEW OF VISION-R1

  • Vision-R1 cold-starts on multimodal CoT data to learn human-like reasoning, then applies RL to guide the initialized model toward correct reasoning processes.The cold-start initialized model is named Vision-R1-CI.Source: §3.2 · p. 5

3.3.1 MODALITY BRIDGING TO OBTAIN HIGH-QUALITY MULTIMODAL COT DATA

  • Modality Bridging uses MLLMs to generate Pseudo-CoT and detailed textual image descriptions, helping DeepSeek-R1 produce multimodal reasoning chains.This approach addresses DeepSeek-R1’s difficulty processing multimodal inputs directly while exposing more image details in textual form for reasoning.Source: §3.3.1 · pp. 5–6
  • Answer-aligned CoT samples are filtered for logical consistency, paired with their images as Vision-R1-cold, and used to cold-start Vision-R1.The resulting reasoning processes are described as natural and human-like.Source: §3.3.1 · p. 6

3.3.2 OVERTHINKING OPTIMIZATION PROBLEM

  • Cold-start SFT transfers complex reasoning modes to the base MLLM, but Vision-R1-CI can overthink: its prolonged chains exceed the shorter chains where correct reasoning is concentrated.Extending the allowed thought length directly to 16K led to longer answers without performance improvement, complicating subsequent RL optimization.Source: §3.3.2 · p. 6

3.4 PROGRESSIVE THINKING SUPPRESSION TRAINING

  • PTST initially suppresses reasoning length, then relaxes constraints; Vision-R1 was selected after stage two, with stage limits of 4K, 8K, and 16K tokens and group counts of 16, 8, and 4.The final model did not undergo the third stage. GRPO uses a hard formatting-result reward: 1 only when both format and final-answer correctness are satisfied, otherwise 0; no system-prompt formatting constraint is imposed.Source: Figure 3 · p. 5, §3.4 · p. 7

4 EXPERIMENTS

Vision-R1 achieves strong multimodal math reasoning results, while the ablations attribute gains to high-quality cold-start reasoning data and PTST’s staged length constraints.

4.1 MAIN RESULTS

  • Vision-R1-7B scores 73.5% on MathVista, only 0.4% below OpenAI O1.The paper describes O1 as the most widely recognized reasoning model.Source: §4.1 · p. 8
  • On MathVista’s GEO, ALG, and GPS tasks, Vision-R1-7B scores 80.3%, 79.0%, and 83.2%, respectively, improving over Qwen-2.5-VL-7B by over 10% on average.Vision-R1-7B also ranks Top-1 on MathVerse and Top-2 on MM-Math, behind only Qwen-2.5-VL-72B on MM-Math.Source: §4.1 · p. 8

4.2 ABLATION STUDY

  • Vision-R1-cold contains more questioning, reflection, and inspection than Mulberry and LLaVA-CoT, and its SFT model outperforms both on general and math reasoning benchmarks.The paper presents these cognitive processes as a source of high-quality cold-start initialization for RL training.Source: §4.2 · pp. 8–9, Table 4 · p. 9
  • Vision-R1 balances CoT complexity and accuracy: direct RL yields insufficiently long reasoning, cold-start-only reasoning is excessively long and error-prone, and further RL on it is difficult to optimize.The comparison evaluates Vision-R1-Zero, Vision-R1-CI, Vision-R1-Long, and Vision-R1.Source: Table 3 · p. 8, §4.2 · p. 9
  • The 4K×16→8K×8 PTST schedule reaches 55.4% Avg., beating fixed 4K×16 at 54.3% by +1.1 Avg.; fixed 16K setups score 47.7% and 47.9%.Increasing Stage 2 sampling to 16 yields 55.3%, while adding a stage yields 55.1%; neither improves meaningfully under matched training time. The results support limiting early response length, then relaxing it after correct thinking is established.Source: §4.2 · p. 9, Table 5 · p. 9
  • Cold-starting on Vision-R1-cold with PTST reaches 55.4% Avg., versus 51.8% for Zero+PTST and 50.7% for Vision-R1-Zero; Zero+SFT+PTST reaches 39.8%.The authors conclude that cold-start CoT priors are essential for RL to learn correct reasoning patterns.Source: §4.2 · p. 9, Table 6 · p. 10
  • Vision-R1-7B’s visualized reasoning exhibits an “Aha moment” involving questioning and reflection.The paper describes this capability as analogous to human cognitive processes and links it to improved inference on complex reasoning tasks.Source: §4.2 · p. 9

5 CONCLUSION

The paper explores using reinforcement learning to incentivize reasoning in multimodal large language models and presents Vision-R1, which achieves strong mathematical reasoning performance comparable to state-of-the-art MLLMs.

  • Vision-R1 uses RL training to incentivize MLLM reasoning and achieves strong math reasoning performance comparable to state-of-the-art MLLMs.Source: §5 · p. 10

A EXPERIMENT SETTINGS

The experiments use a filtered multimodal CoT cold-start dataset followed by GRPO on multimodal math data, with evaluation across math and general multimodal benchmarks. The final Vision-R1 checkpoint uses two PTST stages; extending training to a third stage adds reasoning complexity without significant performance gains.

Dataset and Benchmarks

Implementation Details

  • Qwen2.5-VL-72B (Bai et al., 2025) and DeepSeek-R1 (DeepSeek-AI, 2025) process the VQA data for cold-start preparation.Qwen2.5-VL (Bai et al., 2025) is the base model, supervised-fine-tuned for two epochs using Llama-Factory (Zheng et al., 2024); GRPO uses Verl (Sheng et al., 2024).Source: A EXPERIMENT SETTINGS · p. 16
  • The final Vision-R1 model uses two PTST stages, ending after 200 RL steps with an 8K-token generation limit.Stage 1 runs 100 steps from Vision-R1-CI with a 4K-token limit and 16 samples per input; Stage 2 adds 100 steps with an 8K-token limit and 8 samples per input. The 32B and 72B models continue training on additional data under Stage 2 settings.Source: A EXPERIMENT SETTINGS · p. 16
  • Vision-R1-Zero tests RL without cold-start initialization, while Vision-R1-Long uses 16K-token generations from Vision-R1-CI.Vision-R1-Zero uses a 4K-token limit, 16 samples per input, and 300 steps; Vision-R1-Long uses four samples per input and 300 steps.Source: A EXPERIMENT SETTINGS · p. 16
  • Adding a third training stage yields no significant performance improvement but produces more complex reasoning processes.The extension continues for 100 steps using Vision-R1-Long’s parameter settings.Source: A EXPERIMENT SETTINGS · p. 16

B COT COMPARISON FOR DATA CONSTRUCTION STRATEGY

The comparison shows that naive image descriptions without “Pseudo-Co” often yield incorrect reasoning trajectories for VQA data because they omit image information relevant to reasoning.

B COT COMPARISON FOR DATA CONSTRUCTION STRATEGY

  • Without “Pseudo-Co,” naive image descriptions often lead DeepSeek-R1 to generate incorrect VQA reasoning trajectories because key reasoning-relevant image information is missing.The passage attributes the missing information to the MLLM’s inability to identify which image details matter for reasoning, producing unusual CoT data.Source: B COT COMPARISON FOR DATA CONSTRUCTION… · p. 17

C ADDITIONAL DATASET ILLUSTRATIONS

The section illustrates examples from the Vision-R1-cold dataset and response instances generated by Vision-R1, including their reasoning processes and final answers.

C ADDITIONAL DATASET ILLUSTRATIONS

D DATA SOURCES

Vision-R1-cold draws on diverse multimodal datasets spanning mathematical reasoning, general question answering, science and medicine, and figure understanding.

Mathematical Data

General QA Data

Science and Medical Data

Figure Understanding Data

Related papers

Loading 2503.06749v4…