Source-linked AI summary
R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, Cho-Jui Hsieh
TL;DR
Multimodal efforts have struggled to reproduce DeepSeek R1’s emergent reasoning characteristics. This report applies reinforcement learning directly to a non-SFT Qwen2-VL-2B model, achieving 59.47% accuracy on CVBench while finding that instruct-model RL produces superficial reasoning and naive length rewards are ineffective.
Problem
Prior multimodal efforts struggled to reproduce DeepSeek R1’s emergent “aha moment,” increased response length, and autonomous development of sophisticated reasoning strategies.
Method
The report applies reinforcement learning directly to the non-SFT Qwen2-VL-2B base model and examines failed RL attempts with instruction-tuned models.
Results
59.47% accuracy on CVBench, approximately ~30% above the base model and ~2% above the SFT model; instruct-model RL produced superficial reasoning, while naive length rewards were ineffective.
Takeaways & Limitations
Non-SFT multimodal RL can reproduce R1-like emergent characteristics in a 2B model, while instruction-tuned RL and naive length rewards do not reliably elicit genuine reasoning.
Takeaways & Limitations
The report is a work in progress with preliminary findings and planned further investigations and updates.
Abstract
from arXiv · showhide
Recently DeepSeek R1 demonstrated how reinforcement learning with simple rule-based incentives can enable autonomous development of complex reasoning in large language models, characterized by the "aha moment", in which the model manifest self-reflection and increased response length during training. However, attempts to extend this success to multimodal reasoning often failed to reproduce these key characteristics. In this report, we present the first successful replication of these emergent characteristics for multimodal reasoning on only a non-SFT 2B model. Starting with Qwen2-VL-2B and applying reinforcement learning directly on the SAT dataset, our model achieves 59.47% accuracy on CVBench, outperforming the base model by approximately ~30% and exceeding both SFT setting by ~2%. In addition, we share our failed attempts and insights in attempting to achieve R1-like reasoning using RL with instruct models. aiming to shed light on the challenges involved. Our key observations include: (1) applying RL on instruct model often results in trivial reasoning trajectories, and (2) naive length reward are ineffective in eliciting reasoning capabilities. The project code is available at https://github.com/turningpoint-ai/VisualThinker-R1-Zero
1 Introduction
Multimodal RL efforts often failed to reproduce DeepSeek R1’s self-reflective “aha moment” and increasing response length. This report applies RL directly to a non-SFT 2B model and reports both emergent characteristics alongside improved CVBench accuracy.
- Background: DeepSeek R1’s “aha moment” involved self-reflection and spontaneously increasing response length during training.These behaviors appeared as the model developed increasingly sophisticated problem-solving strategies.
- Problem: Multimodal efforts struggled to reproduce DeepSeek R1’s emergent “aha moment,” increased response length, and autonomous reasoning strategies.The report identifies this reproduction gap as a central challenge for multimodal reasoning.
- Contribution: 59.47% accuracy on CVBench was achieved by the non-SFT 2B model, approximately ~30% above the base model and ~2% above the SFT model.The approach directly applies reinforcement learning to Qwen2-VL-2B without supervised fine-tuning.
- Contribution: RL on a non-SFT 2B model reproduced the “aha moment” and increased reasoning length in multimodal reasoning.The report presents this as a successful replication of DeepSeek R1’s key characteristics.
- Failed Attempts: RL on instruction-tuned models produced superficial reasoning, while naive length rewards failed to induce deeper reasoning capabilities.The report includes these failed attempts as insights into the difficulty of reproducing R1-like reasoning.
- Contributions: The report’s contributions include improved vision-centric spatial reasoning and an open-source project for future multimodal reasoning studies.The project code is identified as a resource for further investigation.
2 Related Works
Prior work established RL-based emergent reasoning in language models, but multimodal replications have been evaluated against whether they reproduce the same characteristic phenomena. The related work defines the “aha moment” and increasing response length as central comparison criteria.
- Multimodal Reasoning: Post-training multimodal reasoning commonly relies on sophisticated prompting or large amounts of reasoning training data.This motivates interest in methods that use neither extensive supervised data nor complex prompting techniques.
- DeepSeek R1: DeepSeek R1 showed that rule-based RL can elicit reflection and self-correction without supervised reasoning data.The reported “aha moment” was the model’s autonomous development of advanced problem-solving strategies.
- Key Phenomena: The “aha moment” denotes autonomous development of advanced problem-solving strategies during training.Increasing response length indicates that the model naturally learns to allocate more thinking time for reasoning tasks.
- Comparison: Table 1 compares DeepSeek R1 with multimodal replications using the “aha moment” and increasing response length as highlighted phenomena.The passage questions whether replications lacking these characteristics qualify as valid R1 implementations.
3 VisualThinker R1 Zero
VisualThinker R1 Zero applies GRPO directly to the Qwen2-VL-2B base model, training on SAT spatial-reasoning questions with rule-based format and correctness rewards. The method is designed to encourage prolonged reasoning and the multimodal “aha moment” without supervised fine-tuning.
- Model and Algorithm: VisualThinker R1 Zero applies GRPO directly to a non-SFT Qwen2-VL-2B model for multimodal reasoning.The authors contrast this setting with instruction-tuned visual models that failed to reproduce DeepSeek R1’s key characteristics.
- Training Recipe: The model is trained directly on SAT questions to explore spatial reasoning for each image-based question.Each generated response is optimized using the RL objective.
- Training Recipe: The chat template prompts the assistant to reason step by step inside <think> tags before providing an answer.This template structures the intended reasoning and answer format.
- RL Algorithm: The method produced prolonged reasoning trajectories and an “aha moment” when GRPO was applied to the non-SFT model.The authors identify this setting as central to their multimodal R1-like reasoning result.
- RL Algorithm: GRPO uses sampled-response rewards as the advantage baseline, avoiding an additional value-function approximation model.The policy is then optimized with a KL objective.
- Reward Modeling: The rule-based reward assigns +1 for a correct final answer and +1 for the required thinking and answer tags.Responses receiving neither reward receive 0, and the reward function is reported to promote rapid convergence to the desired format.
4 Experiments
Experiments train Qwen2-VL-2B directly with GRPO on SAT spatial reasoning data and evaluate generalization on CVBench and other vision-centric benchmarks. The method combines improved benchmark performance with increasing response length and observed self-revision during training.
- Training Configuration: Direct GRPO training used SAT’s static subset, containing questions about relative spatial relationships, relative depth, and object counting.SAT comprises 218k question-answer pairs synthesized with a photo-realistic physics engine.
- Training Configuration: The experiments used a maximum response length of 700, sampled 8 responses per GRPO step, and applied a KL coefficient of 0.04.Training ran for 1500 steps with a learning rate of 1 × 10^-6 and temperature 1.0.
- Benchmark Setup and Results: Around 27% advantage over the SFT-trained model was observed on the BLINK and VSR benchmarks.The reported evaluation used spatial subsets of BLINK covering multiview reasoning, relative depth, and spatial relations.
- Benchmark Setup and Results: ~2% improvement over Qwen2-VL-2B SFT and ~5% over Qwen2-VL-2B-Instruct were reported for Qwen2-VL-2B + R1.The comparison is presented as an RL-versus-SFT result in Figure 2.
- Emergent Reasoning Behavior: Increasing response length and an “aha moment” emerged during training, with the model revisiting previous judgments and exploring alternative options.The reported behavior parallels self-reflective reasoning patterns associated with DeepSeek R1.
5 Challenges of Applying RL to Supervised Fine-Tuned Models
Applying RL to supervised fine-tuned multimodal models improves benchmark performance but produces trivial reasoning trajectories, and naive length rewards fail to induce deeper reasoning.
- 5 Challenges of Applying RL to Supervised Fine-Tuned Models: The investigation was motivated by the stronger instruction-following capability of supervised fine-tuned models and examined why they failed to reproduce R1-like reasoning.The report explicitly frames these experiments as failed cases intended to provide insights for future research.
- 5.1 Emergence of Trivial Reasoning Patterns: RL on supervised fine-tuned models improves CVBench performance but can degenerate into generic reasoning followed by question-specific answers.The reported trajectory places trivial reasoning inside <think></think> tags and the answer between <answer></answer> tags.
- 5.2 Preliminary Investigation of Trivial Reasoning Trajectories: The observed performance gains may not indicate genuine reasoning capability, highlighting the complexity of RL training dynamics in multimodal models.The report presents this as a hypothesis and calls for further analysis.
- 5.2 Preliminary Investigation of Trivial Reasoning Trajectories: Freezing either the vision encoder or language model yields greater improvement than vanilla fine-tuning, yet responses remain short and trivial.The comparison covers full fine-tuning and settings that freeze either language or vision components.
- 5.3 Failed Attempts with Length Reward: Length-rewarded RL increases response length without improving reasoning quality and can trigger reward hacking through repetitive, meaningless content.The auxiliary reward adds +0.001 for each additional generated token.
6 Conclusion
VisualThinker R1 Zero reports a multimodal replication of DeepSeek R1’s emergent reasoning characteristics using RL on a non-finetuned Qwen2-VL-2B model. The report also documents trivial reasoning under instruct-model RL and remains an ongoing preliminary effort.
- 6 Conclusion: VisualThinker R1 Zero applies RL directly to a non-finetuned Qwen2-VL-2B model and reports both the “aha moment” and increased response length.These are presented as indicators of autonomous reasoning development.
- 6 Conclusion: 59.47% accuracy on CVBench is reported without supervised fine-tuning, outperforming both base and instruction-tuned models.
- 6 Conclusion: RL on supervised fine-tuned models is reported to produce trivial reasoning trajectories rather than genuine problem-solving strategies.
- 6 Conclusion: The report is a work in progress, with further updates planned to expand and refine its preliminary findings and technical roadmap.