Source-linked AI summary
Improved Visual-Spatial Reasoning via R1-Zero-Like Training
Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, Zhijie Deng
TL;DR
Video-based visual-spatial reasoning is a crucial but underperforming capability of MLLMs. The paper studies R1-Zero-like GRPO training of Qwen2-VL with VSI-100k, finding that CoT prompting is ineffective and reporting gains over base models and competitive performance against larger models.
Problem
Video-based visual-spatial reasoning is important for MLLMs but existing models often fall short, and small- to medium-sized Qwen2-VL models do not improve this capacity with CoT prompts.
Source: §1 · p. 1Method
Results
12.1%: vsGRPO-2B outperforms Qwen2-VL-2B by this amount, while vsGRPO-7B achieves performance comparable to LLaVA-NeXT-Video-72B.
Source: §1 · p. 2, §3.3.1 · p. 6Takeaways & Limitations
Experiments show that GRPO improves Qwen2-VL visual-spatial reasoning and outperforms SFT and DPO baselines.
Source: §1 · p. 2, §4 · p. 7Length rewards can be exploited with extra think/answer tags that do not produce meaningful thinking, motivating more reasonable reward functions.
Source: §3.3.3 · p. 7
Abstract
from arXiv · showhide
Increasing attention has been placed on improving the reasoning capacities of multi-modal large language models (MLLMs). As the cornerstone for AI agents that function in the physical realm, video-based visual-spatial intelligence (VSI) emerges as one of the most pivotal reasoning capabilities of MLLMs. This work conducts a first, in-depth study on improving the visual-spatial reasoning of MLLMs via R1-Zero-like training. Technically, we first identify that the visual-spatial reasoning capacities of small- to medium-sized Qwen2-VL models cannot be activated via Chain of Thought (CoT) prompts. We then incorporate GRPO training for improved visual-spatial reasoning, using the carefully curated VSI-100k dataset, following DeepSeek-R1-Zero. During the investigation, we identify the necessity to keep the KL penalty (even with a small value) in GRPO. With just 120 GPU hours, our vsGRPO-2B model, fine-tuned from Qwen2-VL-2B, can outperform the base model by 12.1% and surpass GPT-4o. Moreover, our vsGRPO-7B model, fine-tuned from Qwen2-VL-7B, achieves performance comparable to that of the best open-source model LLaVA-NeXT-Video-72B. Additionally, we compare vsGRPO to supervised fine-tuning and direct preference optimization baselines and observe strong performance superiority. The code and dataset will be available soon.
1 Introduction
Video-based visual-spatial reasoning is an important but weak capacity of MLLMs. The paper studies whether prompting or R1-Zero-like GRPO training can improve it in Qwen2-VL models.
1 Introduction
- Video-based visual-spatial reasoning is crucial for MLLMs used by physical-world agents, yet existing models often fall short on this capacity.The paper cites VSI-bench as evidence of the shortfall.Source: §1 · p. 1
- Small- to medium-sized Qwen2-VL models do not improve visual-spatial reasoning through CoT prompting; vanilla prompts perform best on VSI-bench.This suggests these models cannot trade additional inference FLOPs for better visual-spatial reasoning.Source: §1 · p. 1
- The study applies GRPO to Qwen2-VL and constructs VSI-100k, a video question-answering dataset of more than 100k samples from ScanNet scans and object-level 3D annotations.The dataset supports spatial question-answer pairs and follows the VSI-bench protocol.Source: §1 · p. 2
- GRPO turns Qwen2-VL-2B into vsGRPO-2B, which outperforms the base model by 12.1% after 120 GPU hours, while vsGRPO-7B performs similarly to the leading open-source 72B model.The authors also report GRPO's superiority over SFT and DPO for improving spatial-visual reasoning.Source: §1 · p. 2
2 Can Visual-spatial Reasoning Capacities Be Activated by Prompting?
The section tests whether reasoning-oriented prompts activate visual-spatial reasoning in Qwen2-VL. On VSI-bench, think-mode and observe-mode underperform vanilla prompting, and an example suggests a perception error may persist despite a reasoning trace.
2 Can Visual-spatial Reasoning Capacities Be Activated by Prompting?
- The comparison evaluates think-mode and observe-mode CoT prompts against a non-CoT vanilla-mode prompt.Think-mode asks the model to think before answering; observe-mode asks it to observe the video first.Source: §2 · pp. 2–3
- Think-mode and observe-mode underperform vanilla-mode on VSI-bench, despite producing longer responses.The authors conclude small- to medium-sized Qwen2-VL cannot trade inference FLOPs for improved visual-spatial reasoning.Source: §2 · p. 3
- The example's think-mode response follows the instruction to reason but gives the same wrong answer as vanilla prompting, possibly because it fails to perceive the sofa.The authors infer the possible perception failure from the exposed chain of thought.Source: §2 · p. 3, Figure 1 · p. 3
3 R1-Zero-like Training for Visual-spatial Reasoning
The paper turns to fine-tuning Qwen2-VL for improved visual-spatial reasoning, focusing on GRPO because of its success in building DeepSeek-R1-Zero.
3 R1-Zero-like Training for Visual-spatial Reasoning
- The section motivates fine-tuning Qwen2-VL after prompting fails to improve visual-spatial reasoning.It focuses on GRPO (Shao et al., 2024), citing its success in building DeepSeek-R1-Zero.Source: §3 · p. 3
- The training approach adapts GRPO (Shao et al., 2024) to Qwen2-VL for visual-spatial reasoning.The authors identify DeepSeek-R1-Zero as the motivating precedent for focusing on GRPO.Source: §3 · p. 3
- GRPO is selected as the fine-tuning focus because of its success in building DeepSeek-R1-Zero.The paper's stated objective is improving Qwen2-VL's visual-spatial reasoning capacity.Source: §3 · p. 3
3.1 Training Data Construction
VSI-100k is a video question-answering dataset for visual-spatial reasoning, built from ScanNet scans with object-level 3D annotations. It covers six spatial topics while holding out two VSI-bench topics for generalization testing.
3.1 Training Data Construction
- VSI-100k contains more than 100k video-based question-answer samples, constructed from high-fidelity ScanNet scans with detailed object-level 3D annotations.The annotations support creating question-answer pairs about spatial information.Source: §3.1 · p. 3
- The dataset covers object count, relative direction and distance, object size, room size, and absolute distance, while holding out route planning and appearance order to test task generalization.Those held-out topics are difficult to construct from static 3D information and may require expensive manual annotation.Source: §3.1 · p. 3
- Figure 2 provides examples of the constructed dataset.Source: Figure 2 · p. 4
- The six constructed tasks use annotation labels or geometric measurements, including point-cloud centers, 3D bounding boxes, and alpha shapes.The dataset contains 6.4k object-count, 75k absolute-distance, 13k relative-distance, 8k relative-direction, 13k object-size, and 1.5k room-size samples.Source: §3.1 · p. 4
3.2 GRPO
This section describes GRPO policy optimization over grouped outputs and specifies the rewards, prompting modes, and training setup used for VSI-bench experiments.
3.2 GRPO
- GRPO samples grouped outputs per question and optimizes the policy without a critic model, reducing training costs.The objective includes a clipping hyperparameter and a coefficient for the KL penalty.Source: §3.2 · p. 4
- The reward design combines format and accuracy rewards, with the format reward omitted in vanilla-mode.Format reward scores whether responses follow the specified format; accuracy uses character matching for non-NA tasks and a normalized absolute difference for NA tasks.Source: §3.2 · pp. 4–5
- Table 2 reports VSI-bench results for GRPO prompt variants and evaluation modes alongside open- and closed-source model baselines.The caption identifies LLaVA-NeXT-Video and InternVL2 as open-source examples, and GPT-4o and Gemini-1.5 Pro as closed-source examples.Source: Table 2 · p. 5
- GRPO training considers think-mode, observe-mode, and vanilla-mode prompts, with specified response formats for each mode.Think-mode requests step-by-step reasoning; observe-mode requests video observations and analysis; vanilla-mode requests a short answer.Source: §3.2 · p. 5
- Training uses Qwen2-VL-2B/7B base models, 14 rollouts per question, and a KL coefficient β of 0.0001 unless otherwise specified.LoRA learning rates are 10^-5 for Qwen2-VL-2B and 5 × 10^-6 for Qwen2-VL-7B; the default sampling temperature is 1.Source: §3.2 · p. 6
3.3 Results and Analyses
The GRPO-trained models show strong results against the reported baselines, while training analyses identify the importance of KL regularization and unresolved reward-design challenges.
3.3.1 Main Results
- vsGRPO-2B with vanilla-mode prompting outperforms GPT-4o, while vsGRPO-7B performs comparably to LLaVA-NeXT-Video-72B and improves Route Plan performance.The authors suggest Route Plan may benefit from decomposition into relative-direction subtasks, indicating inter-task generalization.Source: §3.3.1 · p. 6
3.3.2 Importance of KL Penalty
- Removing the KL penalty can lead to training collapse, whereas a positive β—even 0.0001—can address the issue.The authors suggest this may reflect the specific nature of VSI reasoning problems.Source: §3.3.2 · p. 6
3.3.3 Reward Hacking
- Length reward mitigation remains vulnerable to reward hacking: extra empty think and answer tags can exploit it without meaningful thinking.The authors call for more reasonable reward functions.Source: §3.3.3 · p. 7
3.3.4 Dynamics of Various Rewards
- Format reward quickly converges to 1, but accuracy reward increases slowly and may have an upper bound that remains an open problem.These dynamics are reported during GRPO training.Source: §3.3.4 · p. 7
3.3.5 Comparison to Other Training Approaches
- SFT and DPO both improve over the base model on VSI-bench but lag behind GRPO-V; DPO’s improvement is minor, possibly due to suboptimal preference pairs.SFT uses VSI-100k for tuning; DPO pairs correct answers with modified wrong answers as less-preferred responses.Source: §3.3.5 · p. 7
4 Conclusion
The work studies video-based visual-spatial intelligence in MLLMs, finding that CoT prompts do not activate Qwen2-VL-2B/7B reasoning on VSI-bench. It constructs VSI-100k and adapts GRPO, with experiments reporting gains over same-size models and superiority to SFT and DPO.
4 Conclusion
- Experiments find vsGRPO-2B and vsGRPO-7B outperform same-size models, with GRPO outperforming SFT and DPO.The study uses Qwen2-VL as its base model and focuses on video-based visual-spatial intelligence.Source: §4 · p. 7