Source-linked AI summary
Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models
Xiaoyu Zhan, Wenxuan Huang, Hao Sun, Xinyu Fu, Changfeng Ma, Shaosheng Cao, Bohan Jia, Shaohui Lin, Zhenfei Yin, Lei Bai, Wanli Ouyang, Yuanqi Li, Jie Guo, Yanwen Guo
TL;DR
MLLMs have advanced in 2D visual understanding, but whether they capture the fine-grained cross-view information needed for robust 3D reasoning remains unclear. The paper introduces Viewpoint Learning, Viewpoint-100K, and two-stage fine-tuning with hybrid cold-start initialization. It reports improved spatial reasoning on in-domain and out-of-domain tasks.
Problem
It remains unclear whether MLLMs capture the fine-grained cross-view 3D information needed for robust real-world spatial reasoning.
Source: §1 · p. 2Method
The method trains on 100K viewpoint image pairs with SFT and hybrid cold-start initialization, then uses GRPO-based reinforcement learning to enhance generalization across spatial tasks.
Source: §1 · p. 2, §4.4 · p. 7Results
The approach significantly improves MLLM performance on both in-domain and out-of-domain reasoning tasks.
Source: §6 · p. 10Takeaways & Limitations
The authors identify foundational viewpoint learning as a task for advancing MLLMs’ three-dimensional understanding and informing applications such as robotics and autonomous systems.
Source: §1 · p. 3Some pseudo CoTs contradict 3D knowledge or do not match the input image, so their proportion is kept small.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved 2D visual understanding, prompting interest in their application to complex 3D reasoning tasks. However, it remains unclear whether these models can effectively capture the detailed spatial information required for robust real-world performance, especially cross-view consistency, a key requirement for accurate 3D reasoning. Considering this issue, we introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs. Our approach employs a two-stage fine-tuning strategy: first, foundational knowledge is injected to the baseline MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in significant improvements across multiple tasks; second, generalization is enhanced through Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm on a broader set of questions. Additionally, we introduce a hybrid cold-start initialization method designed to simultaneously learn viewpoint representations and maintain coherent reasoning thinking. Experimental results show that our approach significantly activates the spatial reasoning ability of MLLM, improving performance on both in-domain and out-of-domain reasoning tasks. Our findings highlight the value of developing foundational spatial skills in MLLMs, supporting future progress in robotics, autonomous systems, and 3D scene understanding.
1 Introduction
The introduction asks whether MLLMs can capture fine-grained 3D spatial information across views, beyond their progress in 2D visual understanding. It presents Viewpoint Learning, a dataset, and two-stage fine-tuning, reporting improved in- and out-of-domain spatial reasoning.
1 Introduction
- Cross-view consistency remains a challenge: it is unclear whether MLLMs capture the fine-grained 3D information needed for robust real-world visual-spatial performance.Prior work reports difficulty with cross-view consistency, while additional spatial information and visual prompts can improve spatial reasoning.Source: §1 · p. 2
- Viewpoint Learning evaluates and activates spatial reasoning through 100K real-world, object-centric image pairs with distinct viewpoints and ego- or object-centric question-answer pairs.The task is presented as straightforward to acquire data for, calculate ground truth, and evaluate.Source: §1 · p. 2
- The two-stage strategy uses SFT on Viewpoint-100K for foundational knowledge, then GRPO-based reinforcement learning to improve generalization across broader spatial tasks.Hybrid cold-start initialization with human-assisted pseudo chain-of-thoughts is used to maintain coherent reasoning and instruction-following.Source: §1 · pp. 2–3
- Viewpoint Learning improves MLLMs’ spatial reasoning across multiple benchmarks, including out-of-domain inference tasks.The authors frame it as a foundational task for visual-spatial reasoning.Source: §1 · p. 3
- The paper’s contributions are viewpoint learning, Viewpoint-100K, and two-stage fine-tuning with hybrid cold-start initialization.The dataset contains auto-generated ego-centric or object-centric QAs based on real-world object-centric image pairs.Source: §1 · p. 3
2 Related Work
Prior MLLM work adds cross-view information, 3D inputs, or structured visual features, while many approaches emphasize high-level reasoning over foundational viewpoint skills. Existing benchmarks cover varied spatial capabilities; this paper investigates activating those capabilities in MLLMs.
2.1 Multimodal Large Language Models for Spatial Reasoning.
- Prior work identifies cross-view consistency as a difficulty for MLLMs [21], while 3D inputs, features, and structured representations have improved spatial reasoning [14] [9] [27] [5] [41] [15] [33].SpatialMM reports particular gains on tasks requiring fewer reasoning steps [33].Source: §2.1 · p. 3
- Many existing studies emphasize high-level reasoning but overlook foundational spatial tasks such as viewpoint estimation and spatial transformation.The paper positions targeted training strategies as a response to specific visual-spatial deficiencies.Source: §2.1 · p. 3
2.2 Benchmarks for Spatial Reasoning
- Spatial benchmarks assess reference frames, multi-view data, scene graphs, and other capabilities; SAT uses synthetic multi-task data, while VSI-Bench highlights relational and perspective-transformation errors [10] [48] [23] [30] [45] [18] [34] [12] [20] [44] [35].Other cited benchmarks include BLINK, 3DSRBench, VSR, and COMFORT [12] [25] [23] [48].Source: §2.2 · pp. 3–4
- The Figure 2 contrast shows that smooth 2D continuity does not ensure preserved 3D spatial and geometric relationships across frames.The paper aims to investigate how spatial reasoning can be activated and leveraged for complex visual and spatial tasks.Source: Figure 2 · p. 4, §2.2 · p. 4
3 Overview
The overview distinguishes smooth 2D frame-to-frame continuity from 3D consistency, which preserves spatial integrity and geometry. It presents Actial as a two-stage pipeline for teaching MLLMs to treat images and videos as projections of 3D space.
3 Overview
- 2D continuity alone does not guarantee 3D consistency, which preserves depth, scale, and object positions across frames.The distinction matters for 3D reconstruction, SLAM, and pose estimation.Source: §3 · p. 4
- MLLMs trained mainly on 2D data and limited video frame rates face challenges in reliable 3D reasoning without explicit spatial supervision or enriched multimodal inputs.Video-generation models also struggle with real-world 3D properties such as perspective consistency and vanishing points.Source: §3 · p. 4
- The overview frames images and videos as continuous projections of 3D space, with 3D consistency supporting spatial interpretation.Actial combines knowledge injection using hybrid cold-start initialization and pseudo CoTs with a generalization-enhancement stage.Source: §3 · pp. 4–5, Figure 3 · p. 5
4 Method
The method teaches viewpoint understanding as a foundation for 3D consistency, injects that knowledge with SFT, and then broadens spatial-task generalization with reinforcement learning. Its hybrid cold-start initialization pairs viewpoint training with reasoning templates.
4 Method
- The method frames multi-view images and videos as projections of 3D-consistent objects, with viewpoint understanding as a step toward reasoning about spatial structure.The method section comprises viewpoint learning, knowledge injection, hybrid cold-start initialization, and generalization enhancement.Source: §4 · p. 5, §4.1 · p. 5
4.1 Viewpoint Learning
- The approach simplifies camera-pose prediction into separate multiple-choice translation and rotation questions, generating Viewpoint-100K from MVImgNet with 100K object-centric image pairs whose horizontal viewpoint gaps range from 20 to 100 degrees.The set covers 10,813 objects across 205 classes.Source: §4.1 · pp. 5–6
4.2 Foundational Knowledge Injection
- SFT on Viewpoint-100K substantially improves spatial reasoning and reduces dependence on superficial 2D cues, while straightforward GRPO proves inadequate to overcome pretrained models’ entrenched 2D reasoning bias.Source: §4.2 · p. 6
4.3 Hybrid Cold-Start Initialization
- Hybrid cold-start initialization combines CoT templates with Viewpoint-100K SFT to learn viewpoint representations while maintaining coherent reasoning.The authors use 1K pseudo CoTs and keep their proportion small because some analyses conflict with 3D knowledge or the image.Source: §4.3 · pp. 6–7
4.4 Generalization Enhancement
- The generalization-enhancement phase uses GRPO on synthetic SAT data to broaden spatial reasoning while retaining viewpoint knowledge; it follows concern that viewpoint-only tuning risks overfitting.The authors report improved adaptability and performance on in-domain and out-of-domain tasks.Source: §4.4 · p. 7
5 Experiments
Actial is evaluated on 3DSRBench, CV-Bench, and BLINK, with results indicating improved spatial reasoning across multiple tasks. The ablations show that viewpoint SFT can transfer beyond its training tasks, while GRPO retains and extends some gains but can reduce performance on some metrics.
5.1 Datasets and Details
- Actial uses Viewpoint-100K for knowledge injection and SAT [30] for generalization, and is evaluated on 3DSRBench, CV-Bench, and BLINK.The three benchmarks assess spatial reasoning abilities.Source: §5.1 · pp. 7–8
5.2 Evaluations
- Both knowledge injection and generalization enhancement improve 3DSRBench performance, although Actial remains slightly behind previous state-of-the-art models.The authors attribute the remaining gap to the baseline method’s performance.Source: §5.2 · p. 8
- Actial substantially improves over the baseline on CV-Bench and outperforms existing proprietary models.The authors present this as evidence of improved visual-spatial task performance.Source: §5.2 · p. 8
- Actial’s experiments show improved spatial reasoning across multiple tasks, supporting the value of explicitly training foundational spatial skills.The reported improvements compared with baseline are consistent across multiple tasks.Source: §5.2 · p. 9
- Viewpoint-task performance reaches 99.2, while existing large models perform near random on BLINK’s multi-view task.The authors use these findings to distinguish viewpoint skill potential from the pre-training performance of existing models.Source: §5.2 · pp. 8–9
5.3 Ablation Studies
- SFT improves performance on most tasks and also benefits out-of-domain Height, Depth, and Relation, despite occasional marginal declines on other tasks.The authors note that task-specific SFT and limited data diversity can harm performance outside the fine-tuning distribution.Source: §5.3 · p. 9
- GRPO retains SFT gains, addresses some declines, and adds improvements, but can lower metrics such as 3DSRBench height and location.The authors caution that result-based rewards can affect previously injected foundational knowledge.Source: §5.3 · p. 9
6 Conclusion, Limitation and Impact
The paper presents Viewpoint Learning and a two-stage SFT-then-GRPO approach, reporting improved in-domain and out-of-domain reasoning. Its object-centric dataset covers basic viewpoint tasks rather than more complex settings such as camera pose estimation.
6 Conclusion, Limitation and Impact
- The two-stage approach combines SFT with hybrid cold-start initialization on Viewpoint-100K and GRPO to improve generalization.The authors report improved performance on both in-domain and out-of-domain reasoning tasks.Source: §6 · p. 10
- The dataset is constrained to object-centric scenarios and basic viewpoint tasks, which are simpler than settings such as camera pose estimation.The paper frames these foundational skills as relevant to advancing toward more complex visual tasks.Source: §6 · p. 10
Appendix
- The appendix provides additional experiments, training-process analysis, dataset examples, and prompts.Source: Appendix · p. 13
A.1 Results on Viewpoint-100K
On the 1,000-example Viewpoint-100K test set, accuracy rises from 12.9% for the Qwen base model to 92.2% after SFT, then falls to 81.4% after GRPO.
A.1 Results on Viewpoint-100K
- 92.2% accuracy after SFT, up from 12.9% for the Qwen base model; GRPO after SFT yields 81.4% on the 1,000-example test set.The authors suggest the RL objective may not align with the downstream task or may need reward refinement; the human evaluation averaged 97.67% across three annotators on 100 sampled instances.Source: §A.1 · p. 13
A.2 Results on MMSI-Bench
On MMSI-Bench, Actial improves on the baseline across many subtasks and reaches performance comparable to larger models and GPT-4o on some tasks. Long reasoning chains can be truncated on MSR before the correct answer is reached.
A.2 Results on MMSI-Bench
- Actial improves on the baseline across many MMSI-Bench subtasks and is comparable to larger models and GPT-4o, even outperforming them on some tasks.The comparison uses a 32K maximum token limit because of the long CoT template.Source: §A.2 · p. 13
- On MSR, excessively long reasoning chains can be truncated before the correct answer, preventing complete and accurate responses.The authors identify reasoning efficiency and output-length management as future work.Source: §A.2 · p. 13
A.3 Additional Ablation Study
An SFT-only ablation mixed Viewpoint-100K, pseudo CoTs, and SAT; its performance gap highlights the two-stage framework and GRPO’s gains on OOD tasks.
A.3 Additional Ablation Study
- The SFT-only ablation’s significant gap highlights the two-stage framework and GRPO’s performance gains on OOD tasks.The comparison used Qwen2.5-VL-7B-Instruct without GRPO.Source: §A.3 · p. 13
- The ablation mixed Viewpoint-100K, pseudo CoTs, and SAT, using SFT without GRPO.It used the same training parameters as previous experiments.Source: §A.3 · p. 13
- The SFT-only ablation trained for 2,000 steps, approximately 1.5 epochs.Source: §A.3 · p. 13
B Training Analysis
Training analysis tracks response length, KL divergence, and SFT loss, relating their changes to cold-start initialization and viewpoint knowledge injection.
B Training Analysis
- With hybrid cold-start initialization, response length increases during GRPO, unlike the stagnation attributed mainly to SFT without it.The authors relate that stagnation to Viewpoint-100K’s multiple-choice questions lacking reasoning templates, which make format rewards difficult to achieve.Source: B Training Analysis · pp. 13–14
- After knowledge injection, KL divergence becomes smoother; direct Viewpoint-100K GRPO instead shows a growing offset from the initial strategy.The authors attribute the offset to inadequate baseline spatial reasoning for viewpoint-specific tasks.Source: B Training Analysis · p. 14
- Viewpoint-100K produces a sudden SFT loss decrease within one epoch, while hybrid cold-start initialization smooths the curve.The authors caution that three-choice questions can yield correct answers without sound reasoning, and that viewpoint tasks require shifting from 2D cues to 3D spatial cues.Source: B Training Analysis · p. 14
C Dataset Examples
Viewpoint-100K examples pair distinct views of an object with questions about camera or object-relative movement. The prompts guide detailed trajectory analysis and structured answers.
C Dataset Examples
- Viewpoint-100K includes questions about camera-perspective horizontal translation, object-perspective horizontal translation, and rotation angle, calculated from camera parameters.The dataset also includes accurate rotation angles computed from camera parameters.Source: C Dataset Examples · p. 14
D Prompts
- The human-assisted CoT template asks for a camera trajectory across front/back, left/right, up/down, and rotation, then asks movement questions from camera and object perspectives.It also asks for rotation around the camera’s vertical axis and an angle estimate.Source: D Prompts · pp. 16–17
- The pseudo-CoT instructions direct analysis toward three-dimensional movement rather than relative image-position changes.The prompt organizes the response into image identification, visual analysis, movement analysis, trajectory synthesis, and answers.Source: D Prompts · pp. 16, 19