Source-linked AI summary
SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, Alan Yuille
TL;DR
Existing LVLMs often reason about 3D spatial questions implicitly and can fail despite multi-step reasoning. SpatialReasoner instead shares explicit 3D representations across perception, computation, and reasoning, using supervised fine-tuning followed by reinforcement learning. It improves benchmark performance, generalizes to novel questions, and provides interpretable traces for studying failures.
Problem
Current LVLMs often use implicit 3D reasoning and fail on questions that are trivial for humans, motivating more reliable explicit spatial computation.
Method
SpatialReasoner predicts and uses explicit 3D representations across perception, computation, and reasoning through a two-stage SFT-then-RL training pipeline.
Results
SpatialReasoner improves 3D spatial reasoning across multiple benchmarks and generalizes better to novel 3D reasoning tasks.
Takeaways & Limitations
Explicit 3D representations support compositional reasoning, interpretable reasoning traces, and analysis of LVLM bottlenecks and failure modes.
Takeaways & Limitations
Performance drops on CVBench-3D distance questions, where abundant 2D shortcuts encourage spurious correlations despite stronger results on similar 3DSRBench questions.
Abstract
from arXiv · showhide
Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models. Recent studies explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual question-answering data. However, these methods typically perform spatial reasoning in an implicit manner and often fail on questions that are trivial to humans, even with long chain-of-thought reasoning. In this work, we introduce SpatialReasoner, a novel large vision-language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages--3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability to novel question types. Furthermore, by analyzing the explicit 3D representations in multi-step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMs. Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2.0 by 9.2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning.
1. Introduction
SpatialReasoner targets limited 3D spatial reasoning by combining explicit 3D representations with multi-stage perception, computation, and reasoning. Its training and analysis aim to improve generalization, interpretability, and robustness.
- Motivation: 3D spatial reasoning remains limited in current large multimodal models despite its importance for robotics and embodied AI.These models fall behind human-level performance on 3D spatial reasoning.
- Challenges: Prior models can decompose complex questions but often lack reliable explicit 3D computation needed for correct answers.Gemini 2.0 uses organized multi-step reasoning but ultimately fails because its 3D computation is unreliable.
- Approach: SpatialReasoner uses explicit 3D representations as an interface across 3D perception, computation, and reasoning.The model predicts representations such as 3D locations and orientations and uses them across multiple stages.
- Training: Its two-stage training applies supervised fine-tuning for explicit 3D perception and computation, followed by reinforcement learning for generalizable 3D thinking.The method also synthesizes 3D perception, computation, and multi-step spatial reasoning data.
- Analysis: Explicit intermediate 3D predictions make reasoning traces interpretable and support analysis of LVLM failure modes.The authors report that downstream errors still primarily arise from 3D perception rather than 3D computation.
- Results: SpatialReasoner improves performance across benchmarks and generalizes better to novel 3D spatial reasoning questions.The authors also find that standard LVLMs often use 2D shortcuts, whereas SpatialReasoner consistently reasons with explicit 3D representations.
2. Related Works
Prior work studies 3D spatial reasoning through simulated or real-world benchmarks, structured visual modules, post-training, and test-time scaling. These approaches expose progress but also leave challenges in interpretability and generalization.
- 3D Spatial Reasoning: 3D spatial reasoning research has progressed from simulated and object-annotated settings toward benchmarks based on real-world imagery.Recent methods also improve performance with synthetic question-answering data.
- Explicit 3D Representations: Structured explicit 3D representations can simplify reasoning and expose model failures through interpretable visual modules.Simulation-based neural-symbolic methods and PO3D-related systems demonstrate this direction.
- Explicit 3D Representations: Calibrated camera 3D space is used to simplify subsequent spatial computations by aligning coordinates with the 3D world space.This representation makes height and above/below relationships easier to estimate.
- Post-training: Post-training commonly uses supervised fine-tuning and reinforcement learning to align pretrained models with downstream objectives.SFT stabilizes outputs but often overfits, whereas RL improves adaptability; combining both is also explored.
- Test-time Scaling: Test-time scaling improves reasoning without retraining through methods such as beam search, best-of-N sampling, MCTS, and chain-of-thought prompting.The paper builds on chain-of-thought reasoning by fine-tuning models to generate step-by-step rationales for 3D spatial tasks.
3. SpatialReasoner
SpatialReasoner uses explicit 3D representations as a shared interface across perception, computation, and reasoning, combined with two-stage training to improve generalizable spatial reasoning. It is evaluated with 3D-aware data and achieves state-of-the-art performance on 3DSRBench.
- Explicit 3D Representations: SpatialReasoner uses calibrated camera 3D space, where z-coordinates correspond to object heights and some spatial relationships simplify to 2D problems.This representation also simplifies estimating above-and-below relationships and objects arranged on planes parallel to the ground.
- Explicit 3D Representations: Explicit 3D locations and orientations provide a shared interface across 3D perception, computation, and reasoning.The model predicts 3D vectors, computes distances or angles, and aggregates these representations to answer spatial questions.
- Training Strategy: SFT-trained models tend to memorize spatial templates and struggle with novel compositions or combinatorial variations in 3D queries.RL is introduced specifically to address this limitation and improve generalization beyond training-specific distributions.
- Training Strategy: The two-stage strategy uses SFT to learn explicit 3D representations, followed by RL to develop robust and generalizable 3D thinking.SFT initializes the model with 3D-annotated data, while RL uses structured rewards for answer correctness and intermediate computation quality.
- Training Data: The data pipeline generates 3D pseudo-annotations, optionally verifies them by humans, and constructs perception, spatial-relation, and explicit chain-of-thought VQA data.Training variants include Basic3D-QA, SR-QA, and SR-CoT, which target different stages and forms of 3D spatial reasoning.
4. Results
SpatialReasoner is evaluated against generalist, specialist, and proprietary models across three spatial reasoning benchmarks. Results show strong benchmark performance, improved novel-task generalization with RL, and distinct trade-offs between SFT and RL under data and computation scaling.
- Benchmark performance: SpatialReasoner achieves 60.3% mean accuracy on 3DSRBench, outperforming prior open-source and proprietary models.It exceeds Gemini 2.0 Flash at 49.8% and Claude 3.5 Sonnet at 48.2%.
- Generalization: RL generalizes better than SFT to unseen multi-object questions, while combining SFT and RL reaches 43.4% versus 40.0% for SFT alone.RL-only reaches 46.6% in the no-multi-object-training setting, exceeding the combined model’s 43.4%.
- Generalization: Outcome-based RL preserves mean performance without multi-object examples at 54.0% versus 53.7%, whereas SFT declines from 58.3% to 52.2%.The authors associate this pattern with RL developing transferable reasoning strategies and SFT overfitting training distributions.
5. Conclusions
SpatialReasoner performs explicit, generalizable 3D spatial reasoning by using intermediate 3D representations across perception, computation, and reasoning stages. The authors also identify training and inference limitations that constrain the system’s current efficiency and initialization.
- SpatialReasoner predicts and leverages intermediate 3D representations across perception, computation, and reasoning stages.
- A two-stage pipeline combining supervised fine-tuning and reinforcement learning advances spatial reasoning and generalization to novel 3D reasoning tasks.
- Explicit 3D reasoning improves accuracy, produces interpretable reasoning traces, and highlights bottlenecks in 3D perception.
- Supervised fine-tuning data with chain-of-thought reasoning and explicit 3D representations remains crucial as a warm-up despite reinforcement learning.
- SpatialReasoner consistently performs explicit multi-step 3D reasoning, whereas an ideal system would reserve such computation for more challenging questions to improve inference efficiency.
A. Implementation detail
The implementation trains SpatialReasoner through curated supervised fine-tuning followed by reinforcement learning, using different datasets and objectives for the model and ablation variants.
- Training starts from Qwen2.5-VL-7B and applies supervised fine-tuning with 24k SR-CoT examples plus 24k randomly sampled LLaVA examples.
- The final SpatialReasoner further trains the supervised model with reinforcement learning on 1.2k SR-QA examples.
- The SFT (+HQ SFT) ablation instead applies supervised fine-tuning on the same 1.2k SR-QA set.
- SFT training uses 10 epochs, approximately 20K steps, and batch size 6, while RL uses 100 epochs, approximately 13K steps, and batch size 12.
- All training experiments use 4×NVIDIA H100 80GB HBM3 GPUs, with RL using 1 GPU and vLLM for inference acceleration.
B. 2D Reasoning as a Shortcut
Distance questions in CVBench-3D can be solved through 2D bounding-box shortcuts, whereas 3DSRBench avoids these correlations and better tests explicit 3D reasoning. This creates a benchmark-dependent trade-off between shortcut exploitation and robust performance.
- 21.5% improvement on 3DSRBench contrasts with a 9.9% performance drop on CVBench-3D multi-object 3D distance questions.
- CVBench-3D bounding boxes permit answering some 3D distance questions by comparing 2D distances between object boxes.
- 3DSRBench manually avoids spurious correlations such as objects being closer in 3D space and also closer in the 2D image plane.
- A 2D bounding-box heuristic reaches 80.2% accuracy on CVBench-3D distance questions but 34.3% on 3DSRBench.
- Figure 9 compares multi-object distance-related questions across CVBench-3D and 3DSRBench.
- SpatialReasoner’s explicit 3D representations can reduce performance on shortcut-rich data while improving robustness on challenging real-world datasets.
C. Open Access
The paper states that its code, data, and models are released to support reproducibility and research use, with links available on the project page.
- The authors release all code, data, and models to support reproducibility and benefit the research community.
- The project page provides the release links.
- Table 7 compares Qwen2.5-VL and SpatialReasoner on multi-object 3D distance questions in CVBench-3D and 3DSRBench.