Source-linked AI summary
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, Peter Grasch
TL;DR
MLLMs remain limited in 3D spatial perception despite strong 2D abilities, especially for metric geometry and precise grounding. The paper introduces CA-VQA and trains MM-Spatial with high-quality 3D data, multi-view inputs, and metric depth. MM-Spatial achieves state-of-the-art spatial benchmark performance, while depth and multi-view inputs further improve results; the evaluation also has a data-overlap caveat for DepthPro.
Problem
MLLMs remain limited on 3D object perception tasks such as relative depth, metric distance or size estimation, and precise 3D bounding boxes.
Method
The paper builds the CA-VQA dataset and benchmark from high-quality 3D scenes, covering spatial relations, metric estimation, 3D grounding, multi-view images, and multiple depth inputs, then uses it for MM-Spatial SFT.
Results
MM-Spatial achieves state-of-the-art performance on spatial understanding benchmarks while retaining competitive performance across other benchmark categories; depth and multi-view inputs further improve spatial results.
Takeaways & Limitations
The results show that CA-VQA supports a generalist MLLM with strong 3D spatial understanding and that supervised data can produce strong monocular depth perception capabilities.
Takeaways & Limitations
DepthPro’s training data includes ARKitScenes, which also underlies CA-VQA and MM-Spatial’s training data, so the comparison is not zero-shot for either model.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models.
1. Introduction
MLLMs handle many 2D visual tasks but remain limited on 3D perception, including metric depth, distance, size, and precise 3D grounding. The paper addresses this gap with CA-VQA, a broad indoor spatial dataset and benchmark built around richer 3D inputs and tasks.
- Motivation: MLLMs succeed on 2D referring, grounding, and spatial relations but struggle with relative depth, metric distance or size, and precise 3D bounding boxes.These capabilities are relevant to visual comprehension and applications such as robotics and AR/VR.
- Research gap: Existing MLLM work on 3D object perception covers comparatively few tasks and does not comprehensively evaluate depth and multi-view inputs.
- Contributions: CA-VQA is an indoor spatial-understanding dataset generated from high-quality 3D scene data, combining diverse tasks with multi-view images and sensor-based and monocular metric depth maps.
- Contributions: The accompanying benchmark includes relative and metric distance or size estimation and 3D grounding, uses rich visual inputs, and is designed to reduce reliance on language priors.The paper reports that even GPT-4o struggles on this benchmark.
2. Related Work
Prior MLLM research has expanded 2D and scene-level capabilities, while object-level 3D spatial perception remains a distinct and incomplete direction. Existing approaches differ in their use of images, depth, tools, and scene representations, motivating a broader dataset and benchmark.
- MLLM research directions: MLLM research has advanced through larger training datasets, higher-resolution vision, improved encoders and connectors, mixture-of-experts, and region-, pixel-, multi-image, UI, and video understanding.
- Scene-level 3D understanding: Scene-level 3D methods process entire scenes for captioning or VQA, often using multiple views and depth information.Examples include ScanReason, 3D-CLR, 3D-LLM, ConceptGraphs, LLaVA-3D, and related systems.
- Object-level 3D perception: Object-level methods use image-only VLMs, encoded relative depth, or depth queried through tools for spatial relations, metric distance estimation, and 3D grounding.
- Positioning CA-VQA: CA-VQA differs by combining high-quality 3D ground truth, sensor and monocular depth maps, multi-view images, diverse spatial tasks, and both SFT and benchmark splits.
3. Data
CA-VQA is generated from richly annotated 3D indoor scenes and converts geometric and semantic annotations into diverse, automatically generated spatial questions. Its samples combine viewpoint-aware geometry with multi-view and multiple metric-depth sources.
- Source data: CA-1M provides gravity-aligned 7-DOF 3D boxes for every ARKitScenes object plus open-set labels, material, primary color, and shape annotations.The annotations cover approximately 3.3k noun labels for approximately 350k objects.
- Data generation: The pipeline subsamples videos at 1 FPS for training and 0.1 FPS for evaluation to reduce data volume and redundancy.
- Data generation: Ground-truth processing transforms boxes into camera space, retains all visible amodal boxes, and constructs point clouds from depth maps and camera intrinsics.
- Data generation: Template-based open-ended and multiple-choice QA pairs are generated automatically from 3D and semantic annotations, with checks for unambiguous questions and blind-answer filtering.
- Spatial tasks: CA-VQA covers counting, viewpoint-dependent relations, metric size and distance regression, 2D and 3D referring or grounding, and binary comparison or presence questions.Metric distances use object point clouds, while sizes use 3D bounding boxes.
- Input modalities: Each reference frame has up to four support views triggered by camera motion, and every frame includes FARO ground-truth, ARKit, and monocular metric depth sources.Support views also include relative pose and camera intrinsics.
4. Model
MM-Spatial uses an MM1.5-based multimodal architecture and extends its inputs with multi-view images or depth. Training combines the original MM1.5 SFT mixture with CA-VQA spatial data, while depth can also be accessed through tool use.
- Architecture: MM-Spatial uses the mobile-friendly MM1.5 3B architecture with a DFN-CLIP image encoder, decoder-only LLM, C-Abstractor connector, and 672×672 fine-tuning resolution.Static image splitting adds four sub-images plus an overview image.
- Model variants: The multi-view model uses up to four support frames with camera intrinsics and relative poses, while the depth-tool variant receives textual metric depth at test time.
- Input variants: The multi-view variant concatenates preceding support frames with the reference image, while depth variants encode normalized colorized depth maps through the image encoder.
- Training: Supervised fine-tuning starts from the MM1.5 mixture and adds CA-VQA data as a new Spatial category, with mixture ratios selected through ablations.
5. Experiments
Experiments show that MM-Spatial improves spatial understanding while preserving generalist performance, with multi-view inputs, depth tool-use, and chain-of-thought further strengthening results. The model outperforms larger baselines across spatial evaluations and exhibits strong metric-depth estimation, though outdoor generalization and monocular depth remain challenging.
- 5.2. Overview of Benchmark Category Results: MM-Spatial improves strongly on the Spatial category while remaining competitive with MM1.5 across other benchmark categories.This supports improving spatial reasoning without meaningful compromise to generalist capability.
- 5.3. Results on our CA-VQA Benchmark: MM-Spatial-3B significantly outperforms much larger open-source and commercial models across CA-VQA tasks.The result includes comparisons against GPT-4o and SpatialRGPT-VILA-1.5-8B, whose training data and spatial representations differ from CA-VQA.
- 5.3. Results on our CA-VQA Benchmark: Multi-view inputs, depth tool-use, and CoT consistently improve 3D perception, with multi-view competitive overall and ground-truth depth strongest on Regression.The reported comparisons include Multi-view 19 vs. Single-view 13 and Multi-view 19 vs. GT-depth tool-use 16.
- 5.3. Results on our CA-VQA Benchmark: MM-Spatial’s monocular metric-depth accuracy can rival the specialist DepthPro model despite MM-Spatial being trained only on indoor scenes.The comparison is not zero-shot because DepthPro’s training data includes ARKitScenes, which also underlies CA-VQA.
- 5.4. CV-Bench Results: MM-Spatial nearly solves indoor 3D tasks on CV-Bench and generalizes strongly to outdoor splits, while monocular depth can underperform ground-truth depth and the RGB-only baseline on Ego-Distance.The benchmark also reports that vision input improves CA-VQA Counting by approximately 13 points over blind evaluation.
- 5.5. SpatialRGPT-Bench Results: On SpatialRGPT-Bench, indoor-focused CA-VQA training struggles with outdoor metric scales, while scale augmentation substantially improves outdoor Distance performance.The analysis attributes the gap mainly to differences between indoor and outdoor metric scales, especially object distances.
6. Conclusion
The paper contributes datasets, benchmarks, and modeling advances for object-centric 3D spatial understanding in MLLMs, while preserving general capabilities and identifying future extensions.
- CA-VQA is an SFT dataset for 3D perception tasks, including multi-view and depth inputs.
- The paper introduces a 3D spatial understanding benchmark covering spatial relationships, metric estimation, and 3D grounding.
- MM-Spatial achieves state-of-the-art performance on spatial reasoning benchmarks while preserving general MLLM capabilities.
- Adding multi-view and depth inputs further improves spatial perception, while SFT gives MLLMs strong monocular depth estimation capabilities.
- Future work aims to extend the indoor-scene scope to outdoor scenes.
A.1. Spatial Task Categories
CA-VQA constructs object-centric spatial questions from 2D and 3D scene annotations, spanning viewpoint relations, counting, metric estimation, grounding, and depth-based reasoning.
- Binary: Viewpoint-dependent questions classify left/right using 2D box centers and in-front/behind using camera distances to 3D box centers.
- Binary: Object presence questions include negative samples for absent objects, while counting questions count class-specific boxes and include zero-count negatives.
- Regression (Metric Estimation): Metric regression estimates egocentric and inter-object distances plus object width, length, and height from 3D geometry or point clouds.
- Grounding: 2D grounding projects 3D boxes into image space, whereas 3D grounding directly uses the CA-1M 3D boxes.
- Depth: Chain-of-Thought (CoT) / Tool-Use: Depth-based CoT samples target binary depth relations and metric questions, using median depth within each object’s 2D box during training.
- Depth: Chain-of-Thought (CoT) / Tool-Use: The dataset excludes CoT samples for 3D grounding after preliminary experiments found no improvement or slight regression.
B. Optimal Data Mixture for MM-Spatial
The authors tune the spatial-data mixture to improve spatial understanding while retaining the generalist model’s performance across other capabilities.
- The mixture weight is selected to balance spatial performance against general, knowledge, text-rich, and 2D referring-and-grounding categories.
- A 2:1 spatial-to-general mixture provides a good performance trade-off and is used for MM-Spatial.
- The mixed MM-Spatial model significantly improves spatial performance while maintaining similar results across most other categories.
- Training only on CA-VQA yields a small spatial improvement but substantially regresses on other benchmark categories.
C. Results on Further Benchmark Categories
MM-Spatial retains performance similar to vanilla MM1.5 on general and knowledge, text-rich, and 2D referring-and-grounding benchmarks.
- General and Knowledge: MM-Spatial maintains performance similar to the vanilla MM1.5 baseline across general and knowledge benchmarks.
- Text-rich: The model preserves a similar performance level on text-rich benchmarks.
- 2D Referring & Grounding: MM-Spatial maintains similar performance on 2D referring-and-grounding benchmarks while adopting the new spatial capability.
D. Analysis of Blind Filtering Procedure
Blind filtering makes CA-VQA more dependent on visual input, unlike some comparison benchmarks where blind models can perform strongly. Evaluation also requires care because CA-VQA and SpatialRGPT-Bench use different 3D box conventions.
- CA-VQA tasks become more reliant on vision after blind filtering, whereas some CV-Bench and SpatialRGPT-Bench tasks remain strong for blind models.The authors suggest these comparison benchmarks could also benefit from blind filtering.
- AABB–OBB mismatches can affect object dimensions when evaluating models across benchmarks with different 3D bounding-box ground truths.MM-Spatial uses gravity-aligned yaw-oriented OBB ground truth, while SpatialRGPT-Bench uses AABB ground truth.
- Blind filtering makes blind models perform substantially worse and increases the vision-versus-blind performance gap for GPT-4/V and MM-Spatial.The analysis compares vision-evaluated and blind-evaluated models before and after filtering.