Source-linked AI summary
3D Aware Region Prompted Vision Language Model
An-Chieh Cheng, Yang Fu, Yukang Chen, Zhijian Liu, Xiaolong Li, Subhashree Radhakrishnan, Song Han, Yao Lu, Jan Kautz, Pavlo Molchanov, Hongxu Yin, Xiaolong Wang, Sifei Liu
TL;DR
SR-3D addresses the difficulty of applying strong 2D VLM priors to 3D spatial reasoning while specifying regions across multi-view scenes. It connects single-view and multi-view data through shared visual representations, canonical 3D positional embeddings, and flexible region prompting. Experiments report state-of-the-art performance across 2D vision-language and 3D spatial benchmarks, with applicability to in-the-wild videos.
Problem
Existing 2D VLMs lack mechanisms for complex 3D relationships, while 3D VLMs use different representation spaces and region specification across views is cumbersome.
Method
SR-3D unifies single-view and multi-view data in a shared representation space by integrating canonical 3D positional embeddings and dynamic tiling-based region extraction into a foundational VLM.
Results
SR-3D achieves state-of-the-art performance across general 3D question answering, 3D video spatial understanding, video region-level tasks, and broader 2D vision-language and 3D spatial benchmarks.
Takeaways & Limitations
The shared representation enables flexible region prompting across single-view and multi-view settings while supporting spatial reasoning in in-the-wild videos.
Takeaways & Limitations
The method is designed for multi-view static data, and accurately interpreting object orientation remains challenging because scaling suitable data is difficult.
Abstract
from arXiv · showhide
We present Spatial Region 3D (SR-3D) aware vision-language model that connects single-view 2D images and multi-view 3D data through a shared visual token space. SR-3D supports flexible region prompting, allowing users to annotate regions with bounding boxes, segmentation masks on any frame, or directly in 3D, without the need for exhaustive multi-frame labeling. We achieve this by enriching 2D visual features with 3D positional embeddings, which allows the 3D model to draw upon strong 2D priors for more accurate spatial reasoning across frames, even when objects of interest do not co-occur within the same view. Extensive experiments on both general 2D vision language and specialized 3D spatial benchmarks demonstrate that SR-3D achieves state-of-the-art performance, underscoring its effectiveness for unifying 2D and 3D representation space on scene understanding. Moreover, we observe applicability to in-the-wild videos without sensory 3D inputs or ground-truth 3D annotations, where SR-3D accurately infers spatial relationships and metric measurements.
1 Introduction
SR-3D addresses the difficulty of transferring 2D vision-language priors to 3D spatial reasoning and of specifying regions across multi-view scenes. It unifies 2D and multi-view representations with canonical 3D positional embeddings and flexible region prompts, achieving state-of-the-art results across spatial tasks.
- The approach targets limitations of 2D VLMs’ 3D reasoning, 3D VLMs’ separate representation spaces, limited 3D data, and cumbersome language-only region specification.
- SR-3D integrates depth-derived 3D positional embeddings into a foundational VLM and trains region tokens across single-view and multi-view stages.
- Flexible region prompting lets users specify regions on any chosen frame, while canonical positional embeddings support coherent correspondences across frames.
- Evaluations cover single-view and multi-view region-level and global question answering, with improvements in region tasks that do not compromise overall VQA performance.
- SR-3D establishes new state-of-the-art results across general 3D question answering, 3D video spatial understanding, and video region-level spatial tasks.
- The model unifies single-view and multi-view representations so 2D-trained region embeddings generalize to multi-view context.
2 Related Work
Prior work explores region-level VLMs, multi-view spatial reasoning, and 3D multimodal models, but commonly uses separate pathways or specialized representations. SR-3D instead provides a unified architecture and representation space for image and video spatial understanding.
- Region-level VLMs use coordinate text, visual markers, or RoI-aligned features to focus fine-grained visual understanding on image and video regions.
- Multi-view spatial reasoning remains less explored than 2D spatial understanding, with VSI-Bench introduced to evaluate 3D video-based spatial understanding.
- 3D multimodal models represent scenes through multi-view pixel-aligned points, point-cloud encoders, or segmented point-cloud objects.
- LLaVA-3D and Video-3D-LLM incorporate 3D position-aware features into 2D VLMs but use separate pathways or specialized 3D fine-tuning.
- SR-3D uses a unified architecture and 3D representation space for image and video data, improving alignment and generalization across spatial understanding tasks.
3 Methodology
SR-3D unifies single-view and multi-view spatial reasoning through canonical 3D positional features integrated into a foundational VLM. Its dynamic tiling and region extraction support spatially consistent prompts across diverse input and annotation formats.
- 3.1 Canonical 3D Positional Representation: SR-3D directly integrates 3D positional embeddings into a foundational 2D VLM, creating a shared representation for single-view and multi-view spatial understanding.Depth or point maps are canonicalized into a normalized space before enriching visual tokens.
- 3.1 Canonical 3D Positional Representation: The model estimates depth for single-view images, back-projects pixels into 3D, canonicalizes positions, and injects learned embeddings into vision tokens.A sinusoidal encoding and learnable point-wise MLP produce embeddings aligned with token dimensions.
- 3.1 Canonical 3D Positional Representation: Multi-view training aligns uniformly sampled frames in the shared canonical space, extending spatial reasoning beyond individual images.The multi-view procedure uses ground-truth depth during training and transforms point maps into a common coordinate system.
- 3.2 Dynamic Tiling-based Region Extractor: The tile-and-stitch region extractor preserves high-resolution details by tiling images and masks, stitching visual tokens, and pooling features over the region.In multi-view inputs, each frame acts as a tile while preserving consistency for the same region across frames.
- 3.4 Inference: A unified pipeline accepts boxes, segmentation masks, sparse-frame masks, single-frame masks, and projected 3D boxes without separate single- and multi-view pathways.Canonicalized positions also allow estimated point maps from off-the-shelf 3D models to replace ground-truth depth at inference.
4 Experiments
Experiments evaluate SR-3D across 2D recognition and spatial tasks, 3D benchmarks, region-level video reasoning, and ablations. The results report strong or state-of-the-art performance while preserving general VLM capabilities and supporting sparse or estimated geometry.
- 4.1 Evaluation on 2D Benchmarks: On COCO-2017 region classification, SR-3D achieves 78.0 mAP and 88.6% accuracy, validating its region extractor.The evaluation uses ground-truth boxes for positional information.
- 4.1 Evaluation on 2D Benchmarks: SR-3D attains 90% accuracy on BLINKDepth, outperforming SpatialRGPT on point-level depth understanding with region prompts.The benchmark tests whether a 3D-injected VLM can use auxiliary spatial information.
- 4.1 Evaluation on 2D Benchmarks: SR-3D achieves state-of-the-art results across Scan2Cap, ScanQA, and SQA3D spatial scene-understanding benchmarks.Table 4 reports performance across all metrics, including zero-shot comparisons marked with †.
- 4.4 Analysis and Ablation Study: The single-view model maintains comparable math, general-understanding, and OCR performance while improving spatial understanding, including RealWorldQA.Ablations further show that single-view pretraining improves multi-view performance, and estimated Cut3R geometry preserves performance close to ground-truth results.
- 4.3 Video Spatial Intelligence.: SR-3D outperforms all baselines across SR-3D-Bench region-level spatial categories, while blind language models and language-referral VLMs remain competitive mainly on quantitative tasks.Region-level VLMs using SoM struggle to track marks across frames.
5 Conclusion
SR-3D unifies single-view and multi-view data in a shared space while leveraging pretrained 2D visual-language priors. Its region features support flexible prompts, and experiments report state-of-the-art performance across 2D and 3D spatial benchmarks.
- 5 Conclusion: SR-3D unifies single-view and multi-view data in a shared space, leveraging pretrained 2D priors for complex 3D spatial tasks.The tile-and-stitch design extracts high-resolution region features for flexible prompts across both settings.
- 5 Conclusion: Experiments on 2D vision-language and 3D spatial benchmarks report state-of-the-art performance, supporting SR-3D’s unified representation design.The conclusion specifically connects benchmark performance with unified spatial reasoning.
A More Quantitative Results on 3D General Benchmarks
The supplementary evaluation reports full validation or testing results for Scan2Cap, ScanQA, and SQA3D. These tables provide expanded metric coverage beyond the main reported comparisons.
- Scan2Cap: Full Scan2Cap validation results are provided in Table 10 for the 3D dense-captioning evaluation.The table supplements the main benchmark comparison with additional metrics.
- ScanQA: Full ScanQA validation results are provided in Table 11 for 3D question answering.The supplementary table expands the reported ScanQA evaluation.
- SQA3D: Full SQA3D testing results are provided in Table 12 for spatial question answering.The supplementary table extends the main SQA3D benchmark results.
B More Quantitative Results on VSI-Bench
Additional VSI-Bench results evaluate SR-3D on ScanNet++ scenes, a challenging dataset with high-fidelity environments and coarse 3D alignment. SR-3D shows superior relative-direction determination in this setting.
- SR-3D demonstrates superior relative-direction reasoning on ScanNet++ scenes in VSI-Bench.The results are presented in Figure 6.
- ScanNet++ provides a distinct evaluation setting because it is absent from EmbodiedScan’s annotations.This makes it an independent dataset for testing the method.
- ScanNet++ offers higher-fidelity, more diverse indoor environments than ScanNet, but its 3D annotations are only coarsely aligned to walls and floors.
C More Ablation Study
The ablations show that the fully trained model benefits from combining 2D pre-training with 3D spatial information. Removing pre-training substantially harms complex reasoning, while a narrower Wide-and-Big discrepancy likely reflects differing width definitions across 2D and 3D.
- The ablation study reports full results on Scan2Cap, ScanQA, and SQA3D, with results organized in Table 13.
- The fully trained model consistently outperforms baselines on 3D general question-answering benchmarks.This supports leveraging both 2D and 3D spatial information.
- The 3D spatial-focused dataset shows a slight Wide-and-Big decline, likely because width is defined differently in 2D and 3D.
- Removing pre-training causes a substantial performance drop on complex reasoning tasks, especially multi-choice complex questions.The model struggles without prior exposure to large-scale 2D pre-training.
D Statistics of SR-3D-Bench
SR-3D-Bench is designed to increase the complexity and diversity of spatial reasoning evaluation through prior spatial-reasoning templates and situated EmbodiedScan annotations.
- SR-3D-Bench follows template designs from SpatialRGPT and SpatialVLM for spatial-reasoning evaluation.
- Situated annotations from EmbodiedScan make the benchmark more complex, diverse, realistic, and challenging.
- The benchmark includes spatial relations such as thin-wide, tall-short, and big-small.The passage introduces these as basic geometric comparisons.
E Implementation Details of SR-3D
SR-3D uses PaliGemma and Qwen-2-7B backbones, with specified image, patch, tile, frame, optimization, and data-setting choices. Additional VSI-Bench visual results and the data recipe are provided for implementation context.
- The model pairs a PaliGemma visual backbone with a Qwen-2-7B language-model backbone.PaliGemma uses 448 input size and 14 patch size.
- Training uses up to 12 tiles per image for the 2D VLM and 32 uniformly sampled frames for the multi-view VLM.The frame setting is intended to support fair comparison with previous methods.
- Figure 6 provides additional SR-3D visual outputs on VSI-Bench alongside ground-truth numerical answers.
- The training data excludes spatial and region-related datasets from a subset of 2D data to preserve original vision-language capabilities.The recipe is detailed in Table 15.
F Limitations
SR-3D has limitations in dynamic environments, object-orientation questions, OCR-related tasks, and combining its single- and multi-view models.
- Orientations: Accurately interpreting object-orientation questions remains challenging because scaling the required data is difficult.The authors leave orientation understanding to future work.
- Dynamic Videos: SR-3D is designed for multi-view static data, while dynamic real-world environments remain outside its current setting.The authors identify incorporating positional embeddings for both static and dynamic inputs as non-trivial.
- OCR Tasks: The 2D foundation model shows a consistent slight performance drop on OCR-related tasks despite comparable general-benchmark performance.The authors suggest adding more OCR-related tasks to the training data pipeline.
- Unified Checkpoint: The single- and multi-view models are not yet effectively combined into one unified checkpoint.Future directions include an agentic flow between models or direct training across both settings.