SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

1 Introduction

• VLMs struggle with basic and complex spatial relations, and global-image parsing makes it hard to specify instances; direction and distance also require more than RGB pixels alone.The cited applications motivating spatial awareness include robotics navigation and manipulation, and augmented-reality interaction.Source: §1 · p. 2
• Its pipeline builds 3D scene graphs from images using instance detection and segmentation, metric depth estimation, and camera calibration, then converts them into region-aware spatial QA tasks.The pipeline combines template-based and LLM-based generation and is designed to handle similar objects without extensive hand-crafted rules or post-processing.Source: §1 · p. 2
• SpatialRGPT adds region proposals as inputs and a flexible plugin for relative-depth information, letting pre-trained visual encoders use depth when available and function without it.This lets the LLM use both local-region and global-scene context; the paper reports improved spatial reasoning over RGB-only VLMs.Source: §1 · p. 2
• SpatialRGPT enhances region-level spatial reasoning by representing regional information and flexibly integrating depth into VLMs for improved 3D perception.The framework is designed to acquire spatial knowledge and reason about local regions while retaining scene context.Source: §1 · p. 2
• The pipeline produces the Open Spatial Dataset, containing 8.7M spatial concepts grounded in 5M unique regions, while SpatialRGPT-Bench evaluates cognition with 3D annotations across indoor, outdoor, and simulated environments.The benchmark addresses the stated lack of a spatial-cognition evaluation benchmark for VLMs.Source: §1 · pp. 2–3
• SpatialRGPT can serve as a region-aware dense reward annotator for robotics and as a stand-alone complex spatial reasoner capable of multi-hop reasoning.The paper presents these as downstream applications of the model’s region capabilities.Source: §1 · p. 3

Spatial Reasoning via Large Language Models

• Prior systems integrate multi-view 3D features or scene graphs into LLMs [20] [21] [22], while SpatialVLM [17] reasons implicitly but relies on textual object descriptions and lacks precise region specification.Multi-view reconstruction can be resource-intensive and suffer from a 3D-language modal gap, while LLMs may struggle to use textual coordinates; text cues can also let them answer spatial queries without visual grounding [23].Source: §2 · p. 3

Region-level Visual Language Models

• Region-level MLLMs use textual boxes [24] [25] [26] [27] [28] [29] or ROI-aligned features [30] [31] [32] [33] [34] [35], while RegionGPT [19] and Osprey [36] use pixel-level features, but none specifically targets region-level spatial reasoning.Bounding boxes can include background that misaligns region descriptions and text; SpatialRGPT builds on RegionGPT’s pixel-level inputs to deepen spatial reasoning in region VLMs.Source: §2 · p. 3

3.1 3D Scene Graph from Single 2D Images

• The pipeline turns single images into 3D scene graphs by grounding open-vocabulary objects, recovering metric geometry, and encoding object-level spatial relations.It combines object tagging, detection, and mask refinement with Metric3Dv2 depth estimation and camera calibration; graph nodes store object attributes, while edges capture relative and metric relations.Source: §3.1 · pp. 3, 5, Figure 1 · p. 4

3.2 Learning Spatial-aware VLMs from 3D Scene Graph

• The Open Spatial Dataset contains 1M images, 5M open-vocabulary regions, 8M template-based QAs, and 700K LLM-based QAs for spatial learning.Templates teach basic attributes and relations; Llama3-70B generates complex QAs from language-form spatial descriptions and region tags, rather than directly interpreting scene-graph coordinates.Source: §3.2 · p. 5

3.3 VLM Architecture

• A depth-to-language connector lets the visual encoder use relative-depth features while remaining usable when depth inputs are absent.The same image encoder processes RGB and depth, and the depth connector is initialized from the RGB connector and trained only on spatial QAs; region and depth tokens represent prompted regions in the LLM input.Source: §3.3 · p. 6

3.4 Training and Inference Paradigm

• Training proceeds through connector feature alignment, visual-language pretraining, and visual instruction-tuning, while modality sampling supports box- or mask-based inference.The depth connector is pretrained on OSD after the RGB connector and LLM are frozen; at inference, available masks are used, or boxes are segmented with SAM.Source: §3.4 · p. 6

4.1 3D Spatial Reasoning Benchmarks

• SpatialRGPT significantly outperforms baselines on qualitative success rates and achieves the lowest quantitative error on SpatialRGPT-Bench.Its depth variant improves on the RGB variant especially when depth helps resolve front/behind and wide/thin ambiguity or estimate distances.Source: Table 1 · p. 7, §4.1 · p. 9

4.2 Public Vision-language Benchmarks

• SpatialRGPT variants perform similarly to VILA-1.5-3B on general VQA benchmarks and slightly better on VQA-v2 and MMVet.These results indicate that spatial VQA training improves spatial reasoning without compromising general VQA performance.Source: Table 2 · p. 7, §4.2 · p. 9

4.1 3D Spatial Reasoning Benchmarks

• SpatialRGPT-Bench contains 657 qualitative and 749 quantitative VQA pairs across 88 classes, built from urban, indoor, and simulated scenes with 3D cuboid annotations.All samples come from validation or test splits and were unseen during SpatialRGPT training.Source: §4.1 · p. 7

4.2 Public Vision-language Benchmarks

• SpatialRGPT outperforms baselines on COCO-2017 region classification and gains over 20% accuracy on BLINK Relative Depth versus GPT-4V-Turbo.The point-level depth inputs and questions were not specifically included in training, demonstrating generalization to a new task without explicit training.Source: Table 3 · p. 9, Table 4 · p. 9, §4.2 · p. 9

4.3 Real-world Applications

• SpatialRGPT directly answers complex real-world spatial questions using its own spatial knowledge, rather than relying on a separate model for reasoning.It also handles multi-hop questions despite not having been specifically trained on multi-hop tasks.Source: Figure 4 · p. 8, §4.3 · pp. 9–10, Figure 5 · p. 10
• In a real-robot experiment, SpatialRGPT’s estimated fingertip-to-cube distance decreased monotonically as the fingertip moved toward the target.The region-aware setup specifies fingertip and cube regions with bounding boxes, and the depth variant performs slightly better than the RGB variant.Source: Figure 6 · p. 10, §4.3 · p. 10

Discussion

• SpatialRGPT integrates region representations and depth information, with scene-graph data curation and SpatialRGPT-Bench supporting spatial cognition across diverse environments.The framework perceives spatial arrangements at local and global scopes.Source: §5 · p. 10
• The results show improved spatial reasoning, complex spatial-relation reasoning, and dense reward annotation capabilities for robotic applications.Source: §5 · p. 10
• AABBs can make labels inaccurate; OBBs require challenging object-pose estimation, while human labeling is more accurate but effort-intensive [74].The authors leave these alternatives for future work.Source: §5 · p. 10

A Ablation Study on Augmented SpatialRGPT-Bench

• The evaluation augments and rephrases both benchmark questions and answers using GPT-4.The reported metrics are success rate and absolute relative error.Source: A Ablation Study on Augmented… · p. 17, Table 5 · p. 17
• The baseline advantage persists when the questions and answers differ from the training data.The paper presents this as evidence of consistent performance under the augmented and rephrased evaluation.Source: A Ablation Study on Augmented… · p. 17
• SpatialRGPT consistently outperforms baseline models on the augmented SpatialRGPT-Bench evaluation.Table 5 reports success rates and absolute relative error for the comparison.Source: Table 5 · p. 17, A Ablation Study on Augmented… · p. 17

B Ablation Study on Metric-Scale Width and Height Data

• Width and height data slightly improves size-question accuracy but slightly worsens horizontal and vertical object-distance accuracy, suggesting size cues may obscure distance measurements.Table 6 reports success rates across question categories.Source: Table 6 · p. 17, B Ablation Study on Metric-Scale Width… · p. 17

C Ablation Study on Bounding Box Types

• Table 7 evaluates axis-aligned versus oriented bounding boxes by MSE against Omni3D ground truth.Source: Table 7 · p. 17

D Ablation Study on Different Input Modalities

• Boxes yield a slight performance drop versus masks on SpatialRGPT-Bench, but overall performance remains very close, supporting the effectiveness of random-modality training.The ablation evaluates box and mask inputs during inference.Source: Table 8 · p. 18, D Ablation Study on Different Input… · p. 18

• The benchmark includes 12 QA types, evenly divided between relative relationships and metric measurements.Some datasets, such as SUNRGBD, emphasize closeobject scenarios.Source: E Statistics and Samples of… · p. 18
• To reduce bias, the authors source benchmark data from a diverse range of datasets following [63].Source: E Statistics and Samples of… · p. 18
• Figure 7 summarizes QA category counts, data-source counts, and object counts, while Figure 8 shows six benchmark samples.Source: E Statistics and Samples of… · p. 18, Figure 7 · p. 18

F.1 Filtering.

• The pipeline filters unsuitable OpenImages using CLIP-based open-vocabulary classification, excluding 700K of 1.7M images.The excluded categories include paintings, collages, screenshots, text, close-ups, and product images; retained labels include indoor and outdoor scenes.Source: §F.1 · p. 19

F.2 Metric Depth Estimation

• Metric3Dv2 estimates metric depth, with WildCamera intrinsics; predicted normal maps serve as a proxy for reconstructed-geometry edge quality.The authors report these components perform well on images from natural environments.Source: §F.2 · p. 19, Figure 9 · p. 20

F.3 Point Cloud Processing

• Point-cloud processing canonicalizes orientation from estimated pitch and roll, denoises object points, then creates axis-aligned bounding boxes.Denoising removes statistical outliers, downsamples, and applies DBSCAN; object areas with fewer than ten clustered points are excluded.Source: §F.3 · p. 20

F.4 Open Spatial Dataset QA Templates

• Open Spatial Dataset QA templates vary distance questions and answers, viewer-perspective left/right questions, and their corresponding responses.The templates also include direction questions answered with an approximate clock-face direction.Source: §F.4 · p. 21

G Implementation Details for SpatialRGPT Architecture

• SpatialRGPT uses a 336×336 CLIP-L visual backbone, refined region features with MaskPooling, and separate linear connectors for image and depth embeddings.A separate depth feature-refinement module is initialized from the RGB refinement module.Source: §G.1 · p. 21, §G.2 · pp. 21–22, §G.3 · p. 22

H Implementation Details for Training SpatialRGPT

• Stage-three instruction tuning combines general and LAN-style data with region-level data; depth-connector pretraining uses a 1e-4 maximum learning rate and batch size 32.Instruction fine-tuning uses a 5e-5 maximum learning rate and batch size 16, with other hyperparameters unchanged from pretraining.Source: §H.1 · p. 22, Table 10 · p. 22, Table 11 · p. 22

I Experimental Setting and Details

• OpenImages conversion to 3D scene graphs took within a day, and QA construction another half-day; SpatialRGPT training stages used 16-node and 2-node A100 GPU setups.Benchmark data came from ground-truth 3D cuboids and human-annotated labels; after QA synthesis, 1,500 samples were randomly selected, and human verification took a day.Source: §I.1 · p. 23

J Benchmark Evaluation Details

• GPT-4 evaluates qualitative answers as correct or incorrect and standardizes quantitative answers to a common unit before measuring accuracy and error.The evaluator receives the question, correct answer, and model response; prompts are provided in Tables 12 and 13.Source: J Benchmark Evaluation Details · p. 23

K More Discussion on Limitations

• OBBs are preferred for accurate object detection because AABB-derived dimensions can differ, as Figure 11 illustrates.Figure 11 compares bounding-box types.Source: K More Discussion on Limitations · p. 23, Figure 11 · p. 25
• PCA on a convex hull gives an approximate minimal OBB, but noisy, incomplete single-view point clouds can make it inaccurate.Source: K More Discussion on Limitations · p. 23
• OBB estimation still fails for partially elevated objects, while pose-based OBBs are difficult without accurate poses or open-vocabulary pose estimation.The paper names human labeling in Q-Spatial Bench [74] as another direction for addressing this limitation.Source: K More Discussion on Limitations · p. 24

L Broader Impacts

• SpatialRGPT risks hallucinations, inherited base-model biases, and energy use; spatial evaluation remains difficult, especially for safety-focused robotics, yet release can advance robotics research.Source: L Broader Impacts · p. 24

M Licenses

• The training data, OpenImages, is released under Apache License 2.0.Source: M Licenses · p. 24
• Images from Unsplash are released under the Unsplash License, which permits free commercial use without photographer or Unsplash attribution.Source: M Licenses · p. 25