PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, Sifei Liu

arXiv:2406.01584v3cs.CV

TL;DR

VLMs remain limited in reasoning about spatial arrangements, especially when interpreting object regions and 3D relations. SpatialRGPT addresses this with 3D scene-graph data, region-aware representations, and a flexible depth plugin, and reports improved spatial reasoning together with benchmark and robotic applications.

  • Problem

    VLMs perform well on many 2D tasks but remain limited in reasoning about spatial arrangements and accurately parsing object-instance regions.

  • Method

    SpatialRGPT combines a scalable 3D scene-graph data pipeline, region representations, and a flexible plugin that integrates depth into existing VLM visual encoders.

  • Results

    SpatialRGPT significantly improves spatial reasoning across tasks, including a relative-depth benchmark where it achieves over 20% accuracy gain over the state of the art.

  • Takeaways & Limitations

    The model generalizes to complex spatial relations and supports region-aware dense reward annotation for robotic tasks and stand-alone spatial reasoning.

  • Takeaways & Limitations

    Accurate oriented 3D bounding boxes remain constrained by incomplete, noisy single-view point clouds and the lack of open-vocabulary object-pose estimation.

Abstract

from arXiv · show

Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (1) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (2) a flexible plugin module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT

1 Introduction

SpatialRGPT addresses VLMs’ limited spatial reasoning by combining region-aware representations, depth integration, and large-scale 3D spatial supervision. It also introduces a benchmark and demonstrates applications in complex reasoning and robotics.

  • VLMs struggle with basic and complex spatial relations, including left/right, above/below, behind/in front, inside/outside, and near/far.
  • Existing spatial-VQA training addresses missing spatial knowledge, but VLMs still struggle to parse object-instance regions in cluttered images.
  • SpatialRGPT combines a data curation pipeline with region-aware and 3D-aware visual encoding to improve VLM spatial reasoning.
  • The pipeline constructs 3D scene graphs with object-instance nodes and spatial-relation edges, then converts them into template-based and LLM-based spatial QA tasks.
  • A region module combines local proposals with global image context, while a flexible plugin adds relative-depth information to existing VLM visual encoders.
  • The Open Spatial Dataset contains 8.7M spatial concepts grounded in 5M unique regions.
  • SpatialRGPT-Bench evaluates spatial cognition across indoor, outdoor, and simulated environments, while downstream demonstrations include dense robotic rewards, complex reasoning, and multi-hop reasoning.

2 Related work

Prior work uses multi-view imagery and dense semantic 3D representations to support LLM spatial reasoning. These approaches can be resource-intensive and depend on multi-view images, which are not always available.

  • Recent approaches reconstruct scenes from multi-view images as point clouds or neural fields and integrate dense semantic features into LLMs.
  • Explicitly constructing 3D representations with dense semantic features is resource-intensive, and multi-view images are not always available.
  • Integrating 3D representations into LLMs also introduces a modal gap.

3 Method

SpatialRGPT builds 3D scene graphs from single images, converts them into region-aware spatial QA data, and integrates regional and depth representations into a VLM. Its pipeline combines open-vocabulary perception, metric geometry, camera calibration, and multimodal tokenization.

  • SpatialRGPT builds 3D scene graphs from single images and supports 2D and 3D spatial understanding through region-aware visual representation learning.
  • The scene-graph pipeline filters images, detects and segments open-vocabulary objects, estimates metric depth, calibrates cameras, and lifts objects into 3D.
  • 3.1 3D Scene Graph from Single 2D Images: Figure 1’s curation pipeline uses image tagging, GroundingDino detection, and segmentation models to obtain precise object masks and boxes.
  • 3.1 3D Scene Graph from Single 2D Images: Scene-graph edges encode relative relations such as left and behind, plus metric direction, direct distance, horizontal distance, and vertical distance.
  • 3.2 Spatial Question Answering: Template-based and LLM-based QA generation is blended to increase instruction diversity and support generalized, complex spatial reasoning.
  • 3.2 Spatial Question Answering: The Open Spatial Dataset contains 1M unique images, 5M open-vocabulary regions, 8M template-based QAs, and 700K LLM-based QAs.
  • 3.3 VLM Architecture: The architecture combines a visual encoder, region-feature extractor, linear connectors, and an LLaMA2-7B language model.
  • 3.3 VLM Architecture: A relative-depth plugin supplies depth maps alongside RGB images, while special region and depth tokens interleave their embeddings with image and text tokens.

4 Experiments

Experiments evaluate SpatialRGPT on 3D spatial reasoning, general VLM benchmarks, region cognition, and real-world applications. Across these settings, it outperforms baselines on spatial tasks, preserves general VQA performance, and supports complex reasoning and region-aware reward annotation.

  • 3D Spatial Reasoning Benchmarks: SpatialRGPT-Bench evaluates qualitative and quantitative 3D spatial reasoning across diverse environments and model categories.The benchmark uses ground-truth 3D annotations, while Table 1 reports success rates and absolute relative error for blind LLMs, VLMs, and region-aware VLMs.
  • 3D Spatial Reasoning Benchmarks: SpatialRGPT significantly outperforms baselines on qualitative success rate and achieves the lowest absolute relative error on quantitative questions.Quantitative success is defined as answers within ±25% of ground truth; the evaluation excludes non-numeric answers from relative-error calculations.
  • Public Vision-language Benchmarks: SpatialRGPT variants perform similarly to or slightly better than VILA-1.5-3B on general VQA benchmarks, without compromising general VQA performance.The reported comparison includes VQA-v2 and MMVet, where the variants are slightly better than the baseline.
  • 3D Spatial Reasoning Benchmarks: SpatialRGPT achieves over 20% accuracy gain compared to GPT-4V-Turbo on BLINK’s Relative Depth Benchmark despite no task-specific training.The benchmark tests point-level depths, while training did not specifically include point-level region inputs or questions.
  • Real-world Applications: SpatialRGPT demonstrates complex spatial reasoning and multi-hop reasoning, including identifying objects through chained spatial relations.Examples compare its complex spatial responses with GPT-4V and show multi-hop reasoning despite the absence of specifically trained multi-hop tasks.
  • Real-world Applications: SpatialRGPT functions as a region-aware dense reward annotator for robotics, with estimated distance decreasing monotonically as a fingertip approaches a target.Region specification addresses ambiguity in language-based targeting when scenes contain identical objects or small specific regions.

5 Discussion

SpatialRGPT improves VLM spatial reasoning by combining region representation, flexible depth integration, and scene-graph-based 3D knowledge. It supports complex spatial relations and robotic dense-reward annotation, but its AABB representation can be inaccurate.

  • SpatialRGPT combines local region representation with flexible depth integration to perceive spatial arrangements at local and global scopes.Its data pipeline supports learning 3D spatial knowledge from scene graphs.
  • The framework improves performance on spatial reasoning tasks and reasons about complex spatial relations.
  • SpatialRGPT can function as a dense reward annotator for robotic applications.
  • Axis-Aligned Bounding Boxes can produce inaccurate label representations.Oriented bounding boxes require precise object pose estimation, while human labeling is more accurate but resource-intensive.

A Ablation Study on Augmented SpatialRGPT-Bench

Augmenting and rephrasing SpatialRGPT-Bench questions and answers tests performance under wording changes. SpatialRGPT consistently outperforms baseline models in this setting.

  • SpatialRGPT consistently outperforms baseline models when benchmark questions and answers differ from the training data.The evaluation augments and rephrases both questions and answers using GPT-4.

B Ablation Study on Metric-Scale Width and Height Data

Adding metric-scale width and height data creates a trade-off between size-related and object-distance questions. It slightly improves size accuracy but slightly worsens horizontal and vertical distance accuracy.

  • Adding width and height data slightly improves accuracy on size questions.The affected categories include big/small, tall/short, and wide/thin.
  • Adding width and height data slightly worsens accuracy on horizontal and vertical object-distance questions.The ablation suggests size information can make distance measurements less clear.

C Ablation Study on Bounding Box Types

The bounding-box ablation compares axis-aligned and oriented boxes using mean squared error against Omni3D ground truth.

  • The study evaluates axis-aligned versus oriented bounding boxes using MSE against Omni3D ground truth.

D Ablation Study on Different Input Modalities

The ablation compares box and mask region inputs, finding nearly equivalent benchmark performance, while random modality training appears effective.

  • Using boxes caused a slight performance drop, but results remained generally very close to mask-input performance.The comparison evaluates alternative region modalities on SpatialRGPT-Bench.
  • SpatialRGPT supports both bounding boxes and masks during inference.
  • The similar box and mask results suggest that random modality training is effective.

E Statistics and Samples of SpatialRGPT-Bench

SpatialRGPT-Bench summarizes QA categories, data sources, and object counts, and includes samples spanning its benchmark data.

  • SpatialRGPT-Bench organizes its QA data into 12 distinct types, evenly divided between relative relationships and metric measurements.
  • Some datasets, including SUNRGBD, emphasize closeobject scenarios.
  • Figure 8 presents six samples from SpatialRGPT-Bench.
  • The benchmark statistics report category counts, source counts, and object counts.

F Implementation Details for Data Pipeline

The data pipeline filters images, reconstructs and cleans point clouds, canonicalizes geometry, and synthesizes region-aware spatial QA with multimodal representations.

  • Filtering: The pipeline filters unsuitable images using CLIP-based open-vocabulary classification, excluding screenshots, paintings, collages, and text images.The filtering labels also identify indoor and outdoor photographs and close-up or product images.
  • Depth and calibration: Metric3Dv2 depth estimation and WildCamera camera intrinsics support reconstructed geometry in natural images.Predicted normal maps serve as a proxy for assessing reconstructed geometry edges.
  • Point-cloud processing: Point clouds are denoised and processed into scene graphs before spatial QA generation.The denoising steps include outlier removal, downsampling, and DBSCAN-based filtering.
  • Canonicalization: Canonicalization uses pitch and roll to transform point clouds with an inverse rotation, aligning the ground surface with the z-axis.This makes axis-aligned bounding boxes represent object height and vertical distance more accurately.
  • QA synthesis: Template-based QA synthesis covers distance, left-right, and clock-direction questions with corresponding answer templates.The appendix provides examples of question and response templates for generated spatial QA.
  • Model and training implementation: The model uses separate linear connectors for image and depth embeddings, while region features are extracted through refinement and MaskPooling.Additional instruction-tuning data comes from general, LAN-style, and region-level datasets.
  • Dataset construction: The Open Spatial Dataset pipeline processes OpenImages through filtering, calibration, depth estimation, QA synthesis, and verification.The reported processing setup uses 8 GPUs for preprocessing and human verification for benchmark annotations.

K More Discussion on Limitations

The discussion identifies geometric limitations in bounding-box estimation and broader risks and evaluation challenges associated with SpatialRGPT as a visual assistant.

  • Geometric limitations: Oriented bounding boxes are preferred for accurate detection, but PCA-based estimates can be inaccurate with incomplete, noisy single-view point clouds.Axis-aligned boxes may produce dimensions that differ from oriented boxes.
  • Geometric limitations: Current OBB methods cannot handle extreme cases where objects are partially elevated.More precise pose-based OBB estimation remains difficult because accurate open-vocabulary object poses are unavailable.
  • Broader impacts: SpatialRGPT inherits risks including hallucinations, base-model biases, and energy consumption during scaling.
  • Broader impacts: Accurately evaluating spatial information remains challenging, especially for safety-sensitive robotics applications.
  • Data and licensing: The paper reports that OpenImages is released under Apache License 2.0 and includes Unsplash images under the Unsplash License.
Loading 2406.01584v3…