PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialBot: Precise Spatial Understanding with Vision Language Models

Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, Bo Zhao

arXiv:2406.13642v7cs.CV

TL;DR

VLMs remain limited in spatial understanding because they are mainly trained on RGB imagery and lack depth-focused training data. SpatialBot combines RGB and depth inputs with multi-level SpatialQA training and SpatialBench evaluation, then applies the learned capability to embodied tasks. Experiments report improved spatial understanding, general VLM benchmark performance, and more accurate robot manipulation with depth information.

  • Problem

    VLMs perform strongly on 2D image understanding but struggle with spatial information needed for embodied tasks, while existing training datasets lack depth maps and depth-related tasks.

  • Method

    SpatialBot combines RGB and depth images, trains progressively on multi-level SpatialQA and SpatialQA-E datasets, and evaluates spatial capabilities with SpatialBench.

  • Results

    SpatialBot shows promising performance across general visual recognition, spatial understanding, and robot manipulation, with depth improving pick-and-place accuracy.

  • Takeaways & Limitations

    The experiments support using depth-aware VLMs for spatial Visual QA and embodied robot manipulation.

  • Takeaways & Limitations

    Text-only VLM outputs are not fully prepared for accurate pixel-level monocular depth estimation, for which an extra decoder may be needed.

Abstract

from arXiv · show

Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose SpatialBot for better spatial understanding by feeding both RGB and depth images. Additionally, we have constructed the SpatialQA dataset, which involves multi-level depth-related questions to train VLMs for depth understanding. Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities in spatial understanding at different levels. Extensive experiments on our spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks, demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The model, code and data are available at https://github.com/BAAI-DCAI/SpatialBot.

1. Introduction

VLMs perform well on general 2D understanding but struggle to comprehend spatial information from images, limiting their use in embodied tasks. SpatialBot addresses this gap with RGB-D training, evaluation benchmarks, and robot manipulation experiments.

  • VLMs struggle to comprehend spatial information from 2D images, despite strong general visual understanding and reasoning.
  • Depth understanding is limited because popular VLMs are trained primarily on RGB images and existing tuning datasets lack depth maps and depth-related tasks.
  • Indoor navigation and manipulation require millimeter-level precision, while outdoor tasks require broader depth ranges.
  • SpatialBot progressively trains general spatial understanding with SpatialQA and embodied manipulation with SpatialQA-E.
  • SpatialBench evaluates spatial understanding and reasoning, while experiments cover spatial benchmarks, general VLM benchmarks, and robot manipulation.

2. Related work

Related work establishes that spatial understanding requires information beyond 2D RGB images, especially for precision robotics. SpatialBot uses depth maps and RGB-D data to address this need in general and embodied tasks.

  • Existing VLMs primarily address perception, reasoning, and OCR, while some encoder-decoder models perform pixel-level grounding.
  • Spatial understanding extends beyond 2D RGB images and is particularly important for precision tasks such as robotic grasping.
  • Monocular RGB methods can estimate depth and detect objects, but their metric-depth accuracy is limited.
  • SpatialVLM and Spatial-RGPT infer spatial relationships from 2D images, whereas robotic tasks require sensor-based depth information.

3. SpatialBot

SpatialBot incorporates depth into VLM inputs through RGB-D processing, depth encoding, object-level depth descriptions, and staged spatial reasoning. Its design preserves metric depth while accommodating indoor and outdoor ranges.

  • Depth information is added to RGB inputs because depth maps are easier to collect and process than point clouds and are widely available from inexpensive RGB-D cameras.
  • 3.1. Depth Map Encoding: The encoding stores raw depth in millimeters from 1mm to 131.071m using uint24 or three-channel uint8 representations.
  • 3.1. Depth Map Encoding: The three-channel representation distributes depth across channel units of 2^0, 2^5, and 2^10 millimeters to preserve information across ranges.
  • 3.1. Depth Map Encoding: When raw sensor depth is unavailable, ZoeDepth estimates metric depth for indoor and outdoor scenes rather than using relative-depth models.
  • 3.2. Depth Description of an Object: SpatialQA describes objects with maximum, minimum, mean, and center depth values when masks are available.
  • 3.2. Depth Description of an Object: Because standard VLM output cannot provide pixel-level information, inaccurate masks trigger fallback to the bounding-box center depth.

3.3. Image Sources

SpatialQA is built by extending existing RGB VQA resources with depth-related questions and a progressive pipeline for learning depth understanding, RGB-depth alignment, and higher-level spatial reasoning. SpatialQA-E extends this focus to robot manipulation with spatially specified instructions.

  • Image Sources: SpatialQA builds on Bunny 695k, using COCO and VG image-question data and adding depth-related question-answer pairs.Bounding boxes are used with SAM to obtain masks while preserving bounding-box limits.
  • Dataset Pipeline: The dataset pipeline progressively teaches VLMs to understand depth, align depth with RGB, and apply depth to complex reasoning tasks.The pipeline is designed so existing RGB datasets can be converted into RGB-Depth datasets.
  • Low Level: Low-level tasks ask models to query point depth, associate coordinates with image pixels, and describe depth maps or depth-only images.The encoder and projector are also expected to learn joint RGB-depth representations.
  • Middle Level: Middle-level tasks use depth for proximity judgments and object or region descriptions based on center, minimum, maximum, and mean depth.These tasks also train relationships between image and depth inputs.
  • High Level: High-level tasks apply depth understanding and RGB-depth alignment to grounding, counting, enumeration, and spatial reasoning.The depth map is intended to clarify object boundaries and surroundings.
  • SpatialQA-E: SpatialQA-E contains 2,000 robot-manipulation episodes involving grasping, obstacle avoidance, and placement under spatial language instructions.Relationships include positional, size, comparative, superlative, and perspective-dependent descriptions, plus printed-object illusions.

3.6. SpatialBot in Embodiment Tasks

For embodiment tasks, SpatialBot operates as a vision-language-action model over multi-frame RGB or RGB-D observations and predicts robot actions for spatially specified manipulation. Its data and interfaces support depth-aware perception, dataset construction, and benchmark evaluation.

  • Model Interface: SpatialBot predicts robot actions from history and current image observations, supporting multi-frame RGB or RGB-D inputs.The model is described as a Vision-Language-Action model for embodiment tasks.
  • Action Representation: Actions are represented as a 7-DoF vector covering translational and rotational pose changes plus gripper closure.The model outputs textual action parameters that are decoded into robot-control signals frame by frame.
  • Dataset Construction: The SpatialQA conversion pipeline requires RGB-depth information, image conversations or captions, object boxes or masks, and detailed object descriptions.These components support conversion of image sources into RGB-Depth training data.
  • Evaluation: SpatialBench evaluates spatial understanding through object contact or reachability, spatial relationships, counting, enumeration, and size comparison questions.The benchmark uses yes/no or multiple-choice formats and includes depth and proximity questions.
  • Depth API: SpatialBot uses a Depth API that queries a point’s depth from the corresponding depth map when the model emits the Depth(point) format.The queried depth value is then fed back into the model.

4. Experiments

Experiments evaluate SpatialBot on depth understanding, spatial benchmarks, general VLM benchmarks, and robot manipulation. The reported results show comparable SpatialBench performance to GPT-4o for multiple model sizes and more accurate RGB-D manipulation than RGB-only input.

  • Overall Findings: The experiments collectively report depth comprehension, improvements on general VQA tasks, and benefits for robot manipulation.These evaluations span SpatialBench, general benchmarks, and RT-X-related embodied tasks.
  • Implementation: SpatialBot is built on Bunny-based VLMs with multiple base language models and is trained using large-scale image-text pretraining followed by fine-tuning.The implementation uses SigLIP or CLIP vision encoders depending on the task and adds LoRA during fine-tuning.
  • Depth Understanding: SpatialBot is evaluated on low-, middle-, and high-level depth understanding, including metric depth, proximity relationships, and high-level spatial tasks.Depth API answers are reported for depth and proximity evaluation.
  • SpatialBench: 3B, 4B, and 8B SpatialQA-trained models reach results comparable with GPT-4o on SpatialBench.SpatialBench covers positional relationships, object existence, reaching, and size comparison tasks.
  • General Benchmarks: Depth-aware training is evaluated across general VLM benchmarks including MME, MMBench, SEED Bench Image, VQAv2, GQA, and POPE.RGB-D inputs are used on MME perception and GQA, which include counting, existence, and position questions.
  • Embodiment Tasks: With depth information, SpatialBot can pick and place objects more accurately than its RGB variant on real-robot manipulation tasks.The robot experiments use SpatialQA-E fine-tuning and multi-frame history to predict end-effector position changes.

5. Monocular Depth Estimation by VLM

The paper tests whether VLMs can estimate metric depth from monocular RGB images using text-only outputs. It reports that this capability remains insufficiently accurate and does not add a separate decoder because that would limit SpatialQA’s generality.

  • Evaluation: SpatialBot is asked to predict the depth of points or objects for monocular depth estimation.The evaluation compares estimated depth with ground-truth depth values using a depth-accuracy measure.
  • Limitation: VLMs are not fully prepared for accurate monocular depth estimation through text-only outputs.The paper cites extra decoders as potentially necessary for accurate pixel-level depth generation.
  • Design Boundary: The method does not add a separate depth decoder because the new structure would limit SpatialQA’s generality.This preserves the standard VLM setup despite the limitation in text-only metric depth prediction.

6. SpatialBench Metadata

SpatialBench evaluates spatial understanding through positional relationships, paired consistency, and object counting. Its protocols use generated questions and human annotations to define correctness.

  • SpatialBench measures positional relationships with 34 GPT-4o-generated multiple-choice questions, using the correct selection ratio as accuracy.
  • A paired question is correct only when the model answers both positive and negative questions correctly.
  • For object counting, human experts annotate object quantities and models are asked to count them.

7. Data Generation in SpatialQA

SpatialQA is generated from approximately 50,000 images covering depthmap, spatial, and robot scene understanding. Separate prompts, data protocols, and human annotations support dataset construction.

  • SpatialQA uses about 50k images for depthmap understanding, spatial understanding, and robot scene understanding.
  • The dataset documents data sources and selection protocols in Table 3 and task-specific GPT prompts in Table 4.
  • Sample images and generated questions are shown in Figures 9, 10, and 11, while human experts annotate three bounding boxes per RTX image.

8. Dataset, Model and Benchmark Usage

The paper releases SpatialBot resources, datasets, benchmarks, and checkpoints through official repositories and Hugging Face. These materials cover implementation, training data, evaluation data, and pretrained models.

  • The official SpatialBot repository provides metadata, code, scripts, checkpoints, licenses, and resource links.
  • SpatialQA, SpatialQA-E, and SpatialBench are available for download through Hugging Face.
  • A SpatialBot-3B checkpoint based on Phi-2 and SigLip is available, alongside pretrained checkpoints in the Bunny model zoo.

9. Conclusion

SpatialBot is presented as a family of VLMs trained on SpatialQA and SpatialQA-E for depth understanding and robot manipulation. Experiments across benchmarks and deployment settings support its reported effectiveness against competitors.

  • SpatialBot is trained on SpatialQA and SpatialQA-E to support depth understanding and precise robot manipulation in embodied AI.
  • The model uses depth inputs for spatial understanding and reasoning in visual question answering and embodiment tasks.
  • SpatialBench evaluates spatial knowledge across multiple aspects, while experiments cover the benchmark, general VLM benchmarks, and robot manipulation deployment.
Loading 2406.13642v7…