Source-linked AI summary
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, Fei Xia
TL;DR
VLMs remain limited in 3D spatial reasoning, and the paper attributes this gap to insufficient spatial knowledge in scalable training data. SpatialVLM automatically generates spatial annotations and VQA data from internet-scale real-world images, then trains VLMs on the resulting mixture. The trained model improves qualitative and quantitative spatial reasoning and supports chain-of-thought reasoning and robotics applications.
Problem
VLMs still lack reliable 3D spatial reasoning, while common scalable image-caption training data contains limited spatial information.
Method
SpatialVLM combines off-the-shelf vision models to generate dense 3D annotations from real-world images and trains VLMs on captioning, VQA, and spatial reasoning data.
Results
SpatialVLM improves qualitative spatial answering and quantitative estimation and supports spatial chain-of-thought reasoning and robotics applications.
Takeaways & Limitations
The framework demonstrates that internet-scale real-world spatial data can support VLM capabilities in complex reasoning and robotics tasks.
Takeaways & Limitations
The direct spatial queries use a finite set of templates, and the approach assumes that current architecture is not the fundamental limitation on spatial reasoning.
Abstract
from arXiv · showhide
Understanding and reasoning about spatial relationships is a fundamental capability for Visual Question Answering (VQA) and robotics. While Vision Language Models (VLM) have demonstrated remarkable performance in certain VQA benchmarks, they still lack capabilities in 3D spatial reasoning, such as recognizing quantitative relationships of physical objects like distances or size differences. We hypothesize that VLMs' limited spatial reasoning capability is due to the lack of 3D spatial knowledge in training data and aim to solve this problem by training VLMs with Internet-scale spatial reasoning data. To this end, we present a system to facilitate this approach. We first develop an automatic 3D spatial VQA data generation framework that scales up to 2 billion VQA examples on 10 million real-world images. We then investigate various factors in the training recipe, including data quality, training pipeline, and VLM architecture. Our work features the first internet-scale 3D spatial reasoning dataset in metric space. By training a VLM on such data, we significantly enhance its ability on both qualitative and quantitative spatial VQA. Finally, we demonstrate that this VLM unlocks novel downstream applications in chain-of-thought spatial reasoning and robotics due to its quantitative estimation capability. Project website: https://spatial-vlm.github.io/
1. Introduction
VLMs perform well across many vision-language tasks but still struggle with 3D spatial reasoning, which limits their use in spatially demanding applications. SpatialVLM addresses this gap by generating large-scale spatial data from real-world images and training VLMs to support qualitative, quantitative, and multi-step spatial reasoning.
- Motivation: VLMs still struggle to understand 3D object positions and spatial relationships despite broad progress across vision-language tasks.Spatial reasoning is relevant to VQA, robotics, AR, reward annotation, and success detection.
- Motivation: The authors hypothesize that limited spatial reasoning primarily reflects insufficient spatial information in scalable training datasets rather than a fundamental architectural limitation.Common image-caption datasets contain limited spatial information, while embodied data and high-quality 3D annotations are difficult to obtain.
- Approach: The framework targets real-world diversity by extracting spatial information directly from internet images rather than relying mainly on rendered scenes with semantic annotations.The authors identify object richness and 3D relationships as limitations of many prior data-generation efforts.
- Approach: SpatialVLM combines open-vocabulary detection, metric depth estimation, semantic segmentation, and object-centric captioning to create dense 3D annotations from real-world images.The generated data is converted into a mixture of captioning, VQA, and spatial reasoning training data.
- Results: Experiments show improved qualitative spatial answering, reliable quantitative estimation despite noisy data, and support for spatial chain-of-thought reasoning and robotics applications.The model can serve as an open-vocabulary reward annotator for rearrangement tasks and coordinate with a language model on complex spatial problems.
- Contributions: The paper contributes an automatic internet-scale framework for 3D spatial VQA data, studies training recipes, and demonstrates new reasoning and robotics capabilities.It specifically examines data quality, training pipelines, and visual-encoder freezing or unfreezing.
2. Related Work
Related work spans spatial reasoning methods, grounding studies, and vision-language datasets, while SpatialVLM’s pipeline synthesizes spatial supervision from internet images. The pipeline combines image filtering, expert-model annotations, 3D lifting, ambiguity reduction, and question generation.
- Spatial reasoning: Prior spatial reasoning approaches use SLAM, depth estimation, explicit spatial memories, or structured scene graphs to represent spatial relationships.Scene graphs provide interpretable structures for learning statistical relations, but VQA systems must use them to answer spatial questions.
- Model grounding: Research on grounding notes that language and vision-language models can lack grounding for physical, embodied, physics, social, and spatial reasoning tasks.Interactive world experience has been studied as a way to improve grounding.
- Datasets: Vision-language datasets commonly emphasize VQA benchmarks, scene understanding, detection, segmentation, or object identification, with some works targeting spatial relations directly.These datasets cover tasks such as object relations, but the passage does not characterize their scale or metric-space coverage.
- SpatialVLM pipeline: Figure 2’s synthesis pipeline filters internet images, applies expert models for segmentation, depth, and captions, lifts images into 3D point clouds, resolves ambiguity, and generates spatial questions and answers.The pipeline uses CLIP filtering and caption clustering before synthesizing millions of examples.
3. SpatialVLM
SpatialVLM builds a large-scale pipeline that converts real-world 2D images into 3D spatial supervision and trains VLMs on qualitative and quantitative spatial questions. The resulting system supports direct spatial reasoning and enables multi-step reasoning when coordinated by an LLM.
- Spatial grounding: The pipeline combines open-vocabulary detection, metric depth estimation, semantic segmentation, and object-centric captioning to extract spatial contexts from images.It lifts 2D information into metric-scale 3D point clouds and canonicalizes them using horizontal-surface segmentation.
- Spatial grounding: Ambiguity resolution uses fine-grained, variable-length object captions and semantic post-processing to distinguish similar objects.The approach avoids coarse fixed-category detectors and can augment or reject ambiguous captions.
- Spatial VQA dataset: The dataset synthesizes qualitative and quantitative questions about spatial relations, with answers computed from segmented point clouds and 3D bounding boxes.It covers 38 question types, approximately 20 question templates and 10 answer templates per type, and applies human-aligned rounding.
- Spatial VQA dataset: 2 billion direct spatial reasoning QA pairs are generated from 10 million images, split evenly between qualitative and quantitative questions.Diverse object captions, distance units, question types, and phrasing support mixing the examples with other captioning or question-answering datasets.
- Learning spatial reasoning: SpatialVLM is trained as a VLM that answers spatial queries directly, including both binary predicates and quantitative estimations.The model is trained with a mixture of the original PaLM-E dataset and the synthetic spatial dataset.
- Learning spatial reasoning: An LLM can coordinate SpatialVLM by decomposing complex questions into simpler spatial queries and composing the answers into multi-hop reasoning.This Chain-of-Thought Spatial Reasoning setup supports tasks such as determining whether objects form a geometric shape or satisfy spatial constraints.
4. Experiments
The experiments evaluate SpatialVLM’s spatial VQA performance, general VQA effects, training choices, and downstream reasoning capabilities. Across these studies, spatial supervision improves qualitative and quantitative spatial reasoning while preserving comparable general VQA performance.
- Evaluation setup: The benchmark evaluates binary qualitative and quantitative spatial questions using human-annotated answers on unseen WebLI images.Qualitative answers are judged by human raters, while quantitative answers are assessed by numeric validity and closeness to ground truth.
- Spatial VQA performance: SpatialVLM significantly outperforms baselines on qualitative spatial VQA, including GPT-4V, after training with synthetic spatial VQA data.The results suggest that large, high-quality spatial reasoning data is important for spatial capabilities absent from standard pretraining mixtures.
- Spatial VQA performance: 99.0% of SpatialVLM outputs have a valid quantitative format, and its distance estimates are closer to human ground truth than baseline estimates.The quantitative evaluation uses the percentage of answers within half to twice the ground-truth value as an accuracy measure.
- Robotics applications: SpatialVLM achieves accurate and monotonically decreasing distance estimates as a robot gripper approaches a coke can.This sequence tests whether estimated distance changes consistently with the observed motion.
- General VQA effects: SpatialVQA co-training preserves comparable OKVQA performance and slightly improves VQA-v2 performance relative to vanilla PaLM 2-E.The table reports a 2.4% VQA-v2 improvement for a same-parameter PaLM 2-E model trained with SpatialVLM data, while OKVQA does not benefit.
- Training choices: Unfreezing the ViT is considerably better for fine-grained distance estimation, although frozen and unfrozen models are comparable for rough half-to-twice estimates.SpatialVLM reaches 8.4% accuracy for predictions within the 0.9× to 1.1× range of human annotation, despite noisy annotations.
- Downstream reasoning: SpatialVLM supports multi-step spatial reasoning, including using GPT-4 with the model as a submodule to assess whether objects form an isosceles triangle.The experiment tests whether enhanced elemental spatial answers support more complex reasoning tasks.
5. Conclusion
The paper addresses VLM spatial-reasoning limitations by generating 3D spatial VQA data from Internet-scale real-world images and training VLMs on it. SpatialVLM supports more complex spatial reasoning and robotics applications, while the authors note limitations from template-based queries and incomplete geometric grounding.
- Contribution: The paper constructs a framework for automatically generating 3D spatial reasoning VQA data from Internet-scale real-world images.The framework and training recipe include large noisy datasets and experiments with unfreezing the ViT.
- Applications: SpatialVLM extends beyond template-based direct queries to chain-of-thought reasoning and robotics reward annotation.The conclusion identifies these as demonstrated applications of a 3D spatial-aware VLM.
- Limitations: The direct spatial queries use a finite set of templates, and richer geometric primitives are needed to fully ground spatial reasoning in 3D geometry.This marks the paper’s scope boundary for query diversity and geometric coverage.
A.1. Additional Experiments and Details
The additional material describes human-annotated spatial VQA benchmarks and a chain-of-thought setup in which an LLM queries SpatialVLM for visual information. Examples show the system combining spatial measurements and relations to answer multi-step questions and produce robotic instructions.
- Human-annotated benchmark: 546 qualitative and quantitative question pairs were manually labelled for WebLi and robotic manipulation VQA.Annotators could accept a sampled question, write a more suitable one, or skip the image before entering an answer.
- Benchmarks: SpatialVLM’s qualitative and quantitative benchmarks cover spatial questions involving object relations, distances, and measurements.The supplementary figures provide example question-answer pairs for both VQA settings and robotic manipulation.
- Chain-of-thought reasoning: The chain-of-thought implementation prompts a large language model to query SpatialVLM for visual information while reasoning through a question it cannot directly see.The LLM decomposes the task into questions for a friend-like visual module, whose answers support the original response.
- Chain-of-thought reasoning: A trash-bin example combines left-right and depth relations to generate detailed instructions for moving a hand and cleaning a table.The model identifies the objects, determines the trash bin’s position relative to the coke can, and compares their depth.
A.2. Implementation Details
The implementation filters suitable images, extracts and cleans object-centered 3D point clouds, canonicalizes coordinates when possible, and synthesizes human-aligned spatial data. It also uses ambiguity handling and sampling strategies to support reliable metric estimates.
- Data filtering: Semantic filtering removes images unsuitable for spatial questioning and favors views close to the depth model’s effective field of view.CLIP labels identify scenes while excluding products, artworks, screenshots, text, and other unsuitable image types.
- 2D context extraction: The pipeline extracts object regions with a region proposal network, non-max suppression, and class-agnostic segmentation rather than fixed-category object detectors.This supports fine-grained descriptions such as “cake shaped like a house” and “cup cake in plastic container.”
- 3D context lifting: ZoeDepth and field-of-view estimation lift 2D images into metric-scale 3D point clouds, which are cleaned using outlier removal and DBSCAN clustering.The resulting structured points support subsequent shape and geometry analysis.
- Coordinate canonicalization: Coordinate canonicalization detects horizontal surfaces and transforms camera-frame points toward world coordinates for elevation and other ground-referenced questions.When insufficient horizontal-surface points are detected, canonicalization fails and dependent questions are not synthesized.
- Ambiguity removal: Ambiguity removal compares caption embeddings and augments or rejects captions when object references are too similar.This process uses CLIP cosine similarity to distinguish multiple objects with similar categories.
- Human alignment: Human alignment rounds distance answers to human-preferred values, while averaging multiple samples can provide finer-grained estimates for robotics.The authors note that depth and field-of-view estimation contain irreducible errors and that prompting for fixed precision is left for future work.
A.3. Question and Answer Template
The question-and-answer template system converts object descriptions into diverse qualitative and quantitative spatial queries. Templates cover viewer-relative relations, canonicalized vertical relations, object dimensions, and metric distance estimates with natural-language answers.
- Template generation: Template-based synthesis fills object captions into question and answer patterns for multiple spatial question types.The pipeline derives answers from descriptions of object pairs such as “the yellow banana” and “the cake in the shape of a house.”
- Predicates: Predicate templates represent binary spatial relations such as left, right, above, below, behind, and in front.Above and below require coordinate canonicalization, while viewer-relative predicates use the viewer’s perspective.
- Choice questions: Choice templates ask which object is more behind, more in front, taller, shorter, wider, or thinner.Canonicalization is required for tall and short comparisons, while the answer is an object name or an uncertainty phrase.
- Quantitative estimation: Estimation templates ask for distances, elevations, widths, and vertical distances using answers expressed in natural language with human-like rounding.Elevation and vertical-distance estimation require coordinate canonicalization.
- Distance questions: Distance templates vary wording across ordinary, vertical, and horizontal distance questions while preserving object placeholders and answer slots.The template sets include direct measurements, approximate estimates, and multiple paraphrases for the same relation.
- Quantitative questions: Quantitative questions provide fine-grained answers with numbers and units, using disambiguated object captions as their entity references.Examples ask how far one object is from another or where it lies relative to the camera.