RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

Chan Hee Song; Valts Blukis; Jonathan Tremblay; Stephen Tyree; Yu Su; Stan Birchfield

Source-linked AI summary

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, Stan Birchfield

arXiv:2411.16537v5Nov 25, 2024cs.CVcs.AIcs.CLcs.RO

TL;DR

Vision-language models still struggle with robotics-relevant spatial reasoning because existing data underrepresent nuanced relationships and reference frames. RoboSpatial addresses this gap with a large-scale 2D/3D dataset and automated annotations, and trained models outperform baselines across spatial benchmarks and robotic tasks.

Problem
VLMs struggle with nuanced spatial relationships and reference-frame reasoning needed for robotics, while existing training data provide insufficient spatial supervision.
Method
RoboSpatial constructs a large-scale 2D/3D robotics dataset from annotated indoor and tabletop scenes, covering multiple spatial relations and ego-, world-, and object-centric frames.
Results
Models trained with RoboSpatial consistently outperform baselines across held-out, manually collected, and external spatial reasoning benchmarks, including robotic tasks.
Takeaways & Limitations
RoboSpatial supports spatial relationship understanding, generalization to unseen relations, nuanced reference-frame inference, and real-world robotic applications.
Takeaways & Limitations
The top-down occupancy-map approach does not support containment questions requiring detailed volumetric modeling.

Abstract

from arXiv · show

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial information relevant to robotics. The dataset includes 1M images, 5k 3D scans, and 3M annotated spatial relationships, and the pairing of 2D egocentric images with 3D scans makes it both 2D- and 3D- ready. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robot manipulation.

1. Introduction

RoboSpatial addresses shortcomings in VLM spatial reasoning by providing robotics-specific data with multiple reference frames and spatial question types. Models trained on it outperform baselines across spatial reasoning and robotic evaluation settings.

Motivation: VLMs struggle with nuanced object-object and object-placement reasoning despite strong recognition and description abilities.Existing training data often lack the spatial information needed for embodied interaction.
Dataset: RoboSpatial targets this gap by converting annotated indoor and tabletop RGBD scenes into robotics-focused spatial question-answer pairs.The approach is motivated by the scarcity of suitable training data for robotic spatial reasoning.
Dataset: Questions cover spatial context, compatibility, and configuration, including point prediction and binary reasoning about placement and object relations.Examples include identifying usable table regions, checking whether an object fits, and testing relative positions.
Dataset: Each question is posed in ego-centric, world-centric, and object-centric reference frames to support flexible spatial reasoning.The dataset pairs approximately 1M images and 5k 3D scans with about 3M annotated spatial relationships and is both 2D- and 3D-ready.
Results: Models trained on RoboSpatial consistently outperform baseline methods across held-out, manually collected, and external spatial reasoning benchmarks.The evaluations include robotic rearrangement and indoor contextual question answering.

2. Related Work

Prior VLM spatial reasoning work addresses relationships in generic or simulated settings, while RoboSpatial extends this focus to actionable spatial understanding for robotics. Its automated 2D/3D dataset and evaluation resources target both in-domain and out-of-domain reasoning.

VLMs for Robotics: VLMs support perception, planning, manipulation, and action generation in robotics, but precise spatial reasoning remains a central integration challenge.Related applications include grounded scene understanding, task planning, keypoint prediction, and grasp pose prediction.
Spatial Understanding with VLMs: Existing spatial reasoning benchmarks often focus on simulations or generic images rather than robotics-specific settings.This limits direct evaluation of actionable spatial relationships relevant to robotic workflows.
Novelty: RoboSpatial introduces actionable spatial context and compatibility relations alongside spatial configuration for robotics-specific reasoning.The stated aim is direct application to task planning and verification.
Contribution: The paper provides a large-scale 2D/3D-ready training dataset and automated generation pipeline for teaching VLMs spatial reasoning.The resources support experiments on both in-domain and out-of-domain spatial reasoning datasets.

3. Approach

RoboSpatial automatically converts annotated RGBD scenes into spatial relations and question-answer pairs grounded in 3D geometry, camera parameters, and multiple reference frames. Its pipeline produces binary relation answers, image-space coordinates, grounding supervision, and roughly 3 million examples.

Dataset Generation: The pipeline converts scene datasets containing images, camera parameters, and oriented 3D boxes into image-question-answer-frame records.It also generates object-grounding annotations linking descriptions to projected 2D bounding boxes.
3.2.1. Stage 1: 3D Spatial Relation Extraction: Stage 1 extracts object-object and object-free-space relations from oriented 3D bounding boxes in ego, world, and object frames.Box locations and heading vectors define directional relations such as left, right, front, and behind.
3.2.1. Stage 1: 3D Spatial Relation Extraction: Camera extrinsics transform coordinates between frames, while intrinsics and extrinsics support consistent projection between 3D relations and 2D views.Visible object pairs are evaluated for spatial configuration while duplicate instances are avoided.
3.2.2. 2D Target Generation: Stage 2 projects free-space and compatibility targets into image coordinates using occupancy maps, raycasting, and virtual object placement.Compatibility requires nonintersection with existing boxes and at least a 10 cm margin along each axis.
Question-Answer Generation: Structured templates produce binary answers for configuration and compatibility, coordinate lists for context, and language-grounded questions designed to reduce linguistic shortcuts.The templates follow a target-relation-anchor-reference-frame format.
Scale: The pipeline generates around 3 million spatial relationships and associated question-answer pairs, an order of magnitude larger than prior spatial reasoning datasets.

4. Experiments

Experiments evaluate ROBOSPATIAL across held-out, out-of-domain, cross-dataset, and robotic manipulation settings. Models trained on ROBOSPATIAL show improved spatial reasoning, generalization, and real-world task performance, while evaluation criteria and modality comparisons have stated limitations.

Experimental Setup: Approximately 3M spatial QA pairs span 5k 3D scans and 1M images from indoor and tabletop environments.The dataset supports larger scenes for navigation and smaller object-centric setups for manipulation.
Experimental Setup: ROBOSPATIAL-Val contains 6,000 held-out questions, split evenly across three spatial relation types and evaluated with accuracy or geometric point inclusion.Coordinate predictions are counted correct when they fall within the convex hull of a reference point set derived from scene geometry.
In-Domain Evaluation: Fine-tuning existing 2D and 3D VLMs on ROBOSPATIAL improves every reported task on the held-out validation split.Table 3 compares RGB-only and specialized models, including VILA-1.5-8B, LLaVA-NeXT-8B, SpaceLLaVA-13B, RoboPoint-13B, Molmo-7B, and GPT-4o.
Evaluation Caveats: The convex-hull criterion is conservative, and comparisons between 2D and 3D VLMs remain confounded by differences in pretraining data and base architectures.Predictions just outside the geometric boundary are marked incorrect, while some 3D models may have prior exposure to overlapping indoor scan environments.
Cross-Dataset Generalization: Models trained on one environment type improve spatial reasoning on held-out datasets from the other environment type.The cross-dataset evaluation compares indoor-scene and tabletop training and test subsets.
Out-of-Domain Evaluation: ROBOSPATIAL-trained models generalize to ROBOSPATIAL-Home, BLINK, and SpatialBench, including spatial relationships and language configurations not explicit in training.The authors attribute this generalization partly to coverage of all six principal 3D directions and proximity relationships.
Real Robot Experiments: LLaVA-NeXT fine-tuned on ROBOSPATIAL achieves the highest robot-task success rate across all evaluated models.ROBOSPATIAL-trained models better interpret placement instructions and reduce failures from inaccurate spatial predictions; 2D-to-3D projection errors can turn a 2-pixel image shift into a 5–10 cm physical error.

5. Conclusion

The paper introduces a robotics-focused 2D/3D spatial understanding dataset and evaluation suite. Experiments report improved spatial reasoning, generalization to unseen relationships, nuanced reference-frame inference, and real-robot applicability.

Contribution: ROBOSPATIAL, ROBOSPATIAL-Val, and ROBOSPATIAL-Home form a large-scale 2D/3D spatial understanding training and evaluation suite tailored for robotics.The paper also describes an automatic pipeline for extending the dataset to new data sources and spatial relations.
Findings: Models trained with ROBOSPATIAL understand spatial relationships, generalize to unseen relationships, and infer nuanced reference frames.These capabilities are demonstrated across tasks requiring spatial understanding.
Robotics Relevance: Robot experiments demonstrate the applicability of ROBOSPATIAL to tasks requiring spatial understanding.The conclusion presents robotics as a broader application scope for the dataset.

Appendices

The supplementary material provides omitted details and clarifications, including limitations, dataset information, implementation details, and additional results.

Appendix Contents: The appendices cover limitations, dataset details, implementation details, and more results.These materials supplement the main text due to space constraints.

A. Limitations

ROBOSPATIAL has scope and integration limits: its spatial annotations emphasize horizontal placement, modular robot use, household indoor/tabletop scenes, and a constrained training subset. It also deliberately excludes some visual reasoning tasks while adding grounding supervision to reduce reference errors.

Scope boundaries: Containment questions are unsupported because top-down occupancy maps model horizontal placement rather than detailed volumetric relations.The limitation includes questions such as whether an object fits inside or under another object.
System integration: The models are deployed modularly, but joint training with robot trajectories is not explored.The authors identify tighter integration as a direction for future work.
Data coverage: ROBOSPATIAL covers indoor and tabletop household scenes but excludes humans and animals.This limits coverage of social or dynamic interaction scenarios, although the authors report transfer to BLINK.
Data scale and diversity: Training uses 900,000 sampled spatial relationships because of computational constraints, and tabletop sources have limited diversity.The authors plan to add more tabletop datasets.
Task selection: The benchmark excludes object counting and distance-only tasks from its spatial-understanding scope.The authors describe both as possible complementary or auxiliary tasks for future extensions.
Grounding support: An auxiliary grounding dataset supplies 100k question-answer pairs to reduce reference ambiguity and improve object identification.Descriptions are paired with 2D boxes projected from annotated 3D boxes using camera intrinsics and extrinsics.

B.4. Dataset Generation Details

The dataset-generation pipeline computes 3D spatial relationships and projects them into 2D image space, while annotating multiple reference frames and supporting point-based outputs. It also includes generated top-down maps from 3D bounding boxes.

Pipeline: The pipeline uses a two-stage process to compute 3D spatial relationships and project them into 2D image space.The expanded description clarifies the reasoning logic used for spatial annotation.
Map generation: Figure 5 illustrates a generated top-down map derived from 3D bounding boxes.The map is an example of the representation produced during dataset generation.
Reference frames: Reference-frame annotations label spatial configurations from egocentric, object-centric, and world-centric perspectives.Object-centric directions use each oriented 3D bounding box’s heading vector to define front, left, right, and behind.
Output formats: The pipeline uses point prediction for integration with robot setups and also supports mask-based outputs for future extensions.Mask-based outputs are described as an available extension rather than the primary representation.

C. Implementation Details

Implementation uses instruction tuning of 2D and 3D VLMs, with benchmark and ablation evaluations covering frame-specific accuracy, general multimodal capabilities, and auxiliary grounding. Baselines include models used zero-shot when fine-tuning was unavailable.

Model training: VILA, LLaVA-NeXT, SpaceLLaVA, and LEO are fine-tuned with repository scripts or checkpoints and mostly default hyperparameters.LEO receives both image and 3D point-cloud inputs, while model selection uses validation accuracy where specified.
Evaluation: ROBOSPATIAL fine-tuning improves every reported ROBOSPATIAL-Val task for all methods in the per-frame evaluation.Table 8 reports accuracy by reference frame and marks higher performance as better.
Additional evaluations: Tables 10 and 11 evaluate general multimodal capability and the auxiliary grounding dataset’s effect on ROBOSPATIAL-Val.The table captions identify the evaluated benchmarks and ablation focus.
Baselines: Molmo and GPT-4o serve as zero-shot baselines because fine-tuning scripts were unavailable.The limitation concerns implementation availability rather than model capability.
Reference-frame behavior: Before training, 2D VLMs struggle with world-centric frames while 3D VLMs comparatively excel there, but 3D models lag on ego-centric and object-centric frames.The authors attribute the latter gap to 3D models’ lack of visually injected perspective information from complete point clouds.

D.2. Data Scaling

Increasing the number of spatial annotations while keeping images fixed improves ROBOSPATIAL-Val performance, and the authors plan to expand environmental diversity. General multimodal performance is maintained or slightly improved after spatial fine-tuning.

Annotation scaling: Increasing annotations with images fixed improves average accuracy on ROBOSPATIAL-Val.The scaling experiment is conducted with LLaVA-NeXT and reports average validation accuracy.
Future scaling: The authors plan to apply the data-generation pipeline to more diverse indoor and tabletop environments.The stated goal is to further improve model performance.
Broader capabilities: ROBOSPATIAL-trained models maintain or slightly improve performance on MMMU, MME, and MMBench.These benchmarks assess whether spatial fine-tuning affects broader commonsense and factual multimodal reasoning.

D.4. Ablation of the Auxiliary Grounding Dataset

The auxiliary grounding dataset provides only a small improvement over the base model, while ROBOSPATIAL yields substantially larger gains; combining both datasets performs best. The robot experiments use point-based picking and depth-based placement without cuRobo failures.

+2.1 improvement comes from training on the auxiliary dataset alone, far below ROBOSPATIAL’s gains.
Combining the auxiliary dataset with ROBOSPATIAL produces the best performance, suggesting object localization complements spatial supervision.
Picking maps a predicted point to an object with SAM 2, while placing converts the pixel’s depth value into a 3D coordinate.
The robot experiments report no failures caused by cuRobo and were designed around behaviors the system can execute.

D.5.2. Additional Results

ROBOSPATIAL-trained models generally outperform baselines in robot experiments and qualitative spatial-reasoning evaluations, including generalization to unseen relationships. However, stacked items and cluttered-scene object grounding remain challenging, and some mixed training cases hurt performance.

Robot experiments: ROBOSPATIAL-trained models consistently outperform baselines in most robot experiments despite prompts not being optimized for them.
Robot experiments: The models struggle with stacked items, motivating further augmentation with diverse layouts.
Robot experiments: Mixing RoboPoint and ROBOSPATIAL training data can adversely affect performance, particularly for RoboPoint, possibly through grounding side effects.
Qualitative comparisons: ROBOSPATIAL-trained models show spatial understanding on ROBOSPATIAL-Home and can outperform GPT-4o, but cluttered-scene grounding often limits improvement.
Qualitative comparisons: The trained model generalizes to unseen spatial relationships in BLINK-Spatial, including distance-related relations such as touching.
Additional examples: The additional robot examples include questions about containment, stacks, empty space, object height, and fitting objects into rows or boxes.