Source-linked AI summary
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox
TL;DR
Robots need precise action points, while VLMs struggle to express such actions through language. ROBOPOINT uses automatically generated synthetic affordance data to instruction-tune a VLM, which outperforms prior methods across spatial affordance and downstream tasks.
Problem
VLMs struggle to precisely articulate robotic actions from language instructions, despite their growing use for robot control.
Method
ROBOPOINT instruction-tunes a VLM with real-world VQA data and automatically generated synthetic data for relational image-keypoint affordance prediction.
Results
ROBOPOINT outperforms state-of-the-art VLMs and visual prompting methods on spatial affordance prediction and downstream robotic applications.
Takeaways & Limitations
The model supports precise action suggestions across manipulation, navigation, and augmented reality without requiring large-scale expert demonstrations.
Takeaways & Limitations
ROBOPOINT provides no confidence estimates for point predictions, and the number of output points is not controllable.
Abstract
from arXiv · showhide
From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Project website: https://robo-point.github.io.
1 Introduction
ROBOPOINT addresses the challenge of converting relational language instructions into precise spatial action points for robotics. Its autonomous synthetic-data pipeline supports a scalable VLM applicable to manipulation, navigation, and augmented reality.
- VLMs provide useful semantic guidance for robotics but struggle to articulate precise actions using language.
- ROBOPOINT predicts affordance points that satisfy spatial relations specified by language instructions.
- An autonomous pipeline generates diverse ground-truth action points by computing camera-perspective relations and sampling object and surface intersections.
- The resulting model outperforms GPT-4o, LLaVA-NeXT, Qwen-VL, and SpatialVLM on relational references and cluttered object rearrangement without losing standard VQA accuracy.
2 Related Work
Prior work studies spatial reasoning, affordance representations, and language-based robotic planning, but existing approaches often use coarse relations, action-specific keypoints, or external execution mechanisms. ROBOPOINT instead targets general spatial affordance prediction with 2D keypoints.
- Spatial Reasoning: Spatial reasoning benchmarks commonly test coarse relations, while state-estimation and symbolic methods generalize poorly to novel objects.
- Affordance Prediction: Affordance links visual observations to possible manipulations and can be represented through segmentation, dense descriptors, or keypoints.
- Affordance Prediction: ROBOPOINT uses 2D keypoints because they can be readily converted into language format.
- Zero-shot Language Models for Robotics: Language-based robotics methods produce plans or fine-grained outputs, but alternatives rely on predefined primitives, iterative image sampling, 3D value maps, or action-specific keypoints.
3 Method
ROBOPOINT is instruction-tuned to predict image-space affordance points satisfying relational language prompts. Its training combines synthetic and real-world data, while deployment projects predicted 2D points into 3D targets for robot execution.
- Training: ROBOPOINT is instruction-tuned from Vicuna-v1.5-13B using a mixture of synthetic and real-world spatial-affordance data.
- Spatial Affordance Prediction: Spatial affordance prediction outputs image-space coordinates whose points satisfy relations expressed in a language prompt.
- Spatial Affordance Prediction: Point outputs provide more precise, directly actionable targets than fuzzy language actions or bounding boxes containing camera-induced clutter.
- Instruction Fine-tuning: Instruction tuning uses an image encoder, MLP projector, tokenizer, and transformer language model to process images and language.
- Co-finetuning with Synthetic Data: The data mix combines VQA, object detection, object reference, and free-space reference sources so the model retains language reasoning and grounds instructions to regions.
- Deployment: During deployment, predicted 2D action points are projected into 3D with a depth map, then used by a motion planner to navigate to targets.
4 Dataset
The dataset is generated procedurally in simulation to diversify scenes, objects, and viewpoints, including affordances in visually unmarked free space. Labels are created from spatial relations and object-surface intersections.
- Procedural Scene Generation in Simulation: Procedural randomization of scene layouts, objects, and camera viewpoints creates a diverse simulated affordance dataset.
- Dataset Composition: The instruction-tuning dataset combines object reference, free-space reference, VQA, and object-detection data.
- Generating Affordance in Free Space: Free-space affordances enable predictions for regions without distinct visual cues, such as the left part of a pizza box.
- Generating Affordance in Free Space: To create free-space labels, the pipeline computes relations, removes the target object, re-renders the scene, and samples points at mesh-surface intersections.
5 Experimental Results
ROBOPOINT outperforms VLM and visual-prompting baselines on spatial affordance prediction, generalizes beyond seen relations, and supports manipulation, navigation, and augmented-reality applications. Its predictions also remain physically valid, view-consistent, and compatible with preserved conversational ability.
- Spatial Affordance Prediction: ROBOPOINT achieves significantly higher spatial affordance accuracy than all evaluated VLM and visual-prompting baselines on object and free-space reference.Accuracy is measured as the percentage of predicted points within the ground-truth target mask.
- Spatial Affordance Prediction: ROBOPOINT remains accurate on combinations of seen relations and novel relation types absent from fine-tuning, maintaining its advantage over baselines.The synthetic training data used templated language and a fixed relation set, yet the model handled relations such as “in the middle” and “rightmost.”
- Spatial Affordance Prediction: ROBOPOINT satisfies spatial relations while avoiding obstacles, whereas baselines fail to respect the physical constraints illustrated in Figure 3.The target points avoid obstacles such as the bowl in the evaluated scene.
- Downstream Applications: ROBOPOINT retains common-sense VQA performance on par with LLaVA-v1.5-13B while enabling conversational visual guidance for manipulation and AR tasks.Its AR demonstrations include setting a dining table, playing tic-tac-toe, and reaching a carpool lane.
- Downstream Applications: In downstream evaluations, ROBOPOINT outperforms the best manipulation baseline by 39.5% in average success rate and outperforms PIVOT and GPT-4V in 2 of 3 navigation scenarios.The manipulation evaluation contains 7 language-conditioned tasks, while navigation uses 3 room scenes and a simple path planner.
- Data Composition: Each data source contributes to overall WHERE2PLACE accuracy, while fine-tuning on only 10% of the data causes a significant performance drop.The evaluated sources include real-image VQA, LVIS object detection, and synthetic object and free-space reference data.
- Viewpoint Consistency: ROBOPOINT maintains consistent predictions across camera viewpoints, making its action suggestions suitable for moving-camera settings such as mobile platforms and AR.Figure 4 illustrates the response to finding free space right of a blue cup from different views.
6 Conclusion
ROBOPOINT predicts spatial affordances from relational language instructions by integrating real-world VQA data with automatically generated synthetic data. The authors identify missing confidence estimates and uncontrollable output-point counts as future-work limitations.
- ROBOPOINT predicts spatial affordances in images from relational language instructions.
- The model integrates real-world VQA data with automatically generated synthetic data to generate precise action points respecting spatial and physical constraints.
- ROBOPOINT is presented as applicable to complex tasks including relational free-space reference and object rearrangement in cluttered environments.
- ROBOPOINT lacks confidence estimates for point predictions, and its number of output points is not controllable.
A Instruction Tuning
ROBOPOINT is instruction-tuned using a Vicuna-v1.5-13B language model with a CLIP-pretrained ViT-L/14 image encoder and a pretrained projector.
- ROBOPOINT uses Vicuna-v1.5-13B as its language-model base and ViT-L/14 at 336px as its CLIP-pretrained image encoder.
- Its projector is a 2-layer MLP pretrained on a 558K LAION-CC-SBU subset with BLIP captions.
- Instruction tuning took 40 hours on 16 A-100 GPUs with batch size 16 per GPU and learning rate 4e-5.
B Data Generation
The synthetic data pipeline procedurally generates diverse kitchen scenes, samples object placements and viewpoints, and computes relational annotations from 3D scene geometry.
- Around 660K image-relation pairs are generated from 10K scenes using procedural kitchen-environment layouts.
- Furniture and object assets are placed in random semantic layouts, with stable poses and collision rejection determining valid object configurations.
- Cameras are randomized and retained when scenes contain at least three visible objects and one valid relation between visible-object pairs.
- The pipeline uses 3D bounding boxes of objects, surfaces, and containers to compute pairwise spatial relations.
C Qualitative Examples
The qualitative examples illustrate relational object and free-space references across RoboRefIt and WHERE2PLACE, including unseen relations and cases where GPT-4o performs better.
- Figure A compares ROBOPOINT with baselines on RoboRefIt and WHERE2PLACE qualitative examples.
- They also include free-space references involving regions above, between, inside, or to the right of marked objects or surfaces.
- Synthetic-dataset examples use red and ground boxes as reference prompts and cyan dots as ground-truth affordance points excluded from model inputs.
- The examples include object references such as items left of, below, or behind other objects.
- Some examples demonstrate relations unseen during training, while others show cases where GPT-4o performs better.