PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics

Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox

arXiv:2406.10721v1cs.ROcs.AIcs.CV

TL;DR

Robots need precise action points for navigation and task execution, while current VLM approaches often rely on predefined motion primitives or large-scale expert demonstrations. RoboPoint uses automatically generated synthetic data alongside real-world VQA data to predict spatial affordances from language, outperforming baselines in pointing accuracy and state-of-the-art methods on complex tasks.

  • Problem

    Robots need precise action points for navigation and task execution, while current VLM approaches often rely on predefined motion primitives or large-scale expert demonstrations.

    Source: §1 · p. 1, §6 · p. 8

  • Method

    RoboPoint is instruction-tuned using automatically generated synthetic action-point data alongside real-world VQA data to predict image affordances from language.

    Source: §1 · p. 2, §3 · p. 3, §6 · p. 8

  • Results

    RoboPoint significantly outperforms baselines in object and free-space pointing accuracy and state-of-the-art methods on relational free-space reference and cluttered object rearrangement.

    Source: §5.1 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    RoboPoint’s spatial-affordance predictions have applicability to robot navigation and augmented-reality assistance.

    Source: §6 · p. 8

    RoboPoint provides no confidence estimates for point predictions, and the number of output points is not controllable.

    Source: §6 · p. 8

Abstract

from arXiv · show

From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Project website: https://robo-point.github.io.

1 Introduction

ROBOPOINT addresses the imprecision of language for robotic actions by predicting image points from instructions, using a scalable synthetic-data pipeline to support manipulation, navigation, and AR applications.

1 Introduction

  • ROBOPOINT predicts action points in RGB images and transforms them to 3D with depth, avoiding predefined action primitives [9] [10], external object detectors [15] [16], and iterative visual prompting [17].Spatial affordance prediction specifies actions by pointing at where to act, addressing the imprecision of language for spatial relations and object-free locations.Source: §1 · p. 1
  • The work restores pointing as a precise way to communicate navigation and manipulation plans, after recent VLM-based approaches shifted toward language instructions [5] [6].Prior work finds that robot waypoints can mimic human pointing and make plans more intuitive.Source: §1 · p. 1
  • ROBOPOINT significantly outperforms GPT-4o, LLaVA-NeXT, Qwen-VL [6], and SpatialVLM [22] on relational reference, free-space reference, and cluttered real-world rearrangement, without losing standard VQA accuracy.The authors also evaluate free-space reference on WHERE2PLACE, a manually annotated, challenging real-world benchmark.Source: §1 · p. 2
  • An autonomous pipeline generates a large, diverse dataset of ground-truth action points from camera-perspective spatial relations and samples within object masks and surface intersections, making it easier to scale than human demonstrations [18] [19] [20].Although the added data contained simulated images and templated language, the resulting model improved on real images with natural-language commands.Source: §1 · p. 2
  • ROBOPOINT also provides visual action suggestions in interactive augmented reality, guiding users through tasks by predicting target points based on common sense.Figure 1 presents manipulation, augmented reality, and navigation as application domains for the model.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

RoboPoint is positioned as a point-based approach to spatial reasoning and affordance prediction, distinct from zero-shot language-model methods. Related work motivates 2D keypoints as a language-compatible representation and contrasts RoboPoint with prior robotic planners and VLMs.

Spatial Reasoning

  • SpatialVLM [22] predicts metric-space relations, while RoboPoint locates affordances as points and achieves better performance on real-world spatial reasoning tasks.Earlier spatial reasoning work often focused on coarse-grained relations; state estimation plus symbolic reasoning generalizes poorly to novel objects, while SORNet [31] generalizes zero-shot to unseen objects.Source: §2 · p. 2

Affordance Prediction

  • Affordances link object observations to possible actions; RoboPoint uses 2D keypoints because they can be readily converted into language format.Affordances have been represented as part segmentation [38], dense image feature descriptors [39], and keypoints [41] [12], and support manipulation methods for 6-DoF grasping [32] [33] [34] and stable placement [35] [36] [37].Source: §2 · p. 3

Zero-shot Language Models for Robotics

  • Language-model planners [9] [10] use structured plans requiring predefined action primitives; VoxPoser [15], PIVOT [17], and MOKA [16] produce finer-grained outputs but rely on external object detectors.VoxPoser generates 3D value maps, PIVOT iteratively samples and evaluates image-space actions, and MOKA predicts action-type-specific keypoints.Source: §2 · p. 3

3 Method

RoboPoint is instruction-tuned from Vicuna-v1.5-13B [43] to predict image-space action points from language instructions. Its point formulation supports multiple robotic tasks, while a co-training mix combines language reasoning, visual grounding, and synthetic affordance data.

Spatial Affordance Prediction

  • Image-space target points satisfy language-specified relations and can serve as navigation waypoints, grasp contact points, or placement proposals.Unlike fuzzy language actions or bounding boxes that may include perspective-related clutter, points are precise outputs that can be directly converted to actions.Source: §3 · p. 3

Instruction Fine-tuning

  • The model projects image tokens into language-token space, concatenates them with instruction tokens, and autoregressively predicts response tokens and an instruction-response boundary token.The projector and transformer are updated during fine-tuning, while the pretrained vision encoder and tokenizer remain frozen; the pipeline follows Liu et al..Source: §3 · p. 3
  • The paper reports that instruction-tuned RoboPoint achieves much higher precision than in-context-learning baselines [17] [22].The supplied passage gives no numerical precision values.Source: §3 · p. 3

Co-finetuning with Synthetic Data

  • Co-training combines 665K VQA conversations, LVIS bounding-box supervision, and synthetic object- and free-space-reference data to teach reasoning, grounding, and precise action-point generation.All sources are converted to a common format and trained together.Source: §3 · pp. 3–4
  • Synthetic instruction-point pairs come from procedurally rendered 3D scenes, with affordance points sampled within object masks and object-surface intersections.At deployment, predicted 2D points are projected into 3D using a depth map, then used by a motion planner for robot navigation.Source: Figure 2 · p. 4

4 Dataset

RoboPoint’s dataset uses procedurally randomized simulated scenes and viewpoints, with affordance labels that include free space lacking distinct visual cues.

Procedural Scene Generation in Simulation

  • Around 660K image–relation pairs from 10K procedurally randomized scenes provide diverse layouts, objects, and camera viewpoints for training.Scenes use a repository of over 8K object instances across 262 categories, with articulated bodies and objects placed on support surfaces.Source: §4 · p. 4
  • The instruction-tuning dataset combines object and space references with VQA and object-detection data to support combinatorial generalization.Source: Table 1 · p. 5

Generating Affordance in Free Space

  • Free-space affordance labels let ROBOPOINT predict regions without distinct visual cues, such as the left part of a pizza box.The pipeline removes a target object, re-renders the scene, and samples points in the intersection of its mesh and supporting surface, relating them to other scene entities.Source: §4 · pp. 4–5

5 Experimental Results

ROBOPOINT outperforms baselines on spatial affordance prediction and real-world manipulation, while also demonstrating generalization, view consistency, navigation, and augmented-reality capabilities.

5.1 Spatial Affordance Prediction

  • ROBOPOINT significantly outperforms all baselines on object-reference and free-space-reference accuracy, including WHERE2PLACE examples with unseen relations.Accuracy is the percentage of predicted points within the target mask; object reference is evaluated on RoboRefIt and free-space reference on WHERE2PLACE.Source: §5.1 · pp. 5–6, Table 2 · p. 6
  • ROBOPOINT predicts accurately for combinations of seen relations and novel relation types, and avoids obstacles that baseline predictions may target.The paper illustrates obstacle avoidance with a bowl and reports that ROBOPOINT maintains its advantage over baselines on novel relations.Source: §5.1 · p. 6, Figure 3 · p. 6
  • ROBOPOINT performs on par with LLaVA-v1.5-13B on VQA benchmarks, retaining common-sense knowledge from pretraining.The paper characterizes this as evidence that ROBOPOINT remains a general-purpose VLM rather than becoming domain-specific.Source: §5.1 · p. 6, Table 3 · p. 6
  • Combining all data sources achieves the best WHERE2PLACE results, and using only 10% of the data causes a significant performance drop.The evaluated sources include real-image VQA, LVIS object detection, and synthetic object and free-space reference data.Source: §5.1 · pp. 6–7, Table 4 · p. 6
  • ROBOPOINT maintains consistent affordance predictions as the camera moves, supporting use on mobile platforms and in augmented reality.The paper illustrates consistent responses to “find free space right of the blue cup” across different viewpoints.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Downstream Applications

  • ROBOPOINT achieves a 39.5% higher average success rate than GPT-4V, the best-performing baseline, across seven real-world manipulation tasks.The system converts predicted 2D target points into 3D points using depth, then plans and executes trajectories; success requires collision-free execution and accurate placement.Source: §5.2 · p. 7, Figure 5 · p. 7
  • In three simulated room scenarios, ROBOPOINT outperforms PIVOT [17] and GPT-4V [5] in two, supporting navigation to language-specified regions.A simple path planner uses predicted points to navigate the YouBot platform to the target region.Source: §5.2 · p. 8, Figure 6 · p. 8
  • ROBOPOINT provides visual affordance guidance for real-world tasks including winning tic-tac-toe and reaching a carpool lane.Co-training with VQA data retains natural-language conversational capabilities, allowing users to query the model and receive visual action suggestions.Source: §5.2 · p. 8, Figure 7 · p. 8

6 Conclusion

ROBOPOINT combines real-world VQA and automatically generated synthetic data to predict spatially and physically constrained action points from relational instructions, with reported superiority on complex tasks and applicability to augmented reality and robot navigation. It does not provide confidence estimates or allow control over the number of output points.

6 Conclusion

  • ROBOPOINT combines real-world VQA and automatically generated synthetic data to predict constrained action points from relational instructions, outperforming state-of-the-art methods on complex tasks.The reported tasks include relational free-space reference and object rearrangement in cluttered environments; the model also applies to augmented reality and robot navigation.Source: §6 · p. 8
  • ROBOPOINT provides no confidence estimates for point predictions, and the number of output points is not controllable.Both limitations are identified as directions for future work.Source: §6 · p. 8

Appendix · A Instruction Tuning

RoboPoint’s instruction-tuning setup uses a Vicuna-based multimodal model and specifies its projector pretraining and tuning compute configuration.

A Instruction Tuning

  • RoboPoint is instruction-tuned from Vicuna-v1.5-13B with a CLIP-pretrained ViT-L/14 336px image encoder [64].Its 2-layer MLP projector is pretrained on the 558K LAION-CC-SBU subset with BLIP captions from. Instruction tuning took 40 hours on 16 A-100 GPUs, with batch size 16 per GPU and learning rate 4e-5.Source: A Instruction Tuning · p. 13

B Data Generation

RoboPoint’s synthetic data pipeline generates image-relation pairs from procedurally arranged kitchen scenes. It derives image-coordinate point targets from spatial relations and uses visual prompts during training to compensate for sparse object descriptions.

Scene generation

  • About 660K image-relation pairs are generated from 10K scenes, with cameras selected to show at least three objects and one valid relation.Kitchen assets are arranged in random but semantic layouts; object placements that collide with the scene are rejected, and views require over 100 segmentation-mask points per visible object.Source: B Data Generation · p. 13

Relations and point targets

  • Relations computed from scene-layout 3D bounding boxes support object-reference targets, with 1–50 sampled points per image encoded as normalized image coordinates.The relation set includes spatial and part-based labels such as left, inside, and on the front part.Source: B Data Generation · p. 13

Visual prompts

  • Colored boxes mark instruction-referenced objects during synthetic-data training, but are omitted at test time so the model does not require object detection.The prompts compensate for objects having sparse text descriptions; the authors expect the model to learn object detection from other data and focus on relational reasoning in this dataset.Source: B Data Generation · p. 13

C Qualitative Examples

Qualitative examples compare ROBOPOINT with baselines on RoboRefIt [55] and WHERE2PLACE, showing generalization to unseen relations alongside cases where GPT-4o performs better.

C Qualitative Examples

  • Synthetic-dataset examples teach relational object and free-space reference through prompts about locations behind, above, between, and inside outlined objects.Red and ground boxes indicate reference objects; cyan dots visualize ground truth and are not included in model inputs.Source: C Qualitative Examples · p. 14
  • Figure A presents ROBOPOINT comparisons on RoboRefIt and WHERE2PLACE, including unseen relation types and examples where GPT-4o performs better.The RoboRefIt examples are (a, b), and the WHERE2PLACE examples are (c–h).Source: C Qualitative Examples · pp. 13, 15
  • ROBOPOINT handles relations unseen during training, including locating points on a stair, behind a sink, between blocks, and in front of a mug.These are examples (d, e, f, h), respectively.Source: C Qualitative Examples · p. 15
  • GPT-4o performs better in WHERE2PLACE examples (g) and (h), involving a location between AirPods and a black lid and a point in front of a mug.The figure caption identifies (g, h) as the cases where GPT-4o performs better.Source: C Qualitative Examples · p. 15

Related papers

Loading 2406.10721v1…