PaperlayerView exact revision on arXiv

Source-linked AI summary

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

Fangchen Liu, Kuan Fang, Pieter Abbeel, Sergey Levine

arXiv:2403.03174v3cs.ROcs.AI

TL;DR

Robotic systems need to control diverse, open-world manipulation tasks from free-form language while existing VLM capabilities are difficult to connect directly to robot actions. MOKA uses point-based affordances and mark-based visual prompting to generate motions, achieving strong performance across tabletop tasks and improving with in-context learning, while remaining limited by VLM understanding and affordance coverage.

  • Problem

    Open-world robotic manipulation requires connecting VLM understanding of physical scenes and free-form commands to executable robot control.

  • Method

    MOKA uses compact point-based affordances and hierarchical mark-based visual prompting to convert VLM affordance reasoning into visual question-answering and robot motions.

  • Results

    MOKA demonstrates effective and robust zero-shot and few-shot manipulation across tabletop tasks, with consistent improvements from in-context learning and policy distillation.

  • Takeaways & Limitations

    Point-based visual affordances provide an interface through which pre-trained VLMs can guide open-world robotic manipulation and reuse successful trajectories for further improvement.

  • Takeaways & Limitations

    Existing VLMs lack profound understanding of 3D space, contact physics, and robotic control, while the current affordance representation does not cover all useful robot skills.

Abstract

from arXiv · show

Open-world generalization requires robotic systems to have a profound understanding of the physical world and the user command to solve diverse and complex tasks. While the recent advancement in vision-language models (VLMs) has offered unprecedented opportunities to solve open-world problems, how to leverage their capabilities to control robots remains a grand challenge. In this paper, we introduce Marking Open-world Keypoint Affordances (MOKA), an approach that employs VLMs to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. By prompting the pre-trained VLM, our approach utilizes the VLM's commonsense knowledge and concept understanding acquired from broad data sources to predict affordances and generate motions. To facilitate the VLM's reasoning in zero-shot and few-shot manners, we propose a visual prompting technique that annotates marks on images, converting affordance reasoning into a series of visual question-answering problems that are solvable by the VLM. We further explore methods to enhance performance with robot experiences collected by MOKA through in-context learning and policy distillation. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.

I. INTRODUCTION

MOKA addresses open-world robotic manipulation by connecting VLM predictions to robot motion through compact point-based affordances and mark-based visual prompting. It demonstrates zero-shot and few-shot manipulation across varied tasks, with improvements from in-context examples.

  • MOKA demonstrates robust manipulation across variations in instructions, objects, and initial arrangements, including tool use and other tabletop tasks.The approach is evaluated in zero-shot settings and supports trajectory collection for later learning.
  • MOKA achieves state-of-the-art performance across four evaluation tasks and eight subtasks, with consistent improvements from in-context learning.The reported evaluation counts successes over 10 trials per task and notes that MOKA works without example prompts, unlike the baselines.
  • MOKA connects VLM predictions on RGB images to robot motion through a compact point-based affordance representation.The representation uses keypoints and waypoints to specify desired motions for open sets of objects and tasks.
  • Mark-based visual prompting converts affordance reasoning into visual question-answering problems by annotating candidate points, grids, and captions on observed images.The VLM selects points that produce the desired motion, simplifying point generation into multiple-choice questions.
  • MOKA uses hierarchical prompting to decompose free-form task descriptions into subtasks and then predict keypoints and affordance attributes.The high-level stage summarizes subtasks, while the low-level stage produces the affordance representation.

III. PROBLEM STATEMENT

MOKA targets tabletop manipulation tasks with unseen objects and goals specified by free-form language, where the robot must infer and execute subtasks sequentially. It uses structured visual and textual prompting to make VLM outputs parseable as motion affordances for a robot.

  • The robot must decompose free-form instructions into feasible subtasks because their sequence is not provided beforehand.Tasks may involve object interaction, environmental interaction, or tool use across one or multiple stages.
  • MOKA operates in a tabletop setting with a robotic arm, RGBD cameras, proprioception, and 6-DoF gripper-pose actions.Each observation combines RGBD images with robot proprioception, while each action includes gripper pose and finger status.
  • MOKA designs text and visual prompts that produce structured responses parseable into point coordinates and attributes for downstream robot motion.The VLM receives language and visual inputs and generates text responses controlled by prompt descriptions, output formats, and examples.
  • The approach uses a 2D-image affordance representation as the interface between VLM predictions and the motions performed by the robot.The representation is produced by the VLM and specifies the desired motion.

A. Point-Based Affordance Representations

MOKA represents affordances with keypoints and waypoints that connect VLM image predictions to executable robot motion. Hierarchical prompting first decomposes free-form instructions into subtasks, then predicts each subtask’s affordance representation from visual observations.

  • A. Point-Based Affordance Representations: Keypoints and waypoints provide a unified affordance representation that specifies robot motion across manipulation tasks.The predicted 2D points are converted into SE(3) poses, followed by smooth trajectory generation.
  • A. Point-Based Affordance Representations: Object-centric motion uses grasping and manipulation phases for in-hand and unattached objects, with either phase skippable when unnecessary.The framework supports interactions involving both an object held by the robot and an unattached target object.
  • A. Point-Based Affordance Representations: The representation defines grasp, function, target, pre-contact, and post-contact points to specify contact locations and free-space motion.During manipulation, the function point follows a path through the pre-contact, target, and post-contact waypoints.
  • A. Point-Based Affordance Representations: MOKA uses hierarchical VLM prompting to convert affordance reasoning into visual question-answering problems.The VLM is prompted to predict the affordance representation from the task description and image observations.
  • A. Point-Based Affordance Representations: High-level reasoning produces structured subtask descriptions, object identities, and motion directions before low-level affordance prediction.The high-level output is a list of dictionaries that conditions detailed motion generation for each subtask.
  • A. Point-Based Affordance Representations: Low-level reasoning predicts image-plane coordinates and lifts them into 3D using depth values and camera parameters.Object-surface points use corresponding depth measurements, while free-space waypoints also require predicted heights.

C. Mark-Based Visual Prompting

MOKA uses marks, candidate points, and image grids to make continuous affordance prediction more tractable for VLMs. The selected 2D affordances are then lifted into SE(3) motions using depth, height, orientation, and analytical grasping components.

  • C. Mark-Based Visual Prompting: Mark-based prompting converts continuous keypoint selection into a visual reasoning problem better suited to VLMs.The marks include dots, grids, and text annotations that encourage attention to target objects.
  • C. Mark-Based Visual Prompting: Candidate keypoints are plotted on segmented in-hand and unattached objects, with distinct labels and colors for each object.GroundedSAM is used to segment the objects before candidate points are presented to the VLM.
  • C. Mark-Based Visual Prompting: A 5×5 image grid restricts free-space waypoint search, after which exact waypoint locations are sampled within selected tiles.The VLM selects tiles for pre-contact and post-contact waypoints rather than sampling the entire workspace directly.
  • D. Motion Generation with Predicted Affordances: Selected 2D keypoints and waypoints are lifted into SE(3) space to produce executable robot motion.Object points use registered depth, while free-space points require a height estimate.
  • D. Motion Generation with Predicted Affordances: Waypoint height is constrained to match the target point’s height when deprojecting free-space locations into 3D.This simplifies the conversion of unanchored image-grid selections into spatial waypoints.
  • D. Motion Generation with Predicted Affordances: The vector from grasp to function specifies object orientation, which the VLM predicts from a finite set of directional options.Example orientation choices include forward, backward, upside, downside, left, and right.
  • D. Motion Generation with Predicted Affordances: Analytical grasping supplements VLM predictions because robust grasping depends on contact physics and gripper design.A grasp sampler proposes candidates from local point-cloud geometry, and the closest candidate is used instead of directly executing the predicted grasp point.

E. Bootstrapping through Physical Interactions

MOKA uses successful real-world trajectories to improve VLM control through in-context learning or to train a distilled visuomotor policy. Experiments evaluate zero-shot performance, task execution, and improvement from physical trials across varied manipulation tasks.

  • E. Bootstrapping through Physical Interactions: MOKA collects successful robot trajectories from physical rollouts and labels their success using VLMs or humans.These experiences support both in-context learning and policy distillation.
  • E. Bootstrapping through Physical Interactions: Three successful trajectories are appended as annotated image-response examples to guide high-level and low-level VLM reasoning.The approach uses in-context examples without changing the high-level or low-level prompts.
  • E. Bootstrapping through Physical Interactions: Policy distillation records multi-view images and proprioception during successful trajectories to train a visuomotor policy by behavior cloning.MOKA thereby provides demonstrations for real-world robot learning.
  • E. Bootstrapping through Physical Interactions: The experiments examine affordance reasoning on unseen tasks, task performance after motion translation, and improvement from real-world trials.The evaluation compares baseline methods across four manipulation tasks in zero-shot and in-context learning settings.
  • E. Bootstrapping through Physical Interactions: Each evaluation task contains two stages and varied object-interaction or tool-use scenarios, with additional open-world tasks reported in the appendix.The task suite is summarized in Table II.

A. Experimental Setup

MOKA is evaluated against zero-shot baselines across four two-subtask manipulation tasks, with additional in-context learning and policy-distillation variants. The experiments examine success rates, generated trajectories, failure types, and robustness to task and object variations.

  • Experimental Settings: Two annotated examples are provided to GPT-4V for the in-context setting, using scenes with varied objects and instructions.
  • Qualitative Results: Fig. 3 illustrates marked images, point-based affordances, and three trajectory keyframes for subtasks including pressing, closing, rearranging, and tool use.
  • Policy Distillation: MOKA-generated successful trajectories are used to fine-tune Octo, with 50 language-annotated trajectories collected per task.This policy-distillation setup transfers MOKA’s generated demonstrations to a learned robot policy.
  • Quantitative Evaluation: MOKA achieves state-of-the-art performance on all 8 subtasks across 4 tasks, with consistent improvements from in-context learning.Each task reports successes out of 10 trials; the evaluation compares MOKA with Code-as-Policies and VoxPoser.
  • Failure Analysis: Failure analysis separates affordance-prediction reasoning failures from low-level execution failures, and both policy distillation and in-context learning reduce total failures.Distilled policies eliminate reasoning failures because the VLM is no longer part of the execution pipeline.
  • Robustness Analysis: Robustness is assessed across instruction variations, initial arrangements, and object variations, including alternative geometries, colors, materials, and some deformable objects.

C. Qualitative Evaluation

MOKA produces plausible affordance-driven motions across diverse manipulation skills and remains robust to changes in instructions, objects, and initial arrangements. Its trajectories can also support in-context learning and policy distillation, although current VLM and representation limits constrain finer control.

  • Qualitative results: MOKA successfully generates motions for pressing, closing, rearranging, and tool-use skills from annotated visual marks.The qualitative examples show selected keypoints, waypoints, and resulting trajectories.
  • Robustness: MOKA consistently predicts keypoints and waypoints across variations in language instructions, object geometry, color, material, and initial arrangements.The reported robustness includes deformable and transparent alternative objects.
  • Limitations: Existing VLMs limit fine-grained SE(3) motions and dynamic manipulation, while the current point representation does not cover all complex skills.The paper identifies 3D understanding, contact physics, latency, and representation coverage as the main constraints.

APPENDIX A EXPERIMENT DETAILS

The experiments use a real-world tabletop setup with a 7-DoF robot, RGBD cameras, and diverse daily-object tasks. MOKA is evaluated on multi-stage tasks through VLM motion predictions, with successful trajectories retained for policy distillation.

  • Environment: The evaluation uses a 7-DoF Franka arm with a 2F-85 gripper, RGBD cameras, and 5Hz interaction in a real-world tabletop environment.The action space comprises a 6-DoF end-effector twist plus gripper position.
  • Task design: The task suite contains diverse daily-object tabletop manipulation tasks, mostly organized into two stages.The drawer-closing task is the stated exception to the two-stage structure.
  • Evaluation tasks: Comparative evaluation covers Table Wiping, Laptop Packing, Gift Preparation, and Ultrasound Cleaning, with additional tasks shown in supplementary material.The supplied passage names four main-paper tasks and four supplementary-video tasks.
  • Evaluation procedure: Failure analysis separates incorrect VLM motion predictions from unsuccessful executions, while successful trajectories are saved as policy-distillation demonstrations.A human expert checks whether each executed motion completes the task.
  • Pipeline: Algorithm 1 takes a VLM, task instruction, prompts, and an initial observation, then decomposes the task and predicts low-level motion for each subtask.Each subtask uses a top-down observation, annotated candidates, and a low-level VLM query.

A. High-Level Task Reasoning

MOKA first uses high-level VLM reasoning to decompose a free-form instruction into structured subtasks and identify involved objects and motions. It then proposes visual point candidates for low-level affordance selection.

  • High-Level Task Reasoning: The high-level VLM query decomposes the language instruction into subtasks containing object descriptions and desired motion information.The response includes fields for grasped objects, unattached objects, and motion direction.
  • High-Level Task Reasoning: Because VLMs cannot directly generate keypoints and waypoints, MOKA proposes candidates and asks the VLM to select them through visual question answering.This converts low-level affordance selection into a visually grounded choice among annotated candidates.
  • High-Level Task Reasoning: Object keypoints comprise sampled boundary points plus a center point plotted on the image for VLM selection.The boundary points are obtained by farthest point sampling, while the center is the segmentation-mask geometric mean.
  • High-Level Task Reasoning: Waypoints are selected from a 5 × 5 image grid whose predicted tile supplies free-space waypoint candidates.Columns are labeled a–e and rows 1–5 from bottom to top.

C. Low-Level Motion Reasoning

MOKA uses annotated object keypoints and grid cells to prompt low-level VLM motion reasoning. The selected points are converted into grasping and contact motions, with optional grasping skipped for direct-interaction tasks.

  • Low-Level Motion Reasoning: Low-level reasoning prompts the VLM with an annotated image, keypoint and waypoint definitions, an output schema, and step-by-step guidance.The prompts organize visual inputs and constrain the motion-reasoning response.
  • Low-Level Motion Reasoning: MOKA decomposes each manipulation subtask into an optional grasping phase followed by a manipulation phase.Direct-interaction tasks omit grasping, whereas tool-use tasks can require it before contact.
  • Low-Level Motion Reasoning: The grasping phase lifts the predicted 2D grasp point into the robot frame and selects the nearest sampled antipodal grasp proposal.Thirty 4-DoF proposals are sampled from the cropped depth image.
  • Low-Level Motion Reasoning: The manipulation phase samples pre-contact and post-contact waypoints from predicted tiles, assigns heights, and executes a sequential contact motion.The function keypoint moves through the pre-contact waypoint, target keypoint, and post-contact waypoint.
  • Low-Level Motion Reasoning: After each subtask, MOKA returns the robot to a neutral pose and captures a clean top-down image before predicting the next motion.Execution continues until the multi-stage task is complete.

E. Policy Distillation

MOKA is evaluated with zero-shot language changes, in-context examples, and prompt ablations to assess how its prompting designs affect manipulation reasoning. Results show that simple successful examples improve performance, while hierarchical prompting is especially important.

  • Policy Distillation: The student-policy experiments use Octo with full-model fine-tuning and largely reuse its default hyperparameters.The customized hyperparameters are listed in Table VIII.
  • Zero-Shot Evaluation: In zero-shot evaluation, the prompts remain unchanged while only the free-form language task description changes.Examples include sweeping garbage, picking up perfume, and pressing a button.
  • In-Context Learning: Two successful predictions from scenes with object and scene variations improve MOKA through simple in-context examples without intricate prompt engineering.The examples are collected from successful VLM predictions.
  • Ablation Study: The ablations remove hierarchy, keypoint and waypoint descriptions, or chain-of-thought guidance to isolate the contribution of each prompting design.The no-hierarchy variant directly asks GPT-4V for low-level motion reasoning from an annotated image.
  • Ablation Study: Removing hierarchical prompting decreases performance by a large margin, indicating that subtask decomposition is central to MOKA's reasoning.Removing keypoint descriptions or chain-of-thought prompting preserves most performance but remains worse than MOKA on most tasks.

B. Additional Robustness Analysis

Additional analyses examine whether MOKA's predicted affordances remain consistent across object appearances and initial poses. The reported robustness results cover drawer closing and fur removal, while the ablation visual summarizes prompt-design effects.

  • Drawer Closing: MOKA consistently predicts drawer-closing waypoints and target points that follow the correct motion across object poses and appearances.Columns use similar initial arrangements, while rows use different objects.
  • In-Context Learning: The in-context examples depict object-pose variations, supporting robustness evaluation with varied visual conditions.Figure 7 shows the examples used in MOKA's in-context prompting.
  • Prompt Robustness: The ablation table reports reasoning success rates across four tasks, each containing two subtasks, and states that all three prompt designs consistently benefit MOKA.The table summarizes the comparative effect of the prompting choices.
  • Prompt Robustness: Figure 8 shows that removing keypoint descriptions or chain-of-thought prompting can cause keypoint and waypoint prediction mistakes.The figure contrasts these failures with the full prompting design.
  • Fur Removal: The robustness analysis also evaluates fur removal under varied object conditions.Figure 10 is identified as the robustness analysis for the fur removal task.
Loading 2403.03174v3…