PaperlayerView exact revision on arXiv

Source-linked AI summary

CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models

Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, Yang Gao

arXiv:2403.08248v1cs.RO

TL;DR

Foundation-model robotics methods often leave low-level control to task-specific learned policies. CoPa uses VLMs to ground task-relevant object parts and derive manipulation poses from spatial constraints, achieving a 63% success rate across ten tasks and significantly outperforming VoxPoser.

  • Problem

    Foundation-model robotics methods generally address high-level task planning, while low-level control continues to rely on task-specific learned policies.

    Source: §I · p. 1

  • Method

    CoPa uses VLM common-sense knowledge, coarse-to-fine grounding of task-relevant parts, and spatial constraints to generate robotic manipulation poses.

    Source: §I · p. 2

  • Results

    63% success rate across ten different tasks, significantly outperforming the VoxPoser baseline and ablation variants.

    Source: B. CoPa for Real-World Manipulation · p. 5

  • Takeaways & Limitations

    CoPa handles open-set instructions and objects with minimal prompt engineering and no additional training, and integrates with high-level planners for long-horizon tasks.

    Source: §I · p. 2

    Current VLMs are pretrained on large-scale 2D images and lack genuine 3D grounding, hampering accurate spatial reasoning.

    Source: §V · p. 7

Abstract

from arXiv · show

Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object's grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how CoPa can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that CoPa possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training. Project page: https://copa-2024.github.io/

I. INTRODUCTION

CoPa uses VLM common-sense knowledge to ground task-relevant object parts and translate their spatial constraints into robot poses. Real-world experiments show it can perform fine-grained manipulation and integrate with high-level planning for long-horizon tasks.

I. INTRODUCTION

  • Task-specific low-level policies remain brittle in unseen scenarios, while prior foundation-model approaches [16], can fail on tasks needing fine-grained physical understanding.The introduction motivates this gap with human manipulation that adapts grasp and orientation to an object's task-relevant parts.Source: §I · pp. 1–2
  • CoPa grounds task-relevant object parts, derives spatial constraints with VLMs, solves for robot poses, and uses traditional motion planning between adjacent poses.Its coarse-to-fine grounding and spatial constraints bridge language-based VLM reasoning with low-level control.Source: §I · p. 2
  • Real-world experiments show CoPa completes everyday manipulation tasks with high success and significantly surpasses VoxPoser [16] and other baselines.The paper attributes its fine-grained physical understanding and precise 6-DoF poses to coarse-to-fine grounding and constraint generation.Source: §I · p. 2
  • CoPa handles open-set instructions and objects with minimal prompt engineering and no additional training, and integrates with high-level planners for complex, long-horizon tasks.Examples include making pour-over coffee and setting up a romantic table.Source: §I · p. 2

II. RELATED WORK · III. METHOD · A. Promblem Formulation

CoPa uses foundation-model common sense to structure open-world manipulation as task-oriented grasping followed by task-aware motion planning, generating end-effector poses from language instructions and RGB-D observations. Its related-work discussion contrasts this approach with learning-based manipulation and other foundation-model robotics methods.

II. RELATED WORK

  • Imitation learning [14], [15], – requires expert demonstrations and struggles with out-of-distribution samples, while reinforcement learning is sample-inefficient and often relies on sim-to-real transfer [27]–.The paper says end-to-end learned policies often fail to generalize to new tasks, whereas CoPa targets open-world scenarios without additional training.Source: §II · p. 2
  • Prior foundation-model robotics work uses vision representations [20], –[34], specifies RL rewards [35]–[40], plans at high level [7], [8], [10], –[49], or supports low-level control [14], [15], [21],.The paper contrasts CoPa with VLM fine-tuning, which needs extensive expert data, and Code as Policies and VoxPoser [16], which rely on complex prompting and offer coarse scene understanding.Source: §II · p. 2

III. METHOD

  • The method section presents two framework components: task-oriented grasping and task-aware motion planning.These components are described in Sections III-B and III-C, respectively.Source: §III · p. 2

A. Promblem Formulation

  • The grounding module first segments and labels scene objects with SoM, then uses GPT-4V and the instruction to select relevant objects and ground specific parts.It supports both grasping-part identification and task-relevant-part identification.Source: Figure 2 · p. 3
  • CoPa divides manipulation into task-oriented grasping and task-aware motion planning, reflecting the grasp-then-act structure of tasks such as pulling a drawer or lifting a glass.The transition between adjacent target poses is achieved through motion planning.Source: A. Promblem Formulation · pp. 2–3
  • Given instruction l and initial RGB-D observation O0, CoPa generates grasp pose P0=f(l,O0), then post-grasp poses {P1,…,PN} from l and O1 for motion planning.The end-effector can reach the target poses using planners such as RRT* [50] or PRM* [51].Source: A. Promblem Formulation · p. 3

B. Task-Oriented Grasping

CoPa generates task-oriented grasp poses by combining GraspNet candidates with VLM grounding of the instruction-relevant grasping part. A coarse-to-fine process identifies the target object and part, then filters candidates to select the highest-scoring grasp within that part.

Grasping Part Grounding

  • VLMs such as GPT-4V identify the appropriate grasping part using common-sense knowledge, matching the object part to the intended use described by the instruction.The paper illustrates the principle with grasping a knife by its handle rather than its blade.Source: B. Task-Oriented Grasping · p. 3

Grasp Pose Proposals

  • The method selects a task-specific grasp by retaining GraspNet candidates whose projected grasp points fall within the grounded grasping-part mask, then choosing the candidate with the highest GraspNet confidence score.GraspNet receives a point cloud backprojected from RGB-D images and outputs 6-DOF grasp candidates with grasp-point and graspness information.Source: B. Task-Oriented Grasping · pp. 3–4, Figure 3 · p. 3

Grasping Part Grounding

  • VLM grounding proceeds coarse-to-fine: Set-of-Mark labels scene objects for instruction-guided target selection, then the cropped target is used to ground its specific grasping part.Set-of-Mark uses segmentation to divide the image into marked regions, supporting visual grounding by VLMs.Source: B. Task-Oriented Grasping · pp. 3–4

C. Task-Aware Motion Planning · IV. EXPERIMENTS · A. Experimental Setup

CoPa grounds task-relevant parts, uses VLM-generated spatial constraints to compute post-grasp poses, and evaluates manipulation in a real-world tabletop setup. The experiments use varied tasks and environments and compare CoPa with VoxPoser.

C. Task-Aware Motion Planning

  • CoPa grounds multiple task-relevant parts through coarse-to-fine grounding and filters robot-arm masks that could affect VLM part selection.The parts can include functional regions such as a hammer’s striking surface and handle or a nail’s surface.Source: C. Task-Aware Motion Planning · p. 4
  • VLMs generate spatial constraints from images annotated with part geometry, representing slender parts as vectors and other parts as surfaces with centers and normals.The constraints can express relations such as collinearity between vectors or perpendicularity between a vector and a surface.Source: C. Task-Aware Motion Planning · p. 4
  • A nonlinear constraint solver computes SE(3) poses by minimizing constraint losses, assuming the manipulated object part and end-effector form a rigid body.Subsequent VLM-specified actions produce additional poses, such as moving the current pose 7 cm downward; motion-planning algorithms facilitate transitions between adjacent poses.Source: C. Task-Aware Motion Planning · pp. 4–5

IV. EXPERIMENTS

  • The experiments evaluate CoPa on real-world manipulation, compare it with VoxPoser, analyze key-module ablations, and test integration with high-level planning for long-horizon tasks.The paper introduces these evaluations in Sections IV-B through IV-E, respectively.Source: §IV · p. 5

A. Experimental Setup

  • The real-world tabletop setup uses a 7-DoF Franka Emika Panda arm, a 1-DoF parallel-jaw gripper, and two calibrated Intel RealSense D435 RGB-D cameras.Source: A. Experimental Setup · p. 5
  • CoPa is tested on 10 real-world manipulation tasks across 10 environment variations per task, varying object types and arrangements.Source: A. Experimental Setup · p. 5
  • CoPa uses GPT-4V with minimal few-shot prompts and chain-of-thought prompting; the VoxPoser baseline uses GPT-4, Owl-ViT, and Segment Anything.The paper describes VoxPoser as synthesizing closed-loop robot trajectories without additional training.Source: A. Experimental Setup · p. 5

B. CoPa for Real-World Manipulation · C. Understanding Properties of CoPa

CoPa achieves a 63% success rate across ten real-world tasks, outperforming VoxPoser and ablation variants. Its results and comparisons highlight fine-grained part understanding, minimal prompt requirements, and rotation-aware motion planning.

B. CoPa for Real-World Manipulation

  • VLM common-sense knowledge supports part grounding and constraint generation, including grasping an eraser cover and aligning a stick with a button.CoPa also identifies the flower stem and vase rim as critical parts, and understands that a spoon can be inserted vertically into a cup.Source: B. CoPa for Real-World Manipulation · p. 5

C. Understanding Properties of CoPa

  • CoPa grounds grasping and task-relevant parts coarse-to-fine and derives their spatial constraints, while VoxPoser treats objects as wholes.This finer-grained physical understanding supports precise operations that can fail with whole-object perception.Source: C. Understanding Properties of CoPa · p. 5
  • For flower insertion, CoPa grasps the stem rather than petals; for hammering, it aligns the hammer with the nail, unlike VoxPoser.VoxPoser overlooks the hammer’s fine-grained physical constraint and treats it as a single rigid body.Source: C. Understanding Properties of CoPa · p. 5

B. CoPa for Real-World Manipulation

C. Understanding Properties of CoPa

  • CoPa uses three examples to guide VLMs, whereas VoxPoser uses 85 hand-crafted examples and nearly fails when reduced to three per module.The passage attributes VoxPoser’s reasoning predominantly to its prompts, which limits generalization to new scenarios.Source: C. Understanding Properties of CoPa · p. 6
  • CoPa derives 6-DoF poses from spatial constraints for continuous rotation control, while VoxPoser selects from limited discrete rotation options.In spoon insertion, CoPa rotates the spoon vertically; VoxPoser points the end-effector toward the cup, causing a collision.Source: C. Understanding Properties of CoPa · p. 6

D. Ablation Study … 3) CoPa w/o constraint:

The ablations show that foundation-model knowledge, coarse-to-fine grounding, and constraint-based pose planning each address challenges in open-world manipulation.

D. Ablation Study

  • The ablation studies assess the significance of foundation models, coarse-to-fine grounding, and constraint generation in CoPa.Results are reported in Table I.Source: D. Ablation Study · p. 6

1) CoPa w/o foundation:

  • The no-foundation variant replaces GPT-4V grounding with Owl-ViT and removes constraint generation, using predefined rules for post-grasp poses instead.The rule-based details are in the Appendix.Source: 1) CoPa w/o foundation · p. 6
  • Replacing GPT-4V with Owl-ViT and rule-based post-grasp poses yields only 11% success across all tasks.The Sweep nuts task illustrates the difficulty of choosing a suitable sweeping tool without VLM common-sense knowledge.Source: 1) CoPa w/o foundation · p. 6

2) CoPa w/o coarse-to-fine:

  • Removing coarse-to-fine grounding causes zero success on Hammer nail tasks because the variant cannot accurately identify the hammer’s striking surface.The variant instead uses fine-grained SoM and GPT-4V directly to select object parts.Source: 2) CoPa w/o coarse-to-fine · p. 6

3) CoPa w/o constraint:

  • Directly generating post-grasp pose values from scene images is challenging for most tasks, whereas solving poses from VLM-generated constraints is more viable.In Pour water, the variant struggles to generate precise values for tilting the kettle correctly.Source: 3) CoPa w/o constraint · p. 6
  • In the no-constraint variant, VLMs output numerical post-grasp end-effector poses directly rather than object-manipulation constraints.The experiments indicate that precise pose values are difficult to derive directly from scene images.Source: 3) CoPa w/o constraint · p. 6

E. Integration with High-Level Planning · V. DISCUSSION & LIMITATIONS

CoPa executes long-horizon tasks when paired with high-level planning, while its limitations include restricted geometric modeling, weak 3D grounding in VLMs, and discrete textual outputs.

E. Integration with High-Level Planning

  • CoPa combined with high-level planning effectively completed long-horizon tasks, including making pour-over coffee and setting up a romantic table.The paper presents these real-world task rollouts as evidence of the combination’s potential for complex tasks.Source: Figure 6 · p. 7, E. Integration with High-Level Planning · p. 7
  • VILA [10] decomposed each high-level instruction into low-level control tasks, which CoPa then executed sequentially.The two evaluated tasks were making pour-over coffee and setting up a romantic table; each required actionable decomposition and understanding task-relevant objects’ physical properties.Source: E. Integration with High-Level Planning · pp. 6–7

V. DISCUSSION & LIMITATIONS

  • CoPa uses foundation VLMs to generate manipulation pose sequences with simple prompt engineering, no training, and generalization to open-set instructions and objects.The paper also describes CoPa as naturally combinable with high-level planning algorithms for complex, long-horizon tasks.Source: §V · p. 7
  • CoPa’s reliance on surfaces and vectors limits its ability to process complex objects; incorporating more geometric elements could improve its modeling.Source: §V · p. 7
  • VLMs trained on large-scale 2D images lack genuine 3D physical grounding, hampering accurate spatial reasoning; incorporating point clouds during training may help.Source: §V · p. 7
  • Current VLMs produce discrete textual outputs, but CoPa needs continuous values such as object-part coordinates.The paper identifies foundation models with these capabilities as a needed advancement.Source: §V · p. 7

APPENDIX … C. VLMs and Prompting.

The appendix describes CoPa’s real-world robot setup, ten manipulation tasks and their evaluation variations, and the VLM and prompts used for grounding and constraint generation.

A. Hardware Setup

  • CoPa’s real-world setup uses a 7-DoF Franka Panda arm, a 1-DoF parallel-jaw gripper, Franka ROS, MoveIt1, and its default RRT-Connect planner.Two Intel RealSense D435 RGB-D cameras are mounted at opposite ends of the tabletop and calibrated.Source: A. Hardware Setup · p. 9

B. Tasks and Evaluations.

C. VLMs and Prompting.

  • CoPa uses GPT-4V and minimal few-shot prompts, with chain-of-thought prompting to support VLM scene understanding.Source: C. VLMs and Prompting · p. 9
  • The appendix provides prompts for coarse-grained object grounding and fine-grained grasp-part grounding.Source: C. VLMs and Prompting · pp. 9–10
  • It also provides prompts for coarse-grained task-relevant object grounding, fine-grained task-relevant part grounding, and constraint generation.Source: C. VLMs and Prompting · p. 10

D. Baselines. … G. Details of Constraints.

CoPa’s implementation compares against Voxposer and prepares task-relevant parts for spatial-constraint planning by filtering the robot from the camera view, modeling and annotating parts, and optimizing an SE(3) transform against VLM-derived constraints.

D. Baselines.

  • For Voxposer, the baseline uses GPT-4, Owl-ViT, Segment Anything, and the real-world prompt specified by Huang et al [16].Voxposer synthesizes closed-loop robot trajectories without additional training.Source: D. Baselines · p. 10

E. Robotic Arm Filtering.; F. Part Modeling and Annotation.

F. Part Modeling and Annotation.

  • Vector parts are fitted to 2D masks by linear regression and represented by endpoints, while surface parts use RANSAC-derived 3D centers and normals projected onto the image.The parts are color-masked and translucent; vectors are drawn between endpoints and surfaces are marked with their center and normal.Source: F. Part Modeling and Annotation · p. 10

G. Details of Constraints.

  • CoPa solves for an SE(3) matrix by minimizing cumulative loss over VLM-derived task constraints using optimization algorithms such as BFGS or Trust-Region Constrained Optimization.Point A and Vector A belong to the manipulated object and require transformation in the loss calculation; other points and vectors are static.Source: G. Details of Constraints · p. 10
  • The constraints specify object points and vectors that undergo the solved transformation, whereas static points and vectors do not.This distinction determines which geometric elements are transformed when their constraint losses are calculated.Source: G. Details of Constraints · p. 10
  • The solved transformation belongs to SE(3), with rotation R in SO(3) and translation t in R3, and maps points as Rp + t and vectors as RV.The transformation applies translation to points but not vectors.Source: G. Details of Constraints · p. 10

H. Details of Subsequent Actions. · I. Predefined Rule-Based Post-Grasp Pose Generation.

The section describes methodologies for calculating new poses for experimental subsequent actions and a rule-based alternative for generating post-grasp poses from instruction formats.

H. Details of Subsequent Actions.

I. Predefined Rule-Based Post-Grasp Pose Generation.

  • A rule-based method replaces the constraint-generation module with a prescribed post-grasp pose calculation protocol for each instruction format, detailed in Table V.Source: §I · p. 10

J. More Visualization.

This section presents additional visualizations of CoPa’s grounding, task-oriented grasping, and task-aware motion-planning modules.

J. More Visualization.

  • Additional grounding-module visualizations are presented in Fig. 7.The figure caption identifies the visualization as covering the grounding module.Source: J. More Visualization · p. 10, Figure 7 · p. 12
  • Additional visualizations for task-oriented grasping are presented in Fig. 8.The figure caption identifies the visualization as covering task-oriented grasping.Source: J. More Visualization · p. 10, Figure 8 · p. 12
  • Additional visualizations for task-relevant motion planning are presented in Fig. 9.The figure caption identifies the visualization as covering task-aware motion planning.Source: J. More Visualization · p. 10, Figure 9 · p. 13
  • Table III describes constraints and their corresponding loss calculations, including table coordinates and normals and how points and normals correspond to parts.The table defines the l2-norm notation and specifies the table coordinate as (0.5, 0, 0.07) and normal vector as (0, 0, 1).Source: Table III · p. 11
  • Table IV describes subsequent actions and methods for calculating their corresponding new poses.Source: Table IV · p. 11
  • Table V describes predefined rule-based pose-calculation methods, with instruction objects A and B allowed to refer to any object.Source: Table V · p. 11

Related papers

Loading 2403.08248v1…