Source-linked AI summary
CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models
Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, Yang Gao
TL;DR
CoPa addresses the difficulty of translating foundation-model knowledge into generalizable low-level robotic control. It uses coarse-to-fine grounding and VLM-generated spatial constraints to produce grasp and post-grasp poses, achieving a 63% success rate across ten real-world tasks while retaining limitations in geometric modeling and 3D spatial reasoning.
Problem
Existing robotic manipulation approaches rely on task-specific learning or extensive demonstrations, limiting generalization, while foundation models mainly address higher-level task planning.
Method
CoPa uses VLMs for coarse-to-fine grounding of grasping and task-relevant parts, then converts their spatial constraints into 6-DoF pose sequences with a solver and motion planning.
Results
63% success rate across ten real-world manipulation tasks, significantly outperforming the VoxPoser baseline and ablation variants.
Takeaways & Limitations
CoPa handles open-set instructions and objects with minimal prompt engineering and no additional training, and integrates with high-level planning for long-horizon tasks.
Takeaways & Limitations
CoPa is constrained by simplistic geometric elements, limited VLM grounding in the 3D physical world, and discrete textual outputs despite requiring continuous values.
Abstract
from arXiv · showhide
Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object's grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how CoPa can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that CoPa possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training. Project page: https://copa-2024.github.io/
I. INTRODUCTION
CoPa addresses the gap between high-level foundation-model task planning and brittle, task-specific low-level robotic control by using VLM common-sense knowledge for fine-grained manipulation. It combines coarse-to-fine grounding, spatial constraints, and pose planning to support open-set tasks and long-horizon execution.
- CoPa uses VLM common-sense knowledge to provide low-level robotic control for manipulation requiring fine-grained physical understanding.The framework is designed to handle open-set instructions and objects with minimal prompt engineering and without additional training.
- CoPa grounds task-relevant object parts from coarse object selection to fine-grained part identification, then converts their spatial constraints into robot poses.Traditional motion planning connects adjacent poses for precise execution.
- CoPa achieves a high success rate in extensive real-world manipulation experiments and significantly surpasses the VoxPoser baseline.The supplied introduction reports this outcome without providing a numerical success rate.
- CoPa integrates with high-level planning methods to execute complex, long-horizon tasks such as making pour-over coffee and setting up a romantic table.
II. RELATED WORK
Prior robotic manipulation methods often depend on demonstrations, trial-and-error learning, or task-specific policies that limit data efficiency and generalization. CoPa instead decomposes manipulation into task-oriented grasping and task-aware motion planning, using grounded parts and target poses as the interface to motion planning.
- Imitation learning requires expert demonstrations and struggles with out-of-distribution samples, while reinforcement learning is sample-inefficient and often relies on sim-to-real transfer.End-to-end learned policies also often lack generalization to new tasks.
- The grounding module identifies grasping or task-relevant parts through coarse object grounding followed by fine-grained part grounding.SoM labels scene objects before VLM-based selection and part localization.
- CoPa structures manipulation as task-oriented grasping followed by task-aware motion planning because tasks require an initial grasp and subsequent object motion.Robot execution is represented as generating a series of target end-effector poses, with motion planning connecting adjacent poses.
- The formulation maps an instruction and initial RGB-D observation to a grasp pose, then maps the post-grasp observation to a sequence of poses reached with RRT* or PRM*.
B. Task-Oriented Grasping
Task-oriented grasping combines model-generated grasp proposals with instruction-conditioned grounding of the object part that should be grasped. CoPa selects the final pose by filtering proposals according to the grounded part and grasp quality.
- GraspNet generates grasp pose candidates from a scene point cloud, while grounding identifies the instruction-relevant grasping part.The final grasp pose is selected by filtering candidates using the grasping-part mask and GraspNet scores.
- The method uses task-specific physical knowledge to select functional grasp regions, such as a knife handle rather than its blade.
- CoPa grounds the grasp target in two stages: coarse object grounding followed by fine-grained part grounding.SoM provides visual prompting and object-level labels before the selected object is cropped for part-level grounding.
- The grounding module supports both grasping-part identification and task-relevant-part identification for later motion planning.
C. Task-Aware Motion Planning
Task-aware motion planning identifies multiple task-relevant parts, represents them geometrically, and asks VLMs to generate spatial constraints. A solver converts those constraints into post-grasp poses that are connected by motion planning.
- Task-aware planning identifies multiple task-relevant parts, models them in 3D, and uses VLM-generated spatial constraints to compute post-grasp poses.The process includes grounding, geometric modeling, image annotation, constraint generation, and solver-based pose calculation.
- Slender task-relevant parts are represented as vectors, while other parts are represented as surfaces with centers and normal vectors.These geometric elements are projected and marked on the scene image before VLM constraint generation.
- The resulting pose sequence contains P1 through PN, with motion planning facilitating transitions between adjacent poses.
- The evaluation includes ten real-world tabletop manipulation tasks using a 7-DoF Franka Panda arm, a parallel-jaw gripper, and two RGB-D cameras.
B. CoPa for Real-World Manipulation
CoPa achieves a 63% success rate across ten real-world manipulation tasks and surpasses VoxPoser by grounding actions in task-relevant object parts and spatial constraints. Its comparisons demonstrate finer physical understanding and more accurate rotation control.
- 63% success across ten real-world tasks, significantly outperforming VoxPoser and ablation variants.The evaluation uses quantitative real-world experiments summarized in Table I.
- Fine-Grained Physical Understanding: CoPa grounds manipulation in fine-grained object parts and their spatial geometry, unlike VoxPoser’s whole-object perception.Examples include grasping a flower stem, identifying a vase rim, and recognizing spoon insertion geometry.
- Handling Rotation DoF: CoPa computes 6-DoF poses from spatial constraints, enabling accurate continuous rotation control where VoxPoser directly specifies rotation values from prompts.The Pour water task illustrates the need to rotate a kettle to a precise angle.
- Generalizability: CoPa uses minimal prompt engineering, whereas VoxPoser relies on 85 hand-crafted examples and nearly fails when reduced to three examples per module.This comparison supports CoPa’s broader generalizability across scenarios.
1) CoPa w/o foundation:
Ablations show that foundation VLMs, coarse-to-fine grounding, and constraint generation are important components of CoPa’s manipulation performance. Removing foundation-model reasoning reduces success to 11%, while removing the other designs causes task-specific failures.
- 11% success across all tasks after removing foundation VLMs and replacing constraint generation with rules.The ablation replaces part grounding with Owl-ViT and uses predefined post-grasp rules.
- Removing coarse-to-fine grounding causes performance declines, especially when accurately identifying important object parts is difficult.The Hammer nail variant cannot accurately identify the hammer’s striking part.
- Directly predicting post-grasp pose values is extremely challenging for most tasks, including tilting a kettle correctly in Pour water.The alternative asks VLMs to output numerical poses rather than object constraints.
E. Integration with High-Level Planning
CoPa’s low-level control integrates with high-level planning to execute complex long-horizon tasks. Using VILA to decompose instructions, the system completes pour-over coffee and romantic-table tasks sequentially.
- CoPa combined with high-level planning effectively completes Make pour-over coffee and Set up romantic table.These tasks require both reasonable decomposition and execution of each low-level action.
- VILA decomposes high-level instructions into sequential low-level control tasks, which CoPa then executes.This connects high-level task planning with CoPa’s low-level manipulation framework.
- The framework is presented as naturally compatible with high-level planning algorithms for complex, long-horizon manipulation.The conclusion extends the integration claim beyond the two demonstrated tasks.
APPENDIX
The evaluation uses a real-world tabletop setup with a Franka Panda, parallel-jaw gripper, RGB-D perception, and ten manipulation tasks varied across object types and arrangements. Prompts and implementation details are provided for reproducibility.
- Experiments use a 7-DoF Franka Emika Panda arm, a 1-DoF parallel-jaw gripper, and two calibrated Intel RealSense D435 RGB-D cameras.Franka ROS and MoveIt are used for control, with RRT-Connect as the default planner.
- The study evaluates ten real-world manipulation tasks across ten environmental variations involving object types and arrangements.The task list and descriptions are provided in Table II.
- CoPa uses GPT-4V with minimal few-shot prompts and chain-of-thought prompting to support VLM scene understanding.The paper provides prompts for coarse and fine grounding and constraint generation.
D. Baselines.
The baseline setup models task-relevant parts geometrically and uses VLM-generated spatial constraints to optimize robot poses. Voxposer is reproduced with GPT-4, Owl-ViT, and Segment Anything for comparison.
- Voxposer is evaluated using GPT-4, Owl-ViT, and Segment Anything with its real-world prompt.
- Task-relevant parts are represented as vectors or surfaces based on their geometry.Slender parts are modeled as vectors, while other parts are modeled as surfaces.
- VLM-generated constraints are solved by optimization algorithms for an SE(3) transformation minimizing cumulative constraint loss.The described solvers include BFGS and Trust-Region Constrained Optimization.
- The SE(3) transformation uses rotation R and translation t to transform points and vectors.
H. Details of Subsequent Actions.
CoPa supports subsequent actions by calculating new poses from action-specific methodologies, while also offering rule-based pose generation for instruction formats.
- Subsequent actions are paired with methods for calculating new robot poses.
- A rule-based alternative generates post-grasp poses using prescribed calculations for each instruction format.The supported instruction formats and pose calculations are listed in Table V.
J. More Visualization.
The supplementary visualizations cover grounding, grasping, motion planning, and the constraint and pose-calculation tables used by the framework.
- The supplementary figures visualize grounding, task-oriented grasping, and task-aware motion planning.
- Tables III–V document constraint losses, subsequent-action pose calculations, and predefined rule-based pose calculations.
- The grounding visualization distinguishes coarse-grained scene understanding, fine-grained scene understanding, and grasping-part selection.
- The framework overview connects environment images and task-relevant 3D components to constraints, post-grasp poses, and execution.