Source-linked AI summary
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation
Lucas Manuelli, Wei Gao, Peter Florence, Russ Tedrake
TL;DR
Category-level pick-and-place must place varied, novel object instances into desired states despite limitations of fixed-template pose representations and goal specifications. kPAM uses semantic 3D keypoints with geometric costs and constraints in a factorized perception-to-action pipeline, and hardware experiments demonstrate reliable manipulation across shape variation.
Problem
Fixed-template pose representations and category-level target poses may not capture large intra-category variation or specify physically feasible desired states for novel instances.
Method
kPAM represents objects with semantic 3D keypoints, specifies goals as geometric costs and constraints, and factors manipulation into perception, planning, grasping, and execution.
Results
Hardware experiments show the pipeline generalizes to novel category instances and accomplishes tasks requiring centimeter-level precision.
Takeaways & Limitations
Ignoring task-irrelevant geometric details makes the keypoint-based pipeline robust to large intra-category shape and topology variations.
Takeaways & Limitations
The current data-collection pipeline requires human annotation, although 3D reconstruction somewhat alleviates this manual labor.
Abstract
from arXiv · showhide
We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and size of a coffee mug relative to some canonical mug is not sufficient to successfully hang it on a rack by its handle. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Using this formulation, we factor the manipulation policy into instance segmentation, 3D keypoint detection, optimization-based robot action planning and local dense-geometry-based action execution. This factorization allows us to leverage advances in these sub-problems and combine them into a general and effective perception-to-action manipulation pipeline. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations.
1 Introduction
kPAM formulates category-level manipulation around semantic 3D keypoints, enabling interpretable goals and a factorized pipeline that handles shape variation and novel instances.
- Motivation: Category-level manipulation must generalize across objects differing in shape, size, appearance, and topology while converting natural-language goals into robot actions.The paper motivates tasks including upright placement, hanging mugs by their handles, and placing shoes on racks.
- Motivation: Fixed-template pose representations may fail to capture large intra-category variation, producing ambiguous estimates and unsuitable manipulation actions.The paper contrasts template-based pose estimation with cases such as aligning a shoe template to a boot observation.
- Approach: kPAM represents objects with semantic 3D keypoints and specifies manipulation targets through geometric costs and constraints on those keypoints.This representation is presented as a flexible generalization of pose-based manipulation targets.
- Approach: The manipulation policy factors into instance segmentation, 3D keypoint detection, optimization-based action planning, and geometric grasping and execution.This modular design combines established solutions for the constituent submodules.
- Results: The keypoint representation ignores task-irrelevant geometry, supporting robustness to shape and topology variation and generalization to novel category instances.Hardware experiments evaluate category-level pick-and-place tasks with centimeter-level precision requirements.
2 Related Work
Related work includes pose-based perception, dense descriptors, keypoint-based methods, grasping systems, and end-to-end reinforcement learning; the paper emphasizes their limitations for flexible category-level goal specification.
- Object representations: Category-level pose estimation extends instance-level pick-and-place pipelines but can become ambiguous under large shape variation and yield infeasible target states.A template pose may not represent the geometry of every category instance.
- Object representations: Dense visual descriptors offer category-level generalization, but self-occlusions complicate representing the full object configuration.The paper contrasts this with sparse 3D keypoints and their geometric target constraints.
- Object representations: Prior keypoint methods commonly use detections as grasp points or components of other shape parameterizations rather than as direct task-goal representations.The paper positions geometric costs and constraints on 3D keypoints as a way to define class-general configuration changes.
- Grasping: Grasping algorithms address reliable object pickup, but grasp success alone does not determine how to place novel category instances into desired goal states.The paper therefore treats post-grasp manipulation as outside the scope of those grasping works.
- End-to-end reinforcement learning: End-to-end reinforcement learning can learn pick-and-place policies from experience, yet its desired-configuration specification is not generally flexible or interpretable across tasks and categories.The burden of handling shape variation is transferred to the reward-function design.
- Positioning: The paper’s formulation combines keypoint perception, geometric goal specification, optimization-based planning, and grasp-based execution for category-level manipulation.The concrete mug example detects bottom- and top-center keypoints before solving the action transform.
3 Manipulation Formulation
The formulation represents category-level objects with semantic 3D keypoints and specifies manipulation goals through geometric costs and constraints. It then factors execution into perception, optimization-based action planning, grasping, and trajectory execution.
- 3.1 Concrete Motivating Example: The mug task uses bottom-center and top-center semantic keypoints, allowing upright placement to be specified without requiring surface-only keypoints.The keypoint detector receives RGBD images or point clouds and outputs 3D keypoint locations.
- 3.1 Concrete Motivating Example: The robot action is a rigid transformation Taction applied to the manipulated object’s keypoints, followed by grasping and trajectory execution.The trajectory can impose approach-direction requirements, such as approaching an upright mug from above.
- 3.1 Concrete Motivating Example: The mug placement objective constrains the transformed bottom-center keypoint to a target location and penalizes misalignment of the bottom-to-top direction with the upright axis.The upright target orientation is vtarget axis = [0,0,1]T.
- 3.1 Concrete Motivating Example: The resulting inverse-kinematics problem optimizes Taction subject to the placement constraint and orientation cost, then applies the solution using robotic grasping and trajectory planning.The optimization can be solved with off-the-shelf optimization solvers.
- 3.2 General Formulation: For arbitrary category-level tasks, a modeler selects semantic 3D keypoints that are defined across all possible test instances and specifies geometric costs and constraints over them.The formulation is analogous to selecting planning costs and constraints or specifying a reinforcement-learning reward function.
- 3.2 General Formulation: The complete policy is factored into instance segmentation, category-level 3D keypoint detection, kinematic optimization for Taction, and grasping with action execution.The pipeline overview processes an RGBD image, detects keypoints, plans the rigid action, and executes it with an object-agnostic grasp planner.
- 3.2 General Formulation: The detector produces probability heatmaps and depth predictions, recovers 3D keypoints using camera intrinsics, and transforms them into world coordinates using camera extrinsics.Training annotations are obtained from manually labeled 3D reconstructions projected into calibrated camera images; object meshes are not required.
- 3.2 General Formulation: The optimization treats Taction as the decision variable, with detected keypoints entering parametrically and user-defined costs and constraints encoding varied manipulation tasks.Supported primitives include target-distance costs, half-space constraints, point-to-plane contact costs, workspace constraints, and collision avoidance.
4 Comparison and Discussions
The paper compares semantic 3D keypoints with pose representations for category-level manipulation, arguing that keypoints better accommodate shape variation and support flexible geometric targets.
- Keypoint Representation vs Pose Representation: Pose estimators align observations to fixed geometric templates, which cannot adequately capture large intra-category shape variation.Ambiguous suboptimal alignments can produce different manipulation actions and final object configurations.
- Keypoint Representation vs Pose Representation: Semantic 3D keypoints provide a sparse, task-specific representation that can generalize to novel category instances while ignoring task-irrelevant geometry.The paper leverages existing advances in 3D keypoint detection to build a manipulation pipeline.
- Keypoint Target vs Pose Target: Keypoint targets generalize pose-based manipulation and additionally express geometric costs and constraints such as support and upright orientation.Assigning target positions to three or more keypoints is equivalent to pose-based manipulation, while constraints enable more flexible goals.
- Keypoint Target vs Pose Target: Applying a reference pose to a scaled mug can produce physically infeasible states, including penetration of the table.Keypoint optimization instead returns a mug state resting stably on the table.
- Keypoint Target vs Pose Target: Applying a reference pose to a scaled-down mug can miss the rack handle even when the pose matches, whereas a handle-keypoint constraint yields a valid hanging state.The comparison is illustrated for the mug-on-rack task.
- Experiments: The experiments evaluate semantic keypoints on shoes and mugs across shelf-placement and mug-rack hanging tasks.The three tasks are putting shoes on a shelf, putting mugs on a mug shelf, and hanging mugs by their handles.
5 Results
Hardware experiments evaluate the keypoint-based pipeline on shoes and mugs across shelf-placement and rack-hanging tasks, including objects with varied geometry and appearance. The reported results show high task success and placement accuracy.
- Experimental Setup: The hardware system uses a 7-DOF Kuka IIWA LBR arm, a Schunk WSG 50 gripper, and a wrist-mounted RGBD sensor.The demonstrations target category-level manipulation without instancewise tuning or specification.
- Put shoes on a shoe rack: Shoes with different appearance and geometry are used to evaluate policy generality and robustness on the shoe-rack task.The task uses six semantic keypoints with optimization costs and constraints for target placement and surface support.
- Quantitative Results: Figure 7 reports shoe placement errors, mug-shelf bottom-keypoint errors, and mug-rack success rates across mug sizes.Small mugs are defined by handles whose height or width is less than 2cm.
- Put shoes on a shoe rack: Greater than 97% success rate is reported for placing shoes on the rack.The experiments used a held-out set of 20 shoes, with five trials per shoe.
- Put mugs upright on a shelf: 97% of upright mug trials and 88% of horizontal trials had bottom-keypoint final location errors below 3cm.Most error arose from slight object movement during grasping, with the remainder attributed to keypoint detection.
- Hang mugs on a rack: The mug-rack experiment autonomously hangs mugs by their handles, using a handle-center target constraint and top- and bottom-keypoint costs.An intermediate approach goal is shifted 10cm away from the rack before the final straight-line placement.
6 Limitations and Future Work
The paper identifies human annotation and rigid-action assumptions as limitations, and points to synthetic data and broader manipulation policies as future directions.
- The current data collection pipeline requires human annotation, although 3D reconstruction reduces the associated manual labor.
- Synthetic data is proposed as a future direction for training the keypoint detector.
- The rigid transformation action representation applies to robotic pick-and-place but not deformable objects or dexterous rigid-object manipulation.
- Future work could combine keypoint representations with learning- or model-based approaches for category-generalizing manipulation policies.
7 Conclusion
The paper concludes that semantic 3D keypoints provide an interpretable category-level manipulation formulation that supports a factored perception-to-action pipeline. Hardware experiments demonstrate robustness to shape variation and centimeter-level task precision.
- Semantic 3D keypoints represent objects while geometric costs and constraints specify manipulation targets.
- The formulation factors manipulation into keypoint detection, optimization-based action planning, and grasp-based action execution.
- The pipeline combines advances in component sub-problems into a general perception-to-action manipulation system.
- Hardware experiments show robustness to large intra-category shape variation and success on tasks requiring centimeter-level precision.
- The experimental setup uses a 7-DOF Kuka IIWA LBR arm, a Schunk WSG 50 gripper, and a wrist-mounted Primesense Carmine 1.09 RGBD sensor.
B Dataset Generation and Annotation
The dataset pipeline reconstructs static scenes, annotates 3D keypoints, projects labels into images, and composites single-object examples to train multi-instance segmentation.
- The pipeline reconstructs static scenes from multiple RGBD images and uses robot forward kinematics with TSDF fusion to build aligned 3D data.
- Background subtraction and 3D-to-2D label propagation provide pixelwise labels for the generated training images.
- Single-object annotated images are composited into multi-object training images so instance segmentation can handle multiple test-time instances.
- A custom keypoint annotation tool labels 3D keypoints on reconstructed meshes, which are projected into camera images using calibration.
- The dataset contains 117 labeled scenes, including 43 shoe scenes.
C Neural Network Architecture and Training
The neural pipeline uses Mask R-CNN for instance segmentation and an integral-network-based detector for probabilistic 3D keypoint prediction, trained with coordinate, depth, and regression losses.
- Instance segmentation uses a COCO-pretrained Mask R-CNN with an R-101-FPN backbone, fine-tuned for 40,000 iterations on 10,000 generated images.
- The 3D visualization colors the top-center, bottom-center, and handle-center keypoints green, red, and purple, respectively.
- The keypoint detector takes Mask R-CNN crops and predicts a normalized spatial probability map for each keypoint.
- Expected pixel coordinates are recovered from each keypoint’s spatial probability distribution.
- Per-pixel depth predictions are combined to compute each keypoint’s depth coordinate.
- The detector uses integral and heatmap regression losses with a 34-layer ResNet backbone.
D Experiments
Figures 11–13 show tiled before-and-after images from the shoe-on-rack, mug-on-rack, and mug-on-shelf experiments, while the images support discussion of the mug experiments.
- Figures 11–13 present initial RGB images alongside post-manipulation images for the shoe-on-rack and mug experiments.The mug experiments cover both rack and shelf placements.
D.1 Mugs Upright on Shelf
The mug-on-shelf task achieved upright placement within 5 cm of the target in 116 of 118 trials, with two failures attributable to sensing and grasp-related issues.
- 116 of 118 trials succeeded under the criterion that the mug be upright with its bottom-center keypoint within 5 cm of the target.Two trials failed.
- One failure combined inaccurate keypoint detection with grasp-induced mug torque that could not be corrected using only the wrist-mounted camera.An externally mounted camera is proposed to enable re-perception after object motion during grasping.
- The second failure placed the mug upside down because side-view keypoint detection confused the mug’s top and bottom when the handle was occluded.The paper suggests confidence estimates and additional viewpoints as possible remedies.
D.2 Hang mug on rack by its handle
The experiments included mug categories separated by handle size and visualized before-and-after outcomes across rack and shelf placements, including an upside-down failure.
- Mugs were divided into regular and small groups according to handle size.A mug was classified as small when either handle height or width was less than 2 cm.
- Figures 11–13 show before-and-after images for 100 shoe-on-rack, 120 mug-on-rack, and 118 mug-on-shelf trials.
- Figure 14 documents the mug-on-shelf trial that failed because the mug was placed upside down after top-and-bottom keypoint confusion.
- Figure 15 compares five small test mugs with four mugs from the regular category.
- Small handles challenged the pipeline because hanging a mug on the rack requires increased precision.