PaperlayerView exact revision on arXiv

Source-linked AI summary

GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts

Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, He Wang

arXiv:2211.05272v2cs.CV

TL;DR

Cross-category object perception and manipulation remain underexplored because existing methods mainly operate within object categories. The paper introduces GAParts and GAPartNet, develops domain-generalizable perception, and uses part poses for manipulation, outperforming prior methods while retaining a stated pose-dependence limitation.

  • Problem

    Existing work approaches object perception and manipulation intra-category, motivating part-level generalization to unseen object categories.

  • Method

    The paper defines Generalizable and Actionable Parts, builds GAPartNet, learns domain-invariant perception features, and designs pose-based manipulation heuristics.

  • Results

    The approach outperforms previous methods in segmentation and pose estimation and enables effective, generalizable manipulation in simulation and the real world; segmentation reaches 76.5% AP50 on seen and 37.2% AP50 on unseen categories.

  • Takeaways & Limitations

    GAPart semantics and poses provide a shared basis for cross-category perception and manipulation with minimal human interaction-guidance effort.

  • Takeaways & Limitations

    The manipulation heuristics rely on precise part pose predictions, and cross-category generalizability still has room for improvement.

Abstract

from arXiv · show

For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this work, we propose to learn such cross-category skills via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (lids, handles, etc.) in 27 object categories, we construct a large-scale part-centric interactive dataset, GAPartNet, where we provide rich, part-level annotations (semantics, poses) for 8,489 part instances on 1,166 objects. Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation. Given the significant domain gaps between seen and unseen object categories, we propose a robust 3D segmentation method from the perspective of domain generalization by integrating adversarial learning techniques. Our method outperforms all existing methods by a large margin, no matter on seen or unseen categories. Furthermore, with part segmentation and pose estimation results, we leverage the GAPart pose definition to design part-based manipulation heuristics that can generalize well to unseen object categories in both the simulator and the real world. Our dataset, code, and demos are available on our project page.

1. Introduction

GAPartNet reframes cross-category perception and manipulation around functional parts whose shapes and actionability transfer across object categories. It introduces a part-centric dataset, three cross-category tasks, and domain-generalizable perception and manipulation methods.

  • Existing perception and manipulation research mainly generalizes within known object categories, whereas functional parts support recognition and interaction across unseen categories.
  • GAParts are part classes whose members share similar shapes and aligned actionability, reducing human effort when designing robust manipulation guidance.
  • GAPartNet contains 1,166 articulated objects and 8,489 annotated GAPart instances, providing semantic and pose labels for part-centric perception and manipulation.
  • The paper investigates cross-category part segmentation, part pose estimation, and part-based object manipulation on known and unseen object categories.
  • 76.5% AP50 is achieved on seen categories and 37.2% AP50 on unseen categories using part-oriented domain adversarial techniques for segmentation.
  • Part-pose-based interaction policies enable effective and generalizable manipulation in simulation and the real world with minimal interaction-guidance effort.

2. Related Work

Related work covers 3D part segmentation, domain generalization, category-level pose estimation, and generalizable manipulation, while emphasizing their predominantly intra-category scope.

  • Part Instance Segmentation from Point Cloud Observations: Existing 3D part segmentation datasets and networks support shape learning but generally approach object perception within individual categories.
  • Domain Generalization: Domain generalization methods learn from multiple source domains to generalize to unseen domains through data manipulation, learning strategies, or invariant representations.
  • Category-level Object Pose Estimation: Category-level pose methods use normalized coordinate spaces, canonical latent shapes, shape priors, or specialized networks to estimate object poses.
  • Generalizable Object Manipulation: Generalizable manipulation research includes articulated-object skills for known categories and object-agnostic grasping for simple rigid objects, but novel categories remain relevant.

3. GAPart Definition and GAPartNet Dataset

The paper defines GAParts through geometric similarity and actionability alignment, then constructs GAPartNet with standardized semantics and poses across diverse object categories.

  • GAPart Definition: GAPart classes require both geometric similarity within each class and aligned actionability across class members.
  • GAPart Semantics: The dataset identifies 9 GAPart classes across 27 object categories, including handles, lids, buttons, drawers, doors, hinges, and knobs.
  • GAPart Poses: GAPart poses are canonicalized in Normalized Part Coordinate Space, with innate rotational or mirror symmetries accounted for during pose handling.
  • GAPart Poses: Once GAPart classes and poses are known, simple heuristics can support manipulation across object categories.
  • GAPartNet Dataset: GAPartNet provides semantic and pose annotations for 8,489 GAPart instances on 1,166 objects from 27 categories, averaging 7.3 functional parts per object.
  • GAPartNet Dataset: The dataset combines PartNet-Mobility and AKB-48, adds uniform annotations, and performs mesh fixing, part re-merging, semantic labeling, and pose alignment.

4. Problem Formulation

The paper formulates cross-category perception and manipulation from partial point clouds, requiring models trained on seen categories to operate on unseen categories.

  • The framework studies cross-category generalizable object perception and manipulation using GAPart-based representations.
  • Perception: Perception predicts each GAPart’s segmentation mask, class, rotation, translation, and size from a partial colored point cloud.
  • Perception: Perception networks are trained on seen object categories and evaluated for generalization to unseen categories without seeing them during training.
  • Manipulation: Manipulation requires a pose-based interaction policy that uses a partial point cloud to act on a target part, such as opening a door in an unseen category.

5. Method

The method learns domain-invariant GAPart features for 3D part segmentation and pose estimation through domain-adversarial training, then supports part-based manipulation using predicted part poses.

  • Architecture Overview: The pipeline extracts point-wise features with Sparse UNet, generates mask proposals, scores them, and applies NMS to produce final part masks.The segmentation network uses a Dual Set Grouping module before scoring and suppression.
  • Domain-invariant GAPart Feature Learning: A domain classifier and Gradient Reverse Layer encourage the feature extractor to learn features that cannot distinguish object-category domains.The classifier predicts domain labels, while reversed classification gradients update the feature extractor toward domain-invariant representations.
  • Part-oriented Feature Query: Part-oriented feature queries focus adversarial learning on foreground GAParts, reducing reliance on domain-specific context.Only mask-proposal features whose scores exceed sthre are passed to the domain classifier.
  • Multi-resolution: Multi-resolution features address GAPart size variation, while these features serve domain-adversarial learning rather than mask-proposal grouping.The implementation queries proposal features from three UNet decoder layers and uses them for adversarial training.
  • Distribution-balancing: A part-level focal loss addresses severe distribution imbalance during adversarial training, whose combined design encourages domain-invariant GAPart feature learning.The loss weights depend on part class, domain distribution, mean domain-classification accuracy, and a hyper-parameter.
  • Pose Estimation and Manipulation: For pose estimation, predicted mask features regress point-wise NPCS coordinates, then RANSAC and Umeyama recover each part’s 7-dimensional rigid transformation.A symmetry-aware NPCS loss handles different symmetry patterns, and the GAPart pose definition provides information for joint prediction and manipulation policies.

6. Experiments

Experiments evaluate domain-generalizable part segmentation, pose estimation, and manipulation across seen and unseen object categories. The proposed perception model improves segmentation and pose estimation, while pose-based heuristics transfer manipulation to simulation and real objects.

  • Experimental Setup: The evaluation splits 27 object categories into 17 seen and 10 unseen categories, training on seen categories and testing GAPart understanding on unseen categories.All GAPart classes are represented in both seen and unseen categories.
  • Part Segmentation: AP50 measures part-segmentation average precision at 50% IoU, with AP averaging precision over IoU thresholds from 50% to 95%.
  • Part Segmentation: 7.7 percentage points on seen categories and 6.7 percentage points on unseen categories separate the method from the respective second-ranked AP50 results.The corresponding relative improvements are 11.2% and 22.0%.
  • Part Segmentation: Part-oriented feature querying, multi-resolution processing, and distribution balancing each improve segmentation performance across seen and unseen categories.Domain adversarial training improves unseen-category generalization but somewhat reduces seen-category performance when using object global features.
  • Part Pose Estimation: The method outperforms baselines on most pose-estimation metrics for seen categories and all metrics for unseen categories.Pose evaluation includes rotation, translation, scale, interaction-axis errors, 3D mIoU, and 5°5cm and 10°10cm accuracies; pose is evaluated only for detected parts.
  • Part-based Manipulation: Pose-based heuristics guide the arm toward predicted parts, orient and close the gripper, then follow GAPart-defined trajectories; they generalize to unseen categories in simulation.Manipulation success requires opening the target part through 90% of its motion range within 1,000 time-steps and stopping stably.
  • Part-based Manipulation: The method successfully predicts part segmentation and poses on real objects and enables cross-category robot manipulation despite training on synthetic data.Real-world manipulation results are shown in Figure 6(b).

7. Conclusion

The paper concludes that generalizable and actionable parts support cross-category perception and manipulation. GAPartNet, domain-generalizable perception, and part-pose-based policies produce effective manipulation in simulation and the real world, while robustness remains limited by pose accuracy.

  • GAPartNet annotates cross-category part semantics and poses for studying part segmentation, part pose estimation, and part-based object manipulation.
  • The proposed approach outperforms previous works in segmentation and pose estimation and supports effective, generalizable manipulation in simulation and the real world.The interaction policies use part poses together with the GAPart definition and domain-generalizable perception model.
  • Generalizability still has room for improvement, and heuristic manipulation relies on precise part pose predictions for robust strategies.The authors identify more robust manipulation strategies as future research.

A.1. Data Annotation

GAPartNet is built by cleaning, relabeling, and annotating object data to provide consistent part semantics, poses, and rendered task inputs across 1,166 articulated objects.

  • The annotation pipeline identifies database problems, then systematically processes existing object data for the new part-centric annotations.
  • The source data combines 23 categories from PartNet-Mobility with 4 from AKB-48, retaining overlapping category names separately because their shapes can differ substantially.
  • Existing databases lack consistent cross-category part semantics and sufficiently fine-grained annotations, including for fixed handles.
  • The authors manually re-separate fixed-handle meshes, modify kinematic chains and joints, and repair imperfect meshes; more than 100 object instances are modified.
  • The final 1,166 shapes receive semantic annotations, oriented tight bounding boxes, and GAPart pose annotations for 8,489 parts.
  • SAPIEN rendering produces partial point clouds, semantic and instance masks, NPCS maps, depth and RGB images, and part pose annotations for the three proposed tasks.

B.1. Details on Network Architecture

The network combines PointGroup-style part segmentation, NPCS-based pose prediction, symmetry-aware supervision, and domain adversarial learning, followed by GAPart-specific manipulation policies.

  • Network Architecture: The vision network voxelizes point clouds and uses a seven-depth encoder-decoder backbone to produce point-wise features for mask proposal generation.
  • Network Architecture: Proposal scoring uses a second voxelization, ROI pooling, and confidence prediction, with low-scoring points and proposals filtered during inference.
  • Domain Generalization: Domain adversarial learning adds a gradient reversal layer and three discriminators at different decoder depths to classify domain-specific features.
  • Pose Estimation: NPCS-Net predicts normalized coordinates for each mask using nine semantic groups of multilayer perceptrons, supervising only the ground-truth class channel.
  • Pose Estimation: Symmetry-aware pose supervision tolerates class-specific transformations, including 180° flips, continuous rotations, and no symmetry for slider drawers.
  • Training: Training proceeds in four stages, progressively adding proposal scoring, NPCS prediction, and domain adversarial learning after segmentation proposals stabilize.
  • Pose and Manipulation: RANSAC removes pose outliers and Umeyama estimates a seven-dimensional rigid transformation, while GAPart geometry directly determines joint parameters.
  • Manipulation: The manipulation heuristics map GAPart types to actions such as grasping, pressing, rotating, pulling, or opening along predicted axes.

C.2. Simulation Experiments

Simulation evaluates GAPart-based manipulation under first-frame partial-point-cloud observations, comparing the proposed heuristics with oracle-assisted baselines across seen and unseen categories.

  • Benchmark Settings: The simulator benchmarks opening drawers and doors, manipulating handles, and pressing buttons with a single Franka gripper.
  • Benchmark Settings: Each task uses 20 objects from seen categories and 20 from unseen categories, totaling 160 objects across 6 seen and 6 unseen categories.
  • Part-pose-based Manipulation Heuristics: The method converts predicted part poses into grasping poses, uses motion planning, and executes interaction policies with predicted axes.
  • Baselines: Where2act receives extra handle-center information, while ManiSkill receives per-frame point clouds and ground-truth part masks as oracle inputs.
  • Real-world Evaluation: Real-world tests use KINOVA and FRANKA arms with RGB-D partial point clouds, ArUco calibration, predicted part boxes, and heuristic trajectories.
  • Qualitative Results: Figure 7 provides additional qualitative results for real-world part-based object manipulation.

D. Visualization of GAPartNet Dataset

The paper visualizes GAPartNet examples across seen and unseen categories and provides additional visualizations of part segmentation and pose estimation.

  • Figure 8 shows exemplar objects for each GAPart class from seen and unseen categories in GAPartNet.
  • Figures 9 to 11 visualize additional part segmentation and part pose estimation results.

F. More Results of Part-based Object Manipulation

The paper evaluates GAPart-based manipulation in simulation and the real world, while examining depth quality and outlier part shapes as practical boundaries.

  • Simulation Experiments: Simulation results show that the method significantly outperforms baselines on all four manipulation tasks.The authors attribute this to the generalizability of their part-pose-based manipulation policy.
  • Real-world Experiments: Real-world experiments provide qualitative evidence for part segmentation, pose estimation, and manipulation on unseen objects.Additional results are reported in Figure 7 and the project video.
  • Real Depth Signal and Sim-to-Real Gap: Depth quality is crucial for the perception and downstream manipulation pipeline.Transparent and difficult objects required contrast-aid paint or close structural scanning to obtain detailed geometry.
  • Real Depth Signal and Sim-to-Real Gap: Metallic or transparent objects can produce incomplete depth, which the current work leaves for future geometry prediction or refinement.The authors distinguish this from diffuse objects with acceptable depth, where domain adaptation may help.
  • Outlier Part Shapes: GAParts are defined by similar geometry and actionability, leaving outlier-shaped parts such as curvy or irregular door handles as a limitation.Their poses may be less informative for manipulation heuristics even when their function remains aligned with regular handles.
  • Outlier Part Shapes: The authors suggest learning manipulation policies from actionable information rather than relying on heuristics to handle outlier shapes.This direction is proposed as potentially useful for irregular door handles whose function remains the same as regular handles.
Loading 2211.05272v2…