Source-linked AI summary
Category-Level Articulated Object Pose Estimation
Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A. Lynn Abbott, Shuran Song
TL;DR
The paper addresses category-level articulated pose estimation for unseen object instances from a single depth image, where exact CAD models may be unavailable. It introduces ANCSH and a PointNet++-based predictor, then uses canonicalized joints and combined optimization to improve part estimates and joint-parameter accuracy, with heavy occlusion as a failure boundary.
Problem
Category-level articulated pose estimation must recover part and joint attributes for novel instances despite intra-category variation, physical joint constraints, and methods that often require exact CAD models.
Method
ANCSH uses normalized articulated object and part coordinate spaces, while a PointNet++ network predicts the representation and joint-aware optimization regularizes part poses.
Results
The experiments significantly improve part pose prediction and joint-parameter estimation, including high camera-space accuracy after transforming predictions from NAOCS.
Takeaways & Limitations
Canonicalized joints provide kinematic constraints for part pose and scale estimation and support accurate camera-space joint-parameter prediction.
Takeaways & Limitations
Heavy occlusion can cause inaccurate predictions, with prismatic joints additionally exhibiting ambiguity in the size of heavily occluded parts.
Abstract
from arXiv · showhide
This project addresses the task of category-level pose estimation for articulated objects from a single depth image. We present a novel category-level approach that correctly accommodates object instances previously unseen during training. We introduce Articulation-aware Normalized Coordinate Space Hierarchy (ANCSH) - a canonical representation for different articulated objects in a given category. As the key to achieve intra-category generalization, the representation constructs a canonical object space as well as a set of canonical part spaces. The canonical object space normalizes the object orientation,scales and articulations (e.g. joint parameters and states) while each canonical part space further normalizes its part pose and scale. We develop a deep network based on PointNet++ that predicts ANCSH from a single depth point cloud, including part segmentation, normalized coordinates, and joint parameters in the canonical object space. By leveraging the canonicalized joints, we demonstrate: 1) improved performance in part pose and scale estimations using the induced kinematic constraints from joints; 2) high accuracy for joint parameter estimation in camera space.
1. Introduction
The paper targets category-level articulated pose estimation from a single depth image without exact instance CAD models, introducing ANCSH and joint-aware optimization for unseen objects.
- Existing articulated-object methods often require exact CAD models and joint parameters at test time, limiting generalization to new instances.
- The task estimates per-part segmentation, 6D poses, scales, amodal boxes, joint parameters, and joint states for novel instances in known categories.
- ANCSH combines a normalized articulated object space with normalized part spaces to represent object-level and part-level attributes across category instances.
- The method predicts ANCSH from one depth point cloud and uses joint constraints to regularize part poses through combined optimization.
- Experiments report improved part pose and scale prediction plus high camera-space joint-parameter accuracy after transforming canonical-space predictions.
2. Related Work
Related work spans rigid and articulated pose estimation, category-level canonical coordinates, active manipulation, and body or hand-specific articulated models.
- Rigid object pose estimation: Rigid-object pose methods commonly assume exact 3D CAD models and estimate instance-level pose by alignment or object-coordinate voting.
- Rigid object pose estimation: Category-level pose estimation uses a canonical representation such as NOCS, with normalized sizes and aligned orientations to support intra-category generalization.
- Articulated object pose estimation: Articulated-object methods often use instance CAD models and known kinematic parameters to constrain separate part-pose recovery.
- Articulated object pose estimation: Active-manipulation approaches infer articulation from image sequences observing different object states rather than a single depth image.
- Human body and hand pose estimation: Human body and hand methods rely on class-specific skeletal models, expected joint structures, and strong shape priors.
3. Problem Statement
The system receives a single-depth-derived point cloud of an unknown instance from a known category and predicts part geometry, poses, and joint attributes within a shared kinematic chain.
- The input is a point cloud backprojected from one depth image of an unknown object instance in a known category.
- Objects share a kinematic chain with M rigid parts and K joints whose types are known.
- The output includes part segmentation, 3D rotations, translations, sizes, joint parameters, and joint states.
- For revolute joints, the parameter is a rotation-axis direction and pivot point, while the state is relative rotation angle from a predefined rest state.
- For prismatic joints, the parameter is a translation-axis direction, while the state is relative translation distance from a predefined rest state.
4. Method
ANCSH represents articulated objects with canonical object- and part-level spaces, then predicts this hierarchy from depth and uses it to estimate poses and joints under kinematic constraints.
- ANCSH Representation: ANCSH is a two-level hierarchy with NAOCS as the object-level reference frame and NPCS as a reference frame for each part.NAOCS normalizes object pose, scale, and articulation; NPCS further normalizes each part’s pose and scale.
- ANCSH Representation: NAOCS normalizes articulations to predefined rest states, zero-centers objects, aligns orientations, and uniformly scales them.Examples include right-angle eyeglass revolute joints and closed drawers.
- ANCSH Representation: NPCS normalizes each part’s position and size while preserving orientation, enabling transformations from part space to camera space for per-part pose and scale.NAOCS instead encodes canonical relationships among parts from a holistic object-level perspective.
- ANCSH Network: The PointNet++ network takes a depth point cloud and predicts part segmentation, NPCS coordinates, NPCS-to-NAOCS transformations, and NAOCS joint parameters.The architecture uses two PointNet++ modules and applies to objects with revolute or prismatic joints.
- ANCSH Network: The network predicts NAOCS transformations using 1D scaling and 3D translation, with NAOCS coordinates computed as scaled NPCS coordinates plus translation.Part labels select the corresponding per-point NPCS and transformation predictions.
- ANCSH Network: Joint parameters are predicted through point-to-joint association and dense regression, then aggregated by voting across points associated with each joint.The predicted parameters represent revolute rotation axes and pivots or prismatic translation axes in NAOCS.
5. Evaluation
Evaluation uses synthetic and real depth datasets, ablated baselines, quantitative and qualitative comparisons, and analyses of optimization, joint estimation, generalization, and failure cases.
- Experimental Setup: Evaluation measures part rotation and translation errors, amodal bounding-box 3D IoU, joint-state errors, and joint-parameter errors.Revolute joints use angle and axis-orientation errors; prismatic joints use relative translation and translation-axis measures.
- Experimental Setup: Testing includes unseen synthetic instances generated from Shape2Motion and SAPIEN, plus real-world depth data from four object instances.Synthetic testing averages 3000 images per category; the real dataset contains Kinect video sequences.
- Experimental Setup: The NPCS and NAOCS ablations provide baseline comparisons because no existing category-level articulated pose-estimation methods are available.NPCS estimates parts independently, whereas the full system combines NPCS, joint parameters, and global scaling and translation values.
- Experimental Results: Combined optimization consistently improves predictions across almost all categories and evaluation metrics, especially for thin eyeglass temples with few visible points and ambiguous shapes.NAOCS joint parameters regularize part poses through kinematic-chain constraints during pose optimization.
- Experimental Results: The canonical-space joint-parameter approach significantly improves joint-axis prediction for unseen instances over direct PointNet++ voting.It predicts in NAOCS, reduces noise through voting, and transforms parameters through NPCS and part poses into camera coordinates.
- Experimental Results: 96.25%, 92.3%, 96.9%, and 79.8% AD accuracy are achieved on laptop, cabinet, cupboard, and toy-train kinematic chains, respectively.The method was not trained on real-world depth images and is reported as performing on par with or better than state-of-the-art.
- Failure Cases: Heavy occlusion causes inaccurate predictions, while severe occlusion around prismatic joints creates ambiguity in heavily occluded-part size.NAOCS remains reliable for estimating joint state and translation-axis parameters despite this size ambiguity.
6. Conclusion
The paper presents ANCSH-based category-level articulated pose estimation from a single depth image and reports improved part pose and joint-parameter accuracy. It also evaluates real-world instance-level depth performance and documents representative failure cases.
- 6. Conclusion: ANCSH accommodates unseen articulated object instances with large intra-category variations.The representation supports the paper’s category-level pose estimation setting.
- 6. Conclusion: Table 3 evaluates AD accuracy for the whole kinematic chain and individual parts on a real-world depth benchmark.The benchmark caption reports comparable state-of-the-art performance and improvements for challenging laptop cases.
- 6. Conclusion: Failure cases include heavy occlusion on eyeglasses and drawer shape variation with only the drawer front visible.The drawer prediction is oversized, although joint state and parameters remain reliably predicted through NAOCS.
- 6. Conclusion: A combined optimization problem uses part pose errors and joint constraints to improve part pose prediction and joint-parameter estimation.The optimization jointly considers predicted part poses and induced joint constraints.
A. Implementation Details
The experiments use uniformly sampled depth-point clouds and a fixed TensorFlow training configuration across object categories.
- A. Implementation Details: Inputs contain N=1024 uniformly sampled points from the whole back-projected depth point cloud.The same point-count setting is used across experiments.
- A. Implementation Details: Models are trained on a single Nvidia V100 GPU with batch size 16.This hardware and batch configuration is reported for all experiments.
- A. Implementation Details: Training starts with learning rate 0.001 and decays it by 0.7 every 200k steps.The reported experiments usually converge after more than 150k steps.
B. Data generation and statistics
The study combines synthetic depth data rendered from articulated-object datasets with real RGB-D sequences, while reporting category part definitions and train/test instance counts.
- B. Data generation and statistics: Synthetic depth images are rendered from Shape2Motion and SAPIEN object models containing geometry and articulation information.These descriptions provide ground truths during rendering.
- B. Data generation and statistics: Random joint states are generated within each object’s motion ranges, and depth images with masks are rendered from random camera viewpoints.Camera poses producing complete part occlusion are filtered out.
- B. Data generation and statistics: Real data come from the ICCV2015 Articulated Object Challenge and include laptops, cabinets, cupboards, and toy trains.Each object has two testing sequences of about 1000 RGB-D images, with a fixed articulation state per sequence.
- B. Data generation and statistics: Figure 6 lists part definitions and the numbers of object instances used for training and testing in each category.The statistics cover the synthetic-data experiments.
C. Handling severe occlusion cases
ANCSH retains pose-estimation gains under severe occlusion compared with the NPCS baseline, using occlusion measured by visible surface-area ratio.
- C. Handling severe occlusion cases: ANCSH improves pose estimation over the NPCS baseline under severe occlusion.The evaluation uses data from part 2 of unseen eyeglasses.
- C. Handling severe occlusion cases: Occlusion level is defined as the visible-area ratio relative to the total mesh surface for each part.This definition supports comparison across different occlusion levels.
- C. Handling severe occlusion cases: The analysis examines ANCSH performance across different levels of occlusion.The reported comparison focuses on unseen eyeglasses.
D. Additional results
Additional qualitative results are presented for both synthetic and real-world datasets. The synthetic examples visualize segmentation, normalized coordinates, and joint voting, while the real-world examples show predicted part bounding boxes.
- Figure 8 presents additional qualitative results on the category-level synthetic dataset.
- Figure 9 shows additional qualitative results on a real-world instance-level depth dataset.The examples cover four objects from the ICCV2015 Articulated Object Challenge.
- Synthetic visualizations compare predicted and ground-truth part segmentation masks, NPCS, and NAOCS.RGB channels encode the normalized coordinates in the NPCS and NAOCS visualizations.
- The synthetic results also visualize joint voting for revolute and prismatic joints using offset vectors and joint-axis directions.
- For real-world examples, the method estimates 3D tight bounding boxes for all parts on the kinematic chain.Only depth images are used for pose estimation; RGB images are included for reference.