Source-linked AI summary
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation
Meng Tian, Marcelo H Ang, Gim Hee Lee
TL;DR
Unseen category-level object pose and size estimation is difficult because instances vary in shape and lack known 3D models. The paper learns categorical shape priors, predicts their deformation and dense observation correspondences, and significantly outperforms the state of the art on synthetic and real-world benchmarks.
Problem
Category-level 6D pose and size estimation must handle unseen object instances with unknown 3D models and substantial intra-class shape variation.
Method
An autoencoder learns categorical shape priors from object models, while a deep network predicts deformation to each instance and dense correspondences for pose and size estimation.
Results
The method achieves significantly higher mean average precisions on both synthetic and real-world benchmark datasets.
Takeaways & Limitations
Explicitly modeling deformation from categorical shape priors captures intra-class shape variation for category-level 6D pose estimation.
Takeaways & Limitations
Prediction quality is generally better on synthetic than real data, indicating that observation noise needs further attention; camera reconstruction is less accurate because of its complex, varying geometry.
Abstract
from arXiv · showhide
We present a novel learning approach to recover the 6D poses and sizes of unseen object instances from an RGB-D image. To handle the intra-class shape variation, we propose a deep network to reconstruct the 3D object model by explicitly modeling the deformation from a pre-learned categorical shape prior. Additionally, our network infers the dense correspondences between the depth observation of the object instance and the reconstructed 3D model to jointly estimate the 6D object pose and size. We design an autoencoder that trains on a collection of object models and compute the mean latent embedding for each category to learn the categorical shape priors. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach significantly outperforms the state of the art. Our code is available at https://github.com/mentian/object-deformnet.
1 Introduction
Category-level 6D pose estimation must handle unseen instances with unknown models and substantial within-category shape variation. The paper addresses this by deforming learned categorical shape priors, estimating correspondences, and recovering pose and size.
- Motivation: Instance-level pose methods cannot directly generalize to new category-level instances whose 3D models are unknown.Category, 6D pose, and object size must therefore be estimated concurrently.
- Motivation: High intra-class shape variation makes generalization to unseen object instances extremely challenging, even when other category instances are available as priors.
- Approach: The method reconstructs complete object models in NOCS by estimating deformation from a categorical shape prior and dense correspondences to the observation.The reconstructed model captures instance-specific shape details while the prior encodes shared category geometry.
- Approach: Categorical shape priors are built by decoding the mean latent embedding of object models within each category using an autoencoder.The learning method is applicable across different object categories and data representations.
- Approach: The network uses estimated correspondences between observed depth points and NOCS points with the Umeyama algorithm to recover 6D pose and metric object size.
- Results: The approach achieves significantly higher mean average precisions on both synthetic and real-world benchmark datasets.
2 Related Work
Prior work spans instance-level pose estimation, category-level detection and pose estimation, and shape deformation. This paper’s overview combines segmentation, reconstruction, correspondence estimation, and similarity-transformation fitting.
- Instance-Level Pose Estimation: Instance-level pose methods use voting and refinement, holistic template matching, latent-code nearest neighbors, or direct image-to-pose prediction.
- Category-Level Object Detection: Category-level 3D detection methods generate proposals in 3D or lift 2D proposals, while single-stage detectors directly use 3D data.
- Category-Level Pose Estimation: Early category-level pose work used semantic centers and part-based forests, while later methods introduced generative pose distributions and unified NOCS representations.
- Shape Deformation: Shape-deformation methods apply free-form or progressive deformation to voxel, mesh, and point-cloud representations.
- Our Approach: The proposed pipeline segments each instance, converts masked depth to a point cloud, reconstructs the object, establishes correspondences, and fits a similarity transformation for pose recovery.
3 Our Method
The method learns categorical shape priors from aligned object models, then deforms each prior using RGB-D observations to reconstruct an instance and estimate its pose and size. Dense correspondences, deformation prediction, symmetry handling, and similarity-transform recovery form the main estimation pipeline.
- Problem: The task uses RGB-D observations of unseen instances from known categories, representing each object by a class label and an amodal 3D bounding box with 6D pose and size.The 6D pose is the rotation and translation transforming an object from reference coordinates to the camera frame.
- 3.1 Categorical Shape Prior: A PointNet-like autoencoder reconstructs aligned object point clouds, and category mean latent embeddings are decoded into categorical shape priors.The priors encode shared geometric characteristics across instances within each category.
- 3.2 Our Network Architecture: The network takes an instance point cloud, image patch, and shape prior, then predicts a deformation field D and soft correspondence matrix A.The reconstructed model is M = Mc + D, while each row of A distributes correspondence weights from an observed point to reconstructed-model points.
- 3.2 Our Network Architecture: Instance and prior branches exchange global features to combine instance-specific details with coarse category geometry when predicting deformation and correspondences.RGB-D features combine point-cloud geometry with image features before the prediction branches.
- 3.3 Pose and Size Estimation: The Umeyama algorithm recovers rotation, translation, and metric scale from observed points and their estimated NOCS coordinates, with RANSAC used for robust estimation.Training uses reconstruction, correspondence, entropy, and deformation regularization losses, with smooth L1 supervision for NOCS coordinates.
- 3.4 Loss Functions: Symmetry ambiguity is handled by mapping rotation labels to a canonical representative using the object's appearance-preserving symmetry group.The evaluated datasets assume continuous symmetry around the y-axis of the NOCS.
4 Experiments
Experiments on CAMERA25 and REAL275 evaluate category-level pose, size, and shape reconstruction, showing improvements over the Baseline and benefits from explicit deformation modeling.
- Datasets and metrics: The method is evaluated on CAMERA25 and REAL275, which contain unseen instances across six object categories and use mAP for detection and pose thresholds.CAMERA25 is synthetic, while REAL275 contains real-world images; pose evaluation uses rotation and translation error thresholds.
- Comparison to Baseline: 83.1% mAP for 3D IoU at 0.75 and 54.3% mAP for 6D pose at 5°2cm on CAMERA25 exceed Baseline results by 14% and 22%.Without depth, the method remains above the Baseline by +15.5% and +17.9%, while depth further improves overall performance.
- Comparison to Baseline: 23.1% and 12.1% mAP improvements over the Baseline on REAL275 are reported for 3D IoU at 0.75 and 6D pose at 5°2cm.Under strict comparison, the corresponding improvements are 16.4% and 8.5%.
- Shape reconstruction: 1.97 and 3.17 CD on CAMERA25 and REAL275 improve over shape-prior CD values of 3.70 and 4.41, respectively.The comparison indicates that estimating deformation improves reconstructed-model quality over the undeformed categorical priors.
- Ablation studies: Embedding, NN, and Random shape priors produce generally stable performance, while removing priors decreases real-dataset mAP by 3.7%.The learned Embedding prior gives the best accurate-pose result and is selected as the best model.
- Ablation studies: Direct Regression lowers 6D-pose mAP at 5°2cm by 3.1% on CAMERA25 and 5.6% on REAL275 versus Embedding.Removing deformation or correspondence regularization causes real-dataset pose drops of 5.9% and 3.6%.
- Qualitative results: Qualitative results show reliable pose and size recovery from noisy correspondences, with better predictions on synthetic than real data.Camera objects have less accurate reconstruction because of their more complicated and varying geometry.
5 Conclusions
The paper presents a category-level 6D object pose estimation approach that explicitly models intra-class shape variation through deformation from latent-space shape priors. Experiments on synthetic and real datasets demonstrate its advantage.
- The approach estimates deformation from a shape prior to reconstruct object models and explicitly models intra-class shape variation.
- Shape priors are learned from a collection of object models and constructed in latent space.
- Experiments on synthetic and real datasets demonstrate the advantage of the proposed approach.
A Comparison to CASS
Compared with CASS, the approach estimates deformation from a corresponding shape prior and recovers pose through dense correspondences rather than direct pose regression. It significantly outperforms CASS in pose accuracy on REAL275.
- The method estimates a deformation field for a corresponding shape prior, whereas CASS uses a variational autoencoder to generate the point cloud.
- Pose is recovered by establishing dense correspondences, while CASS directly regresses pose and size from pose-independent and pose-dependent features.
- The approach significantly outperforms CASS in pose accuracy on REAL275.
B Comparison to 6-PACK
The method achieves comparable REAL275 tracking accuracy to 6-PACK without pose initialization or temporal consistency, while its estimation accuracy remains stable without initial sequence frames.
- 30.4% versus 33.3% at 5°5cm on REAL275 shows comparable accuracy with 6-PACK.
- Unlike 6-PACK, the approach requires neither pose initialization nor temporal consistency.
- When the first 40 frames are excluded, 6-PACK accuracy drops below 30%, whereas this method’s accuracy remains stable.
C Qualitative Results
Per-frame pose detection is better on synthetic data than real data. The performance gap is mainly attributed to observation noise, especially for objects with complicated geometry.
- Pose detection results are better on synthetic data than on real data.
- Observation noise mainly induces the synthetic–real performance gap and has greater influence on objects with complicated geometric shapes.
D Runtime Analysis
The implementation runs at approximately 4 FPS on 640 × 480 RGB-D images with a mean of four objects. Runtime is divided among instance segmentation, network inference, and pose alignment.
- Approximately 4 FPS is achieved on 640 × 480 RGB-D images containing a mean of four objects.The reported hardware is an Intel Core i7-5960X CPU and NVIDIA GTX 1080Ti GPU.
- Qualitative results compare the approach with ground truths on CAMERA25 and REAL275.CAMERA25 results occupy the top two rows, while REAL275 results occupy the bottom two rows.
- Instance segmentation takes 130 ms, network inference 100 ms, and pose alignment 20 ms on average.
E Visualization of Shape Priors
The paper visualizes the categorical shape priors used in its ablation studies.
- Different categorical shape priors are visualized for the ablation studies.
F Derivation of the Map Operator
The Map operator is derived from a proposition about proper symmetry groups and is specialized for objects symmetrical around the y-axis. Its solution minimizes a Frobenius norm over the symmetry angle.
- The proper symmetry group consists of rotations that preserve a given object's appearance.
- Proposition 1 establishes when Map(R1) and Map(R2) are equal using the object's proper symmetry group and the 3×3 identity matrix.
- The proof is omitted, with readers directed to reference for details.
- For objects symmetrical around the y-axis, Corollary 1 gives the corresponding Map operator.
- The derivation rewrites the Frobenius norm as an expression involving rotation-matrix entries and the symmetry angle θ.
- The symmetry angle is obtained by minimizing the Frobenius norm over θ.