Source-linked AI summary
Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation
Dengsheng Chen, Jun Li, Zheng Wang, Kai Xu
TL;DR
Category-level 6D pose and size estimation must handle unseen objects and substantial intra-class shape variation without exact CAD models. The paper learns a generative canonical shape space from 3D shapes and RGBD images, then contrasts pose-independent and pose-dependent features in an end-to-end network. On public category-level datasets, it reports state-of-the-art pose accuracy and comparably high size accuracy, while limitations remain for complex shapes and very high precision.
Problem
Category-level pose estimation lacks exact CAD models for unseen instances and must accommodate substantial intra-class shape variation, making dense correspondence difficult.
Method
The method learns CASS as a VAE latent space of pose-normalized, metric-sized 3D shapes and estimates pose by contrasting pose-independent RGBD embeddings with pose-dependent features.
Results
The method achieves state-of-the-art pose accuracy and comparably high size accuracy on public category-level datasets.
Takeaways & Limitations
A learned holistic canonical shape representation provides a correspondence-free approach to category-level object pose and size estimation.
Takeaways & Limitations
The method handles very complex shapes poorly and does not achieve very high precision at the 5°5cm metric.
Abstract
from arXiv · showhide
We present a novel approach to category-level 6D object pose and size estimation. To tackle intra-class shape variations, we learn canonical shape space (CASS), a unified representation for a large variety of instances of a certain object category. In particular, CASS is modeled as the latent space of a deep generative model of canonical 3D shapes with normalized pose. We train a variational auto-encoder (VAE) for generating 3D point clouds in the canonical space from an RGBD image. The VAE is trained in a cross-category fashion, exploiting the publicly available large 3D shape repositories. Since the 3D point cloud is generated in normalized pose (with actual size), the encoder of the VAE learns view-factorized RGBD embedding. It maps an RGBD image in arbitrary view into a pose-independent 3D shape representation. Object pose is then estimated via contrasting it with a pose-dependent feature of the input RGBD extracted with a separate deep neural networks. We integrate the learning of CASS and pose and size estimation into an end-to-end trainable network, achieving the state-of-the-art performance.
1. Introduction
The paper addresses category-level 6D pose and size estimation without exact CAD models by learning a canonical shape space that represents varied category instances. Its end-to-end network estimates pose and size through pose-independent and pose-dependent feature comparison, achieving state-of-the-art pose accuracy and comparably high size accuracy.
- Motivation: Exact CAD-model requirements limit the practical use of high-accuracy instance-level pose-estimation methods.Category-level estimation instead targets unseen instances without corresponding CAD models and must handle substantial intra-class variation.
- Motivation: Dense mappings to a shared coordinate space become difficult to learn when objects exhibit many unknown shape variants.The paper identifies dense correspondence under significant shape variation as an ill-posed problem.
- Canonical Shape Space: CASS represents category instances in a latent space of canonical 3D shapes with normalized pose and actual metric size.A VAE generates canonical 3D point clouds from RGBD images using cross-category training and large 3D shape repositories.
- Pose and Size Estimation: The encoder maps arbitrary-view RGBD images to pose-independent shape representations, while pose is estimated by contrasting them with pose-dependent RGBD features.This feature-contrasting strategy avoids estimating dense correspondences between representations.
- End-to-End Design: The method jointly learns CASS and pose and size estimation in an end-to-end trainable network.Key designs include cross-category shape training, RGBD-and-shape encoder inputs, feature contrasting, distribution matching, and metric-size reconstruction.
- Results: The method achieves state-of-the-art pose accuracy and comparably high size accuracy on public category-level datasets.The paper presents this evaluation outcome as a principal result.
2. Related Work
Related work moves from instance-level pose estimation with known CAD models toward category-level methods that must handle unseen instances and substantial intra-class variation. CASS differs from NOCS by learning an implicit generative shape space and estimating pose from holistic shape representations rather than pixel-wise correspondence.
- Instance-level approaches: Instance-level methods commonly match RGB(D) templates or features against CAD models to retrieve or estimate object pose.Some approaches regress pose directly from RGB(D), while others use feature detection, registration, or pose-hypothesis verification.
- Category-level approaches: Category-level estimation targets unseen objects without available CAD models, making intra-class variation in color, texture, shape, and size a central challenge.Correspondence-based methods become difficult when the target differs substantially from seen instances.
- Category-level approaches: NOCS represents category instances in a shared normalized space and estimates pose through pixel-wise image-to-coordinate correspondence plus shape matching.Its pipeline combines an instance mask and depth map with the inferred correspondence.
- CASS vs. NOCS: CASS is an implicitly learned generative shape embedding, whereas NOCS explicitly aligns category instances in normalized 3D coordinates.CASS is treated as a holistic canonical representation for end-to-end, correspondence-free pose estimation.
- CASS vs. NOCS: Unlike NOCS, CASS reconstructs a complete canonical 3D shape rather than regressing coordinates only for the visible area.The paper describes this complete reconstruction as global shape understanding beneficial to pose estimation.
3. Model
The model jointly learns a canonical shape space and estimates category-level pose and size from calibrated RGBD input. It combines pose-independent canonical-shape features with pose-dependent photometric and geometric features in an end-to-end network trained with synthetic and real-world data.
- Architecture overview: The network takes a calibrated RGBD image and outputs both the object’s 6DoF pose and a metrically sized point cloud in normalized pose.Object detection and segmentation produce an image crop and point patch for each object before the main network processes them.
- Architecture overview: CASS learning, pose-factorizing embedding, point-cloud reconstruction, pose-dependent feature extraction, and pose estimation form three tightly coupled jointly trained modules.The architecture overview distinguishes canonical-shape processing, pose-dependent features, and final pose estimation.
- Canonical shape space and view factorization: The canonical shape space is the latent space of a VAE trained on 500-point, pose-normalized, metrically sized ShapeNet point clouds.The learned space is intended to span many shape variants while preserving actual object size.
- Canonical shape space and view factorization: Correspondence-based projection was suboptimal because correspondences may be incorrect and metric loss can conflict with other objectives.The paper therefore uses joint embedding rather than direct cross-modality correspondence.
- Canonical shape space and view factorization: A joint RGBD-and-3D embedding maps both modalities into a shared latent space without requiring paired object instances.Mixed training matches the two feature distributions and avoids an additional metric loss beyond conventional VAE losses.
- Pose-dependent feature extraction: Pose estimation concatenates view-factorized, photometric, and geometric features, with the latter two designed to encode pose-related color and geometry correlations.Photometric features are pooled from a fully convolutional RGB encoder, while geometric features come from point-based CNNs sharing weights with the CASS geometry encoder.
4. Results and evaluations
The evaluation measures category-level pose, size, and shape reconstruction on NOCS-REAL275, comparing against NOCS and testing key design choices. The method generally improves pose metrics, reconstruction accuracy with batch mixing, and robustness in qualitative comparisons, while size-related IoU and high-precision performance remain weaker.
- Evaluation setup: The evaluation uses NOCS-REAL275 and reports IoU, rotation-translation accuracy, and single-view shape reconstruction metrics.The compared NOCS baseline uses its best-performing 32-bin pose-classification variant.
- Pose and size estimation: The method outperforms NOCS on all reported pose metrics except IoU, whose lower values are attributed to less accurate size calculation from reconstruction.The authors attribute pose success mainly to view-factorized pose-independent features learned through CASS and RGBD embedding.
- Shape reconstruction: Batch mixing produces uniformly higher point-cloud reconstruction accuracy on both Chamfer Distance and Earth Mover’s Distance.The authors associate this improvement with distribution matching between RGBD and canonical point-cloud embeddings.
- Ablation studies: CASS learning is the most important ablated component, while batch mixing also matters substantially for accuracy.The full combination contributes to high-precision 5°5cm pose-and-size estimation.
- Ablation studies: VAE training improves generalization to unseen objects by learning a well-spanned CASS space, but can blur reconstructions and reduce size accuracy.The ablation attributes the trade-off to the normal-distribution prior used by the VAE.
- Qualitative results: Qualitative results show better size-estimation accuracy under object occlusion and background distraction, alongside full-shape reconstruction from single-view RGBD.Figure 7 compares pose and scale against NOCS, while Figure 8 contrasts reconstructed point clouds with depth-unprojected clouds.
5. Conclusion
The paper presents a correspondence-free category-level pose and size estimator based on a learned canonical shape space. It embeds RGBD inputs into pose-independent shape features, compares them with pose-dependent features, and reports state-of-the-art performance, while identifying several limitations.
- 5. Conclusion: The method learns a canonical 3D shape space with normalized pose and metric size for correspondence-free category-level pose and size estimation.The space is learned with a deep generative model.
- 5. Conclusion: RGBD inputs are embedded into the shape space to extract pose-independent features, which are compared with pose-dependent features for pose estimation.This comparison is part of the paper’s end-to-end framework.
- 5. Conclusion: Evaluation reports state-of-the-art performance for the proposed approach.The conclusion states this outcome without specifying a single benchmark metric.
- 5. Conclusion: The method struggles with very complex shapes, does not use reconstructed geometry to guide pose-estimation training, and has relatively low 5°5cm precision.The authors also note that pose or size refinement with ICP was not used.