Source-linked AI summary
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, Leonidas J. Guibas
TL;DR
The paper addresses category-level 6D pose and size estimation for unseen objects without exact CAD models. It introduces NOCS and an RGB CNN that predicts dense correspondences, masks, and classes, then combines them with depth for metric estimation. The method is supported by context-aware mixed-reality and real-world datasets, with experiments demonstrating robust estimation for unseen instances.
Problem
Existing methods require exact CAD models or provide only viewpoint-dependent boxes, while available datasets lack precise 6D pose, size, and NOCS annotations for relevant objects.
Method
NOCS places category instances in a shared canonical space, and a CNN predicts class labels, masks, and NOCS maps that depth-based pose fitting converts into metric pose and size.
Results
The method estimates full metric 6D pose and size for previously unseen object instances using predicted NOCS maps and depth.
Takeaways & Limitations
The shared NOCS representation enables category-level pose and size estimation without requiring complete, high-fidelity CAD models at test time.
Takeaways & Limitations
Pose estimation depends on potentially incorrect region proposals and category predictions, and lifting NOCS predictions to real-world coordinates requires depth images.
Abstract
from arXiv · showhide
The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image. Contrary to "instance-level" 6D pose estimation tasks, our problem assumes that no exact object CAD models are available during either training or testing time. To handle different and unseen object instances in a given category, we introduce a Normalized Object Coordinate Space (NOCS)---a shared canonical representation for all possible object instances within a category. Our region-based neural network is then trained to directly infer the correspondence from observed pixels to this shared object representation (NOCS) along with other object information such as class label and instance mask. These predictions can be combined with the depth map to jointly estimate the metric 6D pose and dimensions of multiple objects in a cluttered scene. To train our network, we present a new context-aware technique to generate large amounts of fully annotated mixed reality data. To further improve our model and evaluate its performance on real data, we also provide a fully annotated real-world dataset with large environment and instance variation. Extensive experiments demonstrate that the proposed method is able to robustly estimate the pose and size of unseen object instances in real environments while also achieving state-of-the-art performance on standard 6D pose estimation benchmarks.
1. Introduction
The paper targets category-level 6D pose and size estimation for unseen objects without exact CAD models. It introduces NOCS, a CNN-based correspondence representation, and annotated mixed-reality and real-world datasets.
- Instance-level methods require exact CAD models, while category-level 3D detection lacks precise, viewpoint-independent orientation and non-uniform scale.
- NOCS provides a shared canonical representation that enables consistent 6D pose and size estimation across unseen instances within a category.
- The CNN jointly predicts class labels, instance masks, and NOCS maps for multiple objects from a single RGB image.
- The predicted NOCS maps are combined with depth using pose fitting to estimate metric 6D pose and object dimensions.
- The mixed-reality data generation method provides 275K training and 25K testing examples, while the real dataset covers 18 scenes, 6 categories, and 42 unique instances.
- The method uses commodity RGB-D input and is designed to handle symmetric and asymmetric objects in applications including robotics, augmented reality, and 3D scene understanding.
2. Related Work
Prior approaches either require exact CAD models or simplify category-level pose estimation, while existing datasets lack sufficient annotated variation. The paper addresses these gaps with context-aware mixed-reality data and a real-world dataset.
- Instance-Level 6 DoF Pose Estimation: Instance-level 6D pose methods infer location and rotation using exact 3D CAD models and object sizes during training.
- Instance-Level 6 DoF Pose Estimation: Existing instance-level approaches also face practical challenges in storing models and capturing complete, high-fidelity models for many objects.
- Category-Level 4 DoF Pose Estimation: Category-level pose methods commonly restrict rotation to gravity-aligned four-degree-of-freedom estimates and focus on large room-scale categories.
- Training Data Generation: Real-world datasets are often small and costly to annotate, whereas exclusively synthetic datasets introduce domain gaps by omitting material, sensor-noise, and lighting variation.
- Training Data Generation: The paper introduces context-aware mixed reality generation and a real-world dataset to improve generalization to real data.
3. Background and Overview
NOCS places canonically aligned category instances in a shared unit-cube coordinate space and represents visible geometry through dense pixel correspondences. An RGB CNN predicts these correspondences, which depth then lifts into metric pose and size.
- Category-level pose and size estimation predicts three rotation, three translation, and three scale parameters for previously unseen instances.
- Normalized Object Coordinate Space: NOCS is a unit-cube space where category CAD models are uniformly scaled, centered, and consistently aligned in orientation.
- Normalized Object Coordinate Space: Each object vertex is represented by normalized (x, y, z) coordinates, visualized as RGB values in the NOCS representation.
- NOCS Map: The CNN predicts a NOCS map that can be interpreted as normalized reconstruction of visible shape or dense pixel–NOCS correspondences.
- Method Overview: The CNN uses only RGB to predict class labels, instance masks, and NOCS maps, allowing depth to be incorporated later for metric estimation.
- Method Overview: Depth lifts the normalized representation into real-world coordinates, followed by robust outlier removal and alignment to estimate full metric pose and size.
4. Datasets
The paper addresses limited ground-truth data for category-level 6D pose and size estimation through context-aware mixed reality and real-world datasets.
- Existing datasets lack 6D pose, NOCS-map annotations, hand-scale tabletop objects, and sufficient examples for this task.
- Context-Aware Mixed Reality: CAMERA combines real background scenes with synthetically rendered foreground objects placed at plausible locations, lighting, and scales.
- Context-Aware Mixed Reality: 300K composited images were rendered, with 25K reserved for validation.
- Context-Aware Mixed Reality: Synthetic compositing provides ground truth for class labels, instance masks, NOCS maps, 6D pose, and size.
- Real-World Dataset: The real-world datasets contain 8K RGB-D frames across 18 scenes, with separate training, validation, and testing splits.
5. Method
The method extends Mask R-CNN to predict object information and NOCS maps, then combines those predictions with depth to recover metric pose and size.
- Pipeline: A CNN predicts class labels, instance masks, and NOCS maps from RGB images, while depth enables metric 6D pose and size estimation.
- NOCS Map Head: Three additional Mask R-CNN heads predict the x, y, and z components of NOCS maps for each proposed region.
- NOCS Map Head: Pixel classification with B = 32 performed better than direct NOCS-map regression in the reported experiments.
- Loss Function: The NOCS heads use softmax classification or soft L1 regression losses, with the latter intended to make learning more robust.
- Object Symmetry: Symmetric loss minimizes prediction loss over predefined rotations around each category’s symmetry axis.
- Pose Fitting: The pose-fitting stage estimates scales, rotation, and translation that transform the NOCS point cloud to the observed object point cloud using the Umeyama algorithm.
6. Experiments and Results
The experiments evaluate category-level detection and pose estimation on synthetic and real data, ablate data and network choices, and test transfer to instance-level benchmarks. Results show strong performance for unseen objects, benefits from context-aware mixed reality data and symmetry-aware training, and substantial gains over baselines.
- Evaluation Protocol: The evaluation uses CAMERA25, REAL275, and OccludedLINEMOD, with separate metrics for 3D detection, dimensions, and 6D pose.The paper reports AP at 3D IoU thresholds and pose-error thresholds for rotation and translation.
- Category-Level Results: 83.9% mAP for 3D IoU at 50% and 40.9% mAP for the (5° , 5 cm) metric are achieved on CAMERA25.The model is trained on 275K CAMERA images and tested on objects and backgrounds unseen during training.
- Category-Level Results: 76.4% mAP for 3D IoU at 50%, 10.2% for (5° , 5 cm), and 23.1% for (10° , 5 cm) are achieved on REAL275.The baseline reaches 43.8% for 3D IoU and 0.8% for both pose metrics, while dense NOCS prediction supplies shape, part, and visibility information.
- Ablation Studies: Training with CAMERA*, COCO, and REAL* performs best, while non-context-aware compositing performs worse than context-aware data.CAMERA* alone suffers from domain gap, whereas REAL* alone or with COCO tends to overfit because of limited dataset size.
- Ablation Studies: Pixel classification outperforms regression on CAMERA25 and REAL275, with 32 bins best for pose estimation and 128 bins better for detection.Removing symmetry loss significantly degrades pose accuracy, particularly for 6D pose.
- Instance-Level Results: On OccludedLINEMOD, the method achieves 94.7% detection rate, 88.4% mAP for 3D IoU at 50%, and 30.2% mAP for 2D projection at 5 pixels.It reports 13.9% mAP for (5° , 5 cm) and 33.5% for (10° , 5 cm), exceeding the cited PoseCNN results.
- Limitations: Pose estimation depends on correct region proposals and category predictions, and lifting NOCS predictions to metric coordinates requires a depth image.The paper identifies direct RGB-only pose and size estimation as future work.
7. Conclusion
The paper presents a method for estimating category-level 6D pose and size for previously unseen object instances using NOCS and depth.
- NOCS provides a shared space with consistent object scaling and orientation across instances.
- A CNN predicts NOCS maps that, combined with depth, enable full metric 6D pose and size estimation.
- The method targets applications in augmented reality, robotics, and 3D scene understanding.
A. Implementation and Computation Times
The network uses a Mask R-CNN-based architecture with FPN and a ResNet50 backbone, processing 640×360 images at an interactive rate of around 4 fps.
- Architecture: The implementation is based on MatterPort’s Mask RCNN and uses Feature Pyramid Network with a ResNet50 backbone.It is implemented in Python 3, Keras, and TensorFlow.
- Computation Times: 640×360 images are processed at an interactive rate of around 4 fps on the reported desktop system.The system uses an Intel Xeon Gold 5122 CPU and NVIDIA TITAN Xp.
- Computation Times: Neural network inference averages 210 ms, while pose alignment using the Umeyama algorithm averages 34 ms.
B. Scanned Real Instances
The real dataset includes six categories and 42 scanned instances, with held-out instances used to evaluate substantial intra-category shape variation.
- Scanned Real Instances: The dataset contains 6 object categories and 42 real scanned unique instances.
- Scanned Real Instances: Each category has 7 instances, with 4 assigned to training and validation and 3 reserved for testing.
- Scanned Real Instances: The instances exhibit large intra-category shape variance, with held-out test instances shown separately from training instances.
C. Result Visualization
Visualizations show promising performance on CAMERA25 and decent performance on REAL275 despite limited real training data, alongside several real-data failure modes.
- Result Visualization: The method achieves very promising performance on the CAMERA25 validation set with sufficient training data.
- Result Visualization: The method achieves decent performance on the REAL275 test set even though the amount of real training data is small.
- Result Visualization: Observed real-data failure modes include missing detection, wrong classification, and inconsistency in predicted coordinate maps.
D. Comparisons on the OccludedLINEMOD Dataset
The paper compares its method with existing methods on the OccludedLINEMOD dataset using the 2D projection metric, with results presented in Figure 13.
- Figure 13 presents a comparison between the proposed method and existing methods on OccludedLINEMOD.
- The comparison uses the 2D projection metric.
- OccludedLINEMOD is the dataset used for this evaluation.