Source-linked AI summary
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield
TL;DR
The paper addresses real-time 6-DoF pose estimation of known household objects from single RGB images, where manually labeled training data is difficult to obtain and synthetic data faces a reality gap. It combines domain-randomized and photorealistic synthetic data to train DOPE, a one-shot network that performs competitively with real-and-synthetic-data methods and supports robotic manipulation.
Problem
Manually generating sufficiently varied labeled data for 6-DoF pose estimation is labor-intensive, while synthetic-data methods must operate reliably on real images.
Method
DOPE combines domain-randomized and photorealistic synthetic data to train a one-shot network that predicts projected 3D bounding-box vertices and uses PnP to estimate object pose.
Results
The synthetic-only network achieves performance comparable to state-of-the-art PoseCNN trained on mixed real and synthetic data, with higher AUC for 4 of 5 objects and better results below 2 cm.
Takeaways & Limitations
The resulting poses are sufficiently accurate for real-world robotic tasks including pick-and-place, object handoff, and path following.
Abstract
from arXiv · showhide
Using synthetic data for training deep neural networks for robotic manipulation holds the promise of an almost unlimited amount of pre-labeled training data, generated safely out of harm's way. One of the key challenges of synthetic data, to date, has been to bridge the so-called reality gap, so that networks trained on synthetic data operate correctly when exposed to real-world data. We explore the reality gap in the context of 6-DoF pose estimation of known objects from a single RGB image. We show that for this problem the reality gap can be successfully spanned by a simple combination of domain randomized and photorealistic data. Using synthetic data generated in this manner, we introduce a one-shot deep neural network that is able to perform competitively against a state-of-the-art network trained on a combination of real and synthetic data. To our knowledge, this is the first deep network trained only on synthetic data that is able to achieve state-of-the-art performance on 6-DoF object pose estimation. Our network also generalizes better to novel environments including extreme lighting conditions, for which we show qualitative results. Using this network we demonstrate a real-time system estimating object poses with sufficient accuracy for real-world semantic grasping of known household objects in clutter by a real robot.
1 Introduction
The paper targets real-time 6-DoF pose estimation for known household objects in clutter from a single RGB image, addressing limited labeled 3D data and the synthetic-data reality gap. It combines domain-randomized and photorealistic synthetic data with a one-shot network for robotic manipulation.
- 6-DoF object pose supports robotic pick-and-place, handoff, and imitation learning for known rigid objects.
- Manual annotation is difficult for 3D detection, so existing approaches often rely on real training data correlated with test conditions.
- Synthetic data offers abundant pre-labeled examples, but networks trained synthetically usually face a reality gap on real images without additional fine-tuning.
- Combining domain-randomized and photorealistic data provides sufficient variation to operate on real data without fine-tuning.
- DOPE estimates 3D poses of known objects in clutter from one RGB image in near real time using projected 3D bounding-box coordinates followed by PnP.
- The integrated robotic system reports pose accuracy sufficient for pick-and-place, object handoff, and path-following tasks.
2 Approach
The approach combines multistage keypoint detection with PnP pose recovery and trains the network using complementary domain-randomized and photorealistic synthetic data. The data-generation pipeline varies scenes, objects, viewpoints, and conditions to support pose estimation in clutter.
- 2 Approach: The two-step pipeline predicts 2D keypoint belief maps, then feeds their peaks into PnP to estimate each object's 6-DoF pose.
- 2 Approach: The fully convolutional network uses multistage belief maps and vector fields to detect projected bounding-box vertices and object centroids.
- 2 Approach: Later stages incorporate preceding outputs and larger receptive fields to resolve ambiguities caused by limited early-stage context.
- 2 Approach: Object instances are extracted by thresholding local belief-map peaks and greedily associating projected vertices with detected centroids.
- 2 Approach: PnP uses detected projected vertices, camera intrinsics, and object dimensions to recover translation and rotation relative to the camera.
- 2.3 Data generation: The training data combines non-photorealistic domain-randomized images with photorealistic images because the two types complement one another.
- 2.3 Data generation: Domain-randomized scenes vary distractors, textures, backgrounds, poses, lighting, and noise.
- 2.3 Data generation: The photorealistic dataset places YCB objects in physically constrained environments and collects views from randomized camera azimuths, elevations, and distances.
3 Experimental Results
Experiments evaluate DOPE on YCB objects, varied datasets, synthetic-data mixtures, network stages, and robotic grasping. DOPE trained only on synthetic data matches PoseCNN while generalizing to challenging conditions and supporting real-world manipulation.
- Datasets and comparison: The evaluation compares DOPE with PoseCNN on YCB-Video and a separately collected dataset using different cameras, backgrounds, and extreme lighting.The YCB-Video test set contains 2,949 frames; the additional dataset uses five YCB objects and multiple simultaneous instances.
- YCB-Video dataset: DOPE trained on synthetic data achieves results on par with PoseCNN trained on mixed synthetic and real data, with higher AUC for 4 of 5 objects.DOPE also performs better at thresholds below 2 cm, approximately the graspability limit for Baxter’s parallel-jaw end effector.
- Extreme lighting dataset: Qualitatively, DOPE generalizes more robustly than PoseCNN to different cameras, extreme poses, severe occlusion, and extreme lighting.The reported failures include a severely occluded soup can and three dark cans.
- Additional experiments: 120k mixed synthetic images produce 77.00 AUC, exceeding the highest DR-only value of 66.64 AUC and photorealistic-only value of 62.94 AUC.The DR-only peak occurs at 300k images, while the photorealistic-only peak occurs at 600k images.
- Additional experiments: Performance remains comparable when at least 40% of either dataset is included, while additional network stages improve accuracy at the cost of greater computation.Stage-count comparisons include total execution time, object extraction, and PnP.
- Robotic manipulation: In cluttered robotic trials, successful grasps numbered 10 cracker, 10 meat, 11 mustard, 11 sugar, and 7 soup out of 12 attempts per object.The soup result increased to 9 when the can was placed on its side; errors included pose estimation, calibration, and robot control.
- Robotic manipulation: The system supports pick-and-place between objects, handoff, and real-time 6-DoF path following beyond table-constrained top-down grasping.The reported pick-and-place demonstration places a can on a box without requiring the destination box to align with the table.
4 Related Work
Recent 6-DoF pose-estimation methods increasingly use deep networks directly on RGB images, while synthetic-data research addresses the need for large labeled datasets through photorealistic rendering and domain randomization.
- Object detection and 6-DoF pose estimation: Deep learning-based RGB methods have recently improved 6-DoF pose estimation, including under occlusion.
- Synthetic data for training: Synthetic datasets address deep learning's demand for labeled data, but photorealistic generation requires substantial skilled 3D modeling effort.
- Synthetic data for training: Domain randomization offers a less expensive alternative that encourages networks to focus on essential visual features.
5 Conclusion
The paper presents a synthetic-data-trained system for 6-DoF pose estimation of known objects, combining iterative image-coordinate refinement with PnP and demonstrating state-of-the-art performance and manipulation-relevant accuracy.
- DOPE refines ambiguous 2D projections of each object's 3D bounding-cuboid vertices through multiple stages before estimating pose with PnP.The method assumes known camera intrinsics and object dimensions.
- A network trained only on synthetic data achieves state-of-the-art performance compared with a network trained on real data.
- The resulting poses are sufficiently accurate for robotic manipulation.The paper identifies increasing the number of objects, handling symmetry, and adding closed-loop refinement as future research directions.