Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

• 6-DoF object pose supports robotic pick-and-place, handoff, and imitation learning for known rigid objects.Source: §1 · p. 1
• Manual annotation is difficult for 3D detection, so existing approaches often rely on real training data correlated with test conditions.Source: §1 · p. 1
• Synthetic data offers abundant pre-labeled examples, but networks trained synthetically usually face a reality gap on real images without additional fine-tuning.Source: §1 · p. 1
• Combining domain-randomized and photorealistic data provides sufficient variation to operate on real data without fine-tuning.Source: §1 · p. 2
• DOPE estimates 3D poses of known objects in clutter from one RGB image in near real time using projected 3D bounding-box coordinates followed by PnP.Source: §1 · p. 2
• The integrated robotic system reports pose accuracy sufficient for pick-and-place, object handoff, and path-following tasks.Source: §1 · p. 2

2 Approach

• The two-step pipeline predicts 2D keypoint belief maps, then feeds their peaks into PnP to estimate each object's 6-DoF pose.Source: §2 · p. 2
• The fully convolutional network uses multistage belief maps and vector fields to detect projected bounding-box vertices and object centroids.Source: §2.1 · p. 2
• Later stages incorporate preceding outputs and larger receptive fields to resolve ambiguities caused by limited early-stage context.Source: §2.1 · p. 2
• Object instances are extracted by thresholding local belief-map peaks and greedily associating projected vertices with detected centroids.Source: §2.2 · p. 3
• PnP uses detected projected vertices, camera intrinsics, and object dimensions to recover translation and rotation relative to the camera.Source: §2.2 · p. 3

2.3 Data generation

• The training data combines non-photorealistic domain-randomized images with photorealistic images because the two types complement one another.Source: §2.3 · p. 3
• Domain-randomized scenes vary distractors, textures, backgrounds, poses, lighting, and noise.Source: §2.3 · p. 3
• The photorealistic dataset places YCB objects in physically constrained environments and collects views from randomized camera azimuths, elevations, and distances.Source: §2.3 · p. 4

Datasets and comparison

• The evaluation compares DOPE with PoseCNN on YCB-Video and a separately collected dataset using different cameras, backgrounds, and extreme lighting.The YCB-Video test set contains 2,949 frames; the additional dataset uses five YCB objects and multiple simultaneous instances.Source: §3 · p. 4, §3.1 · p. 4

YCB-Video dataset

• DOPE trained on synthetic data achieves results on par with PoseCNN trained on mixed synthetic and real data, with higher AUC for 4 of 5 objects.DOPE also performs better at thresholds below 2 cm, approximately the graspability limit for Baxter’s parallel-jaw end effector.Source: §3.3 · p. 5, Figure 2 · p. 6

Extreme lighting dataset

• Qualitatively, DOPE generalizes more robustly than PoseCNN to different cameras, extreme poses, severe occlusion, and extreme lighting.The reported failures include a severely occluded soup can and three dark cans.Source: §3.4 · p. 5

Additional experiments

• 120k mixed synthetic images produce 77.00 AUC, exceeding the highest DR-only value of 66.64 AUC and photorealistic-only value of 62.94 AUC.The DR-only peak occurs at 300k images, while the photorealistic-only peak occurs at 600k images.Source: §3.5 · p. 6
• Performance remains comparable when at least 40% of either dataset is included, while additional network stages improve accuracy at the cost of greater computation.Stage-count comparisons include total execution time, object extraction, and PnP.Source: §3.5 · p. 6

Robotic manipulation

• In cluttered robotic trials, successful grasps numbered 10 cracker, 10 meat, 11 mustard, 11 sugar, and 7 soup out of 12 attempts per object.The soup result increased to 9 when the can was placed on its side; errors included pose estimation, calibration, and robot control.Source: §3.6 · pp. 6–7
• The system supports pick-and-place between objects, handoff, and real-time 6-DoF path following beyond table-constrained top-down grasping.The reported pick-and-place demonstration places a can on a box without requiring the destination box to align with the table.Source: §3.6 · pp. 6–7

Object detection and 6-DoF pose estimation

• Deep learning-based RGB methods have recently improved 6-DoF pose estimation, including under occlusion.Source: §4 · p. 8

Synthetic data for training

• Synthetic datasets address deep learning's demand for labeled data, but photorealistic generation requires substantial skilled 3D modeling effort.Source: §4 · p. 8
• Domain randomization offers a less expensive alternative that encourages networks to focus on essential visual features.Source: §4 · p. 8

• DOPE refines ambiguous 2D projections of each object's 3D bounding-cuboid vertices through multiple stages before estimating pose with PnP.The method assumes known camera intrinsics and object dimensions.Source: §5 · p. 8
• A network trained only on synthetic data achieves state-of-the-art performance compared with a network trained on real data.Source: §5 · p. 8
• The resulting poses are sufficiently accurate for robotic manipulation.The paper identifies increasing the number of objects, handling symmetry, and adding closed-loop refinement as future research directions.Source: §5 · p. 8