Source-linked AI summary
BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth
Mahdi Rad, Vincent Lepetit
TL;DR
The paper addresses 3D object detection and pose estimation from color images alone, where depth is unavailable and rotationally symmetric objects create ambiguous image-to-pose mappings. BB8 combines segmentation with holistic CNN prediction of projected bounding-box corners, using pose-range classification for symmetry and optional rendering-based refinement. It reports improved LINEMOD performance, RGB-only Occlusion results, and T-LESS results below the RGB-D state of the art.
Problem
Color-only 3D pose estimation is desirable when depth cameras fail outdoors or on specular objects, but rotational symmetry makes image-to-pose learning one-to-many on T-LESS.
Method
BB8 segments objects, holistically predicts projected 3D bounding-box corners with CNNs, classifies pose ranges for symmetry, and optionally refines estimates using rendered masks or color images.
Results
BB8 improves LINEMOD results over the compared state of the art by 15.6% for 2D Projection, 12.6% for 6D Pose, and 28.4% for 5cm 5°.
Takeaways & Limitations
The holistic RGB-only approach advances 3D pose estimation on challenging objects, while its full approach remains fast and scalable across multiple objects.
Abstract
from arXiv · showhide
We introduce a novel method for 3D object detection and pose estimation from color images only. We first use segmentation to detect the objects of interest in 2D even in presence of partial occlusions and cluttered background. By contrast with recent patch-based methods, we rely on a "holistic" approach: We apply to the detected objects a Convolutional Neural Network (CNN) trained to predict their 3D poses in the form of 2D projections of the corners of their 3D bounding boxes. This, however, is not sufficient for handling objects from the recent T-LESS dataset: These objects exhibit an axis of rotational symmetry, and the similarity of two images of such an object under two different poses makes training the CNN challenging. We solve this problem by restricting the range of poses used for training, and by introducing a classifier to identify the range of a pose at run-time before estimating it. We also use an optional additional step that refines the predicted poses. We improve the state-of-the-art on the LINEMOD dataset from 73.7% to 89.3% of correctly registered RGB frames. We are also the first to report results on the Occlusion dataset using color images only. We obtain 54% of frames passing the Pose 6D criterion on average on several sequences of the T-LESS dataset, compared to the 67% of the state-of-the-art on the same sequences which uses both color and depth. The full approach is also scalable, as a single network can be trained for multiple objects simultaneously.
1. Introduction
BB8 estimates 3D object poses from color images by combining 2D segmentation with holistic CNN regression, addressing partial occlusion, clutter, and rotational symmetry. A pose-range classifier and optional refinement improve robustness while keeping the approach fast and scalable.
- Depth cameras can fail outdoors or on specular objects and consume mobile-device battery, motivating color-only 3D pose estimation.
- Holistic regression predicts an object’s pose directly from its appearance rather than identifying individual surface points.
- BB8 detects objects with segmentation, then predicts 2D projections of 3D bounding-box corners to recover pose with a CNN and PnP.
- Rotational symmetry makes image-to-pose learning one-to-many because different poses can produce identical images, causing the basic approach to perform badly on T-LESS.
- BB8 handles symmetry by restricting training poses and classifying the pose range at runtime before estimating the pose.
- The optional refinement stage compares the input image with a rendering of the initial pose to improve projected-corner predictions.
- BB8 is fast because it applies deep networks only a few times and can train one network for multiple objects simultaneously.
2. Related Work
BB8 is positioned as an RGB-only alternative to depth- and patch-based 3D object detection methods. Its holistic pose prediction is presented as accurate on recent datasets while addressing occlusion and scalability concerns.
- Earlier keypoint-based methods work well mainly for textured objects, while depth-based methods target untextured objects using RGB-D or depth data.
- Local-patch methods use object-coordinate predictions or sparse correspondences, but their outputs can be noisy or lack demonstrated robustness to partial occlusion.
- Compared with direct translation-and-rotation regression, bounding-box corner projections avoid a meta-parameter balancing position and orientation errors.
- The authors describe BB8 as one of the first methods for RGB-only 3D object detection and pose estimation on recent datasets.
3. Proposed Approach
The approach localizes objects through segmentation, predicts 3D pose holistically from projected bounding-box corners, and addresses rotational symmetry with restricted training ranges and mirroring. An optional CNN feedback stage further refines pose estimates.
- The pipeline first localizes visible objects in 2D, estimates their 3D poses, handles rotational symmetry, and optionally refines the estimates.The same overall approach is described as a sequence of localization, pose estimation, symmetry handling, and refinement.
- 3.2. Predicting the 3D Pose: The pose network predicts 2D projections of 3D bounding-box corners, which are paired with their 3D coordinates and solved using PnP.A single network can handle all target objects, and bounding-box corners provide spread-out 3D points framing the object.
- 3.3. Handling Objects with an Axis of Symmetry: Rotational symmetry creates ambiguous image-to-pose mappings, so training uses a restricted rotation range and test-time mirroring handles the complementary range.The pose range is narrowed to half the symmetry interval; a classifier identifies the complementary range before mirroring and mirroring back predictions.
- 3.4. Refining the Pose: An optional CNN refines an initial pose by comparing the input image with a rendered binary mask or color rendering and predicting a pose update.The rendering is generated from the current pose estimate, and the refinement stage is optional.
4. Experiments
The evaluation measures BB8 across LINEMOD, Occlusion, and T-LESS using established pose metrics, refinement comparisons, and color-only robustness tests. Results show strong gains on LINEMOD, accurate estimation under partial occlusion, and 54% average 6D Pose performance on selected T-LESS scenes.
- Evaluation setup: The experiments evaluate LINEMOD, Occlusion, and T-LESS using 2D Projections, 6D Pose, and 5cm 5° metrics.The 2D Projections metric accepts poses below 5 pixels average projected-vertex error; 6D Pose uses a 10% diameter threshold, while 5cm 5° requires translation and rotation errors below those values.
- Refinement and runtime: 4.5% and 6.3% are the average improvements from mask and color-rendering refinement, with color rendering performing best.Two refinement iterations were evaluated using ground-truth 2D object centers; binary-mask refinement still produced significant improvement.
- LINEMOD: 15.6%, 12.6%, and 28.4% are BB8’s margins over on LINEMOD for 2D Projection, 6D Pose, and 5cm 5° respectively.The comparison uses segmentation-based object centers and reports results with and without RGB refinement.
- Occlusion dataset: About 80% of frames for seven Occlusion-dataset objects achieve a 2D Projection error below 15px using color images only.The method does not report Eggbox because more than 70% of close poses are absent from the training sequence.
- T-LESS dataset: 54% is the average per-scene 6D Pose score on the evaluated T-LESS scenes.Most errors occur along the camera’s z axis, while orientation and translation along the x and y axes are typically well estimated.
- Refinement and runtime: A full run takes 140 ms for segmentation and 130 ms for pose prediction, while a simpler single-object network reduces these times to 20 ms and 12 ms.Each refinement iteration takes 21 ms on the reported desktop hardware, and the simpler network has similar accuracy.
5. Conclusion
The paper concludes that a holistic deep-network approach can significantly advance 3D pose estimation from color images, including on challenging T-LESS objects. The work was funded by the Christian Doppler Laboratory for Semantic 3D Computer Vision.
- Conclusion: A holistic approach enabled significant advances in 3D pose estimation from color images, including on challenging T-LESS objects.The conclusion attributes this advance to deep networks’ regression capabilities.
- Conclusion: The work was funded by the Christian Doppler Laboratory for Semantic 3D Computer Vision.