Source-linked AI summary
Real-Time Seamless Single Shot 6D Object Pose Prediction
Bugra Tekin, Sudipta N. Sinha, Pascal Fua
TL;DR
The paper addresses fast RGB-based 6D object pose estimation without multi-stage processing or refinement. It uses a single-shot CNN to predict projected 3D bounding-box vertices and estimates pose with PnP, achieving state-of-the-art LINEMOD accuracy while running more than five times faster for single objects.
Problem
RGB-based 6D pose estimation must handle weakly textured or untextured objects and low-resolution video, while existing multi-stage or approximate methods require refinement for improved accuracy.
Method
A single-shot, end-to-end CNN directly predicts 2D projections of 3D bounding-box vertices, with pose recovered using PnP and confidence learned from localization error.
Results
State-of-the-art LINEMOD accuracy was achieved while running more than five times faster for single-object estimation, with constant runtime as the number of objects increases on OCCLUSION.
Takeaways & Limitations
Accurate single-shot predictions eliminate the need for pose refinement and avoid requiring a precise textured 3D object model for training.
Takeaways & Limitations
The training procedure assumes that at most 5 objects can occlude one another within a single grid cell.
Abstract
from arXiv · showhide
We propose a single-shot approach for simultaneously detecting an object in an RGB image and predicting its 6D pose without requiring multiple stages or having to examine multiple hypotheses. Unlike a recently proposed single-shot technique for this task (Kehl et al., ICCV'17) that only predicts an approximate 6D pose that must then be refined, ours is accurate enough not to require additional post-processing. As a result, it is much faster - 50 fps on a Titan X (Pascal) GPU - and more suitable for real-time processing. The key component of our method is a new CNN architecture inspired by the YOLO network design that directly predicts the 2D image locations of the projected vertices of the object's 3D bounding box. The object's 6D pose is then estimated using a PnP algorithm. For single object and multiple object pose estimation on the LINEMOD and OCCLUSION datasets, our approach substantially outperforms other recent CNN-based approaches when they are all used without post-processing. During post-processing, a pose refinement step can be used to boost the accuracy of the existing methods, but at 10 fps or less, they are much slower than our method.
1. Introduction
Existing RGB-based pose methods struggle with weak texture and low-resolution imagery, while recent CNN pipelines remain slow or require refinement. The proposed single-shot architecture directly predicts projected 3D bounding-box vertices, enabling fast, accurate 6D pose estimation without post-processing.
- Passive RGB pose estimation is attractive for mobile and wearable cameras because active depth sensors are power hungry.
- Fast keypoint and edge-based methods work well for textured objects but struggle with weakly textured or untextured objects and low-resolution video.
- BB8 uses separate CNN stages for segmentation and projected 3D bounding-box vertices, while SSD-6D predicts rough orientation and depth in one step.
- Both BB8 and SSD-6D require pose refinement for improved accuracy, increasing runtime linearly with the number of detected objects.
- The proposed end-to-end CNN directly detects projected 3D bounding-box vertices and remains accurate without a posteriori refinement.
- The method achieves state-of-the-art LINEMOD accuracy and is more than five times faster than competing techniques for single-object estimation.
- For multiple objects, its runtime remains constant rather than growing proportionally with the number of objects.
2. Related Work
Prior work spans classical feature-based, RGB-D, and CNN-based pose estimation, with limitations involving texture, resolution, staging, and pose representation. This paper extends single-shot 2D detection toward direct 6D pose recovery from a few image coordinates.
- Classical methods: Classical methods use local keypoints and feature matching and are often fast and robust to occlusion and scene clutter.
- Classical methods: Classical methods reliably handle textured objects in high-resolution images but are less suitable for weakly textured or low-resolution inputs.
- RGB-D methods: RGB-D methods support object recognition, pose estimation, grasping, and manipulation, but depend on commodity depth cameras.
- CNN-based methods: CNN approaches have represented pose through discretized viewpoints, direct regression, or multiple-stage prediction of translation and rotation.
- CNN-based methods: Predicting 2D image quantities avoids balancing translation and rotation losses and can make training numerically more stable.
- Single-shot detection: Single-shot detectors such as YOLO and SSD provide a fast alternative to two-stage detection architectures.
- Single-shot detection: The proposed approach extends a YOLO-like architecture to predict 2D coordinates from which full 6D pose is recovered.
3. Approach
The approach uses a single-shot CNN to predict 2D projections of nine 3D control points, then estimates 6D pose with PnP. Its grid-based output jointly represents localization, object classes, and confidence, while training and inference handle localization precision and nearby multiple objects.
- Pose representation: Nine control points comprise the 3D bounding-box corners and object centroid, whose predicted 2D coordinates are converted to 6D pose using PnP.The resulting pose estimate provides the object’s 3D rotation and translation in camera coordinates.
- Model architecture: The YOLO-inspired network divides an RGB image into an S × S grid and predicts control-point locations, class probabilities, and confidence for each cell.The output tensor has dimensions S × S × D, with D = 9×2+C+1 for nine 2D control points, C classes, and one confidence value.
- Confidence modeling: The confidence function assigns confidence from the image-space distance between each predicted control point and its ground-truth location, using a sharp exponential with a cutoff.Confidence is applied to all control points and averaged; low-confidence cells are pruned at test time.
- Coordinate prediction: The coordinate parameterization constrains centroid offsets within the responsible cell but leaves corner offsets unconstrained so projected corners can lie outside it.This makes the network first identify the approximate object cell and then refine the eight corner locations.
- Multiple objects: Up to five candidate predictions are allowed per grid cell to handle multiple nearby or mutually occluding objects.The design assumes that at most five objects may occlude one another within a single cell.
- Inference: At inference, one network evaluation produces object identities, confidence scores, and projected control points; low-confidence detections are pruned before PnP pose estimation.Class-specific confidence combines the class probability with the confidence-function score.
4. Implementation Details
The network is initialized from ImageNet classification training and uses staged confidence-loss weighting during pose training because early pose predictions make confidence targets unreliable.
- Initialization and training: ImageNet initialization is followed by confidence-loss pretraining with λconf set to zero, then full training with λconf = 5 for object cells and 0.1 otherwise.The staged procedure addresses unreliable confidence values produced by inaccurate early pose estimates.
5. Experiments
Experiments on LINEMOD and OCCLUSION evaluate pose accuracy, runtime, and multi-object performance using standard metrics and matched protocols. The approach achieves strong accuracy without refinement while maintaining real-time speed and scalable multi-object processing.
- Datasets and metrics: The evaluation uses LINEMOD and OCCLUSION with standard 6D pose metrics, including reprojection error, IoU, and ADD.LINEMOD measures single-object pose estimation, while OCCLUSION evaluates multiple objects under severe occlusion.
- Datasets and metrics: A pose is counted correct below 5 pixels for reprojection error, above 0.5 IoU, or below 10% of object diameter for ADD.The ADD threshold is approximately 2 cm for most objects and about 1 cm for smaller objects such as ape.
- Comparative accuracy: The method outperforms Brachmann et al. and BB8 on LINEMOD reprojection accuracy both with and without competing refinement stages.The comparison uses the same training/test splits as BB8 and reports percentages of correctly estimated poses.
- Comparative accuracy: The method also exceeds SSD-6D in LINEMOD IoU accuracy with and without refinement, despite SSD-6D using extensive rendered training data.The comparison concerns projected 3D-model overlap rather than 2D detection-box overlap.
- Comparative accuracy: Before refinement, the method surpasses competing methods by at least 12% on ADD and retains better or comparable accuracy after competitors are refined.Refinement improves BB8 and SSD-6D but requires full 3D CAD models and additional computation.
- Runtime and multiple objects: The approach runs at least 5 times faster than competing single-object methods and adds only 0.2 milliseconds per object for PnP in multi-object scenes.Its runtime remains effectively constant as object count grows, unlike competing approaches with linear runtime growth.
6. Conclusion
The paper presents a fast, accurate single-shot CNN for 6D pose prediction that extends 2D detection to 3D object detection. It predicts projected 3D bounding-box corners and computes pose with PnP.
- The network predicts 2D projections of 3D bounding-box corners, extending single-shot 2D detection to 6D object detection.The prediction involves only a few additional 2D points beyond 2D bounding-box regression.
- The method computes 6D pose from the predicted corner projections using an efficient PnP method.
- The architecture is designed for fast and accurate single-shot pose prediction without requiring post-processing.
Supplemental Material: “Real-Time Seamless Single Shot 6D Object Pose Prediction”
The supplemental material describes training augmentation, confidence scoring, confidence-weighted pose prediction, and qualitative evaluations on OCCLUSION and LINEMOD. These materials explain the method’s runtime-oriented confidence design and illustrate robustness across challenging visual conditions.
- Training Images: Training images are augmented by compositing segmented foreground objects onto random PASCAL VOC images and combining multiple objects from different training images.Segmentation masks from LINEMOD are used to extract foreground objects before compositing.
- Confidence function: The confidence function closely approximates 3D cube IoU while being much faster to compute.Its value and runtime are compared directly with 3D cube IoU in Figure 7.
- Confidence-weighted prediction: Confidence-weighted averaging combines multiple nearby predictions for control points and the centroid before PnP pose estimation.The procedure scans a 17×17 grid, keeps a 3 × 3 neighborhood around the highest-confidence cell, and averages surviving 18-dimensional prediction vectors.
- Confidence-weighted prediction: Confidence-weighted prediction improved LINEMOD accuracy by 1–2% with the 2D projection metric.The improvement comes from sub-pixel refinement on the prediction grid, particularly for objects spanning adjoining cells.
- Qualitative Results: Qualitative examples on OCCLUSION and LINEMOD show robustness to severe occlusions, rotational ambiguities, reflections, viewpoint changes, and scene clutter.Figures 9–14 visualize projected 3D bounding boxes and pose estimates for multiple objects and object categories.