Source-linked AI summary
Segmentation-driven 6D Object Pose Estimation
Yinlin Hu, Joachim Hugonot, Pascal Fua, Mathieu Salzmann
TL;DR
Global pose regression and keypoint methods treat objects as single entities, leaving them vulnerable to large occlusions. This paper instead combines confidence-weighted local keypoint predictions from segmented visible patches into robust correspondences for PnP. It reports state-of-the-art performance on Occluded-LINEMOD and YCB-Video with real-time operation.
Problem
Existing direct-regression and keypoint-based deep methods produce a single global pose, making them vulnerable to large occlusions and clutter.
Method
The method jointly segments objects, predicts local 2D projections of predefined 3D keypoints with confidence values, and fuses reliable correspondences before RANSAC-based PnP.
Results
The method outperforms state-of-the-art methods on Occluded-LINEMOD and YCB-Video while predicting poses of multiple objects in real time.
Takeaways & Limitations
Combining local pose estimates enables accurate pose estimation without a refinement step even when multiple poorly-textured objects occlude one another.
Takeaways & Limitations
The method cannot yet handle the most extreme occlusions or tiny objects, and it does not match an oracle selecting the best keypoint predictions.
Abstract
from arXiv · showhide
The most recent trend in estimating the 6D pose of rigid objects has been to train deep networks to either directly regress the pose from the image or to predict the 2D locations of 3D keypoints, from which the pose can be obtained using a PnP algorithm. In both cases, the object is treated as a global entity, and a single pose estimate is computed. As a consequence, the resulting techniques can be vulnerable to large occlusions. In this paper, we introduce a segmentation-driven 6D pose estimation framework where each visible part of the objects contributes a local pose prediction in the form of 2D keypoint locations. We then use a predicted measure of confidence to combine these pose candidates into a robust set of 3D-to-2D correspondences, from which a reliable pose estimate can be obtained. We outperform the state-of-the-art on the challenging Occluded-LINEMOD and YCB-Video datasets, which is evidence that our approach deals well with multiple poorly-textured objects occluding each other. Furthermore, it relies on a simple enough architecture to achieve real-time performance.
1. Introduction
Global pose methods treat each object as a single entity, making them vulnerable to occlusions and clutter. The paper introduces a segmentation-driven framework that combines local predictions to produce robust, real-time pose estimates.
- Motivation: Classical 3D-to-2D correspondence methods can fail for featureless objects and scenes containing mutually occluding objects.These methods normally use Perspective-n-Point (PnP) to compute the six pose parameters.
- Motivation: Deep networks either regress pose directly or predict keypoints for PnP, but global treatment makes them vulnerable to large occlusions.Overlapping bounding boxes can introduce irrelevant information from other objects into the pose estimate.
- Approach: The framework assigns local 2D keypoint predictions to visible object patches and combines reliable predictions using confidence values.The fused correspondences are used with RANSAC-based PnP to infer one pose per object.
- Approach: Joint local-patch reasoning provides rough object segmentation while combining detection and pose estimation in a simple architecture suitable for real-time performance.This avoids separating object detection from pose estimation.
- Reported contribution: The method is reported to produce accurate poses without post-processing under multiple poorly-textured objects occluding each other and to outperform state-of-the-art benchmarks.The named benchmarks are Occluded-LINEMOD and YCB-Video.
2. Related Work
Earlier pose-estimation approaches use local features, templates, or global deep-network predictions, but clutter and occlusion remain difficult. Recent work explores segmentation and local keypoint evidence, while rigid objects permit local predictions to be combined more robustly than non-rigid human poses.
- Classical methods: Classical RGB-only methods match image features to a known model and use PnP, while descriptor design targets robustness to transformations.The paper specifically excludes access to a depth map.
- Classical methods: Template- and edge-based techniques help with poorly-textured objects but often fail under mild occlusions and cluttered backgrounds.Hausdorff- and Chamfer-distance strategies are cited as examples.
- Deep methods: Deep methods either regress the pose directly or predict keypoints for PnP, but global bounding-box reasoning lets background and other objects contaminate predictions.These approaches produce a single global pose and commonly require refinement.
- Beyond global prediction: Segmentation-based 3D-coordinate regression has disappointing performance, whereas local keypoint heatmaps assemble multiple predictions for PnP.The paper identifies these as recent attempts to move beyond global prediction.
- Beyond global prediction: Rigid objects allow reliable local visible-part predictions to be combined, unlike non-rigid human poses that require more global receptive-field information.The paper connects this distinction to differing occlusion sensitivity.
3. Approach
The approach jointly segments objects and predicts their 2D keypoint projections through local grid-cell predictions. Confidence-guided fusion then selects correspondences for RANSAC-based pose estimation, with one-shot inference supporting efficient operation.
- 3. Approach: The network jointly detects objects and estimates their 6D poses by having individual image patches predict object identities and 2D projections of predefined 3D keypoints.The objects are assumed rigid and their 3D models are available.
- 3.1. Network Architecture: A shared encoder feeds separate segmentation and regression decoders, each producing outputs over an S×S image grid.The segmentation decoder labels each grid cell, while the regression decoder predicts keypoint locations.
- 3.2. Segmentation Stream: The segmentation stream assigns each grid cell to one of K object classes or background using an output dimension Dseg = K + 1.Ground-truth labels account for object depth and occlusions during training, and the segmentation loss uses Focal Loss for class imbalance.
- 3.3. Regression Stream: The regression stream predicts offsets and confidence values for N predefined 3D keypoints at every grid location, typically using the 8 bounding-box corners.Predicted offsets are added to grid-cell centers, and the regression loss combines position and confidence terms within the foreground mask.
- 3.4. Inference Strategy: At inference, foreground predictions are clustered by object, the n = 10 most confident 2D predictions per 3D keypoint are retained, and RANSAC-based EPnP estimates each pose.This filtering balances speed and accuracy while avoiding RANSAC over all local predictions.
- 3.4. Inference Strategy: The method is designed for real-time operation and reports comparisons with state-of-the-art methods, runtime measurements, and alternative fusion strategies on Occluded-LINEMOD.The evaluation materials include accuracy, runtime, and fusion-strategy comparisons.
4. Experiments
Experiments on Occluded-LINEMOD and YCB-Video show that the method outperforms several state-of-the-art alternatives while maintaining real-time performance. Ablations and qualitative comparisons support local prediction and confidence-based fusion as effective under occlusion, although extreme occlusions and tiny objects remain challenging.
- Datasets and metrics: The evaluation uses Occluded-LINEMOD and YCB-Video, with REP-5px and ADD-0.1d as symmetric pose-accuracy metrics.REP-5px requires reprojection error below 5 pixels, while ADD-0.1d requires 3D pose error below 10% of model diameter.
- Comparison with the state of the art: Our method outperforms global-inference baselines by a large margin and Heatmaps by a smaller margin.The compared global methods include PoseCNN, BB8, and Tekin; Heatmaps combines predictions from multiple relatively large patches.
- Runtime: More than 5 times faster than Heatmaps, the method takes about 50ms per image with five objects on average.Segmentation and 2D reprojection estimation take 30ms per image, while fusion takes 3–4ms per object.
- Fusion strategies: All confidence-based fusion schemes outperform No-Fusion, while the oracle result indicates further gains from better confidence prediction or fusion.No-Fusion always selects the center-grid reprojection; the oracle selects the best predicted location using ground-truth 2D reprojections.
- Comparison with human pose methods: On Occluded-LINEMOD, the local approach yields much more accurate predictions than modified Mask R-CNN and CPM methods.Mask R-CNN and CPM output bounding-box corner locations and perform slightly better than other global-inference methods.
- YCB-Video: On YCB-Video, the method outperforms PoseCNN and Heatmaps, while qualitative results remain better than PoseCNN under large occlusions.The method’s pose estimates are less accurate on YCB-Video than on Occluded-LINEMOD, but still much better than PoseCNN’s.
- Discussion: The method handles neither the most extreme occlusions nor tiny objects, where the grid is too rough a representation.The paper suggests finer or adaptively subdivided grids as possible remedies.
5. Conclusion
The paper concludes that jointly detecting objects and estimating pose from multiple local predictions produces accurate results without refinement, including under large occlusions. It also reports state-of-the-art performance on two challenging datasets and real-time multi-object inference.
- 5. Conclusion: The segmentation-driven approach jointly detects multiple objects and estimates their 6D poses.It combines multiple local pose estimates in a robust fashion.
- 5. Conclusion: The approach produces accurate pose estimates without refinement under large occlusions and outperforms the state of the art on two challenging datasets.The conclusion names Occluded-LINEMOD and YCB-Video as the evaluation datasets.
- 5. Conclusion: Unlike the best competitors, the method predicts poses for multiple objects in real time.