Source-linked AI summary
SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation
Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, Federico Tombari
TL;DR
End-to-end 6D pose estimation is efficient but less accurate than elaborate two-stage methods in cluttered scenes. SO-Pose adds self-occlusion to 2D-3D correspondences in a two-layer representation and directly regresses pose. It consistently outperforms end-to-end competitors and achieves comparable accuracy to two-stage methods while being faster.
Problem
End-to-end pose methods remain less accurate than two-stage approaches, while surrogate correspondence losses limit differentiable processing and downstream integration.
Method
SO-Pose uses self-occlusion and 2D-3D correspondences in a two-layer representation, with cross-layer consistency terms aligning the intermediate geometry and 6D pose.
Results
SO-Pose consistently surpasses other end-to-end competitors and achieves comparable accuracy to state-of-the-art two-stage methods while being much faster.
Takeaways & Limitations
The two-layer model improves performance across multiple 6D pose frameworks and datasets, indicating applicability beyond SO-Pose.
Takeaways & Limitations
The authors identify integrating the two-layer model into self-supervised pose estimation and category-level unseen-object analysis as future work.
Abstract
from arXiv · showhide
Directly regressing all 6 degrees-of-freedom (6DoF) for the object pose (e.g. the 3D rotation and translation) in a cluttered environment from a single RGB image is a challenging problem. While end-to-end methods have recently demonstrated promising results at high efficiency, they are still inferior when compared with elaborate P$n$P/RANSAC-based approaches in terms of pose accuracy. In this work, we address this shortcoming by means of a novel reasoning about self-occlusion, in order to establish a two-layer representation for 3D objects which considerably enhances the accuracy of end-to-end 6D pose estimation. Our framework, named SO-Pose, takes a single RGB image as input and respectively generates 2D-3D correspondences as well as self-occlusion information harnessing a shared encoder and two separate decoders. Both outputs are then fused to directly regress the 6DoF pose parameters. Incorporating cross-layer consistencies that align correspondences, self-occlusion and 6D pose, we can further improve accuracy and robustness, surpassing or rivaling all other state-of-the-art approaches on various challenging datasets.
1. Introduction
End-to-end 6D pose estimation is efficient but remains less accurate than two-stage approaches because noisy correspondence fields create ambiguity. SO-Pose addresses this gap by adding self-occlusion to a two-layer representation and directly regressing pose.
- Two-stage methods establish 2D-3D correspondences before computing pose, but require extra optimization and are not end-to-end differentiable.
- End-to-end methods directly learn pose from geometric representations but remain inferior to approaches using consistency checks, symmetry analysis, or disentangled predictions.
- Textureless surfaces cause unavoidable mismatches, allowing one correspondence field to support multiple poses with similar fitting errors.
- SO-Pose adds self-occlusion information to visible-point correspondences, forming a viewer-centered two-layer representation of the object.
- SO-Pose directly regresses 6D pose and reportedly surpasses end-to-end competitors while achieving comparable accuracy to two-stage methods at higher speed.
2. Related Works
Monocular 6D pose estimation work is organized around direct regression, latent pose embeddings, and correspondence-based RANSAC/PnP solutions. These lines differ in whether pose is predicted directly, retrieved from an embedding, or solved from geometric correspondences.
- Monocular 6D pose methods are broadly divided into direct regression, latent embedding, and 2D-3D correspondence-based approaches.
- Direct-regression methods predict pose from images, including classification-based regression, multiple hypotheses, contour alignment, and point-matching objectives.
- Latent-embedding methods learn low-dimensional pose representations and retrieve poses by comparing detections with a precomputed codebook.
- Correspondence-based methods estimate 2D projections of 3D bounding-box corners or model points before solving pose with RANSAC/PnP.
3. Methodology
SO-Pose combines visible 2D-3D correspondences with analytically derived self-occlusion coordinates in a two-layer representation. Shared features and separate prediction branches feed cross-layer consistency losses and a differentiable pose regressor.
- Pose formulation: Given an image, a neural network maps input features to object rotation R and translation t in the camera frame.
- Framework: SO-Pose uses an encoder and separate decoders to predict a self-occlusion map, 2D-3D correspondences, and an object mask before pose estimation.
- Two-layer representation: The two-layer representation combines visible correspondences with self-occlusion information to capture more complete object geometry than single-layer representations.
- Representation benefits: The self-occlusion coordinate is analytically derived, has two degrees of freedom, and regularizes predictions while reducing rendering and noise errors.
- Self-occlusion construction: For each visible point, the camera ray intersects object coordinate planes at up to three points, with intersections restricted to the object’s minimum bounding cuboid.
- Consistency learning: Two cross-layer consistency losses align self-occlusion, correspondences, and pose in 3D and ensure that P and Q project to the same image point.
- Training objective: The differentiable objective combines pose terms, cross-layer consistency terms, and self-occlusion losses; the pose term includes correspondence, translation, mask, region, and point-matching components.
4. Evaluation
SO-Pose is evaluated on three challenging datasets through state-of-the-art comparisons, ablations, runtime measurements, and qualitative examples. It generally improves accuracy over end-to-end competitors, approaches or exceeds two-stage methods in reported settings, and remains efficient.
- Evaluation setup: SO-Pose is evaluated on LM, LMO, and YCB-V using standard pose metrics and comparisons with state-of-the-art methods.The experiments also include ablations, backbone comparisons, runtime analysis, and qualitative results.
- Comparison with state of the art: 45.9 versus 35.3 under ADD(−S) 0.02d is reported on LM against GDR-Net, an improvement of up to 30%.SO-Pose is reported to outperform all baseline methods for each listed metric, especially ADD(−S) 0.02d and 2°2 cm.
- Comparison with state of the art: 62.3 versus 24.9–56.1 is reported as the average ADD(−S) score on LMO when trained with real+pbr data.The method achieves state-of-the-art performance on 5 of 8 objects in that setting and is comparable to DeepIM when trained with real+syn data.
- Comparison with state of the art: 0.664 mean AR is reported on YCB-V, compared with 0.578 for CDPN-v2 and 0.621 for EPOS, while remaining below CosyPose.The comparison concerns methods using different design choices, including refinement-based approaches.
- Ablation study: Removing either cross-layer consistency loss decreases accuracy, while adding the two-layer model to CDPN significantly improves all reported metrics.The ablations support the usefulness and generalizability of the cross-layer consistency terms and two-layer representation.
- Runtime analysis: 30ms per object and 50ms for all objects are reported on a 640×480 YCB-V image, including 15ms for 2D localization.The measurements use an Intel 3.30GHz CPU and TITAN X GPU; the method achieves AR 0.715 in real time on YCB-V.
5. Conclusion
SO-Pose uses a two-layer representation combining 2D-3D correspondences with self-occlusion information to improve end-to-end 6D pose estimation. The representation improves performance over single-layer competitors and is intended for broader pose-estimation frameworks, while future work targets self-supervised and unseen-object settings.
- SO-Pose combines 2D-3D point correspondences and self-occlusion information to represent an object's spatial cues explicitly.
- The predicted two-layer model includes 2D-3D point matching, self-occlusion coordinates, and corresponding normalized error maps.The error maps use a normalized 0-to-1 color scale.
- The two-layer model achieves significant improvements across challenging datasets over other single-layer competitors.
- The two-layer model is applicable to a wide range of 6D pose-estimation frameworks and consistently benefits performance.
- Future work will integrate the two-layer model into self-supervised 6D pose estimation and category-level unseen-object analysis.