PaperlayerView exact revision on arXiv

Source-linked AI summary

SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, Federico Tombari

arXiv:2108.08367v1cs.CV

TL;DR

End-to-end 6D pose estimation is efficient but less accurate than elaborate two-stage methods in cluttered scenes. SO-Pose adds self-occlusion to 2D-3D correspondences in a two-layer representation and directly regresses pose. It consistently outperforms end-to-end competitors and achieves comparable accuracy to two-stage methods while being faster.

  • Problem

    End-to-end pose methods remain less accurate than two-stage approaches, while surrogate correspondence losses limit differentiable processing and downstream integration.

    Source: §1 · p. 1

  • Method

    SO-Pose uses self-occlusion and 2D-3D correspondences in a two-layer representation, with cross-layer consistency terms aligning the intermediate geometry and 6D pose.

    Source: §1 · p. 2

  • Results

    SO-Pose consistently surpasses other end-to-end competitors and achieves comparable accuracy to state-of-the-art two-stage methods while being much faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two-layer model improves performance across multiple 6D pose frameworks and datasets, indicating applicability beyond SO-Pose.

    Source: §5 · p. 8

    The authors identify integrating the two-layer model into self-supervised pose estimation and category-level unseen-object analysis as future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Directly regressing all 6 degrees-of-freedom (6DoF) for the object pose (e.g. the 3D rotation and translation) in a cluttered environment from a single RGB image is a challenging problem. While end-to-end methods have recently demonstrated promising results at high efficiency, they are still inferior when compared with elaborate P$n$P/RANSAC-based approaches in terms of pose accuracy. In this work, we address this shortcoming by means of a novel reasoning about self-occlusion, in order to establish a two-layer representation for 3D objects which considerably enhances the accuracy of end-to-end 6D pose estimation. Our framework, named SO-Pose, takes a single RGB image as input and respectively generates 2D-3D correspondences as well as self-occlusion information harnessing a shared encoder and two separate decoders. Both outputs are then fused to directly regress the 6DoF pose parameters. Incorporating cross-layer consistencies that align correspondences, self-occlusion and 6D pose, we can further improve accuracy and robustness, surpassing or rivaling all other state-of-the-art approaches on various challenging datasets.

1. Introduction

End-to-end 6D pose estimation is efficient but remains less accurate than two-stage approaches because noisy correspondence fields create ambiguity. SO-Pose addresses this gap by adding self-occlusion to a two-layer representation and directly regressing pose.

  • Two-stage methods establish 2D-3D correspondences before computing pose, but require extra optimization and are not end-to-end differentiable.Source: §1 · p. 1
  • End-to-end methods directly learn pose from geometric representations but remain inferior to approaches using consistency checks, symmetry analysis, or disentangled predictions.Source: §1 · p. 1
  • Textureless surfaces cause unavoidable mismatches, allowing one correspondence field to support multiple poses with similar fitting errors.Source: §1 · pp. 1–2
  • SO-Pose adds self-occlusion information to visible-point correspondences, forming a viewer-centered two-layer representation of the object.Source: Figure 1 · p. 1, §1 · p. 2
  • SO-Pose directly regresses 6D pose and reportedly surpasses end-to-end competitors while achieving comparable accuracy to two-stage methods at higher speed.Source: §1 · p. 2

2. Related Works

Monocular 6D pose estimation work is organized around direct regression, latent pose embeddings, and correspondence-based RANSAC/PnP solutions. These lines differ in whether pose is predicted directly, retrieved from an embedding, or solved from geometric correspondences.

  • Monocular 6D pose methods are broadly divided into direct regression, latent embedding, and 2D-3D correspondence-based approaches.Source: §2 · p. 2
  • Direct-regression methods predict pose from images, including classification-based regression, multiple hypotheses, contour alignment, and point-matching objectives.Source: §2 · p. 2
  • Latent-embedding methods learn low-dimensional pose representations and retrieve poses by comparing detections with a precomputed codebook.Source: §2 · p. 2
  • Correspondence-based methods estimate 2D projections of 3D bounding-box corners or model points before solving pose with RANSAC/PnP.Source: §2 · p. 2

3. Methodology

SO-Pose combines visible 2D-3D correspondences with analytically derived self-occlusion coordinates in a two-layer representation. Shared features and separate prediction branches feed cross-layer consistency losses and a differentiable pose regressor.

Pose formulation

  • Given an image, a neural network maps input features to object rotation R and translation t in the camera frame.Source: §3 · p. 2

Framework

  • SO-Pose uses an encoder and separate decoders to predict a self-occlusion map, 2D-3D correspondences, and an object mask before pose estimation.Source: Figure 2 · p. 3, §3.3 · p. 5

Two-layer representation

  • The two-layer representation combines visible correspondences with self-occlusion information to capture more complete object geometry than single-layer representations.Source: §3 · p. 3, §3.1 · p. 3

Representation benefits

  • The self-occlusion coordinate is analytically derived, has two degrees of freedom, and regularizes predictions while reducing rendering and noise errors.Source: §3.1 · p. 4

Self-occlusion construction

  • For each visible point, the camera ray intersects object coordinate planes at up to three points, with intersections restricted to the object’s minimum bounding cuboid.Source: Figure 3 · p. 4, §3.1 · p. 4

Consistency learning

  • Two cross-layer consistency losses align self-occlusion, correspondences, and pose in 3D and ensure that P and Q project to the same image point.Source: §3.2 · pp. 4–5

Training objective

  • The differentiable objective combines pose terms, cross-layer consistency terms, and self-occlusion losses; the pose term includes correspondence, translation, mask, region, and point-matching components.Source: §3.3 · p. 5

4. Evaluation

SO-Pose is evaluated on three challenging datasets through state-of-the-art comparisons, ablations, runtime measurements, and qualitative examples. It generally improves accuracy over end-to-end competitors, approaches or exceeds two-stage methods in reported settings, and remains efficient.

Evaluation setup

  • SO-Pose is evaluated on LM, LMO, and YCB-V using standard pose metrics and comparisons with state-of-the-art methods.The experiments also include ablations, backbone comparisons, runtime analysis, and qualitative results.Source: §4 · p. 5, §4.2 · pp. 5–6

Comparison with state of the art

  • 45.9 versus 35.3 under ADD(−S) 0.02d is reported on LM against GDR-Net, an improvement of up to 30%.SO-Pose is reported to outperform all baseline methods for each listed metric, especially ADD(−S) 0.02d and 2°2 cm.Source: §4.3 · p. 6
  • 62.3 versus 24.9–56.1 is reported as the average ADD(−S) score on LMO when trained with real+pbr data.The method achieves state-of-the-art performance on 5 of 8 objects in that setting and is comparable to DeepIM when trained with real+syn data.Source: §4.3 · p. 6, Table 3 · p. 7
  • 0.664 mean AR is reported on YCB-V, compared with 0.578 for CDPN-v2 and 0.621 for EPOS, while remaining below CosyPose.The comparison concerns methods using different design choices, including refinement-based approaches.Source: §4.3 · p. 6, Table 2 · p. 6

Ablation study

  • Removing either cross-layer consistency loss decreases accuracy, while adding the two-layer model to CDPN significantly improves all reported metrics.The ablations support the usefulness and generalizability of the cross-layer consistency terms and two-layer representation.Source: §4.4 · pp. 6–7, Table 5 · p. 7

Runtime analysis

  • 30ms per object and 50ms for all objects are reported on a 640×480 YCB-V image, including 15ms for 2D localization.The measurements use an Intel 3.30GHz CPU and TITAN X GPU; the method achieves AR 0.715 in real time on YCB-V.Source: §4.5 · p. 7

5. Conclusion

SO-Pose uses a two-layer representation combining 2D-3D correspondences with self-occlusion information to improve end-to-end 6D pose estimation. The representation improves performance over single-layer competitors and is intended for broader pose-estimation frameworks, while future work targets self-supervised and unseen-object settings.

  • SO-Pose combines 2D-3D point correspondences and self-occlusion information to represent an object's spatial cues explicitly.Source: §5 · p. 8, Figure 6 · p. 8
  • The predicted two-layer model includes 2D-3D point matching, self-occlusion coordinates, and corresponding normalized error maps.The error maps use a normalized 0-to-1 color scale.Source: Figure 6 · p. 8
  • The two-layer model achieves significant improvements across challenging datasets over other single-layer competitors.Source: §5 · p. 8
  • The two-layer model is applicable to a wide range of 6D pose-estimation frameworks and consistently benefits performance.Source: §5 · p. 8
  • Future work will integrate the two-layer model into self-supervised 6D pose estimation and category-level unseen-object analysis.Source: §5 · p. 8

Related papers

Loading 2108.08367v1…