PaperlayerView exact revision on arXiv

Source-linked AI summary

GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator

Xingyu Liu, Ruida Zhang, Chenyangguang Zhang, Gu Wang, Jiwen Tang, Zhigang Li, Xiangyang Ji

arXiv:2102.12145v5cs.CVcs.RO

TL;DR

6D object pose estimation must balance accurate geometric reasoning with fast, end-to-end trainable inference. GDRNPP uses geometry-guided direct regression and depth-enabled differentiable refinement, achieving leading results across RGB and RGB-D settings and surpassing traditional methods in accuracy and speed.

  • Problem

    Direct pose regression has suboptimal performance without geometric priors, whereas correspondence-based pipelines rely on time-consuming traditional techniques and are not end-to-end trainable.

  • Method

    GDRNPP directly regresses monocular 6D pose using geometry-guided features and learnable Patch-PnP, then optionally refines it through predicted 3D-3D correspondences from RGB-D data.

  • Results

    GDRNPP achieved state-of-the-art performance in RGB and RGB-D settings, led the BOP Challenge for two consecutive years, and surpassed traditional methods in accuracy and speed.

  • Takeaways & Limitations

    A fully learning-based geometry-guided pipeline can provide accurate and robust 6D pose estimation while flexibly supporting RGB or RGB-D inputs.

  • Takeaways & Limitations

    The stated goal assumes an RGB(-D) image, detected objects, and corresponding 3D CAD models for pose estimation.

Abstract

from arXiv · show

6D pose estimation of rigid objects is a long-standing and challenging task in computer vision. Recently, the emergence of deep learning reveals the potential of Convolutional Neural Networks (CNNs) to predict reliable 6D poses. Given that direct pose regression networks currently exhibit suboptimal performance, most methods still resort to traditional techniques to varying degrees. For example, top-performing methods often adopt an indirect strategy by first establishing 2D-3D or 3D-3D correspondences followed by applying the RANSAC-based PnP or Kabsch algorithms, and further employing ICP for refinement. Despite the performance enhancement, the integration of traditional techniques makes the networks time-consuming and not end-to-end trainable. Orthogonal to them, this paper introduces a fully learning-based object pose estimator. In this work, we first perform an in-depth investigation of both direct and indirect methods and propose a simple yet effective Geometry-guided Direct Regression Network (GDRN) to learn the 6D pose from monocular images in an end-to-end manner. Afterwards, we introduce a geometry-guided pose refinement module, enhancing pose accuracy when extra depth data is available. Guided by the predicted coordinate map, we build an end-to-end differentiable architecture that establishes robust and accurate 3D-3D correspondences between the observed and rendered RGB-D images to refine the pose. Our enhanced pose estimation pipeline GDRNPP (GDRN Plus Plus) conquered the leaderboard of the BOP Challenge for two consecutive years, becoming the first to surpass all prior methods that relied on traditional techniques in both accuracy and speed. The code and models are available at https://github.com/shanice-l/gdrnpp_bop2022.

1 INTRODUCTION

GDRNPP addresses the accuracy, speed, and trainability limitations of correspondence-based pose pipelines by learning geometry-guided pose estimation and refinement end to end. It combines direct monocular regression with a depth-enabled 3D-3D refinement module and achieves leading performance in RGB and RGB-D settings.

  • Motivation: Direct pose regression methods lack geometric priors, while correspondence-based approaches commonly depend on traditional PnP, Kabsch, or ICP techniques.These pipelines can provide good estimates but are time-consuming and not end-to-end trainable.
  • GDRN: GDRN learns the final 6D pose differentiably from image-organized 2D-3D correspondences, using geometric representations and a learnable Patch-PnP module.The approach avoids the non-differentiable and lengthy PnP/RANSAC process.
  • Depth refinement: When depth is available, GDRNPP predicts 3D optical flow to establish dense 3D-3D correspondences between observed and rendered RGB-D images for pose refinement.This geometry-guided refinement module uses the predicted object coordinates to improve pose accuracy.
  • Pipeline: GDRNPP provides a flexible pipeline for either RGB or RGB-D input while maintaining fully learning-based pose estimation.The pipeline is designed to accommodate the available modality without introducing traditional optimization stages.
  • Results: GDRNPP achieved state-of-the-art performance in both RGB and RGB-D settings and led the BOP Challenge in 2022 and 2023.It was recognized as the Overall Best Method for two consecutive years and surpassed traditional PPF- or ICP-based methods in accuracy and speed.
  • Improvements over GDR-Net: Compared with GDR-Net, GDRNPP adds exploratory improvements and depth-based refinement, strengthening performance on challenging symmetric, textureless datasets.The cited datasets are T-LESS and ITODD.

2 RELATED WORK

Prior 6D pose estimation methods include indirect correspondence-based pipelines, direct pose regression, differentiable indirect methods, and iterative refinement techniques. The literature contrasts traditional correspondence solving with efforts to make pose estimation differentiable and refinement more robust.

  • Indirect methods: Indirect methods establish 2D-3D or 3D-3D correspondences and solve pose using RANSAC-based PnP or Kabsch algorithms.Representative methods use projected control points, segmentation and voting, or 3D keypoint voting to obtain correspondences.
  • Direct methods: Direct methods regress the 6D pose or classify discretized pose spaces without explicitly solving correspondence-based geometric optimization.Some direct approaches use point-matching losses, separate component losses, or proxy tasks.
  • Differentiable pose estimation: GDR-Net replaces PnP/RANSAC with a Patch-PnP module, extending differentiable monocular pose estimation from the conference version of the work.The cited related work positions GDR-Net as an earlier differentiable alternative to traditional pose solving.
  • Differentiable indirect methods: Differentiable indirect methods seek to backpropagate through RANSAC-like procedures but may require complex training and good scene-coordinate initialization.Several approaches share hypotheses or learn inlier probabilities to make the optimization differentiable.
  • Pose refinement: Monocular refinement methods commonly use iterative render-and-compare procedures, feature alignment, correspondence fields, or multi-view scene matching.DeepIM, CosyPose, RePose, and RNNPose represent these refinement strategies.
  • Depth refinement: Depth-based refinement has traditionally relied on ICP and related algorithms, but interpolation errors near object edges can reduce pose accuracy.The cited challenge arises when corresponding points fall between pixels and their depth values must be interpolated.

3 METHODS

GDRN directly estimates 6D object pose from geometry-guided intermediate features, using representations tailored to zoomed-in regions of interest. When depth is available, GDRNPP adds coordinate-guided refinement based on dense 3D correspondences and confidence-weighted features.

  • 3.1 Revisiting Direct 6D Object Pose Estimation: GDRN represents rotation with a continuous 6-dimensional representation and predicts its allocentric form, which is viewpoint-invariant under object translation.The representation uses the first two columns of the rotation matrix and reconstructs the third by normalization and cross products.
  • 3.1 Revisiting Direct 6D Object Pose Estimation: Translation uses SITE parameters that normalize object-center offsets and depth by detected-box geometry, making regression suitable for zoomed-in RoIs.SITE regresses δx, δy, and δz using the detected-box center, size, and zoom ratio.
  • 3.1 Revisiting Direct 6D Object Pose Estimation: The pose loss separately supervises rotation, scale-invariant 2D center, and distance, with a symmetry-aware rotation extension for symmetric objects.This disentangled objective is expressed as the sum of rotation, center, and depth losses.
  • 3.2 Geometry-guided Direct Regression Network: The method unifies direct regression with dense correspondence-based geometry through Patch-PnP, which directly regresses pose from M2D-3D and MSRA features.The geometric features are organized in image-space patches for differentiable pose estimation.
  • 3.2 Geometry-guided Direct Regression Network: GDRN estimates 6D object pose from zoomed-in RoIs by predicting dense correspondence, surface-attention, and visible-mask feature maps.The network uses 256 × 256 RoIs and produces 64 × 64 intermediate maps.

4 EXPERIMENTS

Experiments evaluate Patch-PnP and GDRN components on synthetic data and LM/LM-O, then compare GDRNPP with state-of-the-art methods across BOP datasets. Results show gains in accuracy, efficiency, refinement, and benchmark performance under RGB and RGB-D settings.

  • Ablation Study on LM: Patch-PnP is more accurate than PnP/RANSAC, PointNet-like PnP, and BPnP, while being up to 4× faster in training than BPnP.The comparison is reported in the LM ablation study; Patch-PnP also has faster inference than BPnP.
  • Ablation Study on LM: R6d yields more accurate rotation estimates than unit quaternions, log quaternions, and Lie algebra-based vectors, while allocentric parameterization outperforms egocentric parameterization.The comparison appears in Table 1b across the reported rotation parameterizations.
  • Ablation Study on LM: Scale-invariant tSITE improves translation estimation over directly predicting 3D translation, absolute tz, or the object center.The study identifies scale-invariant translation parameterization as important for zoomed-in regions of interest.
  • Ablation Study on LM: Explicit geometric guidance improves GDRN, while removing M2D causes a significant accuracy drop; direct regression also supports stronger geometric features through pose-loss backpropagation.GDRN without geometric supervision still outperforms CDPN at ADD(-S) 0.1d when using R6d, but explicit guidance performs better.
  • Ablation Study on LM-O: Class-aware geometric outputs improve ARBOP from 67.8 % to 68.6 %, while separate models for each object reach 71.3 % using pure RGB data.These results are reported in the LM-O GDRN ablation study.
  • Ablation Study on LM-O: Feature weighting raises LM-O average recall to 79.9 %, a 2.1 % gain over baseline, while two inner and one outer refinement iteration reduces recall by 0.5% and inference time from 2.48 s to 0.25 s.The refinement ablation also reports a 0.6 % gain from selecting a proper initial pose for symmetric objects.
  • Comparison with State of the Arts: GDRNPP achieves 70.2 % ARBOP using only synthetic RGB training and 85.6 % average recall with real RGB-D data, while the BOP22 version wins “The Overall Best Method” of the BOP 2022 Challenge.With real data on selected datasets, RGB performance reaches 72.8 % without refinement; synthetic-only RGB-D performance reaches 84.4 %.

LM-O YCB-V T-LESS

GDRNPP produces more precise pose estimates than PFA and CosyPose across challenging datasets, while combining high accuracy with fast inference. Its robustness is illustrated under clutter, occlusion, and varying lighting conditions.

  • Qualitative results: GDRNPP shows fewer missing and falsely detected objects than PFA and CosyPose across LM-O, YCB-V, T-LESS, ITODD, IC-BIN, and HB.Qualitative comparisons use rendered 3D models overlaid on grayscale images, with predicted and ground-truth poses shown by colored contours.
  • Qualitative results: GDRNPP produces more precise pose estimations than PFA and CosyPose in the qualitative comparisons.
  • Qualitative results: GDRNPP remains versatile in intricate scenarios involving clutter, occlusion, and varying lighting conditions.
  • Runtime analysis: GDRN combines accurate pose estimation with real-time performance by avoiding the time-consuming PnP/RANSAC procedure used by indirect methods.
  • Runtime analysis: 0.23s per RGB image is GDRN's average speed, compared with 8.89s for SurfEmb and 3.02s for PFA.

5 CONCLUSION

The paper proposes a geometry-guided, fully learning-based pose estimator that directly regresses 6D poses and refines them using differentiable RGB-D correspondences. The pipeline reports competitive performance across challenging scenarios while maintaining fast inference speed.

  • 5 CONCLUSION: GDRN directly regresses 6D poses from monocular images using intermediate geometric features and a learnable 2D convolutional Patch-PnP.
  • 5 CONCLUSION: The differentiable pipeline enables pose solving without the traditional PnP/RANSAC stage.
  • 5 CONCLUSION: The RGB-D refinement module establishes dense 3D-3D correspondences between observed and rendered images and dynamically removes outliers.
  • 5 CONCLUSION: GDRNPP achieves competitive performance in challenging scenarios while maintaining a fast inference speed.
Loading 2102.12145v5…