PaperlayerView exact revision on arXiv

Source-linked AI summary

GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence

Van Nguyen Nguyen, Thibault Groueix, Mathieu Salzmann, Vincent Lepetit

arXiv:2311.14155v2cs.CV

TL;DR

Novel-object pose estimation must avoid costly retraining while handling slow template matching and segmentation errors. GigaPose combines two-degree-of-freedom template matching with single-correspondence estimation of the remaining pose parameters, achieving reported speed, robustness, and accuracy gains and extending to reconstructed 3D models.

  • Problem

    Novel-object coarse pose estimation is limited by template-based inference speed and sensitivity to detection and segmentation errors, while supervised approaches require retraining for new objects.

  • Method

    GigaPose matches rendered templates over azimuth and elevation, then uses patch correspondences and lightweight regressors to estimate the remaining four pose degrees of freedom.

  • Results

    GigaPose is reported as significantly faster, more robust, and more accurate than existing methods, including a 35× speedup over MegaPose and evaluation on seven core BOP datasets.

  • Takeaways & Limitations

    GigaPose can integrate with refinement methods and can use single-image reconstructed 3D models, relaxing the need for accurate CAD models.

  • Takeaways & Limitations

    GigaPose fails under heavy occlusions, low-resolution segmentation, and low-fidelity CAD models, while high-quality single-image reconstruction may require manual reference-image selection.

Abstract

from arXiv · show

We present GigaPose, a fast, robust, and accurate method for CAD-based novel object pose estimation in RGB images. GigaPose first leverages discriminative "templates", rendered images of the CAD models, to recover the out-of-plane rotation and then uses patch correspondences to estimate the four remaining parameters. Our approach samples templates in only a two-degrees-of-freedom space instead of the usual three and matches the input image to the templates using fast nearest-neighbor search in feature space, results in a speedup factor of 35x compared to the state of the art. Moreover, GigaPose is significantly more robust to segmentation errors. Our extensive evaluation on the seven core datasets of the BOP challenge demonstrates that it achieves state-of-the-art accuracy and can be seamlessly integrated with existing refinement methods. Additionally, we show the potential of GigaPose with 3D models predicted by recent work on 3D reconstruction from a single image, relaxing the need for CAD models and making 6D pose object estimation much more convenient. Our source code and trained models are publicly available at https://github.com/nv-nguyen/gigaPose

1. Introduction

GigaPose targets the speed and segmentation-robustness limitations of coarse pose estimation for novel objects. It combines two-degree-of-freedom template matching with patch correspondences to recover complete 6D pose and reports faster, more robust, and more accurate performance.

  • Motivation: Existing coarse pose estimators are slow and sensitive to detection or segmentation errors, especially under occlusion.MegaPose-based methods require more than 1.6 seconds per detection, while erroneous segments create scale, translation, and outlier-matching problems.
  • Method: GigaPose uses templates for azimuth and elevation, while patch correspondences estimate in-plane rotation, 2D translation, and 2D scale from one match.The design assigns complex appearance changes to templates and the remaining four degrees of freedom to correspondences.
  • Results: 35× faster inference than MegaPose is reported, while the two-step design is also described as robust to segmentation errors.The speedup comes from sublinear nearest-neighbor template search rather than evaluating every query-template pair with a network.
  • Extensions: The approach can recover accurate 6D pose from single-image 3D reconstructions despite inaccuracies in predicted models, reducing reliance on accurate CAD models.This result is presented as making novel-object pose estimation more convenient.
  • Results: GigaPose is presented as significantly faster, more robust, and more accurate than existing methods across the seven core BOP datasets.The method is also designed to integrate with existing refinement methods for CAD-based novel object pose estimation.

2. Related Work

Related work spans supervised and category-level pose estimation, feature and template matching, and correspondence-based methods. GigaPose differs by using templates for two rotational degrees of freedom and one 2D-2D correspondence for the remaining pose parameters.

  • Seen and novel objects: Supervised methods can achieve strong accuracy but require hours or days of retraining for newly acquired objects, while category-level methods cannot generalize beyond training categories.GigaPose instead targets novel objects without category-level information.
  • Novel object pose estimation: Template-matching methods compare query images with rendered CAD templates, whereas feature-matching methods use local features and 3D-to-2D correspondences with PnP.These form two broad families of novel-object pose estimation methods.
  • Template matching: MegaPose was highlighted for runtime limitations because it requires a coarse-pose forward pass for every query-template comparison.The related-work discussion identifies this template-comparison cost as a significant limitation.
  • Correspondence-based estimation: GigaPose estimates four remaining degrees of freedom from a single 2D-2D match after template matching, unlike classical PnP approaches requiring at least four 3D-to-2D correspondences.The paper reports that GigaPose outperforms ZS6D, a state-of-the-art method based on 3D-to-2D correspondences.

3. Method

GigaPose decouples pose estimation by using template matching for out-of-plane rotation and patch correspondences for the remaining four degrees of freedom. Its feature extractors and contrastive training are designed to support efficient, rotation-aware matching while preserving robustness to scale, in-plane rotation, and translation changes.

  • 3.1. Generating Templates: GigaPose selects templates to estimate azimuth and elevation, then uses patch correspondences to recover in-plane rotation, scale, and translation.The template stage uses dense features, while the correspondence stage predicts scale and in-plane rotation from matched patches and obtains translation from their locations.
  • 3.2. Predicting Azimuth and Elevation: The method independently extracts dense feature maps from queries and templates, allowing template features to be computed during object onboarding rather than repeatedly at runtime.Each feature-grid location corresponds to a local image patch, and foreground masks identify object regions used for matching.
  • 3.2. Predicting Azimuth and Elevation: Fae is trained so local features vary with out-of-plane rotation while remaining invariant to scale, in-plane rotation, and 2D translation.This design lets template similarity distinguish viewpoint changes without requiring templates for every in-plane rotation and scale.
  • 3.2. Predicting Azimuth and Elevation: For each query patch, GigaPose finds a nearest template-feature neighbor, filters matches below similarity 0.5, and averages the remaining similarity scores.The resulting pairwise similarity metric ranks templates according to their local feature correspondences.
  • 3.3. Predicting the Remaining DoFs: Fist and two lightweight MLPs regress in-plane rotation and scale from each 2D-2D correspondence, while correspondence locations provide 2D translation.Fae cannot regress scale and in-plane rotation because it is intentionally invariant to those factors, so GigaPose uses a separate feature extractor.

4. Experiments

Across seven BOP datasets, GigaPose consistently improves accuracy over MegaPose while substantially reducing coarse-pose runtime, remaining more stable under segmentation errors, and supporting predicted 3D models.

  • Accuracy: Across all settings, GigaPose consistently outperforms MegaPose, with over 6% higher AR for coarse estimation and over 4% higher AR after refinement on T-LESS, IC-BIN, and ITODD.The comparison includes refined and unrefined results across the seven core BOP datasets.
  • Run-time: More than 35× faster coarse pose estimation is achieved by GigaPose, requiring 48 ms per detection versus 1.68 seconds for MegaPose.The speed advantage comes from sublinear nearest-neighbor search rather than evaluating every query-template pair with a network.
  • Predicted 3D models: Using Wonder3D-predicted 3D models from a single reference image, GigaPose outperforms MegaPose and achieves results close to its accurate-CAD performance.The predicted-model setting is evaluated on LM-O using AR scores.
  • Robustness to segmentation errors: GigaPose maintains a stable AR score across segmentation IoU thresholds on T-LESS and YCB-V, unlike MegaPose, which performs well mainly at high IoU.The broader robustness analysis covers LM-O, T-LESS, and YCB-V; improvements are more limited on LM-O because objects are small, especially after occlusion.
  • Ablation study: Using 576 templates instead of 162 improves AR by only 0.6%, indicating that correspondences can reduce template memory without materially hurting accuracy.The comparison uses the same 576-template count as MegaPose.

5. Conclusion

GigaPose is presented as an efficient method for 6D coarse pose estimation of novel objects, emphasizing speed, robustness, accuracy, and compatibility with refinement methods.

  • GigaPose is an efficient method for 6D coarse pose estimation of novel objects.
  • The method is designed to make real-time, accurate pose estimation of novel objects practical.

6. Ground-truth 2D-to-2D correspondences

GigaPose constructs training correspondences between query-image patches and the nearest rendered template through 3D reprojection using depth, camera intrinsics, and relative pose. Color augmentation further reduces the real–synthetic domain gap.

  • Ground-truth correspondence construction: Ground-truth 3D information from BOP training sets, originally sourced from MegaPose, supplies the training correspondences.
  • Ground-truth correspondence construction: 3D reprojection maps each query-image patch center to its corresponding location in the nearest template.The process back-projects the query point with depth and camera intrinsics, transforms it using the ground-truth relative pose, and re-projects it into the template view.
  • Ground-truth correspondence construction: Reversing query and template roles produces correspondences for every template patch as well.
  • Training augmentation: Color augmentation applies blur, contrast, brightness, color, and sharpness filters to narrow the real–synthetic domain gap.The resulting augmented samples are illustrated in Figure 8.

7. Recovering a 6D object pose

GigaPose recovers a 6D pose by combining the nearest template’s out-of-plane rotation with a predicted in-plane rotation, then estimating translation through affine image transformations and camera geometry.

  • Image transformation: The affine transformation between processed template and query images encodes in-plane rotation, 2D scale, and 2D translation.Scaling, cropping, and padding standardize both inputs to 224×224 before this transformation is used.
  • Rotation recovery: The full rotation combines the nearest template’s out-of-plane rotation with the network-predicted in-plane rotation.
  • Translation recovery: The template-to-query affine transform maps the template object-center projection to the query projection, recovering 2D translation coordinates.
  • Translation recovery: Query-depth translation along Z is recovered from template depth, affine scale, and the ratio of query to template focal lengths.The affine scale is the norm of its first column, and the focal-length function supplies the camera term.
  • Translation recovery: The complete query translation is obtained by back-projecting the recovered image center through the inverse query camera intrinsic matrix.

8. “2D” version of the Kabsch algorithm

GigaPose uses a two-dimensional Kabsch procedure to recover the affine transformation from two 2D-to-2D patch correspondences, estimating scale, in-plane rotation, and translation.

  • Affine recovery: Two 2D-to-2D correspondences suffice to recover affine scale, in-plane rotation, and translation with the 2D Kabsch algorithm.The correspondences are obtained from nearest-neighbor feature matching.
  • Affine recovery: The correspondence model transforms each template point into its query point using scale, rotation, and translation.
  • Scale estimation: Scale is computed as the ratio of distances between the query-point pair and template-point pair.
  • Rotation estimation: The in-plane rotation matrix uses cos(α) and sin(α), obtained from normalized dot and cross products of the point-pair vectors.
  • Context: Failure cases in Wonder3D show objects appearing flat from novel angles, motivating the distinct correspondence-based recovery approach.
  • Translation estimation: Translation is the average residual after subtracting the scaled, rotated template points from their corresponding query points.

9. Additional results

Additional experiments evaluate single-image 3D reconstruction and qualitative pose estimation under challenging benchmark conditions. Wonder3D-based reconstructions support the evaluation, while selected visualizations show GigaPose’s matching and alignment behavior.

  • 9.1. Using 3D models predicted by Wonder3D: Wonder3D reconstructs 3D models from single images using predicted views and instant-NGP-based SDF reconstruction.The reconstruction uses a reference front view plus five predicted views: front-left, left, front-right, right, and back.
  • 9.1. Using 3D models predicted by Wonder3D: The Wonder3D visualizations display the input image, predicted RGB views, and corresponding predicted normals for each reconstruction sample.Each sample contains a 2×6 image arrangement with the input outlined in green and predicted views and normals shown in separate positions.
  • 9.1. Using 3D models predicted by Wonder3D: The LM-O evaluation reports CNOS detection and segmentation performance when using 3D models predicted from a single image by Wonder3D.The evaluation follows the standard protocol for the BOP detection and segmentation tasks.

10. Future work

Future work targets failure cases involving severe occlusion, low-resolution segmentation, and low-fidelity CAD models. The authors also identify depth and improved single-image 3D reconstruction as promising directions.

  • 10. Future work: GigaPose fails under heavy occlusion, low-resolution segmentation, and low-fidelity CAD models, as observed on LM-O.These conditions define an explicit boundary of the method’s reported robustness.
  • 10. Future work: Additional modalities such as depth images may improve performance because they capture information about object geometry in these challenging conditions.The proposed direction is motivated specifically by the failures associated with occlusion, segmentation quality, and model fidelity.
  • 10. Future work: High-quality single-image 3D reconstruction remains valuable because the Wonder3D-based setting requires manual reference-image selection.The authors point to more advanced reconstruction methods as a way to reduce this dependency.
Loading 2311.14155v2…