PaperlayerView exact revision on arXiv

Source-linked AI summary

Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images

Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, Wenping Wang

arXiv:2204.10776v2cs.CV

TL;DR

Generalizable 6-DoF pose estimation is constrained by reliance on object models, depth maps, masks, or object-specific training. Gen6D instead matches posed reference images to RGB queries through detection, viewpoint selection, and 3D volume-based refinement, with strong results on model-free datasets and competitive LINEMOD performance.

  • Problem

    Existing generalizable pose estimators require high-quality object models or additional depth maps and masks, while other methods are limited to trained objects or categories.

  • Method

    Gen6D uses posed reference images and RGB query images in a coarse-to-fine pipeline with an object detector, viewpoint selector, and model-free 3D volume-based pose refiner.

  • Results

    Gen6D achieves state-of-the-art results on the model-free MOPED and GenMOP datasets and competitive results with instance-specific estimators on LINEMOD.

  • Takeaways & Limitations

    Gen6D provides an easy-to-use pose estimator for unseen objects that needs posed images rather than a 3D model, depth maps, or masks.

  • Takeaways & Limitations

    Gen6D performs worse with limited training data and may degenerate under severe occlusions because it is not specially designed to handle occlusions.

Abstract

from arXiv · show

In this paper, we present a generalizable model-free 6-DoF object pose estimator called Gen6D. Existing generalizable pose estimators either need high-quality object models or require additional depth maps or object masks in test time, which significantly limits their application scope. In contrast, our pose estimator only requires some posed images of the unseen object and is able to accurately predict the poses of the object in arbitrary environments. Gen6D consists of an object detector, a viewpoint selector and a pose refiner, all of which do not require the 3D object model and can generalize to unseen objects. Experiments show that Gen6D achieves state-of-the-art results on two model-free datasets: the MOPED dataset and a new GenMOP dataset collected by us. In addition, on the LINEMOD dataset, Gen6D achieves competitive results compared with instance-specific pose estimators. Project page: https://liuyuan-pal.github.io/Gen6D/.

1 Introduction

Gen6D addresses restrictive assumptions in generalizable 6-DoF pose estimation with a model-free, RGB-only, coarse-to-fine framework that uses posed reference images for unseen objects. Its detector, viewpoint selector, and volume-based pose refiner support accurate pose estimation without object models, depth maps, or masks.

  • Motivation: Existing generalizable estimators require object models, depth maps, or masks, while many other methods are limited to trained objects or categories.These requirements restrict practical application of 6-DoF pose estimation.
  • Contribution: Gen6D targets arbitrary unseen objects using only posed reference images and RGB query images, without requiring a 3D object model, depth maps, or masks.The reference images define the object coordinate system.
  • Framework: Gen6D estimates pose coarse-to-fine through an object detector, viewpoint selector, and pose refiner.The detector finds object regions, the selector produces a coarse pose, and the refiner searches for an accurate pose.
  • Viewpoint Selection: Pixel-wise viewpoint comparison focuses on object regions and global normalization plus self-attention share information across reference images.These design choices address cluttered backgrounds and sparse viewpoints.
  • Pose Refinement: The 3D volume-based refiner projects nearby reference images into 3D, matches their feature volume with query features, and avoids rendering new images.The method performs pose refinement in 3D space rather than regressing a 3D relative pose only from 2D features.
  • Evaluation: Experiments report that Gen6D outperforms PVNet on GenMOP and MOPED and achieves comparable results to instance-specific estimators on LINEMOD.GenMOP contains objects in different environments and lighting conditions, with objects unseen during training.

2 Related works

Related work includes instance-specific, category-specific, and generalizable pose estimators with varying assumptions about object models, categories, and rendering. Gen6D is positioned as generalizable across unseen instances and categories without requiring a 3D object model.

  • Instance-Specific Estimators: Most object pose estimators are instance-specific and commonly require a 3D model to render extensive training images.Some recent instance-specific methods implicitly reconstruct the object model while remaining model-free.
  • Category-Specific Estimators: Category-specific estimators generalize within known categories but cannot predict poses for objects in unseen categories.Their generalization scope remains tied to category-level training.
  • Gen6D: Gen6D makes no assumption about the object's category or instance and does not require its 3D model.This distinguishes it from the cited instance-specific and category-specific approaches.
  • Generalizable Estimators: Generalizable estimators often use object models for shape embedding, template matching, or rendering-and-comparison.Model-free alternatives based on neural rendering can be sensitive to matching appearance and lighting conditions.
  • Related Tasks: Gen6D differs from instance detection and category-level viewpoint estimation because it estimates full 6-DoF object pose.Its detector is inspired by correlation-based instance detection.

3 Method

Gen6D estimates object pose from posed reference images through a coarse-to-fine pipeline that detects the object, selects a nearby viewpoint, and refines the pose in a 3D feature volume without rendering new images.

  • Overview: Gen6D defines object pose as rotation R and translation t mapping object coordinates into camera coordinates, with known image intrinsics.The object coordinate system is normalized so the object center is at the origin and the object fits inside a unit sphere.
  • 3.1 Detection: The detector estimates the object center projection and compact bounding-box size, then derives depth and an initial translation from these quantities.Depth is computed as d = 2 ˜f/Sq, where Sq encloses the normalized unit sphere.
  • 3.2 Viewpoint Selection: The viewpoint selector compares each reference image with the query to produce similarity scores and relative in-plane rotations, selecting the nearest viewpoint for an initial rotation.Global normalization and a transformer share information across reference views to improve relative comparison.
  • 3.3 Pose refinement: The initial translation and rotation are combined into a coarse pose that is refined using a 3D feature volume constructed from nearby posed reference images and query features.The volume uses six nearby reference images, unprojects their features, and aggregates feature means and variances at volume vertices.
  • 3.3 Pose refinement: The refiner approximates the rigid pose residual with a similarity transformation inside the feature volume and converts the prediction back to a rigid transformation.The approximation includes a 2D in-plane offset, a scale factor, and a residual 3D rotation, avoiding direct regression of translation outside the feature volume.
  • 3.3 Pose refinement: Unlike rendering-based refiners, Gen6D does not render an image at the input pose and uses multiple reference poses to infer how pose changes affect image features.This design supplies pose-dependent 3D structure for unseen objects while avoiding an object model.

4 Experiments

Experiments evaluate Gen6D on GenMOP, LINEMOD, and MOPED using unseen objects and reference/query image protocols. Gen6D generally outperforms model-free and instance-specific baselines, while performance depends on refinement, training data, bounding-box accuracy, and occlusion.

  • Datasets and protocol: GenMOP contains 10 objects with separate reference and query sequences recorded in different environments and lighting conditions.Five objects are used for testing and five for training; each sequence contains approximately 200 images.
  • Datasets and protocol: Gen6D is trained once on rendered ShapeNet and Google Scanned Object data plus GenMOP and LINEMOD objects, then tested on unseen evaluation objects.The training set includes approximately 2,000 ShapeNet models, 1,023 Google Scanned Objects, five GenMOP objects, and five LINEMOD objects.
  • Results on GenMOP: Gen6D outperforms PVNet and RLLG on average on GenMOP because its pose refiner adapts to novel objects with limited reference images.Selecting only the best reference viewpoint is insufficient because reference images do not cover all possible viewpoints.
  • Results on LINEMOD: On LINEMOD, Gen6D outperforms PFS and synthetic-training instance-specific estimators but performs worse than real-training estimators because of inaccurate depth estimation.With ground-truth bounding boxes, Gen6D achieves comparable results to real-training instance-specific estimators.
  • Results on MOPED: Gen6D outperforms Latent-Fusion and PVNet on average on MOPED using only RGB inputs and no training on the test object.Latent-Fusion requires depth and object masks on query images, whereas Gen6D does not.
  • Analysis: Pixel-wise viewpoint comparison outperforms ObjDesc under clutter, while global normalization and reference-view transformers further improve viewpoint selection.The added layers exchange information between reference images to help select the relatively most similar view.
  • Limitations: Gen6D requires diverse training data and may degrade under severe occlusion because its generalization relies on learned image matching.The limitation concerns both viewpoint selection and pose refinement.

5 Conclusion

The paper concludes that Gen6D estimates 6-DoF poses for unseen objects without object models, using only posed object images. Its viewpoint selector and volume-based pose refiner support model-free pose prediction in arbitrary environments.

  • Conclusion: Gen6D predicts poses for unseen objects from posed reference images without requiring an object model.The conclusion describes the estimator as easy to use in the model-free setting.
  • Conclusion: Gen6D combines a novel viewpoint selector with a volume-based pose refiner for unseen-object pose estimation.The conclusion identifies both components as core designs of the estimator.
  • Conclusion: Experiments demonstrate superior performance for predicting unseen-object poses in the model-free setting.This is the paper's stated overall experimental conclusion.

B.1 Data normalization

Gen6D normalizes object coordinates and reference images before detector, selector, and refiner processing. Its modules use multi-scale detection, viewpoint similarity, in-plane rotation, and iterative pose refinement.

  • Data normalization: Object coordinates are normalized around the estimated center so the object fits inside a unit sphere.The center is placed at the origin and the object size is normalized to 1; exact size and center estimates are unnecessary.
  • Data normalization: Reference images are warped with homographies so they look toward the object and enclose the unit-sphere projection.The corresponding camera intrinsics are changed consistently, using known reference-image poses.
  • Detector: The detector correlates reference and query features across three feature levels and five scales, predicting object-center heat and scale maps.It estimates the 2D center projection and a compact square bounding-box size for depth computation.
  • Detector: Detector training classifies heat-map pixels within 1.5 pixels of the projected object center as positive and applies log-space scale loss near that center.The scale loss uses L2 distance and is restricted to pixels within the same 1.5-pixel neighborhood.
  • Selector: The selector scores reference viewpoints, predicts in-plane rotation, and uses global normalization, self-attention, and five rotation angles for comparison.Its similarity loss uses binary cross entropy, while the in-plane angle uses L2 loss against the nearest reference image.
  • Refiner: The refiner combines multi-level 2D features with a 3D feature volume from six neighboring references and is applied iteratively three times.Training samples 323 unit-cube voxels and compares their transformations under predicted and ground-truth similarity transforms.

C Experimental setting

Experiments train Gen6D across real, reconstructed, and rendered objects, using separate reference and query image selections for each dataset. Test splits use held-out images or video sequences to evaluate generalization.

  • Training data: Training uses five LINEMOD objects with approximately 1200 images each and five GenMOP objects with approximately 200 reference and query images each.Additional training objects come from ShapeNet and Google Scanned Objects, rendered or sampled at dataset-specific image counts.
  • Reference/query split: Training references are selected by farthest-point sampling on camera locations, while GenMOP references and queries come from different videos.LINEMOD, ShapeNet, and Google Scanned Objects use 128 reference images; remaining images serve as queries.
  • Reference/query split: Test references use prior LINEMOD training images, separate GenMOP videos, and provided MOPED reference videos, with other images or sequences as queries.This produces object-level or sequence-level reference/query separation across the evaluated datasets.
  • GenMOP: GenMOP contains two one-minute cellphone videos per object recorded in different environments, with approximately 200 sampled images per video.COLMAP recovers camera parameters, while manually labeled keypoints and triangulation provide 3D points.

D Qualitative Results

Qualitative results are additionally reported for LINEMOD, MOPED, and GenMOP, while Table 5 summarizes reference and query image counts across datasets.

  • Qualitative results: Additional qualitative results on LINEMOD, MOPED, and GenMOP are presented in Figures 18, 19, and 20, respectively.The figures provide further visual examples across all three evaluated datasets.
  • Dataset sizes: Table 5 reports the numbers of reference images and test query images for different datasets.The supplied passage identifies the table’s quantities but does not provide the individual dataset values.

E More analysis

Additional analysis compares Gen6D with a version of DeepIM fine-tuned separately on each test object. The comparison tests whether object-specific fine-tuning closes the performance gap.

  • DeepIM comparison: Fine-tuning DeepIM separately on each test object’s reference images significantly improves DeepIM, but it still underperforms Gen6D.Gen6D does not train on the test object in this comparison.
  • GenMOP analysis: Table 6 reports performance on GenMOP and distinguishes generalizable models from models that are not generalizable.The table also defines “DeepIM-Ft” as DeepIM fine-tuned separately on each object’s reference images.

E.2 Fewer reference images

Gen6D remains effective with fewer reference images, but performance depends on both the number and spatial coverage of those views. Refinement substantially improves results, while uncovered viewpoints create a practical limitation.

  • Reference image quantity: With 64 reference images, Gen6D produces results similar to using all approximately 200 images, while performance declines with 16 or 8 images.Farthest point sampling with 128 images can slightly improve performance by distributing views evenly.
  • Reference image distribution: Evenly distributed reference images benefit viewpoint selection and refinement using neighboring views.The even-distribution condition is compared with references restricted to the Y- space in Table 9.
  • Reference image distribution: When references cover only part of the viewpoint space, Gen6D cannot accurately predict poses from uncovered regions.This setting corresponds to reference images whose camera centers occupy the Y- half space.
  • Refinement iterations: One refinement iteration improves GenMOP performance from 17.90 to 38.59 on ADD-0.1d, with further gains from two iterations but not three.The table reports results for different numbers of refinement iterations.

E.5 Ablations on training data

Training data composition affects Gen6D performance, with real data helping substantially and LINEMOD data helping more than GenMOP data. The method also achieves reasonable qualitative and quantitative performance on unseen symmetric objects.

  • Training data: Training only on synthetic datasets suffers from a domain gap, whereas adding real data greatly improves Gen6D results.Table 11 compares synthetic, GenMOP, and LINEMOD training sets while keeping all test objects unseen during training.
  • Training data: LINEMOD training data brings more obvious improvements than GenMOP data because each LINEMOD object has approximately 1200 images versus approximately 200 GenMOP reference and query images.The comparison concerns training-data quantity as well as dataset composition.
  • Symmetric objects: For symmetric objects, matching query images to reference images allows the selector to handle multiple feasible poses caused by symmetry.The paper attributes this behavior to Gen6D’s reference-image matching formulation.
  • Qualitative evaluation: Figures 16–20 compare selected or nearest-view reference images and visualize predicted versus ground-truth poses across LINEMOD, MOPED, and GenMOP.Ground-truth poses are green and predictions are blue in the LINEMOD and MOPED qualitative figures.
Loading 2204.10776v2…