Source-linked AI summary
OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models
Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, Xiaowei Zhou
TL;DR
OnePose++ targets one-shot object pose estimation without CAD models or object-specific training, especially where keypoint-based reconstruction fails on low-textured objects. It reconstructs a semi-dense point-cloud model with keypoint-free SfM and directly matches query images to that model in 2D-3D space. The method substantially improves over CAD-model-free baselines and reaches performance comparable to CAD-model-based instance-level methods, while introducing a low-texture benchmark.
Problem
OnePose’s reliance on repeatable image keypoints makes one-shot pose estimation prone to failure on low-textured objects.
Method
OnePose++ uses keypoint-free coarse-to-fine SfM to reconstruct a semi-dense object point cloud, then directly establishes query-image 2D-3D correspondences for pose estimation.
Results
The method significantly outperforms existing CAD-model-free methods and achieves comparable pose-estimation results to CAD-model-based instance-level methods.
Takeaways & Limitations
OnePose++ supports one-shot CAD-model-free pose estimation for both texture-rich and low-textured objects and provides the OnePose-LowTexture dataset.
Takeaways & Limitations
The method depends on local feature matching, so it is limited by very low-resolution images, extreme scale or viewpoint changes, and the need for a separate rough-region object detector.
Abstract
from arXiv · showhide
We propose a new method for object pose estimation without CAD models. The previous feature-matching-based method OnePose has shown promising results under a one-shot setting which eliminates the need for CAD models or object-specific training. However, OnePose relies on detecting repeatable image keypoints and is thus prone to failure on low-textured objects. We propose a keypoint-free pose estimation pipeline to remove the need for repeatable keypoint detection. Built upon the detector-free feature matching method LoFTR, we devise a new keypoint-free SfM method to reconstruct a semi-dense point-cloud model for the object. Given a query image for object pose estimation, a 2D-3D matching network directly establishes 2D-3D correspondences between the query image and the reconstructed point-cloud model without first detecting keypoints in the image. Experiments show that the proposed pipeline outperforms existing one-shot CAD-model-free methods by a large margin and is comparable to CAD-model-based methods on LINEMOD even for low-textured objects. We also collect a new dataset composed of 80 sequences of 40 low-textured objects to facilitate future research on one-shot object pose estimation. The supplementary material, code and dataset are available on the project page: https://zju3dv.github.io/onepose_plus_plus/.
1 Introduction
One-shot pose estimation avoids CAD models and object-specific training, but OnePose’s keypoint-based reconstruction struggles with low-textured objects. OnePose++ addresses this with keypoint-free reconstruction and direct 2D-3D matching.
- Existing pose-estimation methods often require high-fidelity CAD models or separate training for each object category, limiting real-world applicability.
- OnePose removes CAD-model and category-specific training requirements, but its keypoint-based SfM struggles to reconstruct complete point clouds for low-textured objects.
- LoFTR provides strong low-texture correspondences, but its two-view-dependent matches create inconsistent keypoints and fragmentary tracks unsuitable for direct SfM.
- OnePose++ reconstructs accurate, complete semi-dense object structure through a two-stage coarse-to-fine pipeline and performs sparse-to-dense 2D-3D matching at test time.The reconstruction phase builds repeatable coarse tracks and refines their locations; the matching network establishes correspondences for pose estimation.
- The method outperforms existing one-shot methods by a large margin and is comparable to instance-level CAD-model-based methods on evaluated datasets.
- OnePose-LowTexture contributes 80 sequences covering 40 low-textured objects with ground-truth pose annotations.
2 Related work
Prior work spans CAD-model-based estimation, CAD-model-free approaches, feature matching, and visual localization. OnePose++ adapts detector-free matching to SfM and direct 2D-3D pose estimation while preserving accurate, repeatable features.
- CAD-Model-Based Object Pose Estimation: CAD-model-based methods are categorized as instance-level, category-level, or generalizable according to their ability to generalize across objects.
- CAD-Model-Free Object Pose Estimation: CAD-model-free methods remove known object models but may require object masks, neural representations, or other additional supervision and training.
- Feature Matching: Feature-matching pose methods traditionally detect query-image keypoints before matching, with low-texture and repetitive patterns producing ambiguous correspondences.
- Feature Matching: OnePose++ directly matches a 2D feature map with a 3D model and uses attention to disambiguate features without query-image keypoint detection.
- Structure from Motion and Visual Localization: The proposed system reconstructs a semi-dense object point cloud from posed reference images, then builds coarse-to-fine 2D-3D correspondences and solves pose with PnP.
- Structure from Motion and Visual Localization: Unlike approaches that trade point accuracy for repeatability, OnePose++ refines coarse tracks with two-view patches and geometric optimization.
3 Methods
The method reconstructs a semi-dense object point cloud without fixed keypoints, then estimates query-image pose through coarse-to-fine direct 2D-3D matching.
- Keypoint-Free SfM: LoFTR’s view-dependent matching locations prevent consistent multi-view feature tracks, making direct integration with keypoint-based SfM unsuitable.The proposed framework addresses this inconsistency by separating coarse repeatable matching from fine refinement.
- Keypoint-Free SfM: The pipeline first builds repeatable coarse semi-dense correspondences and uses them with COLMAP to obtain coarse feature tracks and a semi-dense point cloud.Coarse correspondences are pixel-wise dense on downsampled image pairs, providing completeness despite limited accuracy.
- Keypoint-Free SfM: Feature-track refinement applies fine-level matching around source nodes to obtain sub-pixel locations, producing refined multi-view tracks.One node in each coarse track is fixed while the remaining nodes are refined locally.
- Keypoint-Free SfM: The refined tracks are used to optimize the point cloud with reprojection errors, while reducing each point’s optimization to the depth of its reference node.Relative poses and camera projection map reference-node depths to reprojected source locations; optimized points are transformed into canonical object coordinates.
- Object Pose Estimation: At test time, hierarchical query features are matched directly against stored multi-scale 3D point features in a coarse-to-fine network.Coarse matching uses positional encodings, self- and cross-attention, dual-softmax probabilities, and mutual-nearest-neighbor thresholding before fine matching and PnP-RANSAC.
4 Experiments
Experiments evaluate the method on textured and low-textured object datasets, compare it with one-shot and instance-level baselines, and ablate its reconstruction and matching components. The method substantially improves one-shot performance, remains competitive with CAD-model-based methods, and benefits from point-cloud refinement and attention.
- Datasets: The evaluation uses OnePose, LINEMOD, and the newly collected OnePose-LowTexture dataset, which contains 40 low-textured household objects.OnePose contains around 450 real-world video sequences of 150 objects; OnePose-LowTexture provides two videos per object with different backgrounds.
- Evaluation Protocols: The method is compared with one-shot baselines under matched reference-image protocols using cm-degree success rate, ADD(S)-0.1d, and Proj2D metrics.Baselines include OnePose, HLoc variants, and Gen6D; instance-level comparisons include PVNet and CDPN.
- One-shot Baselines: On OnePose-LowTexture, the method surpasses OnePose and HLoc by a large margin, while also outperforming HLoc with LoFTR and running approximately 10× faster.The reported gains are attributed to accurate point-cloud reconstruction and efficient 2D-3D matching.
- Instance-level Baseline: PVNet is outperformed on most scanned low-textured objects despite training on approximately 5000 rendered images per object.The authors attribute this result to PVNet’s domain-gap susceptibility and the matching module’s robustness and generalizability.
- LINEMOD Results: On LINEMOD, the method significantly outperforms OnePose and Gen6D without additional training, while achieving comparable performance with instance-level methods.Gen6D relies on accurate bounding boxes and may lose performance without LINEMOD fine-tuning; the proposed method needs only rough object detection.
- Ablation Studies: Point-cloud refinement improves the strict 1cm-1deg pose-success metric by 7%, while the attention module improves the 5cm-5deg metric by 15%.The refinement increases reconstructed point-cloud precision, and attention makes 2D and 3D features more discriminative for correspondence estimation.
5 Conclusion
The proposed keypoint-free pipeline supports one-shot, CAD-model-free pose estimation for texture-rich and low-textured objects, while remaining subject to resolution, viewpoint, scale, and detection constraints.
- The pipeline enables one-shot CAD-model-free pose estimation for both texture-rich and low-textured objects.
- It reconstructs accurate, complete 3D structures and builds robust 2D-3D correspondences for pose estimation.
- The method achieves significantly better pose estimation accuracy than existing CAD-model-free methods and comparable results with CAD-model-based instance-level methods.
- Dependence on local feature matching limits performance for very low-resolution images and extreme scale or viewpoint changes.
- The current pipeline still requires a separate object detector to provide rough regions of interest.
Checklist
The checklist reports that the paper documents its claims, limitations, societal-impact discussion, assets, and experimental procedures, but does not provide error bars and does not report asset licenses.
- The checklist confirms that the paper describes its contributions and scope, limitations, and potential negative societal impacts.
- The paper reports that code, data, and reproduction instructions are available or will be published, including the OnePose-LowTexture dataset.
- Training details and compute-resource information are reported in the paper.
- The experiments do not provide error bars because of resource constraints.
- The checklist reports that licenses for existing assets were not mentioned.