Source-linked AI summary
OnePose: One-Shot Object Pose Estimation without CAD Models
Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, Xiaowei Zhou
TL;DR
Object pose estimation for augmented reality often assumes accessible CAD models, limiting one-shot estimation for arbitrary everyday objects. OnePose instead builds an SfM model from an RGB video scan and directly matches query-image 2D points to 3D points, achieving better pose accuracy and faster inference than baselines while tracking poses in real time.
Problem
Object pose estimation for augmented reality commonly assumes known CAD models, while one-shot estimation seeks 6D poses for arbitrary-category objects from limited annotated imagery.
Method
OnePose builds a sparse SfM object model from a video scan and uses graph attention networks for direct 2D–3D matching during pose estimation.
Results
OnePose achieves better pose estimation accuracy and faster inference than localization-based, instance-level, and category-level baselines, processing one frame in 58 ms on GPU.
Takeaways & Limitations
OnePose provides a more practical localization-based setting for augmented reality without CAD models or instance- or category-specific network training.
Takeaways & Limitations
OnePose may fail on textureless objects and has difficulty handling extreme scale changes between scanned and testing images.
Abstract
from arXiv · showhide
We propose a new method named OnePose for object pose estimation. Unlike existing instance-level or category-level methods, OnePose does not rely on CAD models and can handle objects in arbitrary categories without instance- or category-specific network training. OnePose draws the idea from visual localization and only requires a simple RGB video scan of the object to build a sparse SfM model of the object. Then, this model is registered to new query images with a generic feature matching network. To mitigate the slow runtime of existing visual localization methods, we propose a new graph attention network that directly matches 2D interest points in the query image with the 3D points in the SfM model, resulting in efficient and robust pose estimation. Combined with a feature-based pose tracker, OnePose is able to stably detect and track 6D poses of everyday household objects in real-time. We also collected a large-scale dataset that consists of 450 sequences of 150 objects.
1. Introduction
OnePose reframes object pose estimation as one-shot localization from a sparse SfM model, avoiding CAD models and instance- or category-specific network training. Its graph-attention 2D–3D matcher enables efficient pose estimation, and evaluation shows higher precision than PVNet and Objectron at 58 ms per GPU frame.
- Motivation: One-shot pose estimation targets arbitrary-category objects from only a few pose-annotated images, unlike instance- and category-level methods requiring CAD models or category-specific training.Category-level methods require many same-category training samples and may generalize poorly to substantially different appearances or shapes.
- Problem setting: OnePose estimates novel-object 6D poses from a simple video scan and sparse SfM model, without CAD models or additional network training.The setting reconstructs a sparse point cloud from the object video and solves pose from query-image 2D features matched to SfM points.
- Method: The method directly matches query-image 2D features with SfM 3D points using graph attention networks that aggregate feature tracks into 3D features and apply self- and cross-attention.This design captures global context while producing 2D–3D correspondences for pose estimation.
- Evaluation: OnePose achieves better validation precision than PVNet and Objectron [4] without training for object instances or categories, processing each GPU frame in 58 ms.The evaluation uses a dataset containing 450 sequences of 150 objects.
- Contributions: The paper contributes a renovated visual-localization pipeline, a graph-attention architecture for robust 2D–3D matching, and a pose-annotated one-shot object dataset.The dataset contains 450 sequences of 150 objects.
2. Related works
Prior object pose estimation methods use CAD-model-based regression or correspondence techniques, category-level shape priors, or CAD-model-free analysis-by-synthesis and correspondence learning. Feature-matching pipelines instead recover poses from 2D–3D correspondences with SfM models, while depth-dependent 2D–2D matching can limit augmented-reality use.
- CAD-Model-Based Object Pose Estimation: CAD-model-based methods either directly regress 6DoF pose parameters from RoI features or establish image-to-3D-coordinate correspondences through regression or voting.
- CAD-Model-Based Object Pose Estimation: NOCS [43] avoids CAD models at test time by learning category-shared normalized object coordinates, with subsequent works refining category-level shape recovery.
- CAD-Model-Free Object Pose Estimation: CAD-model-free methods use analysis-by-synthesis or learned pixel-to-object-coordinate correspondences, but synthesis-based approaches are limited in efficiency and accuracy for AR applications.Neural Object Fitting [8] uses a VAE trained on synthetic data, whereas LatentFusion [23] builds a 3D latent representation from posed RGB-D images of unseen objects; RLLG learns correspondences without CAD models.
- Feature-Matching-Based Pose Estimation: Feature-matching visual localization traditionally finds 2D–3D correspondences between input RGB images and SfM models using hand-crafted features, while learned detection, description, and matching methods have surpassed them.
- Feature-Matching-Based Pose Estimation: A contrasting approach uses 2D–2D matching and requires depth maps to recover 3D information, which can limit its suitability for AR applications.
3. Method
OnePose builds an object-specific sparse SfM representation from a static RGB video scan and estimates query-image poses through direct 2D–3D matching. A graph-attention architecture adaptively aggregates descriptors before matching, enabling fast localization without CAD models or additional network training.
- Data Capture and Annotation: OnePose captures a static object video scan, annotates its bounding box and camera poses, and reconstructs a sparse object point cloud with 2D–3D correspondence graphs.The object is assumed to remain static on a flat surface; ARKit or ARCore can provide camera tracking during capture.
- Pose Estimation through Visual Localization: Direct 2D–3D matching bypasses image retrieval and avoids the runtime–accuracy tradeoff of matching query images against retrieved SfM database images.The method is designed to maintain localization accuracy while being fast, unlike pipelines using image retrieval followed by learning-based matchers such as SuperGlue [32] or LoFTR [36].
- Graph Attention Networks (GATs) for 2D–3D Matching: The aggregation-attention layer adaptively weights multiple 2D descriptors associated with each 3D point, preserving informative features that averaging could lose.Attention coefficients are computed from descriptor similarity, and the resulting 3D descriptors are used for query-image matching.
- Graph Attention Networks (GATs) for 2D–3D Matching: Stacked attention groups interleave aggregation, self-attention, and cross-attention so 2D and 3D descriptors exchange context for globally consistent matching.The proposed GAT architecture contains N stacked attention groups and uses self- and cross-attention layers inspired by [32] [36].
- Match Selection and Pose Calculation: OnePose extracts match confidences with dual-softmax and computes the object pose by solving a PnP problem from the predicted 2D–3D matches.The query descriptors and aggregated 3D descriptors produce match predictions M3D used for pose calculation.
- Dataset: The OnePose dataset contains over 450 video sequences of 150 objects, with multiple recordings, camera poses, and 3D bounding-box annotations across background environments.The dataset was collected because existing large-scale datasets did not fit the one-shot pose-estimation setting.
4. Experiments
Experiments evaluate OnePose against visual-localization, instance-level, and category-level baselines under CAD-free metrics and shared mapping protocols. OnePose matches or slightly exceeds the strongest visual-localization baseline while running about 10× faster, and shows stronger category-level generalization than Objectron.
- Evaluation Protocols: Evaluation uses per-frame pose estimation without tracking, with the same RGB video scan building the SfM map for OnePose and visual-localization baselines.Because object image retrieval does not generalize well, five equally spaced database images are used for feature matching; PVNet receives 3D box corners and auxiliary mask supervision, while Objectron uses its authors’ pretrained models.
- Metrics: CAD-free evaluation replaces ADD and 2D projection with 5cm-5deg, 3cm-3deg, and stricter 1cm-1deg pose criteria.A prediction is correct under 5cm-5deg when its error is below 5cm and 5°; the stricter criteria follow the same definition.
- Comparison with Visual Localization Baselines: OnePose performs on par or slightly better than HLoc (SPP + SPG) while running approximately 10× faster.Table 1 compares visual-localization variants; the runtime excludes 2D detection and PnP and measures query feature extraction plus 2D-3D matching.
- Comparison with the Category-level Baseline Objectron: On all Shoe and Cup objects, OnePose evidently outperforms Objectron in average precision for azimuth and elevation errors, especially for Cups with greater inter-instance variation.The results use the original paper’s metrics, with Objectron evaluated using auxiliary scale adjustment and center alignment; annotation differences explain its lower 2D-projection performance on this dataset.
- Effectiveness of the Aggregation-Attention Layer: Replacing GATs’ aggregation-attention layers with averaging causes significant performance drops for large and medium objects.The matching module retains self- and cross-attention layers, but averaging cannot adaptively select viewpoint-relevant information for different query features.
5. Conclusion
OnePose enables one-shot pose estimation for arbitrary-category objects without CAD models or instance- or category-specific training, achieving higher accuracy and faster inference than several baselines. Its reliance on local feature matching limits performance on textureless objects and under extreme scale changes.
- Conclusion: OnePose achieves better pose estimation accuracy and faster inference than localization-based methods, PVNet, and Objectron.The comparison covers localization-based baseline methods, the instance-level baseline PVNet, and the category-level baseline Objectron.
- Conclusion: OnePose handles arbitrary-category objects without CAD models or instance- or category-specific network training.The method is proposed for one-shot object pose estimation.
- Limitations: Local feature matching can cause OnePose to fail on textureless objects and struggle with extreme scale changes between scans and test sequences.Attention mechanisms enhance the method but do not eliminate its difficulty with extreme scale differences.