Source-linked AI summary
Any6D: Model-free 6D Pose Estimation of Novel Objects
Taeyeop Lee, Bowen Wen, Minjun Kang, Gyuree Kang, In So Kweon, Kuk-Jin Yoon
TL;DR
Any6D addresses the difficulty of estimating poses for unexpected objects without available 3D models or multiview images. It uses single-image object alignment to improve 2D-3D alignment and size estimation, and reports superior performance across five benchmark datasets, while remaining limited by inaccurate initial 3D shapes.
Problem
Existing pose-estimation methods face unexpected objects without available 3D models or multiview images, limiting their effectiveness in real-world applications.
Method
Any6DPose estimates novel-object pose and size from a single anchor RGB-D image by jointly aligning object geometry and metric scale in 2D and 3D.
Results
Any6D demonstrates superior performance to state-of-the-art methods across five benchmark datasets for novel-object pose estimation.
Takeaways & Limitations
The framework supports robust pose estimation when objects are occluded or viewed from substantially different viewpoints, including robotic manipulation scenarios.
Takeaways & Limitations
Any6D remains limited when the initial 3D shape is inaccurate because it does not update the shape.
Abstract
from arXiv · showhide
We introduce Any6D, a model-free framework for 6D object pose estimation that requires only a single RGB-D anchor image to estimate both the 6D pose and size of unknown objects in novel scenes. Unlike existing methods that rely on textured 3D models or multiple viewpoints, Any6D leverages a joint object alignment process to enhance 2D-3D alignment and metric scale estimation for improved pose accuracy. Our approach integrates a render-and-compare strategy to generate and refine pose hypotheses, enabling robust performance in scenarios with occlusions, non-overlapping views, diverse lighting conditions, and large cross-environment variations. We evaluate our method on five challenging datasets: REAL275, Toyota-Light, HO3D, YCBINEOAT, and LM-O, demonstrating its effectiveness in significantly outperforming state-of-the-art methods for novel object pose estimation. Project page: https://taeyeop.com/any6d
1. Introduction
6D pose estimation supports applications such as robotic manipulation and augmented reality, but existing approaches depend on object-specific models, categories, or multiple reference views. Any6D addresses these constraints with single-anchor model-free estimation and joint alignment, showing strong performance across five benchmarks.
- Motivation: 6D pose estimation determines an object’s 3D orientation and translation between reference and camera coordinates.The task is relevant to robotic manipulation and augmented reality.
- Limitations of Existing Methods: Instance-level methods require exact RGB-textured CAD models and generally cannot handle new objects without fine-tuning.
- Limitations of Existing Methods: Category-level methods use category priors but remain restricted to predefined categories and require difficult training data.
- Category-Agnostic Methods: Category-agnostic methods reduce category restrictions, yet model-based variants need textured CAD models and model-free variants need multiview images or video.
- Any6D: Any6D estimates novel-object pose and size from one anchor RGB-D image, jointly improving 2D-3D alignment and metric scale before render-and-compare pose selection.The framework targets occlusions, disjoint views, diverse lighting, and cross-environment variation, and is evaluated on five public datasets.
2. Related Works
Related work progresses from CAD- and category-dependent estimation toward category-agnostic and model-free methods. However, existing model-free systems commonly still require multiview references, while Any6D reconstructs and aligns an object from a single anchor image.
- Category-Agnostic Estimation: Category-agnostic methods estimate poses for arbitrary novel objects without restricting predictions to predefined categories.
- Multiview Model-Free Methods: Gen6D, OnePose, and OnePose++ reconstruct 3D point clouds from videos or multiple views for 2D-3D matching.
- Single-Image Methods: NOPE, LoFTR, and Oryon reduce reference requirements through single-anchor orientation estimation, transformer matching, or language guidance.
- Any6D: Any6D reconstructs normalized object shape from an anchor image, estimates metric-scale shape and alignment, then uses the query image for pose estimation.
3. Method
Any6D reconstructs a normalized shape from an RGB-D anchor, jointly estimates object size and pose, and derives anchor-to-query pose using metric-scale alignment. Its pipeline combines coarse-to-fine alignment, axis alignment, pose hypotheses, and render-and-compare selection.
- Problem Formulation: The task is to estimate the relative 6D transformation between RGB-D anchor and query images, comprising rotation and translation.
- Object Reconstruction: Partial-to-partial matching fails with occlusions or large viewpoint changes, so Any6D reconstructs a complete shape for full-to-partial alignment.
- Framework: The framework reconstructs normalized shape ON, estimates object size s and anchor pose, then combines object-to-anchor and object-to-query transformations.
- Coarse Object Alignment: Because single-view reconstruction produces shape in the normalized range [-1, 1] without scene scale or position, Any6D performs coarse size estimation followed by joint 2D-3D alignment.
- Coarse Object Alignment: An oriented bounding box provides a more reliable center under partial visibility, after which rotated bounding-box IoU guides axis alignment.
- Fine Object Alignment: Pose, size, and axis alignment are refined together by alternating size and pose refinement while sampling size variations in addition to pose hypotheses.
- Relative Pose: The relative pose is computed as TA→Q = (T_OM→A)^-1 · T_OM→Q by composing object-to-anchor and object-to-query transformations.
- Pose Selection: A two-level render-and-compare strategy ranks rendered hypotheses against the cropped observation and uses self-attention to select the optimal pose.
4. Experiments
Any6D is evaluated across five challenging real-world datasets using complementary pose-accuracy metrics and qualitative scenarios involving occlusion, viewpoint changes, lighting variation, and robot interaction. It consistently outperforms prior methods, with especially strong results on HO3D, YCBInEOAT, Toyota-Light, and REAL275, while ablations support its alignment design.
- Datasets: The evaluation covers HO3D, YCBInEOAT, Toyota-Light, REAL275, and LM-O, each representing distinct interaction, viewpoint, occlusion, or environmental challenges.HO3D contains human-hand interactions, while YCBInEOAT contains dual-arm robot manipulation and diverse viewpoints.
- Evaluation metrics: The study reports ADD, ADD-S, and BOP metrics including AR, MSSD, MSPD, and VSD to provide complementary measures of pose accuracy.ADD is used for asymmetric objects and ADD-S for symmetric objects; BOP metrics evaluate recall over multiple thresholds.
- Quantitative comparison: 98.7%, 40.4%, and 38.3% are Any6D’s HO3D mean ADD-S, ADD, and AR scores, exceeding Oryon’s 4.1%, 0%, and 0.2%.The method maintains consistent performance under human-hand interactions and occlusions.
- Quantitative comparison: 89.3, 45.6, and 37.5 are Any6D’s YCBInEOAT mean ADD-S, ADD, and AR scores, compared with Gedi’s 82.7, 7.7, and 14.0.The reported gains include challenging occlusions and non-overlapping viewpoints.
- Quantitative comparison: On Toyota-Light, Any6D reaches 32.2% ADD(-S), 43.3% AR, 55.8% MSSD, and 58.4% MSPD, exceeding Oryon by 9.3%, 9.2%, 12.9%, and 12.9%.The results are reported as robust under varying lighting conditions.
- Quantitative comparison: On REAL275, Any6D achieves 53.5% ADD(-S), 51.0% AR, 56.5% MSSD, and 65.3% MSPD, with improvements over Oryon of 18.6%, 4.5%, 27.1%, and 33.8%.VSD remains competitive at 31.1% versus Oryon’s 32.1%.
- Qualitative results: Qualitative results show successful pose estimation when anchor and query views differ substantially, objects are occluded, or robot grippers obscure objects.Examples include a cleanser, SPAM can, pitcher, cracker box, mustard bottle, and sugar box.
- Ablation studies: The full ablation configuration combining coarse size estimation, refinement, and axis alignment achieves the best results, including ADD 40.4 and AR 38.3.Axis alignment improves object-shape estimation and pose accuracy by reducing proportional distortion along the x, y, and z directions.
5. Conclusion
Any6D reduces dependence on CAD models and multi-view images through model-free pose estimation, while improving pose and size estimation through image-to-3D alignment. Its effectiveness is demonstrated under occlusions and varying viewpoints, though inaccurate initial 3D shapes remain a limitation.
- Any6D is a model-free framework that reduces dependence on CAD models and multi-view images for object pose estimation.
- The method uses image-to-3D alignment to estimate object pose and size, particularly in challenging manipulation scenarios.
- Any6D significantly outperforms state-of-the-art methods under occlusions and varying viewpoints.
- Inaccurate initial 3D shapes limit the method because it does not incorporate shape updating.The authors identify shape refinement as a future direction for improving robustness and applicability.