PaperlayerView exact revision on arXiv

Source-linked AI summary

Any6D: Model-free 6D Pose Estimation of Novel Objects

Taeyeop Lee, Bowen Wen, Minjun Kang, Gyuree Kang, In So Kweon, Kuk-Jin Yoon

arXiv:2503.18673v2cs.CVcs.AIcs.RO

TL;DR

Any6D addresses the difficulty of estimating poses for unexpected objects without available 3D models or multiview images. It uses single-image object alignment to improve 2D-3D alignment and size estimation, and reports superior performance across five benchmark datasets, while remaining limited by inaccurate initial 3D shapes.

  • Problem

    Existing pose-estimation methods face unexpected objects without available 3D models or multiview images, limiting their effectiveness in real-world applications.

    Source: §1 · p. 2

  • Method

    Any6DPose estimates novel-object pose and size from a single anchor RGB-D image by jointly aligning object geometry and metric scale in 2D and 3D.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Any6D demonstrates superior performance to state-of-the-art methods across five benchmark datasets for novel-object pose estimation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports robust pose estimation when objects are occluded or viewed from substantially different viewpoints, including robotic manipulation scenarios.

    Source: §4.4 · p. 7

    Any6D remains limited when the initial 3D shape is inaccurate because it does not update the shape.

    Source: §5 · p. 8

Abstract

from arXiv · show

We introduce Any6D, a model-free framework for 6D object pose estimation that requires only a single RGB-D anchor image to estimate both the 6D pose and size of unknown objects in novel scenes. Unlike existing methods that rely on textured 3D models or multiple viewpoints, Any6D leverages a joint object alignment process to enhance 2D-3D alignment and metric scale estimation for improved pose accuracy. Our approach integrates a render-and-compare strategy to generate and refine pose hypotheses, enabling robust performance in scenarios with occlusions, non-overlapping views, diverse lighting conditions, and large cross-environment variations. We evaluate our method on five challenging datasets: REAL275, Toyota-Light, HO3D, YCBINEOAT, and LM-O, demonstrating its effectiveness in significantly outperforming state-of-the-art methods for novel object pose estimation. Project page: https://taeyeop.com/any6d

1. Introduction

6D pose estimation supports applications such as robotic manipulation and augmented reality, but existing approaches depend on object-specific models, categories, or multiple reference views. Any6D addresses these constraints with single-anchor model-free estimation and joint alignment, showing strong performance across five benchmarks.

Motivation

  • 6D pose estimation determines an object’s 3D orientation and translation between reference and camera coordinates.The task is relevant to robotic manipulation and augmented reality.Source: §1 · p. 1

Limitations of Existing Methods

  • Instance-level methods require exact RGB-textured CAD models and generally cannot handle new objects without fine-tuning.Source: §1 · p. 1
  • Category-level methods use category priors but remain restricted to predefined categories and require difficult training data.Source: §1 · p. 1

Category-Agnostic Methods

  • Category-agnostic methods reduce category restrictions, yet model-based variants need textured CAD models and model-free variants need multiview images or video.Source: §1 · pp. 1–2

Any6D

  • Any6D estimates novel-object pose and size from one anchor RGB-D image, jointly improving 2D-3D alignment and metric scale before render-and-compare pose selection.The framework targets occlusions, disjoint views, diverse lighting, and cross-environment variation, and is evaluated on five public datasets.Source: §1 · p. 2

2. Related Works

Related work progresses from CAD- and category-dependent estimation toward category-agnostic and model-free methods. However, existing model-free systems commonly still require multiview references, while Any6D reconstructs and aligns an object from a single anchor image.

Category-Agnostic Estimation

  • Category-agnostic methods estimate poses for arbitrary novel objects without restricting predictions to predefined categories.Source: §2.1 · p. 2

Multiview Model-Free Methods

  • Gen6D, OnePose, and OnePose++ reconstruct 3D point clouds from videos or multiple views for 2D-3D matching.Source: §2.2 · p. 2

Single-Image Methods

  • NOPE, LoFTR, and Oryon reduce reference requirements through single-anchor orientation estimation, transformer matching, or language guidance.Source: §2.2 · p. 2

Any6D

  • Any6D reconstructs normalized object shape from an anchor image, estimates metric-scale shape and alignment, then uses the query image for pose estimation.Source: Figure 2 · p. 3

3. Method

Any6D reconstructs a normalized shape from an RGB-D anchor, jointly estimates object size and pose, and derives anchor-to-query pose using metric-scale alignment. Its pipeline combines coarse-to-fine alignment, axis alignment, pose hypotheses, and render-and-compare selection.

Problem Formulation

  • The task is to estimate the relative 6D transformation between RGB-D anchor and query images, comprising rotation and translation.Source: §3 · p. 3

Object Reconstruction

  • Partial-to-partial matching fails with occlusions or large viewpoint changes, so Any6D reconstructs a complete shape for full-to-partial alignment.Source: §3 · p. 3

Framework

  • The framework reconstructs normalized shape ON, estimates object size s and anchor pose, then combines object-to-anchor and object-to-query transformations.Source: §3 · p. 3

Coarse Object Alignment

  • Because single-view reconstruction produces shape in the normalized range [-1, 1] without scene scale or position, Any6D performs coarse size estimation followed by joint 2D-3D alignment.Source: §3.1 · p. 3
  • An oriented bounding box provides a more reliable center under partial visibility, after which rotated bounding-box IoU guides axis alignment.Source: §3.1 · pp. 3–4

Fine Object Alignment

  • Pose, size, and axis alignment are refined together by alternating size and pose refinement while sampling size variations in addition to pose hypotheses.Source: §3.2 · p. 4

Relative Pose

  • The relative pose is computed as TA→Q = (T_OM→A)^-1 · T_OM→Q by composing object-to-anchor and object-to-query transformations.Source: §3.2 · p. 4

Pose Selection

  • A two-level render-and-compare strategy ranks rendered hypotheses against the cropped observation and uses self-attention to select the optimal pose.Source: §3.2 · p. 4

4. Experiments

Any6D is evaluated across five challenging real-world datasets using complementary pose-accuracy metrics and qualitative scenarios involving occlusion, viewpoint changes, lighting variation, and robot interaction. It consistently outperforms prior methods, with especially strong results on HO3D, YCBInEOAT, Toyota-Light, and REAL275, while ablations support its alignment design.

Datasets

  • The evaluation covers HO3D, YCBInEOAT, Toyota-Light, REAL275, and LM-O, each representing distinct interaction, viewpoint, occlusion, or environmental challenges.HO3D contains human-hand interactions, while YCBInEOAT contains dual-arm robot manipulation and diverse viewpoints.Source: §4.1 · pp. 4–5, §4.3 · p. 5

Evaluation metrics

  • The study reports ADD, ADD-S, and BOP metrics including AR, MSSD, MSPD, and VSD to provide complementary measures of pose accuracy.ADD is used for asymmetric objects and ADD-S for symmetric objects; BOP metrics evaluate recall over multiple thresholds.Source: §4.2 · p. 5

Quantitative comparison

  • 98.7%, 40.4%, and 38.3% are Any6D’s HO3D mean ADD-S, ADD, and AR scores, exceeding Oryon’s 4.1%, 0%, and 0.2%.The method maintains consistent performance under human-hand interactions and occlusions.Source: §4.3 · pp. 5–6
  • 89.3, 45.6, and 37.5 are Any6D’s YCBInEOAT mean ADD-S, ADD, and AR scores, compared with Gedi’s 82.7, 7.7, and 14.0.The reported gains include challenging occlusions and non-overlapping viewpoints.Source: §4.3 · p. 6
  • On Toyota-Light, Any6D reaches 32.2% ADD(-S), 43.3% AR, 55.8% MSSD, and 58.4% MSPD, exceeding Oryon by 9.3%, 9.2%, 12.9%, and 12.9%.The results are reported as robust under varying lighting conditions.Source: §4.3 · p. 6
  • On REAL275, Any6D achieves 53.5% ADD(-S), 51.0% AR, 56.5% MSSD, and 65.3% MSPD, with improvements over Oryon of 18.6%, 4.5%, 27.1%, and 33.8%.VSD remains competitive at 31.1% versus Oryon’s 32.1%.Source: §4.3 · p. 6

Qualitative results

  • Qualitative results show successful pose estimation when anchor and query views differ substantially, objects are occluded, or robot grippers obscure objects.Examples include a cleanser, SPAM can, pitcher, cracker box, mustard bottle, and sugar box.Source: §4.4 · p. 7

Ablation studies

  • The full ablation configuration combining coarse size estimation, refinement, and axis alignment achieves the best results, including ADD 40.4 and AR 38.3.Axis alignment improves object-shape estimation and pose accuracy by reducing proportional distortion along the x, y, and z directions.Source: §4.5 · p. 8

5. Conclusion

Any6D reduces dependence on CAD models and multi-view images through model-free pose estimation, while improving pose and size estimation through image-to-3D alignment. Its effectiveness is demonstrated under occlusions and varying viewpoints, though inaccurate initial 3D shapes remain a limitation.

  • Any6D is a model-free framework that reduces dependence on CAD models and multi-view images for object pose estimation.Source: §5 · p. 8
  • The method uses image-to-3D alignment to estimate object pose and size, particularly in challenging manipulation scenarios.Source: §5 · p. 8
  • Any6D significantly outperforms state-of-the-art methods under occlusions and varying viewpoints.Source: §5 · p. 8
  • Inaccurate initial 3D shapes limit the method because it does not incorporate shape updating.The authors identify shape refinement as a future direction for improving robustness and applicability.Source: §5 · p. 8

Related papers

Loading 2503.18673v2…