PaperlayerView exact revision on arXiv

Source-linked AI summary

Prior-free relative 6D pose estimation of multiple object instances

Behdad Khodabandehloo, Andrea Caraffa, Davide Boscaini, Fabio Poiesi

arXiv:2609.08949v1cs.CV

TL;DR

Pose estimation for previously unseen objects still typically assumes object-specific prior information, while prior-free relative pose estimation uses multiple co-visible instances as mutual references. PROSE combines foundation features and cycle-consistent correspondence refinement, and outperforms adapted single-reference baselines across three datasets.

  • Problem

    Many pose-estimation methods for unseen objects still assume object-specific prior information, while pose estimation without an external object reference remains an open formulation.

    Source: §1 · pp. 1–2

  • Method

    PROSE fuses appearance and geometry features, refines pairwise correspondences through cycle consistency across instances, and estimates relative poses by 3D registration.

    Source: §3.3 · p. 3, §3.5 · p. 4

  • Results

    PROSE consistently outperforms adapted baselines across IC-BIN, IC-MI, and XYZ-IBD, with average-recall gains of +19.2, +19.5, and +19.7, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The setting removes the need to acquire external object references before deployment, with potential application to robotic manipulation.

    Source: §6 · p. 8

    PROSE assumes instance masks and a predefined anchor are available.

    Source: §6 · p. 8

Abstract

from arXiv · show

Object 6D pose estimation formulations have progressively reduced reliance on object-specific priors, evolving from explicit 3D models to multi-view object captures to single reference images. We take this progression to its extreme by introducing prior-free relative 6D pose estimation, which lifts the assumption of knowing which object is to be posed within the scene. This novel setting aims to estimate the relative poses of multiple instances of an unknown object within the same image, without requiring CAD models, templates, or reference images. We solve this by formulating a novel method (PROSE) that finds coarse correspondences between object instances using multimodal foundation features, thus requiring no training. We refine these correspondences by imposing cycle consistency across tuples of instances, and leverage the resulting globally consistent correspondences to estimate the relative 6D pose between any pair of instances. To enable systematic evaluation, we design a novel benchmark (PRENCH) built from three multi-instance BOP datasets and enriched with task-specific metadata. PROSE consistently outperforms baselines obtained by adapting state-of-the-art single-image methods to the proposed setting, while requiring neither task-specific supervision nor additional learned components. Project website: https://tev-fbk.github.io/PROSE/

1. Introduction

Prior-free relative 6D pose estimation uses co-visible instances of the same unknown object as mutual references, requiring neither a CAD model nor an external reference image. PROSE estimates their relative poses, and PRENCH enables evaluation against adapted single-reference methods.

1. Introduction

  • Earlier pose-estimation approaches reduced required priors from 3D models to multiple views and then a single image, but all still require object-specific information before estimation.That requirement can add onboarding effort for robots encountering many previously unseen objects in small quantities.Source: §1 · p. 1
  • The task estimates relative poses among multiple instances of the same unknown rigid object in one RGBD observation, without a CAD model or additional reference image.A visible instance can serve as the anchor for the others.Source: §1 · p. 2
  • PROSE is training-free: it combines semantic and geometric foundation features, then refines pairwise correspondences through cross-instance cycle consistency before robust 3D registration.The benchmark PRENCH is built from multi-instance BOP datasets, and the paper adapts single-reference methods as baselines.Source: §1 · p. 2
  • PROSE beats adapted baselines across all three datasets, gaining +19.2, +19.5, and +19.7 average recall on IC-BIN, IC-MI, and XYZ-IBD, respectively.Compared with the best baseline per dataset, it also reduces rotation errors by 9.2–36.0 degrees and translation errors by 7.7–15.3 mm.Source: §1 · p. 2

2. Related works

Prior work ranges from CAD-based pose estimation to multi-view and single-reference methods, progressively reducing—but not eliminating—object-specific reference requirements. PROSE instead uses co-visible instances as mutual references within one RGBD observation.

2. Related works

  • CAD-based methods still require object-specific models, multi-view methods require reference views, and single-view methods require an external reference image.The cited approaches span feature matching, template matching, and learned or training-free methods.Source: §2 · p. 2
  • PROSE’s pipeline fuses semantic and geometric features, refines correspondences with cycle consistency across instance tuples, and uses RANSAC-based registration to estimate relative poses.The illustrated overview begins with RGBD input and instance segmentation.Source: Figure 2 · p. 3
  • Single-reference methods estimate a query pose relative to a reference image, whereas PROSE selects its anchor from instances already present in the query image.Examples discussed include POPE, H-/Oryon, NOPE, One2Any, and ConceptPose.Source: §2 · p. 3

3. PROSE

PROSE estimates relative poses among multiple instances of the same unknown rigid object in one RGBD image, without a model or reference image. It extracts multimodal features, refines pairwise correspondences through cross-instance cycle consistency, then registers the instances to estimate relative poses.

3.1. Overview

  • PROSE extracts point-level features with frozen foundation encoders, synchronizes correspondences across instances for cycle consistency, then estimates pairwise poses by RANSAC-based 3D registration.The pipeline comprises feature extraction, correspondence refinement, and relative pose estimation.Source: §3.1 · p. 3

3.2. Problem formulation

  • The task is to estimate a relative rigid transformation between every pair of same-object instances in one RGBD image, without a 3D model or reference image.The sensor intrinsics are assumed known.Source: §3.2 · p. 3

3.3. Multimodal feature extraction

  • For each instance, PROSE concatenates appearance features from its image crop with geometry features from its point cloud into a multimodal representation.The concatenated features are row-wise L2-normalized.Source: §3.3 · pp. 3–4

3.4. Coarse correspondence estimation

  • PROSE computes cosine similarities between normalized instance features, then converts each non-empty similarity-map row into a one-hot match at its maximum.This produces initial, coarse pairwise correspondences.Source: §3.4 · p. 4

3.5. Cross-instance correspondence refinement

  • Cycle consistency compares a direct match with the match composed through another instance; their deviation, the closure gap, identifies correspondence outliers.Jointly minimizing closure gaps across overlapping instance cycles provides redundancy that suppresses isolated matching errors and yields globally consistent correspondences.Source: §3.5 · p. 4, Figure 3 · p. 4
  • PROSE synchronizes pairwise maps with a sparse, low-rank correspondence matrix: feature similarity rewards matches, sparsity favors partial assignments, and low rank enforces cycle consistency.Cycle-consistent maps can be represented through shared latent object points; the method uses a nuclear-norm relaxation of the rank constraint.Source: §3.5 · p. 4
  • To make synchronization tractable, PROSE subsamples each instance to 400 points and keeps the similarity matrix sparse using nearest-neighbor matches.The full-resolution matrix would have tens of thousands of points per dimension.Source: §3.5 · p. 5

3.6. Relative 6D pose estimation

  • For each pair, PROSE estimates the relative 6D pose with RANSAC-based 3D registration, retaining the transformation with the most inlier correspondences.Each iteration samples three correspondences; residual outliers in the refined matches are handled through inlier scoring.Source: §3.6 · p. 5

4. PRENCH

PRENCH adapts three multi-instance BOP datasets for evaluating prior-free relative 6D pose estimation. Its metadata and varied objects, sensors, and difficulty conditions support analysis across task-relevant dimensions.

4. PRENCH

  • PRENCH combines IC-BIN, IC-MI, and XYZ-IBD data with task-specific metadata, defining the most-visible instance as anchor and categorizing other instances by spatial overlap.The overlap categories are easy, medium, and hard, based on intersection-over-union between the object model and the partial instance observed in the image.Source: §4 · p. 5
  • The benchmark contains 1,166 RGBD and grayscale-D images from five sensors and covers 21 objects spanning everyday and industrial categories.Its evaluation reports average recall, ADD(S), rotation error, and translation error, with analysis by difficulty, object category, and sensing condition.Source: §4 · p. 5
  • PRENCH evaluates anchor-to-instance relative pose accuracy across objects with varied appearance and geometry, supporting assessment across sensing conditions.Everyday objects often have richer textures, while industrial objects can have less informative surfaces and complex geometries.Source: §4 · p. 5

5. Experiments

PROSE outperforms adapted One2Any and ConceptPose baselines across PRENCH datasets and metrics, with qualitative examples and ablations examining its refinement and feature choices. The experiments evaluate a training-free pipeline using foundation features and cross-instance correspondence synchronization.

5.1. Implementation details

  • The comparison adapts One2Any and ConceptPose by using the anchor crop from each input image as the reference, and the implementation uses DINOv2-small and dGeDi features with ADMM refinement.The method uses RANSAC for relative pose estimation; ground-truth instance masks are used to crop objects for the adapted baselines.Source: §5.2 · p. 5, §5.1 · p. 5

5.3. Quantitative results

  • PROSE outperforms One2Any and ConceptPose across all three datasets on AR, ADD(S), rotation error, and translation error.AR improves by about 20 points across datasets; ADD(S) improves by 22 points on IC-BIN and XYZ-IBD, while rotation and translation errors improve by 9.2–36.0 degrees and 7.7–15.3 mm, respectively.Source: Table 1 · p. 6, §5.3 · p. 6

5.4. Qualitative results

  • In qualitative examples, PROSE remains more accurate on occluded, densely packed objects and textureless industrial parts where adapted baselines make inaccurate predictions.Cross-instance refinement recovers correct correspondences in the IC-BIN examples; the comparisons also include an IC-MI milk-bottle scene.Source: §5.4 · p. 6, Figure 4 · p. 7

5.5. Ablation study

  • Without synchronization, concatenated DINOv2 and dGeDi features improve pose accuracy across IC-BIN and XYZ-IBD, with visual and geometric features strongest on different object types.The results using frozen foundation features alone outperform the adapted One2Any and ConceptPose baselines reported in Table 1.Source: §5.5 · p. 6, Table 2 · p. 8
  • On XYZ-IBD, any α > 0 yields about a 2-point AR gain, whereas α = 0 degrades the coarse baseline by 3.6 points at λ = 50.Performance is largely insensitive to λ; the authors adopt α = 0.2 and λ = 50.Source: §5.5 · p. 7
  • On XYZ-IBD, synchronization improves average rotation error by 3.2 degrees and translation error by 1.8 mm over the refinement-free baseline.The gain is consistent across scenes and more pronounced in the more challenging scenes; Table 3's average is across scenes, unlike Table 1's average across instances.Source: §5.5 · pp. 6–7, Table 3 · p. 8
  • The refinement's contribution increases with the number of instances, although the task becomes more challenging in that regime.The positive correlation is reported for AR, with a similar trend for ADD(S), RE, and TE.Source: §5.5 · p. 8

6. Conclusion

The conclusion presents prior-free relative 6D pose estimation, PROSE, and PRENCH, reporting that PROSE outperforms adapted single-reference methods. It also identifies assumptions and a geometric limitation of the refinement.

6. Conclusion

  • PROSE refines coarse multimodal-feature correspondences by exploiting information across multiple instances in the input image.The conclusion describes it as the first training-free method designed for the proposed setting.Source: §6 · p. 8
  • On XYZ-IBD, refinement is robust to λ but needs non-zero α; α = 0 can reverse the gain, while α > 0 gives about a 2-point improvement.The reported best setting is α = 0.2 and λ = 50.Source: Figure 5 · p. 8
  • The relative AR improvement over the refinement-free baseline is positively correlated with the number of object instances.The figure plots relative AR improvement against instance count.Source: Figure 6 · p. 8
  • PRENCH extends three multi-instance BOP datasets with task-specific metadata to provide a standardized evaluation framework.The cited datasets are IC-BIN, IC-MI, and XYZ-IBD.Source: §6 · p. 8
  • PROSE substantially outperforms adapted state-of-the-art relative pose methods that rely on a single external reference image.The compared methods are One2Any and ConceptPose.Source: §6 · p. 8
  • PROSE assumes instance masks and a predefined anchor, and its refinement does not explicitly model rigid correspondence geometry.Future work could investigate automatic segmentation and geometry-aware synchronization using pairwise-distance preservation.Source: §6 · p. 8

A. Overview

The supplementary material outlines additional benchmark statistics, ablations, quantitative comparisons, and qualitative analyses.

A. Overview

  • Section B reports PRENCH statistics, including image, instance, anchor-instance pair, object, and per-view instance counts.It also describes the distribution of anchor-instance overlap.Source: A. Overview · p. 11
  • Section C extends the ablations by examining PROSE under different anchor-instance overlap levels and studying refinement effects scene by scene.Source: A. Overview · p. 11
  • Sections D and E add before-and-after refinement results and visualizations of semantic, geometric, fused features, and cycle-consistency examples.The examples include both successful and failed refinement cases.Source: A. Overview · p. 11

B. Benchmark statistics

PRENCH combines three datasets with different instance counts, overlap distributions, and object appearances to evaluate relative pose estimation across varied scene conditions.

B. Benchmark statistics

  • PRENCH contains 1,166 images, 21,833 instances, 21 objects, and 20,667 anchor-instance evaluation pairs.Source: B. Benchmark statistics · p. 11
  • Average instances per view range from 2.7 in IC-MI to 26.5 in XYZ-IBD, with XYZ-IBD reaching 59.IC-BIN averages 9.8 instances per view; the refinement benefit is largest in dense XYZ-IBD scenes and negligible in sparse IC-MI scenes.Source: B. Benchmark statistics · p. 11
  • The datasets span different anchor-instance overlap levels and object appearances, from mostly hard-overlap IC-BIN pairs to textureless industrial parts in XYZ-IBD.IC-MI mainly contains medium-overlap pairs, while XYZ-IBD spans the full overlap range.Source: B. Benchmark statistics · p. 11

C. Extended ablation studies

PROSE performs strongly across overlap-difficulty bands, while the benefit of cycle-consistency refinement varies with dataset and instance count.

C. Extended ablation studies

  • PROSE has the highest AR in almost every overlap bin across all three datasets, with gains persisting into low-overlap cases.On XYZ-IBD, easy/medium AR is 72.9%/48.8% for PROSE, versus 47.0%/25.0% for ConceptPose and 23.0%/10.0% for One2Any; the hard IC-MI split is the exception.Source: C. Extended ablation studies · p. 11
  • Refinement improves mean AR by +2.4 on XYZ-IBD, is nearly neutral on IC-BIN (+0.4), and reduces it on IC-MI (−1.0).Its gains track the number of co-visible instances and available overlapping cycles.Source: C. Extended ablation studies · p. 11
  • XYZ-IBD averages 26.5 instances per image, IC-BIN 9.8, and IC-MI about 3; their overlap-difficulty distributions also differ.IC-BIN skews toward hard, low-overlap pairs; IC-MI concentrates in medium overlap, while XYZ-IBD spans all difficulty bands.Source: Figure 7 · p. 12

D. Additional quantitative results

Dataset-level refinement results are strongest on XYZ-IBD, marginal on IC-BIN, and slightly negative on IC-MI. Per-scene findings likewise show that gains vary across scenes.

D. Additional quantitative results

  • On XYZ-IBD, refinement improves AR in 12 of 15 scenes, including +7.3 in scene 000005, where RE falls from 51.8° to 43.2° and TE from 23.0 to 15.8 mm.The largest reported per-scene AR drop is 2.9 points.Source: D. Additional quantitative results · p. 12
  • On XYZ-IBD, refinement raises AR from 53.2 to 55.8 (+2.6 points), ADD-S from 90.6 to 92.1, and lowers RE from 32.1° to 28.6° and TE from 10.6 to 9.1 mm.The comparison changes only the correspondences supplied to RANSAC; other components are held fixed.Source: D. Additional quantitative results · pp. 12–13
  • Across scenes, mean ΔAR is +2.4 on XYZ-IBD, +0.4 on IC-BIN, and −1.0 on IC-MI.Source: Figure 9 · p. 13
  • IC-BIN changes are marginal (AR +0.4, ADD-S +0.9, RE −0.3°; TE +0.5 mm), while IC-MI is marginally detrimental (AR −0.9, RE +0.1°, TE +0.2 mm; ADD-S remains 100%).Source: D. Additional quantitative results · p. 13

E. Additional qualitative results

The qualitative results show complementary strengths in semantic and geometric features, with fusion supporting matching across object types. Cycle-consistency refinement can correct inconsistent matches, but a consistent cycle may still be wrong.

E. Additional qualitative results

  • Fused features generally yield denser, more spatially distributed inlier correspondences by combining appearance-based matches with geometry-based matches.DINOv2 matches distinctive regions on textured objects; dGeDi uses structures such as teeth, edges, and holes on industrial parts.Source: E. Additional qualitative results · pp. 14–15, Figure 11 · p. 15
  • DINOv2 captures useful semantic and appearance cues on textured objects, while dGeDi captures geometry on textureless industrial objects; fusion stays discriminative across both.The visualizations cover coffee cups, juice boxes, gears, and metal brackets.Source: E. Additional qualitative results · p. 13, Figure 10 · p. 15
  • Cycle refinement closes successful triplets and recovers 3/3 inliers, but can also close a wrong cycle with 0/3 inliers or discard a correct coarse edge.In successful examples, coarse closure gaps are 42–96 mm; closure reaching zero alone does not establish ground-truth correctness.Source: E. Additional qualitative results · p. 15, Figure 12 · p. 16

Related papers

Loading 2609.08949v1…