PaperlayerView exact revision on arXiv

Source-linked AI summary

Back to the Feature: Zero-Shot 6DoF Pose Estimation via Dense Local Features

Ali Rafiaei, Michael Greenspan

arXiv:2609.06726v1cs.CV

TL;DR

Zero-shot 6DoF pose estimation for unseen objects remains challenging because learned methods can falter outside their training objects. B2TFPose uses frozen DINOv3 features in a training-free pipeline with diverse template retrieval, render-guided re-correspondence, and multi-mask selection. It reports state-of-the-art performance among training-free RGB methods on the seven core BOP datasets.

  • Problem

    Learned pose-estimation methods often perform less effectively on previously unseen objects, leaving the question of combining their performance with classical local descriptors’ zero-shot capability.

    Source: §1 · p. 1

  • Method

    B2TFPose uses a single frozen DINOv3 model for viewpoint-diverse template retrieval, render-guided re-correspondence, and multi-mask hypothesis selection without task-specific training.

    Source: §1 · p. 2

  • Results

    B2TFPose establishes state-of-the-art performance among training-free RGB methods on the seven core BOP datasets.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The method demonstrates a training-free route to zero-shot 6DoF pose estimation using frozen foundation-model features and dense local matching.

    Source: §1 · p. 2

    Retaining more than two segmentation candidates could improve robustness, but increases runtime proportionally; two candidates are used as a practical choice.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

We present B2TFPose, a training-free zero-shot method for 6DoF pose estimation of unseen objects from RGB images. Using a single frozen DINOv3 vision transformer as its only pretrained component within the pose estimation pipeline, B2TFPose extracts dense patch-level features that generalize across the synthetic-to-real domain gap without any task-specific fine-tuning, revisiting the classical local feature matching paradigm through the lens of large-scale self-supervised foundation models. Three contributions advance the training-free state of the art. A geodesic non-maximum suppression strategy retrieves a viewpoint-diverse template set for coarse-to-fine correspondence matching. Render-guided Re-Correspondence (RRC) synthesizes object-specific views at the estimated pose and re-establishes dense 2D-3D correspondences to sharpen the initial estimate without additional learned parameters. A multi-mask hypothesis selection strategy jointly scores competing segmentation candidates to resolve segmentation ambiguity. On the seven core datasets of the BOP Benchmark, B2TFPose achieves 40.7 mean AR without refinement and 56.4 with refinement, establishing state-of-the-art performance among training-free RGB methods and outperforming trained counterparts including GigaPose and GenFlow, at competitive inference speed.

1 Introduction

B2TFPose revisits local feature matching to address zero-shot 6DoF pose estimation for unseen objects, where learned methods often falter. It combines viewpoint-diverse retrieval, render-guided re-correspondence, and competing-mask scoring in a training-free pipeline.

1 Introduction

  • Classical local descriptors encoded compact neighborhood patterns and enabled nearest-neighbor spatial matching for applications including Structure-from-Motion and object-level 6DoF pose estimation.Their robustness to in-plane rotation, scale, appearance, and lighting variation motivated their use.Source: §1 · p. 1
  • Learned methods perform less effectively on previously unseen objects, motivating the question of combining supervised methods’ performance with classical local descriptors’ zero-shot capability.Classical local features could match unseen objects without training or tuning, whereas learned methods tended to be global and object-specific.Source: §1 · pp. 1–2
  • Render-guided Re-Correspondence renders the object at the initial pose and rotation variants, then establishes new dense correspondences to improve accuracy without additional learned parameters.This differs from prior render-and-compare refinement, which does not regenerate correspondences for a second geometry solve.Source: §1 · p. 2
  • Multi-mask hypothesis selection retains competing segmentation candidates and scores their pose hypotheses using appearance similarity, projected-model mask overlap, and segmentation confidence.This avoids discarding alternative masks before pose estimation.Source: §1 · p. 2
  • B2TFPose is presented as a training-free zero-shot pipeline using a single frozen DINOv3 model, with state-of-the-art performance among training-free RGB methods and faster inference than the closest comparable baseline.The method adds no task-specific fine-tuning or training of other network components.Source: §1 · p. 2
  • Geodesic-NMS retrieval selects diverse template viewpoints for coarse matching, improving pose recovery over few-template and exhaustive alternatives in ablations.The method focuses on coarse feature matching, pose refinement, and mask hypothesis selection.Source: §1 · p. 2

2 Related Work

Instance-level pose estimation moved from classical templates to learned representations, often requiring object-specific training. Zero-shot methods later used broader training or frozen foundation-model features, while ViT patch tokens provide spatially local representations for matching.

2.1 Instance-Level Pose Estimation

  • Classical LINEMOD templates detected textureless objects in clutter, but classical methods generalized poorly to large viewpoint changes, heavy occlusion, and appearance variation.Learned methods then introduced regression, voting, dense correspondences, and render-and-compare refinement, but typically retained a per-object or per-category training burden.Source: §2.1 · p. 3

2.2 Zero-Shot Object Pose Estimation

  • Zero-shot methods range from MegaPose’s large-scale synthetic training to training-free template matching; training-based approaches can incur substantial data-curation costs, while some alternatives require depth.Co-op achieved the highest RGB-only accuracy among zero-shot methods but required large-scale synthetic training data.Source: §2.2 · pp. 3–4

2.3 Foundation Models for Visual Feature Extraction

  • ViTs encode images as patch tokens with limited receptive fields, and DINOv3 serves as B2TFPose’s sole learned component.DINOv1 established patch-token cross-image semantic correspondence properties; DINOv2 scaled the approach, and DINOv3 advances its generalization and representational capacity.Source: §2.3 · p. 4
  • CNOS combines a fast SAM variant with DINOv2 template scoring to generate zero-shot instance masks from a CAD model, and is used as the detection front-end in the evaluation.Using CNOS across B2TFPose and competing methods is intended to make performance differences reflect pose estimation rather than the detector.Source: §2.3 · p. 4

3 Method

B2TFPose estimates an unseen object’s pose from an RGB image and mesh using an offline template database and an online coarse-to-fine matching pipeline. Its online stages retrieve diverse views, estimate and refine pose from dense correspondences, and select among mask-based hypotheses.

3.1 Overview

  • The method uses one frozen DINOv3 model as its only pretrained component to estimate zero-shot pose from an RGB query and unseen rigid-object mesh.Offline object onboarding takes up to five minutes on one GPU; each online image is segmented, retrieved against templates, and matched for pose estimation and refinement.Source: §3.1 · p. 4

3.2 Offline Template Generation and Encoding

  • The offline stage renders 162 base views and augments each with 12 in-plane rotations, producing 1944 templates with global descriptors and cached local patch features and 3D coordinates.Global features are foreground-pooled final-layer tokens; local features retain individual intermediate-layer patch embeddings and their precomputed object-frame coordinates.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

3.3 Online Stage

  • For each retained CNOS mask, the online stage retrieves up to 125 templates by cosine similarity with at least 20° geodesic rotational separation, then MNN-filters patch matches before RANSAC+PnP.Top-three nearest template patches are considered, and the query patch centers are paired with cached object-frame points.Source: §3.3 · pp. 5–6
  • RRC renders ten rotation variants around the initial pose with translation fixed, matches query patches to the rendered features, and runs another RANSAC+PnP before MegaPose refinement.The variants test axis-aligned alternatives to the current pose hypothesis, and the renders are encoded with the same DINOv3 intermediate layer used for patch matching.Source: §3.3 · p. 6
  • Each retained mask is processed through pose estimation and refinement independently; hypotheses with rendered-mask IoU below 0.2 are discarded before scoring.Survivors are ranked using appearance similarity, mask overlap, and CNOS confidence, z-normalized and combined with weights 0.2, 0.5, and 0.3 respectively.Source: §3.3 · pp. 6–7

4 Experiments

B2TFPose is evaluated as an RGB-only method on the seven core BOP datasets, with comparisons organized by training requirements and use of pose refinement. It achieves the highest mean AR among training-free methods in both refinement settings, while ablations examine its refinement, retrieval, and matching choices.

4.1 Experimental Setup

  • Evaluation uses RGB channels from seven BOP core datasets, reports BOP Average Recall, and compares methods using CNOS instance segmentation and single-hypothesis pose estimates.AR averages recall across thresholds and VSD, MSSD, and MSPD error functions; the comparison table groups methods by training requirements and refinement use.Source: §4.1 · p. 7, §4.2 · p. 7, Table 1 · p. 8

4.2 Comparison with State of the Art

  • 40.7 mean AR without refinement and 56.4 mean AR with refinement make B2TFPose the top training-free method in both settings across the seven BOP core datasets.Without refinement, FoundPose reaches 37.2 mean AR; with refinement, FoundPose reaches 55.0 AR. Co-op is the only trained method reported above B2TFPose with refinement, at 64.0 AR.Source: Table 1 · p. 8, §4.2 · p. 8
  • Without refinement, B2TFPose runs in 0.32 s; only GigaPose is marginally faster at 0.29 s, but has substantially lower accuracy (27.6 versus 40.7 AR, a 13.1-point gap).Source: §4.2 · p. 8

4.3 Ablation Studies and Hyperparameter Tuning

  • RRC alone helps on several occlusion-heavy or textureless datasets, while MegaPose supplies most of the mean-AR gain; together they reach 56.4 mean AR.MegaPose alone raises mean AR from 40.7 to 55.5, and RRC+MegaPose reaches 56.4; RRC’s effects vary by dataset, including small drops on TUD-L, IC-BIN, and HB when used alone.Source: §4.3 · p. 9, Table 2 · p. 10
  • Replacing DINOv3 with DINOv2 lowers AR on TUD-L but raises it on IC-BIN in the PnP-only setting, indicating that the reported gains are not simply inherited from DINOv3.The reported DINOv2 results are 57.3 AR on TUD-L and 23.1 AR on IC-BIN, respectively.Source: §4.3 · p. 9
  • On TUD-L, k=125 reaches 58.1 AR, while using all 1944 templates reaches 57.6 AR; disabling geodesic diversity reduces AR at each tested pipeline stage.The reported reductions without diversity are 0.6, 2.3, and 2.8 AR for PnP-only, RRC, and full refinement; skipping retrieval also reduces AR at all three stages.Source: §4.3 · pp. 9–10
  • On TUD-L, a 4-pixel PnP threshold reaches 58.1 AR, while three patch neighbors reach 58.1 AR and adding more yields only 0.1 AR.The 4-pixel threshold outperforms stricter or looser settings; three neighbors are adopted as the lowest-cost near-saturating choice.Source: §4.3 · p. 10, Figure 3 · p. 10

5 Conclusion

B2TFPose is a training-free, zero-shot method for estimating 6DoF poses of novel objects from RGB images, using a frozen DINOv3 vision transformer. On the seven BOP core datasets, it achieves state-of-the-art performance among training-free RGB methods and outperforms trained methods including GigaPose and GenFlow.

5 Conclusion

  • B2TFPose establishes state-of-the-art performance among training-free RGB methods on seven BOP core datasets, surpassing FoundPose with and without refinement and outperforming trained GigaPose and GenFlow.The conclusion also reports competitive inference speed.Source: §5 · p. 11
  • The training-free, zero-shot method estimates novel-object 6DoF poses from RGB images using only a frozen DINOv3 vision transformer, with no object-specific or task-specific training.Source: §5 · p. 11
  • Geodesic non-maximum suppression retrieves viewpoint-diverse templates, RRC refines correspondences at the initial pose, and multi-mask selection addresses segmentation ambiguity.Source: §5 · p. 11
  • The method requires a 3D CAD model at test time, and its appearance-based matching is less robust under heavy occlusion and dense clutter without depth cues.Source: §5 · p. 11

A Overview

The supplementary material presents qualitative BOP results and visualizes the Render-guided Re-Correspondence (RRC) step. It describes representative success and failure cases and shows how RRC improves an initial pose estimate before MegaPose refinement.

A Overview

  • The supplement covers qualitative results and a visual explanation of the RRC step on BOP benchmark datasets.Source: A Overview · p. 16
  • The qualitative-results section presents representative success and failure cases, with query images, pose overlays, and ten object-specific renders shown for each RRC example.Source: A Overview · p. 16
  • The RRC visualization illustrates improvement of the initial RANSAC+PnP estimate before MegaPose refinement.Source: A Overview · p. 16

B Qualitative Results

The qualitative examples show B2TFPose pose overlays across BOP scenes, including accurate estimates and cases where rotational symmetry or partial occlusion leads to errors. The described examples cover five datasets; two others are excluded because ground-truth poses are unavailable.

B Qualitative Results

  • The examples span TUD-L, T-LESS, LM-O, YCB-V, and IC-BIN; HB and ITODD are excluded because their ground-truth poses are not publicly available.Source: B Qualitative Results · p. 16
  • The overlays show query RGB crops alongside estimated poses before and after RRC and MegaPose refinement, plus ground-truth poses.Source: B Qualitative Results · pp. 16–17
  • The TUD-L example shows accurate pose estimation both without refinement and with refinement.Source: B Qualitative Results · p. 17
  • Rotational symmetry and partial occlusion can combine with patch ambiguity and model noise to produce an incorrect estimate.Source: B Qualitative Results · p. 18
  • Failure cases also include severe occlusion, textureless appearance, and inter-object ambiguity in bin-picking scenarios, where appearance-based matching lacks explicit depth cues.Source: Figure 5 · p. 18

C Render-guided Re-Correspondence (RRC) Visualization

The RRC visualization follows pose overlays through four pipeline stages and shows the object-specific views rendered at the initial estimate for a second correspondence round. Its examples contrast direct MegaPose refinement with the RRC-plus-MegaPose pipeline.

C Render-guided Re-Correspondence (RRC) Visualization

Loading 2609.06726v1…