FoundPose: Unseen Object Pose Estimation with Foundation Features

1 Introduction

• FoundPose requires no object- or task-specific training and achieves state-of-the-art results on the standard BOP benchmark.The method combines modern foundation features with classical computer-vision techniques.Source: §1 · pp. 2–3
• FoundPose integrates DINOv2 descriptors into a bag-of-words representation to retrieve a small set of similarly looking pre-rendered templates.This retrieval approach is reported as 15X faster, though slightly less accurate than the heavy render-and-compare coarse stage.Source: §1 · p. 2
• 100X fewer templates and a 25X lower memory footprint are reported for FoundPose than for previous template-based approaches.The approach uses several hundred templates rather than 90K+; the memory comparison is stated in the contribution summary.Source: §1 · p. 3
• Intermediate-layer DINOv2 descriptors support geometrically consistent correspondences when symmetry or lack of texture makes semantic information ambiguous.The method matches patch descriptors between the real image crop and retrieved synthetic templates, then lifts matched template locations to 3D using rendered depth.Source: §1 · p. 3
• Featuremetric refinement compensates for 2D-3D correspondence discrepancies caused by coarse DINOv2 patch sampling.The refinement applies the idea of classical photometric refinement to DINOv2 patch descriptors.Source: §1 · p. 3

2 Related work

• Classical methods used local-feature or template matching and could rapidly onboard objects when reference images had model-to-camera annotations.RGB-D approaches became popular for accuracy, while model-based estimation remains relevant where CAD models are available.Source: §2 · pp. 3–4
• Deep neural networks improved pose-estimation accuracy, but large-scale training data and lengthy training often limit generalization to new objects.Recent work has returned to unseen-object estimation, though early attempts had lower accuracy and higher computational demands.Source: §2 · p. 4
• Most top-performing BOP Challenge 2023 unseen-object methods require millions of task-specific training images; FoundPose instead uses frozen DINOv2 features without training.FoundationPose, SAM-6D, PoMZ, and ZeroPose are described as RGB-D methods without RGB-only versions; FoundPose is RGB-only.Source: §2 · p. 4
• DINOv2 encodes spatial and semantic information, and FoundPose extends its use from within-category semantic matching to synthetic-to-real correspondences.Goodwin et al. establish semantic correspondences within an object category; their method requires RGB-D inputs at test time and cannot handle symmetric objects.Source: §2 · pp. 4–5

3.1 Method overview

• Given object masks and 3D models, the pipeline renders templates during onboarding, retrieves similar views at inference, estimates poses from patch matches, and refines the best hypothesis.The stated setup assumes known camera intrinsics, limited onboarding, and masks with object identities available at inference.Source: §3.1 · p. 5

3.2 Template-based object representation

• Templates store DINOv2 patch descriptors paired with 3D locations, while bag-of-words vectors summarize their visual-word frequencies for efficient retrieval.The 3D locations come from rendered depth and known camera intrinsics; visual words are k-means cluster centroids of template patch descriptors.Source: §3.2 · pp. 5–7

3.3 Template retrieval by bag-of-words matching

• At inference, a perspective-warped crop is ranked against template bag-of-words descriptors by cosine similarity, and the highest-ranked templates provide approximate orientation hypotheses.The authors describe retrieval as efficient and robust to partial occlusions because visible object regions still contribute visual words.Source: §3.3 · p. 7

3.4 Pose estimation from 2D-3D correspondences

• For each retrieved template, nearest patch-descriptor matches link query-patch centers to 3D template locations; PnP-RANSAC selects the pose with the most inliers.Intermediate-layer positional information helps produce geometrically consistent correspondences on symmetric or texture-less objects, where semantics may be ambiguous.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

3.5 Featuremetric pose refinement

• Featuremetric alignment refines the coarse pose by optimizing template descriptors against sampled query-image features to compensate for patch-center and projected-point discrepancies.It uses Levenberg–Marquardt directly on DINOv2 features without training; the authors note there is no guarantee of reaching a global optimum.Source: §3.5 · pp. 8–9

4.1 Experimental setup

• Evaluation follows the BOP protocol across seven core datasets; experiments use only 3D models and test images, and most compared methods share CNOS masks, with OSOP exceptions.Average Recall combines VSD, MSSD, and MSPD; the datasets contain 108 diverse objects.Source: §4.1 · pp. 9–10

4.2 Main results

• Without refinement, FoundPose averages +10, +14, and +16 AR over coarse GigaPose, GenFlow, and MegaPose; featuremetric refinement adds +5 AR, and MegaPose refinement adds +17 AR.The comparison covers the seven BOP datasets on average.Source: §4.2 · p. 11, Table 1 · p. 12
• 59.6 AR is the best reported score, achieved by refining five FoundPose hypotheses with featuremetric then MegaPose refinement; it beats multi-hypothesis MegaPose and GenFlow and refined GigaPose.The two refinement approaches together achieve the best scores in both single- and multi-hypothesis setups.Source: §4.2 · p. 11, Table 1 · p. 12
• FoundPose is most accurate and second-fastest across evaluated categories; its coarse stage takes 1.7 vs. 15.5s for MegaPose, and object onboarding takes under 5 minutes.The reported times include CNOS segmentation, which takes around 0.3s per image.Source: §4.2 · p. 12, Table 1 · p. 12

4.3 Ablation experiments

• DINOv2 ViT-L layer-18 descriptors give the best tested pose accuracy and more geometrically consistent correspondences than later layers.DINOv2 descriptors also yield significantly higher accuracy than the evaluated alternatives.Source: §4.3 · p. 13, Table 2 · p. 13
• Bag-of-words retrieval outperforms cls-token retrieval, which remains weaker even when background pixels are masked; layer-23 cls-token retrieval performs worse still.The cls token is less robust to occlusions and contains limited object-pose information.Source: §4.3 · pp. 13–14, Table 2 · p. 13
• Using the top retrieved template’s pose reaches accuracy close to MegaPose’s coarse stage while being 15 times faster; direct pairwise patch matching takes 0.64s vs. 0.0008s per query.The pairwise matching time is measured with faiss.Source: §4.3 · p. 14
• Ground-truth masks improve FoundPose by +6–19 AR; tested parameter settings cause only marginal speed and accuracy differences, while an 800-template representation uses 234MB.A 400-template configuration reduces memory to 59MB with under a 1 AR drop.Source: Table 2 · p. 13, §4.3 · p. 14

• FoundPose significantly outperforms existing methods on the standard BOP benchmark for model-based pose estimation of unseen objects.Source: §5 · p. 14
• The method achieves these results without object- or task-specific training, using only a frozen vision foundation model.Source: §5 · p. 14
• Combining FoundPose with an existing render-and-compare refinement approach achieves RGB-only state-of-the-art results.Source: §5 · p. 14
• The authors see these results as encouragement to revisit efficient classical computer vision, often overlooked in modern literature.Source: §5 · p. 14