Source-linked AI summary
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield
TL;DR
Novel-object 6D pose estimation and tracking lacked a unified method spanning CAD-model-based and reference-image-based setups. FoundationPose combines a shared RGBD framework with neural implicit novel-view synthesis and synthetic-data-based generalization, and reports large-margin gains over task-specialized methods while remaining comparable to instance-level methods.
Problem
Existing novel-object pose methods address model-based and model-free setups separately, despite applications providing different types of object information.
Method
FoundationPose unifies 6D pose estimation and tracking for novel objects using RGBD images, neural implicit novel-view synthesis, LLM-aided synthetic training, transformer architectures, and contrastive learning.
Results
FoundationPose outperforms methods specialized for each of four tasks by a large margin and achieves comparable results to methods requiring instance-level training.
Takeaways & Limitations
The framework provides one versatile approach for novel-object pose estimation and tracking across model-based and model-free setups.
Takeaways & Limitations
The method relies on external 2D detection, and false or missing detections frequently bottleneck 6D pose estimation.
Abstract
from arXiv · showhide
We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups. Our approach can be instantly applied at test-time to a novel object without fine-tuning, as long as its CAD model is given, or a small number of reference images are captured. We bridge the gap between these two setups with a neural implicit representation that allows for effective novel view synthesis, keeping the downstream pose estimation modules invariant under the same unified framework. Strong generalizability is achieved via large-scale synthetic training, aided by a large language model (LLM), a novel transformer-based architecture, and contrastive learning formulation. Extensive evaluation on multiple public datasets involving challenging scenarios and objects indicate our unified approach outperforms existing methods specialized for each task by a large margin. In addition, it even achieves comparable results to instance-level methods despite the reduced assumptions. Project page: https://nvlabs.github.io/FoundationPose/
1. Introduction
FoundationPose addresses the need for one method that handles novel-object pose estimation and tracking across both model-based and model-free setups. It combines a unified RGBD framework with neural novel-view synthesis and synthetic-data-driven generalization.
- Object pose estimation is important for applications including robotic manipulation and mixed reality, but instance-level methods cannot handle unseen objects at test time.
- Category-level methods relax instance-specific assumptions but remain limited to predefined training categories and require difficult category-level data collection.
- Model-based setups provide a textured 3D CAD model, whereas model-free setups provide reference images; prior progress addressed these setups separately.
- FoundationPose unifies 6D pose estimation and tracking for novel objects across both setups using RGBD images.
- An object-centric neural implicit representation bridges the setups through novel-view synthesis from approximately 16 reference images, while downstream pose modules remain unified.
- The framework uses LLM-aided large-scale synthetic training, a transformer-based architecture, and contrastive learning to support generalization.
2. Related Work
Prior work develops separate strategies for CAD-model-based and image-based novel-object pose estimation, often retaining setup-specific assumptions. FoundationPose is positioned against this fragmented landscape by targeting the gap between model-based and model-free scenarios.
- CAD-model-based instance-level methods assume a textured CAD model and train and test on the exact same object instance.
- Category-level methods generalize across novel instances within a category but do not address arbitrary objects beyond their training categories.
- Gen6D uses detection, retrieval, and refinement for novel objects but requires fine-tuning to avoid out-of-distribution test-set difficulties.
- The related-work landscape motivates bridging model-based and model-free scenarios rather than treating them as separate problems.
3. Approach
FoundationPose combines scalable synthetic training, object-centric neural rendering, and pose refinement and ranking into a unified framework for novel-object pose estimation and tracking. Its model-free representation supports RGBD rendering from reference images, while hierarchical comparison selects among refined pose hypotheses.
- Language-aided Data Generation at Scale: The framework uses a synthetic data pipeline combining large 3D model databases, LLMs, diffusion models, and physically plausible rendering to diversify training data.LLM-generated texture augmentation addresses variable texture fidelity and enables automated large-scale data generation.
- Neural Object Modeling: An object-centric neural field represents model-free objects and enables efficient novel-view RGBD rendering for downstream render-and-compare pose estimation.The neural field is trained per object without priors, can be optimized within seconds, and is trained once for a novel object.
- Neural Object Modeling: The neural field uses an SDF zero level set for object surfaces and a bell-shaped surface-weighting function for efficient color rendering near observed depth surfaces.Contributions more than λ from the surface are ignored, while integration extends 0.5λ into the surface to model self-occlusion.
- Pose Hypothesis Generation: Pose estimation initializes translation from median detected-box depth and samples rotations over an icosphere with discretized in-plane rotations before refinement.The resulting Ns · Ni global pose initializations are sent to the pose refinement module.
- Pose Selection: Pose selection trains a hierarchical ranking network with pose-conditioned triplet comparisons, retaining pairs whose positive hypothesis is sufficiently close to ground truth.The ranking loss uses ADD-based positive and negative pose assignments and excludes ambiguous pairs where both poses are far from ground truth.
4. Experiments
FoundationPose is evaluated across four combinations of model knowledge and pose task on five challenging datasets, using consistent inference without target-dataset fine-tuning. It outperforms prior methods across model-free estimation, model-based estimation, and tracking, while remaining robust to reference-image count and training-data scale.
- 4.1. Dataset and Setup: Experiments cover model-free and model-based setups for both 6D pose estimation and tracking across five datasets with cluttered, dynamic, multi-instance, textureless, shiny, and symmetric objects.The datasets are LINEMOD, Occluded-LINEMOD, YCB-Video, T-LESS, and YCBInEOAT.
- 4.1. Dataset and Setup: Evaluation uses AUC of ADD and ADD-S, ADD-0.1d recall, and BOP average recall of VSD, MSSD, and MSPD.These metrics follow the protocols of the relevant baselines and benchmarks.
- 4.3. Pose Estimation Comparison: FoundationPose significantly outperforms existing model-free methods on YCB-Video and LINEMOD without target-dataset fine-tuning or ICP refinement.The comparison includes RGBD and RGB baselines under their stated reference-image and detection protocols.
- 4.3. Pose Estimation Comparison: Severe self-occlusion and limited glue texture challenge OnePose++ and LatentFusion, whereas FoundationPose successfully estimates the pose qualitatively.Figure 5 shows the model-free LINEMOD comparison using cropped and zoomed-in images.
- 4.3. Pose Estimation Comparison: On three BOP datasets, FoundationPose outperforms novel-object model-based methods and an instance-level method by a large margin.The model-based comparison reports AR scores for RGBD methods using Mask R-CNN detections.
- 4.4. Pose Tracking Comparison: On dynamic YCBInEOAT tracking, FoundationPose achieves the best performance, surpasses an instance-wise method with ground-truth initialization, and supports end-to-end tracking without external initialization.The evaluation targets abrupt out-of-plane rotations, dynamic external occlusions, and disentangled camera motions.
- 4.5. Analysis: Performance is robust to reference-image count, saturating at 12 images for both ADD metrics; four images still outperform FS6D with 16 images.The training-data gain saturates around 1M synthetic examples.
- 4.5. Analysis: Pose estimation takes about 1.3 s per object, while tracking runs at approximately 32 Hz after initialization because tracking requires only pose refinement.Initialization takes 4 ms, refinement 0.88 s, and pose selection 0.42 s.
5. Conclusion
FoundationPose is a unified foundation model for 6D pose estimation and tracking of novel objects in both model-based and model-free setups. Across four task combinations, it outperforms specialized state-of-the-art methods and approaches instance-level performance.
- FoundationPose supports 6D pose estimation and tracking for novel objects in both model-based and model-free setups.
- Across four task combinations, the unified framework outperforms existing state-of-the-art methods specialized for each task by a considerable margin.
- FoundationPose achieves comparable results to methods requiring instance-level training despite reduced assumptions.
Supplementary Material
The supplementary material reports benchmark performance, implementation details, architectural choices, pose-processing procedures, and limitations of FoundationPose. It documents strong leaderboard results, synthetic-training and rendering components, and dependence on external detection.
- Performance on BOP Leaderboard: FoundationPose ranks #1 on the BOP leaderboard for 6D localization of unseen objects in the model-based pose-estimation task.The evaluation uses 2D detections from CNOS.
- Implementation Details: Training pretrains a neural object field on synthetic reference images, freezes it, and mixes its rendering with OpenGL rendering for refinement and selection.This combination is intended to cover both model-based and model-free setup distributions.
- Implementation Details: The neural object field uses MLP-based geometry and appearance networks with multi-resolution hash encoding for RGBD object rendering.The geometry and appearance networks use ReLU activations, while the final appearance layer uses sigmoid activation.
- Pose Tracking: Pose selection uses K = 5 hypotheses, while tracking refines the current frame using rendering from the previous pose and temporal cues.The tracking procedure adapts the pose-refinement module to video sequences.
- Pose Updates: Translation and rotation are disentangled because they occupy different spaces and can be represented independently in the camera coordinate frame.The paper illustrates this design with a 2D example.