FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

• Object pose estimation is important for applications including robotic manipulation and mixed reality, but instance-level methods cannot handle unseen objects at test time.Source: §1 · p. 1
• Category-level methods relax instance-specific assumptions but remain limited to predefined training categories and require difficult category-level data collection.Source: §1 · p. 1
• Model-based setups provide a textured 3D CAD model, whereas model-free setups provide reference images; prior progress addressed these setups separately.Source: §1 · p. 1
• FoundationPose unifies 6D pose estimation and tracking for novel objects across both setups using RGBD images.Source: §1 · p. 1
• An object-centric neural implicit representation bridges the setups through novel-view synthesis from approximately 16 reference images, while downstream pose modules remain unified.Source: §1 · p. 2
• The framework uses LLM-aided large-scale synthetic training, a transformer-based architecture, and contrastive learning to support generalization.Source: §1 · p. 2

• CAD-model-based instance-level methods assume a textured CAD model and train and test on the exact same object instance.Source: §2 · p. 2
• Category-level methods generalize across novel instances within a category but do not address arbitrary objects beyond their training categories.Source: §2 · p. 2
• Gen6D uses detection, retrieval, and refinement for novel objects but requires fine-tuning to avoid out-of-distribution test-set difficulties.Source: §2 · p. 2
• The related-work landscape motivates bridging model-based and model-free scenarios rather than treating them as separate problems.Source: §2 · p. 2

Language-aided Data Generation at Scale

• The framework uses a synthetic data pipeline combining large 3D model databases, LLMs, diffusion models, and physically plausible rendering to diversify training data.LLM-generated texture augmentation addresses variable texture fidelity and enables automated large-scale data generation.Source: §3.1 · pp. 2–4, Figure 3 · p. 3

Neural Object Modeling

• An object-centric neural field represents model-free objects and enables efficient novel-view RGBD rendering for downstream render-and-compare pose estimation.The neural field is trained per object without priors, can be optimized within seconds, and is trained once for a novel object.Source: §3.2 · p. 4
• The neural field uses an SDF zero level set for object surfaces and a bell-shaped surface-weighting function for efficient color rendering near observed depth surfaces.Contributions more than λ from the surface are ignored, while integration extends 0.5λ into the surface to model self-occlusion.Source: §3.2 · p. 4

Pose Hypothesis Generation

• Pose estimation initializes translation from median detected-box depth and samples rotations over an icosphere with discretized in-plane rotations before refinement.The resulting Ns · Ni global pose initializations are sent to the pose refinement module.Source: §3.3 · p. 5

Pose Selection

• Pose selection trains a hierarchical ranking network with pose-conditioned triplet comparisons, retaining pairs whose positive hypothesis is sufficiently close to ground truth.The ranking loss uses ADD-based positive and negative pose assignments and excludes ambiguous pairs where both poses are far from ground truth.Source: §3.4 · p. 6

4.1. Dataset and Setup

• Experiments cover model-free and model-based setups for both 6D pose estimation and tracking across five datasets with cluttered, dynamic, multi-instance, textureless, shiny, and symmetric objects.The datasets are LINEMOD, Occluded-LINEMOD, YCB-Video, T-LESS, and YCBInEOAT.Source: §4.1 · p. 6
• Evaluation uses AUC of ADD and ADD-S, ADD-0.1d recall, and BOP average recall of VSD, MSSD, and MSPD.These metrics follow the protocols of the relevant baselines and benchmarks.Source: §4.2 · p. 6

4.3. Pose Estimation Comparison

• FoundationPose significantly outperforms existing model-free methods on YCB-Video and LINEMOD without target-dataset fine-tuning or ICP refinement.The comparison includes RGBD and RGB baselines under their stated reference-image and detection protocols.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 7
• Severe self-occlusion and limited glue texture challenge OnePose++ and LatentFusion, whereas FoundationPose successfully estimates the pose qualitatively.Figure 5 shows the model-free LINEMOD comparison using cropped and zoomed-in images.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
• On three BOP datasets, FoundationPose outperforms novel-object model-based methods and an instance-level method by a large margin.The model-based comparison reports AR scores for RGBD methods using Mask R-CNN detections.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Pose Tracking Comparison

• On dynamic YCBInEOAT tracking, FoundationPose achieves the best performance, surpasses an instance-wise method with ground-truth initialization, and supports end-to-end tracking without external initialization.The evaluation targets abrupt out-of-plane rotations, dynamic external occlusions, and disentangled camera motions.Source: §4.4 · p. 7, Table 4 · p. 7

4.5. Analysis

• Performance is robust to reference-image count, saturating at 12 images for both ADD metrics; four images still outperform FS6D with 16 images.The training-data gain saturates around 1M synthetic examples.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
• Pose estimation takes about 1.3 s per object, while tracking runs at approximately 32 Hz after initialization because tracking requires only pose refinement.Initialization takes 4 ms, refinement 0.88 s, and pose selection 0.42 s.Source: §4.5 · p. 8

• FoundationPose supports 6D pose estimation and tracking for novel objects in both model-based and model-free setups.Source: §5 · p. 8
• Across four task combinations, the unified framework outperforms existing state-of-the-art methods specialized for each task by a considerable margin.Source: §5 · p. 8
• FoundationPose achieves comparable results to methods requiring instance-level training despite reduced assumptions.Source: §5 · p. 8

Performance on BOP Leaderboard

• FoundationPose ranks #1 on the BOP leaderboard for 6D localization of unseen objects in the model-based pose-estimation task.The evaluation uses 2D detections from CNOS.Source: §5.1 · p. 13

Implementation Details

• Training pretrains a neural object field on synthetic reference images, freezes it, and mixes its rendering with OpenGL rendering for refinement and selection.This combination is intended to cover both model-based and model-free setup distributions.Source: §5.2 · p. 13
• The neural object field uses MLP-based geometry and appearance networks with multi-resolution hash encoding for RGBD object rendering.The geometry and appearance networks use ReLU activations, while the final appearance layer uses sigmoid activation.Source: §5.2 · pp. 13–14

Pose Tracking

• Pose selection uses K = 5 hypotheses, while tracking refines the current frame using rendering from the previous pose and temporal cues.The tracking procedure adapts the pose-refinement module to video sequences.Source: §5.2 · p. 14

Pose Updates

• Translation and rotation are disentangled because they occupy different spaces and can be represented independently in the camera coordinate frame.The paper illustrates this design with a 2D example.Source: §5.3 · p. 15, Figure 10 · p. 15