Source-linked AI summary
MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, Josef Sivic
TL;DR
MegaPose addresses 6D pose estimation for objects unseen during training, using only an image region and the object’s CAD model at inference. It combines rendered-view refinement, coarse pose classification, and diverse synthetic training data, achieving state-of-the-art ModelNet performance and competitive results across BOP datasets. Its main limitation is sensitivity to inaccurate initial pose estimates because refinement operates within a constrained error range.
Problem
Existing learning-based pose methods require target-object models during training, limiting rapid deployment to objects known only at inference.
Method
MegaPose combines render-and-compare refinement, classification-based coarse estimation, and large-scale diverse synthetic training data for novel-object pose estimation.
Results
MegaPose achieves state-of-the-art performance on ModelNet and competitive performance with target-object-trained methods across the 7 core BOP datasets.
Takeaways & Limitations
The method supports pose estimation for novel objects without retraining and is intended for rapid deployment in robotic manipulation.
Takeaways & Limitations
Refinement can fail when the coarse model’s initial pose error is too large because the refiner corrects poses only within a constrained range.
Abstract
from arXiv · showhide
We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.
1 Introduction
MegaPose targets 6D pose estimation for objects available only at inference, avoiding object-specific retraining. It combines novel-object pose estimation components with a highly diverse synthetic training set and reports strong benchmark performance.
- Motivation: Existing learning-based pose methods require target-object models during training and testing, preventing rapid deployment when objects become known only at inference.Such methods may require hours or days of synthetic-data generation and model training for each object.
- Motivation: Novel-object pose estimation must handle large variation in shape, texture, lighting, and severe occlusion.
- Contributions: MegaPose estimates a novel object’s 6D pose from a CAD model and image region, using render-and-compare refinement and classification-based coarse estimation.The method provides the object’s shape and coordinate system through multiple rendered CAD-model views.
- Contributions: The training dataset contains 2 million photorealistic images depicting over 20K models in physically plausible configurations.
- Results: MegaPose achieves state-of-the-art performance on ModelNet and performance competitive with methods trained on target objects across the 7 core BOP datasets.
2 Related work
Prior work largely addresses known objects or category-level generalization, while MegaPose targets full 6D pose estimation for novel object instances using their CAD models at inference.
- Known objects: Known-object methods increasingly use learned features and trainable render-and-compare refinement, but encode object shape and coordinates in network weights.This encoding leads to poor generalization on unseen objects.
- Novel objects: Category-level methods generalize to new instances within known categories but cannot generalize to novel categories because they rely on shared category properties.
- Novel objects: MegaPose requires a novel object’s 3D model during inference, avoids category-level information, and estimates its full 6D pose.
- Novel objects: Other novel-object approaches estimate only 3D orientation by comparing rendered hypotheses with observations, whereas MegaPose addresses full 6D pose.
3 Method
MegaPose decomposes novel-object pose estimation into detection, coarse estimation, and iterative refinement, using rendered CAD-model views to support inference on unseen objects. Its coarse classifier selects a pose suitable for refinement, which then predicts iterative updates from observed and rendered images.
- Framework: The framework assumes an object detector and estimates a novel object’s pose from an RGB or RGB-D image plus its 3D model.RGB and depth are concatenated when depth is available; detection itself is outside the paper’s scope.
- Coarse pose estimation: The coarse estimator compares observed and rendered images across pose hypotheses and selects the hypothesis that best matches the observed object pose.
- Coarse pose estimation: The coarse model scores whether each pose hypothesis lies within the refiner’s basin of attraction and uses the highest-scoring pose as initialization.Classification can implicitly handle object symmetries because multiple poses may be classified as correct.
- Pose refinement: The refiner consumes observed and rendered images at the current estimate and iteratively predicts updated pose estimates.The pose update uses a rotation-translation parameterization and depends on an anchor point inferred for novel objects.
- Pose refinement: Multiple rendered views aimed at the anchor point provide object shape and anchor information needed for novel-object refinement.Rendered depth and surface normals can add geometric and appearance information to the network.
- Network architecture: The coarse and refiner models use ResNet-34 backbones, while their output heads predict classification logits and nine pose-update values, respectively.
- Training data: Training uses synthetic RGB(-D) images with ground-truth poses and 3D models, generated at large scale from ShapeNet and Google-Scanned-Objects.
- Training: The refiner is trained on randomly perturbed poses to predict the relative transformation back toward the ground-truth pose.Translation and rotation perturbations define the corrections the network learns to make.
4 Experiments
MegaPose is evaluated on novel objects across BOP and ModelNet, including quantitative benchmarks, qualitative examples, runtime, and ablations. Results support the method’s performance, the refiner’s speed, and the importance of rendered views and diverse training objects, while identifying coarse-estimation and runtime limitations.
- 4.1 Dataset and metrics: 132 objects across seven BOP datasets cover clutter, occlusion, texture variation, symmetry, and household or industrial settings.All evaluated objects are novel at test time and were excluded from training.
- 4.2 6D pose estimation of novel objects: On YCB-V, MegaPose improves AR by +10.7 over Zephyr; averaged across YCB-V and LM-O, it reaches 59.7 versus Zephyr’s 55.7.The comparison uses PPF- and SIFT-based detections and pose hypotheses, followed by coarse scoring and five refinement iterations.
- 4.2 6D pose estimation of novel objects: The refiner improves CosyPose coarse estimates by +23.7 average AR for the RGB-D model and outperforms existing approaches across all ModelNet metrics.The RGB-only refiner also performs better on average than CosyPose’s refiner despite not seeing BOP objects during training.
- 4.3 Ablations: Ablations show that increasing rendered views from one to four improves refinement, while performance steadily increases with the number of training objects.Normal maps provide additional appearance information, and GSO objects are particularly important for BOP performance, likely because they include high-quality textured objects.
- 4.4 Limitations: The most common failure arises from inaccurate coarse poses because refinement can fail when the initial error exceeds its constrained correction range.Using 520 pose hypotheses takes around 2.5 seconds to render and evaluate, motivating possible changes to hypotheses, coarse estimation, or refinement range.
5 Conclusion
MegaPose estimates poses for novel objects from CAD models available only at test time, with quantitative evaluation on diverse real-image benchmarks. The method is released with models and a large-scale synthetic dataset, while zero-shot detection remains open.
- MegaPose estimates the 6D pose of novel objects from a CAD model available only at test time.
- The method was quantitatively evaluated on hundreds of objects depicted in cluttered scenes, with ablation studies validating network design choices and training-data importance.
- The released models and synthetic dataset target robotic manipulation settings where rapid deployment to new scenes and objects is important.
- Detecting an unknown object from only a CAD model remains unsolved, and future work targets zero-shot object detection.
A Pose update and anchor point
The refiner predicts a translation update for a 3D anchor point and a rotation update for the object pose. Rotation predictions are independent of object-frame orientation, but translation predictions depend on the anchor point, which the method infers through rendered views.
- The network predicts nine values: a 3-vector for anchor-point translation and two 3-vectors defining a rotation update.
- The pose update changes the 3D reference point attached to the object and the object rotation expressed in the camera frame.
- Rotation corrections do not depend on the chosen object coordinate-frame orientation, whereas translation corrections generally depend on the anchor point.
- Different anchor points produce different network predictions when the initial and target poses differ.
- The method enables generalization by rendering object views in which the anchor point reprojects to the center of each image.
- The refiner loss uses sampled object-surface points to measure pose distance and sums separate xy-translation, relative-depth, and rotation terms over three iterations.
C Depth normalization
For RGB-D refinement and coarse pose hypotheses, MegaPose normalizes depth to reduce sensitivity to absolute object distance and constructs hypotheses from image detections and rendered object views.
- Observed and rendered depth images are normalized so the network becomes invariant to absolute depth values outside the training distribution.
- During coarse-model training, one viewpoint is positive when its pose error lies within the refiner’s basin of attraction; the remaining viewpoints are negative.
- A coarse hypothesis uses the detected bounding-box center as the anchor projection and initializes the anchor depth at 1m.
- The initial depth is updated by projecting the CAD model and matching its projected width and height approximately to the 2D detection.
- Each hypothesis combines a camera-object orientation with the anchor translation, and the procedure generates 103 additional viewpoints around the initial one.
- Repeating the viewpoint construction five times produces 520 pose hypotheses.
E Training details
Training uses millions of photorealistic synthetic images with RGB and depth augmentation. Coarse and refiner models are trained once, and novel-object inference requires no target-object fine-tuning.
- The training set contains 2 million photorealistic images generated with BlenderProc.
- RGB augmentation includes blur, contrast, brightness, color, and sharpness filters, while depth augmentation includes blur, dropout, and noise.
- Training takes 32 hours for the coarse model and 48 hours for the refiner on 32 V-100 GPUs.
- Training is performed once, without fine-tuning on target objects when estimating poses for novel objects.
F Additional experiments.
Additional experiments evaluate MegaPose’s coarse scoring network, runtime–accuracy trade-off, and qualitative robotic applicability on real scenes.
- Coarse network: The coarse network outperforms PPF and Zephyr for selecting pose hypotheses on LM-O and YCB-V.The evaluation uses hypotheses generated by Halcon software implementing PPF.
- Training images: The synthetic training images are generated with BlenderProc from CAD models in ShapeNet and GoogleScannedObjects.The dataset contains randomly sampled images used for training.
- Classification-based coarse network: A regression-based coarse network collapsed during training and achieved performance close to zero on BOP datasets.Its pose errors were too large for the refiner to recover.
- Number of coarse pose hypotheses: Performance improves by 11.4 AR on BOP5 from M=104 to M=520 hypotheses, while the coarse model runs in 1.6 seconds at M=520.Increasing to 4608 hypotheses yields only a further 0.9 AR improvement.
- Robotic grasping experiments: A qualitative robot experiment estimates poses from a single RGB image of YCB-V objects placed among visual distractors.The setup uses a RealSense D415 camera mounted on a Franka Emika Panda robot and combines detection, coarse estimation, and refinement.
G Robustness to illumination conditions
MegaPose produces highly accurate qualitative predictions for a watering can in the TUD-L dataset despite challenging illumination conditions.
- Robustness to illumination conditions: MegaPose shows high-accuracy watering-can pose predictions on real TUD-L images under challenging illumination.Each example compares the observed image with a CAD-model rendering in the predicted pose and an overlay.
H Failure modes and performance on specific types of objects
The YCB-V analysis reports per-object accuracy under joint translation and rotation thresholds and examines failures on lower-performing objects.
- Per-object performance: Per-object YCB-V performance is measured as the percentage of predictions within 5cm translation and 15° rotation error.The analysis covers all 21 dataset objects.
- Failure modes: The reported failure analysis focuses on objects whose performance is below average on YCB-V.Examples are presented in Figure 7.
- Failure modes: Three failure modes involve viewpoint-similar appearances, fine-detail ambiguities in asymmetric objects, and incorrect CAD-model scale.The first two can trap the refiner in a local minimum after an inaccurate coarse estimate, while scale errors create depth errors in RGB images.
I 3D model quality
MegaPose remains applicable with imperfect 3D models, including low-fidelity meshes, poor textures, and geometric discrepancies.
- 3D model quality: MegaPose correctly estimates poses for some objects using low-fidelity CAD models with poor textures or geometric discrepancies.Examples include unreadable model annotations, missing geometry, and low-quality meshes or textures.
- Failure modes: Figure 7 illustrates cases where visually matching contours do not guarantee the correct pose because different viewpoints can look similar.It also shows failures to distinguish asymmetric scissor handles with different dimensions.