Source-linked AI summary
DeepIM: Deep Iterative Matching for 6D Pose Estimation
Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, Dieter Fox
TL;DR
RGB-only 6D pose estimation is difficult, and direct classification or regression can be inaccurate. DeepIM iteratively matches rendered and observed images by predicting relative pose transformations, achieving large improvements over RGB-only state-of-the-art methods and matching unseen objects.
Problem
RGB-only 6D pose estimation remains challenging, while direct pose classification or regression can produce inaccurate estimates.
Method
DeepIM predicts relative SE(3) transformations from rendered–observed image pairs and iteratively refines an initial pose using a disentangled representation of location and orientation.
Results
DeepIM achieves large improvements over state-of-the-art RGB-only methods and demonstrates accurate pose matching for unseen objects.
Takeaways & Limitations
DeepIM shows that accurate 6D pose refinement and matching of unseen objects are possible using color images only.
Takeaways & Limitations
Failures arise when object models and images disagree, visible patterns are sparse, shapes are difficult, initial poses are too distant, or key components are tiny.
Abstract
from arXiv · showhide
Estimating the 6D pose of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the observed image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using an untangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over state-of-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.
1 Introduction
DeepIM addresses challenging RGB-only 6D pose estimation by iteratively matching rendered object views to observed images and refining an initial pose. Its disentangled transformation representation supports accurate refinement, including for textureless and unseen objects.
- Motivation: RGB-only 6D pose estimation remains challenging, despite its importance for robot manipulation and virtual reality.Depth cameras have limitations involving frame rate, field of view, resolution, and depth range.
- Motivation: Pose classification and regression can handle textureless objects but lose accuracy because small classification or regression errors directly cause pose mismatches.Pose refinement addresses this by rendering an object from an initial estimate and matching the rendering against the input image.
- DeepIM: DeepIM predicts a relative SE(3) transformation between a rendered view and the observed image, then repeatedly applies the refinement to improve the pose estimate.The rendered image is regenerated after each update, and iterations continue until convergence or a preset iteration limit.
- Contributions: DeepIM uses a disentangled SE(3) representation and learns image-based refinement without handcrafted features, enabling pose matching for unseen objects.The method is evaluated on LINEMOD and Occlusion LINEMOD, with initial demonstrations on textureless T-LESS and unseen objects.
- Results: DeepIM reports large improvements over state-of-the-art RGB-only methods on both LINEMOD and Occlusion LINEMOD.The paper also reports accurate matching for textureless and unseen objects in initial experiments.
2 Related work
Prior RGB-based methods use local features, templates, keypoints, classification, regression, or learned representations, while depth-based methods perform geometric registration. DeepIM targets the remaining RGB-only refinement gap with iterative matching and a disentangled representation that supports unseen objects.
- RGB-based methods: Local-feature methods recover pose from 2D–3D correspondences but perform poorly on textureless objects because few local features can be extracted.They attach local features to a reconstructed 3D model and match them against features in the input image.
- RGB-based methods: Template-matching methods handle textureless objects by matching rendered or feature-based templates from different viewpoints against the input image.Templates can be generated from a 3D object model or represented using HOG features.
- RGB-based methods: Deep learning methods detect keypoints, learn feature representations, or classify and regress object poses from RGB images.Some approaches solve PnP after keypoint detection, while others detect objects and directly classify or regress their poses.
- Depth-based methods: Depth-based local registration methods such as ICP iteratively establish point correspondences and refine pose, but they are sensitive to inaccurate initial estimates.If the initial pose is sufficiently far from the correct pose, local refinement may converge to a local minimum.
- RGB versus RGB-D: RGB-D systems commonly estimate an initial pose from color and refine it with depth-based local methods such as ICP.This combination contributes to the performance gap between RGB-based and RGB-D-based methods described by the paper.
- DeepIM: DeepIM directly regresses relative SE(3) transformations using disentangled rotation and translation representations, allowing it to match unseen objects.The paper identifies the rotation reference frame as important for this capability.
3 DeepIM Framework
DeepIM refines an initial 6D pose by iteratively matching observed and rendered object images. It combines high-resolution inputs, a FlowNetSimple-based network, disentangled pose transformations, and point matching loss.
- 3 DeepIM Framework: DeepIM takes an observed image, a rendered image from the current pose, and their foreground masks as inputs for iterative pose refinement.At each stage, the object is rendered using the previous pose estimate before the network predicts an update.
- 3.1 High-resolution Zoom In: The observed and rendered images and masks are cropped with an enlarged, aspect-ratio-preserving bounding box and up-sampled to 480 × 640.The expansion ratio is fixed to 1.4, producing a high-resolution patch without image distortion.
- Network Architecture: DeepIM uses FlowNetSimple features to predict relative translation and rotation, with auxiliary optical-flow and foreground-mask branches during training.The pose branch uses fully connected layers for quaternion rotation and 3D translation prediction.
- 3.3 Disentangled Transformation Representation: The disentangled SE(3) representation decouples rotation and translation while making transformations independent of object size, model coordinates, and camera intrinsics.Camera-parallel rotation axes avoid object-dependent model frames, while image-space translation variables represent pixel motion and scale change.
- 3.4 Matching Loss: DeepIM trains pose updates with point matching loss, averaging the ℓ1 distances between model points transformed by ground-truth and estimated poses.Using ℓ1 rather than ℓ2 distances reduces large gradients from outliers and stabilizes training.
4 Experiments
Experiments show that DeepIM improves initial 6D pose estimates across benchmark datasets, initialization methods, design choices, and even unseen object categories. Its gains are strongest when using iterative training, zoomed inputs, disentangled transformations, and auxiliary prediction branches, while failures arise under difficult visual or initialization conditions.
- Robustness and limitations: DeepIM refines poses from different initial networks and can match unseen categories, but failures occur with model-image discrepancies, heavy occlusion, distant initialization, unusual shapes, or tiny components.On symmetric objects, ADD can understate performance because it treats only one valid pose as ground truth; ADD(-S) addresses this issue.
- Ablation studies: Iterative training is important: without it, additional test iterations do not improve results, whereas training and testing with two iterations already yields a large improvement.The authors use four iterations for subsequent experiments because the appropriate iteration count is dataset-dependent.
- Ablation studies: Zooming into the object bounding box provides very significant improvements through the higher input resolution.The comparison contrasts zoomed, up-sampled bounding boxes with full-image inputs.
- Ablation studies: The disentangled transformation representation enables unseen-object refinement and substantially outperforms the common camera-frame representation.Using a shared network across objects also performs better than individual regressors, while mask and flow branches achieve the highest ablation performance.
- State-of-the-art comparisons: DeepIM significantly improves color-only 6D pose estimation over prior methods on LINEMOD, including methods that already use refinement.The comparison includes BB8 and SSD-6D.
- State-of-the-art comparisons: DeepIM greatly improves PoseCNN estimates on Occlusion LINEMOD and surpasses other RGB-only methods by a large margin.It is also competitive with PoseCNN estimates refined using depth-based ICP.
5 Conclusion
DeepIM is a color-only iterative pose-matching framework that refines initial 6D pose estimates and can match unseen objects. The authors report strong performance against color-only methods and identify further applications and extensions.
- Framework: DeepIM directly outputs a relative pose transformation to iteratively improve an initial 6D pose estimate using color images.The network re-renders the object after each improvement so the rendered and observed images become increasingly similar.
- Representation: The disentangled pose representation is independent of object size and model coordinate frame, enabling pose matching for unseen objects.The paper reports this capability in experiments on previously unseen objects.
- Results: DeepIM significantly outperforms state-of-the-art 6D pose estimation methods using color images only.Its reported performance is close to methods using depth images for pose refinement, such as iterative closest point.
- Future directions: The authors suggest stereo DeepIM as a future direction for further improving pose accuracy.They also connect accurate color-only pose estimation with cameras offering high resolution, high frame rates, and a large field of view.