Source-linked AI summary
Deep Model-Based 6D Pose Refinement in RGB
Fabian Manhardt, Wadim Kehl, Nassir Navab, Federico Tombari
TL;DR
RGB-only 6D pose refinement must handle ambiguities, occlusion, and rough initial estimates without relying on depth or explicit appearance models. The paper trains a CNN with a differentiable contour-aligning loss, achieving robust refinement and pose accuracy close to RGB-D or ICP-style methods in the reported evaluations.
Problem
RGB-only 6D pose refinement must address viewpoint ambiguity, occlusion, illumination and appearance differences, while prior regression formulations struggle with multiple correct poses.
Method
A CNN predicts translational and rotational updates using a differentiable, parameter-free visual loss that aligns rendered and scene object contours without explicit correspondences or appearance models.
Results
The method robustly refines strong pose perturbations, reaches less than 5° error in over 60% of extreme ’bvise’ trials, and reports an average RGB-only VSS score of 83%.
Takeaways & Limitations
The approach supports real-time RGB-only pose refinement across varied shapes and textures, including occlusion, symmetry, visual ambiguity, and rough initialization.
Takeaways & Limitations
Exact translation along the Z-axis is difficult to estimate from RGB when object motion changes pixels only slightly, making depth information useful for that distance cue.
Abstract
from arXiv · showhide
We present a novel approach for model-based 6D pose refinement in color data. Building on the established idea of contour-based pose tracking, we teach a deep neural network to predict a translational and rotational update. At the core, we propose a new visual loss that drives the pose update by aligning object contours, thus avoiding the definition of any explicit appearance model. In contrast to previous work our method is correspondence-free, segmentation-free, can handle occlusion and is agnostic to geometrical symmetry as well as visual ambiguities. Additionally, we observe a strong robustness towards rough initialization. The approach can run in real-time and produces pose accuracies that come close to 3D ICP without the need for depth data. Furthermore, our networks are trained from purely synthetic data and will be published together with the refinement code to ensure reproducibility.
1 Introduction
The paper investigates RGB-only 6D pose tracking and refinement under ambiguities, occlusion, and rough initialization. It proposes a CNN with a contour-aligning visual loss that avoids explicit appearance models and hand-crafted tracking components.
- RGB-based 6D tracking must recover metric translation and rotation despite viewpoint ambiguity, occlusion, illumination changes, appearance differences, and large inter-frame motion.
- The work studies whether RGB alone can support 3D tracking and 6DoF pose refinement using a convolutional neural network.
- The proposed differentiable visual loss aligns object contours while implicitly optimizing metric translation and rotation.
- The CNN learns object appearance implicitly from synthetic renderings and scene images under changing illumination and occlusion, without requiring global or local appearance models.
- The method handles varied shapes and textures, geometrical symmetries, visual ambiguities, and very rough pose initializations without manual tweaking.
- Its parameter-free formulation avoids segmentation and correspondence-plus-RANSAC pipelines, while remaining robust to occlusion and approaching RGB-D results without depth data.
2 Related work
Prior RGB CAD-model tracking methods include correspondence-based, edge-based, contour-based, learning-based, and hybrid approaches. The paper relates its network to Spatial Transformer Networks but regresses rigid motion on continuous 3D points rather than affine transformations on discrete lattices.
- Early RGB tracking methods used 2D-3D correspondences or 3D edges with iterative, projective ICP-style updates, while later work tracked contours densely via level sets.
- Contour-based approaches followed projected model contours, with subsequent work improving reconstruction, convergence, GPU speed, and color segmentation.
- Learning-based RGB methods combined detection with pose estimation through pose pools, object-coordinate regression, or projected 3D bounding-box regression, sometimes followed by edge-based ICP.
- The proposed network differs from 2D and 3D Spatial Transformer Networks by regressing rigid-body motion on continuous 3D points to minimize visual error.
3 Methodology
The method learns RGB-only 6D pose refinements from paired scene and rendered patches, using a differentiable contour-alignment loss rather than explicit transformation or appearance matching. It combines shared feature processing with translation and quaternion regression, synthetic training, and optional iterative refinement.
- Training data: Training data pairs randomly rendered ground-truth scenes with hypothesis renderings generated by perturbing the true pose.The scene is composited over a random background, while the perturbed pose provides the refinement input.
- Pose update: The refinement update composes with the current pose through rotation multiplication and translation addition, with rotation represented using unit quaternions.The formulation avoids direct optimization in SO(3) by parameterizing rotations with quaternions.
- Proxy loss for visual alignment: The visual loss aligns projected object contours through a distance transform, reducing 3D alignment to 2D silhouette matching without explicit correspondences.Contour geometry is less sensitive to color and lighting deviations than appearance-based matching.
- Proxy loss for visual alignment: A bi-directional loss aligns hypothesis and scene contours in opposite directions to reduce local-minimum failures under stronger pose perturbations.The final loss was selected for experiments because it also showed faster training overall.
- Network and inputs: The network receives paired scene and rendered patches, processes them through separate InceptionV4 paths, then predicts 3D translation and normalized 4D quaternion updates.Both patches are resized to 224 × 224 × 3 before feature extraction and branch-specific regression.
- Efficiency and deployment: The fully convolutional design targets fast inference, while iterative deployment repeats refinement until the predicted update falls below a threshold.Training from scratch caused synthetic-render overfitting and drastic collapse on real imagery, motivating the reported training strategy.
4 Evaluation
The evaluation tests convergence under pose perturbations, tracking and refinement across several RGB and RGB-D benchmarks, including difficult occlusion, symmetry and appearance conditions. Results show strong robustness and real-time performance, while revealing depth-related and appearance-dependent limitations.
- Evaluation setup: 25ms per iteration enables real-time testing across three synthetic and real datasets.The evaluation covers convergence, Choi tracking, pose refinement on Hinterstoisser, Occlusion and Tejani, plus failure and category-level experiments.
- Pose perturbation: More than 60% of extreme 45° bvise rotations refined below 5° error, while more than 80% finished below 10° error.The method diverged in less than 1% of trials for this perturbation.
- Pose perturbation: Almost 50% of ape trials failed to recover rotation below 10° error, attributed to its small size, poor texture and non-distinctive shape.Despite the pose error, visual alignment was often very good.
- Pose perturbation: Translation error correlated almost linearly with initial and final pose, while difficult viewpoints diverged beyond some perturbation level.Views with sufficient visual overlap could remain easy to align across stronger perturbations.
- Benchmark refinement: On Hinterstoisser, the method outperformed 2D-based ICP by a large margin and was on par with 3D-based ICP using VSS.The method also performed on par with or better than SSD-6D plus 3D refinement except for cam and cat.
- Benchmark refinement: ADD increased from 6.2% to 28.5% on Occlusion despite strong occlusion, alongside reductions of approximately 8° rotational error and 4mm, 4mm and 28mm translational error along X, Y and Z.The reported improvement was relative to the original SSD-6D numbers.
- Detection refinement: On Tejani, the visual loss exceeded MSE-trained networks by 14% in VSS, while RGB-D tracking benefited from its additional depth channel.The dataset contains mostly geometrically symmetric objects, so VSS rather than absolute pose error was reported.
- Category-level tracking: The method learned to metrically track unseen cup, mug and bowl models from the same category, although less accurately than single-instance tracking.The experiment used eight CAD models and kept camera intrinsics constant.
5 Conclusion
The work presents a deep-learning approach to 6D model tracking in RGB and demonstrates it across multiple datasets and tracking scenarios. Future work will investigate generalization to other domains.
- The paper presents a deep-learning approach to 6D model tracking in RGB.
- The approach is demonstrated on multiple datasets for pose refinement and instance/category tracking.
- Future work will investigate generalization to other domains, including visual odometry.