Source-linked AI summary
DPOD: 6D Pose Object Detector and Refiner
Sergey Zakharov, Ivan Shugurov, Slobodan Ilic
TL;DR
RGB-only 6D pose estimation is challenging because of perspective ambiguities and changing object appearance. DPOD predicts dense multi-class 2D-3D correspondences, computes poses with PnP and RANSAC, and refines them with a deep network. It outperforms related detectors across synthetic and real training settings, with dense correspondences supporting accurate poses before and after refinement.
Problem
RGB-only 6D pose estimation remains challenging because perspective ambiguities and viewpoint-dependent appearance changes complicate inference.
Method
DPOD predicts object ID masks and dense 2D-3D correspondence maps, computes initial poses with PnP and RANSAC, and applies a deep learning-based RGB pose refiner.
Results
DPOD outperforms related works with synthetic-only and real-only training, while refined poses surpass other reported deep learning results.
Takeaways & Limitations
Dense correspondences support high-quality 6D poses both before and after refinement, while the complete pipeline operates using RGB data alone.
Takeaways & Limitations
Real training can bias detectors toward training-set lighting, poses, scales, and occlusions, whereas synthetic training faces a domain gap with real data.
Abstract
from arXiv · showhide
In this paper we present a novel deep learning method for 3D object detection and 6D pose estimation from RGB images. Our method, named DPOD (Dense Pose Object Detector), estimates dense multi-class 2D-3D correspondence maps between an input image and available 3D models. Given the correspondences, a 6DoF pose is computed via PnP and RANSAC. An additional RGB pose refinement of the initial pose estimates is performed using a custom deep learning-based refinement scheme. Our results and comparison to a vast number of related works demonstrate that a large number of correspondences is beneficial for obtaining high-quality 6D poses both before and after refinement. Unlike other methods that mainly use real data for training and do not train on synthetic renderings, we perform evaluation on both synthetic and real training data demonstrating superior results before and after refinement when compared to all recent detectors. While being precise, the presented approach is still real-time capable.
1. Introduction
RGB-only 6D pose estimation remains challenging because perspective creates ambiguities and object appearance changes across viewpoints. DPOD addresses this with dense 2D-3D correspondences, pose refinement, and evaluation using both synthetic and real training data.
- RGB-only 6D pose estimation is difficult because perspective ambiguities and viewpoint-dependent appearance changes complicate inference.
- DPOD predicts multi-class object ID masks and dense 2D-3D correspondence maps, producing many pixel-wise matches for pose estimation.The method uses automatically generated UV textures rather than manual correspondence annotations.
- A deep learning-based refinement network takes DPOD’s initial poses and enhances them while supporting training on synthetic or real data.The paper describes the refiner as faster and simpler to train than prior solutions.
- DPOD outperforms related works with either synthetic-only or real-only training, and one real-data refinement iteration surpasses all reported alternatives.The comparison includes SSD6D with depth-based ICP refinement.
- The paper presents an exhaustive experimental validation and comparison with recent approaches.
2. Related Work
Prior RGB 6D pose methods differ in their use of depth, training data, pose representations, pipeline stages, and refinement. DPOD instead integrates detection and pose estimation through dense correspondences in an end-to-end pipeline.
- Depth information helps disambiguate object scale, making RGB-only detection and 6D pose estimation particularly challenging.
- Template matching methods search input images against rendered viewpoint databases, with LineMOD serving as an influential benchmark and dataset.
- Deep learning RGB detectors use varied strategies, including discrete viewpoint classification, bounding-box corner projection regression, segmentation, coordinate regression, and quaternion rotation prediction.
- Existing pose refiners generally operate on poses supplied by external detectors such as PoseCNN or SSD6D.
- DPOD differs by integrating detection and pose estimation based on dense correspondences and training with either real or synthetic data.The paper reports outperforming related approaches on the LineMOD and OCCLUSION datasets.
3. Methodology
DPOD prepares real or synthetic object data, learns dense correspondence maps from textured 3D models, and augments object patches on varied backgrounds. Its inference pipeline predicts masks and correspondences before PnP+RANSAC pose estimation.
- Data Sources: Real and synthetic data offer different trade-offs: real images resemble target objects, while synthetic renderings provide broad pose coverage without requiring pose annotations.Real-data training can bias detectors toward observed conditions, whereas synthetic data introduces a domain gap.
- Synthetic Training Data Generation: Synthetic training data is rendered from poses covering the object, with RGB and depth channels stored for each camera pose.The poses include half-sphere viewpoints and in-plane camera rotations from -30 to 30 degrees.
- Real Training Data Generation: Real training data uses annotated datasets split into 15% training and 85% evaluation, with selected poses covering the object from all sides.
- Detection and Pose Estimation: The pipeline predicts object ID masks and dense correspondence maps with an encoder-decoder network, then computes 6D poses using PnP+RANSAC.The ID mask selects correspondences for each detected object.
- 3.1.1 Correspondence Mapping: Two-channel correspondence textures create a mapping between model vertices and pixels, allowing predicted colors to identify corresponding surface vertices.
- Online Data Generation and Augmentation: Object patches are composited onto MS COCO backgrounds so the detector encounters varied scenes and is less likely to rely on fixed contextual features.Brightness, saturation, contrast, and Gaussian-noise augmentation are also applied.
4. Dense Object Detection Pipeline
DPOD separates inference into correspondence and pose blocks: it predicts object identities and dense UV-based 2D–3D correspondences, then estimates 6D poses from them.
- Correspondence Block: The correspondence block uses an encoder–decoder network with three heads to predict an ID mask and dense U/V correspondence maps from RGB images.The input is 320×240×3; decoder outputs preserve the original image resolution.
- Correspondence Block: Discrete classification of UV coordinates converges faster and produces higher-quality matches than direct continuous coordinate regression.The discretized output space is 256^2, and predicted 3D points remain on the object surface.
- Correspondence Block: The detector is trained with a composite loss combining mask, U-channel, and V-channel losses, with all three weights set to 1.The U and V losses use multiclass cross-entropy, while the mask loss uses weighted cross-entropy.
- Pose Block: The pose block maps detected image pixels to 3D model coordinates and uses PnP with camera intrinsics to estimate the 6D pose.The large correspondence set supports robust pose estimation with RANSAC.
5. Deep model-based pose refinement
The refinement network compares real-image and rendered-object features to update an initial pose, using pose-aware rotation and translation heads initialized for stable identity output.
- Training Objective: The refinement loss measures vertex-to-vertex distance between ground-truth and predicted object poses using a robust L1-norm ADD measure.CAD-model points are resampled each iteration, with sampling limited to 10,000 points for efficiency.
- Refinement Architecture: The refiner receives an image patch and a rendering of the object at the predicted pose through parallel ResNet branches, then subtracts their features before pose regression.The two branches use ImageNet-pretrained early layers.
- Pose Parameterization: Rotation is represented in object coordinates so identical-looking objects share the same rotation representation.This avoids requiring the network to learn the more complex transformation associated with world coordinates.
- Pose Parameterization: The rotation head refines an input quaternion, while the translation head refines image-plane X/Y, using the refined Z translation as an additional input.The rotation head stacks the initial quaternion with intermediate predictions before producing the final rotation.
- Training Initialization: Fully connected layers are initialized to reproduce the input pose at iteration zero, improving training stability and speed.The network therefore begins training with meaningful pose outputs before learning corrections.
6. Training Details
The implementation uses PyTorch, Adam optimization, and ImageNet-style pretrained initialization to address synthetic-to-real domain adaptation.
- Implementation: Experiments used PyTorch on an Intel Core i7-6900K CPU with an NVIDIA TITAN X Pascal GPU.The hardware setup supported the reported training and evaluation experiments.
- Optimization: Training used Adam with a constant learning rate of 3×10^-4 and weight decay of 3×10^-5.These optimizer settings were used throughout the experiments.
- Domain Adaptation: Synthetic-data training faces domain adaptation challenges because networks trained from scratch fail to generalize to real images.The paper addresses this with pretrained real-image classification layers that are frozen during training.
7. Evaluation
The evaluation measures 6DoF pose accuracy, detection performance, and runtime on the LineMOD and OCCLUSION datasets. DPOD achieves strong pose-estimation results across synthetic and real training settings, while refinement and dense correspondences further improve performance.
- Datasets: Evaluation uses LineMOD and OCCLUSION, standard datasets for object detection and pose estimation, with OCCLUSION testing multiple objects and severe occlusions.LineMOD contains single-object sequences with CAD models and ground-truth poses; OCCLUSION extends it with occluded objects.
- Evaluation Metrics: ADD measures the average Euclidean distance between model vertices transformed by predicted and ground-truth poses, with poses correct below 10% of model diameter.For symmetric objects, the metric uses the closest transformed model vertex rather than corresponding vertices.
- Pose Estimation: 50% of synthetic-data poses are correct without refinement, versus 28.65% for the second-best approach on LineMOD.The large correspondence set makes DPOD 5 times better than SSD6D and almost 2 times better than AAE in the reported comparison.
- Pose Estimation: 66.43% of synthetic-data poses are correct after deep-learning refinement, compared with 34.1% for the compared method.For real-data training, DPOD is second best after PVNet without refinement and performs on par with PVNet on most objects.
- Pose Refinement: 95.15% of real-data poses are correct after DPOD refinement, exceeding the compared refinement by 6.55% and outperforming DeepIM on DPOD poses.DeepIM reaches 91.8% when applied to DPOD poses, below the 95.15% result of DPOD’s refiner.
- Runtime: The detector runs at 33 FPS on average, with RANSAC as the bottleneck; one refinement iteration takes 5ms excluding rendering.Two refinement iterations suffice for synthetic data and one for real data.
8. Conclusion
DPOD combines multi-class object masks with dense 2D-3D correspondences for robust 6D pose estimation. The detector and refiner outperform related approaches across real and synthetic training settings, with a lightweight refinement architecture.
- DPOD regresses multi-class object masks and dense 2D-3D correspondences between image pixels and corresponding 3D models.
- Dense correspondences enable more robust and accurate 6D pose estimation than bounding-box projection regression or discrete pose classification.
- For real and synthetic training data, DPOD outperforms related works such as YOLO6D and PoseCNN by a large margin and performs similarly to PVNet.
- The pose refinement approach surpasses other deep learning-based refiners while using a simpler and more lightweight backbone architecture.
A.1. Implementation Details
The detector uses a reduced ResNet-like encoder-decoder architecture. Bilinear interpolation replaces deconvolution during upsampling to reduce parameters and computation.
- DPOD uses a standard ResNet-like encoder-decoder, specifically a reduced ResNet18-style model with fewer layers and pooling operations.
- Bilinear interpolation implements upsampling instead of deconvolution to decrease the number of parameters and required computations.
- The refinement network uses the same backbone architecture as the detector.
A.2. RANSAC Iterations
RANSAC iteration count affects pose quality and runtime, with 150 iterations selected as the best trade-off. Refinement allows strong results even with only 25 iterations.
- 150 RANSAC iterations provide the best trade-off between pose quality and runtime.Larger iteration counts generally did not improve results significantly but increased execution time.
- With one refinement iteration, 25 RANSAC iterations are sufficient to outperform state-of-the-art results.
- Increasing RANSAC iterations beyond this does not produce a considerable increase in pose quality after refinement.
A.3. Runtime analysis
The appendix reports runtime, correspondence quality, qualitative pose behavior, and design trade-offs for DPOD. Results emphasize efficient processing, accurate correspondences, robustness, and known limitations in instance separation and mapping complexity.
- Correspondence quality: The correspondence-quality evaluation reports mean, median, and standard deviation of absolute errors separately for real and synthetic data.Median error is considered more robust because large mean errors can result from outliers.
- Correspondence quality: Most correspondence imprecision occurs at object boundaries and structural edges where correspondence values change rapidly.
- Multiple instance detection: Overlapping instances of the same object class that form one mask region yield only one estimated pose instead of two.An additional contour regression head can distinguish such regions by separating the output contours.
- UVW mapping: UVW mapping improves correspondence quality for complex geometries and is fully automatic, but increases memory use, computational complexity, execution time, and requirements.Ablations showed almost identical LineMOD and OCCLUSION performance with slightly higher execution times and memory requirements.
- Qualitative results: DPOD produces high-quality poses and is reported as robust to occlusions and illumination changes on LineMOD and OCCLUSION.