Source-linked AI summary
SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again
Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, Nassir Navab
TL;DR
The paper addresses RGB-only 3D instance detection and 6D pose estimation for known CAD models, tasks requiring precise localization and poses. It extends SSD with multiscale detection, discrete viewpoint and in-plane-rotation scoring, and synthetic-model training, producing RGB-based results competitive with or better than RGB-D methods while being many times faster.
Problem
3D model localization and 6D pose estimation require precise detection of known CAD-model instances and accurate poses, but RGB-based progress has lagged behind category-level image detection.
Method
The method extends SSD to RGB input, predicts 2D boxes and pools of 6D pose hypotheses from discrete viewpoints and in-plane rotations, and trains using synthetic 3D-model information.
Results
Across multiple benchmark datasets, the color-based approach is competitive with or surpasses state-of-the-art RGB-D detectors while being many times faster.
Takeaways & Limitations
RGB color can support competitive or superior 3D detection and 6D pose estimation without requiring depth as an input modality.
Takeaways & Limitations
Color deviations between synthetic models and real scenes, loose bounding-box regression, and small objects can reduce detection confidence or produce inaccurate poses.
Abstract
from arXiv · showhide
We present a novel method for detecting 3D model instances and estimating their 6D poses from RGB data in a single shot. To this end, we extend the popular SSD paradigm to cover the full 6D pose space and train on synthetic model data only. Our approach competes or surpasses current state-of-the-art methods that leverage RGB-D data on multiple challenging datasets. Furthermore, our method produces these results at around 10Hz, which is many times faster than the related methods. For the sake of reproducibility, we make our trained networks and detection code publicly available.
1. Introduction
The paper adapts SSD-style dense, multiscale detection to RGB-only 3D instance detection and 6D pose estimation, addressing the precision demands of known CAD-model localization. Across challenging benchmarks, the color-based approach is competitive with or surpasses RGB-D methods while being many times faster.
- 6D pose estimation requires precise detection of known 3D CAD-model instances and accurate poses for augmented reality and robotic manipulation.
- View-based 3D detectors sample discrete scene positions and match features against object views, making accuracy dependent on discriminative features and robustness to clutter, illumination, and occlusion.
- SSD reverses this sampling strategy by using dense image input and discretized overlapping boxes across multiple feature-map scales, enabling simultaneous classification in one pass.
- The proposed network uses synthetic 3D-model information, decomposes pose into viewpoints and in-plane rotations, and extends SSD to output 2D detections with 6D poses.
- Depth is treated as optional for hypothesis verification and pose refinement, while RGB-only evaluation is compared with RGB-D results using 2D and 3D error metrics.
- Across multiple benchmark datasets, the color-based approach matches or surpasses RGB-D state-of-the-art detectors while being many times faster.
- The network processes a 299 × 299 RGB image into six multiscale feature maps whose predictions classify objects, refine 2D boxes, and score viewpoint and in-plane rotation hypotheses.
2. Related work
Prior 3D detection methods largely rely on view-based matching, depth, or carefully sampled pose hypotheses. Their coverage, scale handling, computation, and generalization are constrained by discrete views and scene sampling, motivating an SSD-style alternative.
- View-based methods generate discrete object views, compute features at sampled scene positions, and match them to establish correspondences.
- Covering an object’s upper hemisphere can require 3115 template views, while hashing trades speed for accuracy and depth-based methods reduce view counts by roughly an order of magnitude.
- View-based methods cover only a small, predefined 6D pose space, so an untrained placement can fail unless the corresponding view was included.
- Adding views increases computation and matching ambiguity, while coarse scene sampling misses small objects and fine sampling increases computation and false positives.
- Related SSD extensions regress 3D bounding-box corners and azimuth or bin poses for category-level orientation, but differ in their task and training setup.
3. Methodology
The method extends SSD to detect known 3D model instances and estimate 6D poses from RGB images using synthetic training data, discrete pose representations, and pose refinement. It handles object symmetries and generates pose hypotheses from localized 2D detections.
- RGB images produce localized 2D detections, each associated with a pool of likely 6D poses that are refined and verified.Verification can operate in 2D or optionally in 3D when depth is available.
- Network architecture: The SSD-style network predicts object identity, discrete viewpoint, in-plane rotation, and bounding-box corner refinements across six feature-map scales.It produces 21,222 possible bounding boxes through a fully convolutional design.
- Network architecture: Viewpoint and in-plane rotation are classified rather than directly regressed, enabling multiple pose hypotheses and straightforward handling of symmetric or visually ambiguous views.The authors report that classification was more reliable than numerical pose regression in early experiments.
- Training stage: Training renders objects with random transformations over MS COCO backgrounds, assigns boxes with IoU > 0.5 as positives, and targets the nearest discrete viewpoint, in-plane rotation, and tight box corners.Augmentation changes brightness and contrast, while horizontal flipping is avoided because it can confuse views and produce incorrect poses.
- Training stage: The loss combines class, viewpoint, in-plane, and corner-fitting terms, using softmax cross-entropy for classification and smooth L1 for corner regression.Positive and hard-mined negative boxes contribute to the training objective.
- Training stage: Symmetric objects use restricted viewpoint samples, while semi-symmetric objects omit one hemisphere to reduce viewpoint ambiguity during training and testing.The discrete 6D pose space visualizes these allowable viewpoint assignments.
- From 2D bounding box to 6D hypothesis: For each 2D detection, the method parses confident views and in-plane rotations into 6D hypotheses, infers translation from projective geometry, and refines the poses.Offline renders at zr = 0.5m provide bounding-box references for estimating object distance; RGB uses edge-based refinement and RGB-D uses projective ICP.
4. Evaluation
The evaluation measures detection, pose accuracy, robustness across object and dataset conditions, and runtime. The RGB-based method achieves strong results against RGB-D methods, while limitations remain under occlusion, appearance shifts, and small scale.
- Evaluation setup: The method is evaluated on three datasets covering clutter, multiple instances, and varying occlusion conditions.The implementation uses TensorFlow 1.0 and cuDNN 5 on an i7-5820K with an NVIDIA GTX 1080.
- Detection performance: 13.8% higher performance than all related RGB-D methods is reported on the Tejani dataset using color only.Smaller partially occluded instances were sometimes missed, while wrongly sized boxes under stronger occlusion reduced recall.
- Detection performance: The second dataset produces mixed results: the method outperforms RGB-D methods for some objects but performs worse for others.Color and texture benefit larger feature-rich objects, whereas specular effects, exposure changes, and white balancing create appearance mismatches between synthetic and real data.
- Limitations: Small objects such as ape, duck, and cat show reduced recall and precision because box placement can leave blind spots and textureless appearance increases confusion with clutter.Color mismatch between synthetic models and scenes, including specular reflections, can also lower confidence below the detection threshold.
- Pose metrics: VSS evaluates pixelwise mask overlap more strictly than bounding-box IoU and is suited to assessing pose quality for augmented-reality applications.Pose experiments use the threshold yielding the highest F1-score and evaluate pose accuracy on correctly detected instances.
- Pose estimation: Additional refinement improves pose quality: contour-based refinement produces better poses, while depth-based refinement leads to very accurate poses in practice.Parsing more viewpoint samples improves accuracy more than adding in-plane rotations, though it increases refinement workload.
- Pose and multi-object results: On the second dataset, the method comes close to Brachmann et al. and can overtake its results for pure RGB-based poses.For the multi-object dataset, the reported AP is 0.38 versus 0.51 for Brachmann et al.; performance degrades gracefully as the global threshold changes.
- Runtime and scalability: Approximately 85ms is required for a single-object network prediction, with about 24ms of additional refinement and a total runtime around 10Hz.The network prediction scales well as the number of objects increases, while refinement costs more because every pose in each pool is refined.
Conclusion
The paper presents an SSD-style detector for 3D instance detection and full 6D pose estimation trained solely on synthetic model information. It reports that color-based detection can match or surpass RGB-D methods while operating around an order of magnitude faster.
- The method is presented as the first SSD-style detector for 3D instance detection and full 6D pose estimation trained on synthetic model information.
- Color-based detectors can match and surpass current state-of-the-art methods that leverage RGB-D data.
- The method operates around one order of magnitude faster than those related methods.