Source-linked AI summary
Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images
Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, Carsten Rother
TL;DR
RGB-D 6D pose estimation is difficult because occlusion and sensor noise complicate comparisons between observations and pose-specific renderings. The paper learns this comparison with a CNN-based probabilistic pose model trained by maximum likelihood, achieving considerable improvements on two heavily occluded datasets while generalizing across objects and backgrounds.
Problem
Occlusion and complicated sensor noise make observation-rendering comparisons unreliable for RGB-D 6D pose estimation.
Method
A CNN compares rendered and observed images as an energy function modeling the pose posterior, trained with maximum likelihood within the analysis-by-synthesis framework.
Results
The method achieves considerable improvements over the best previously published results on two heavily occluded datasets.
Takeaways & Limitations
Training on a single object is sufficient for the CNN to generalize to different objects and backgrounds.
Takeaways & Limitations
Accuracy is below the competitor on the Tool Box sequences, attributed to downsampling effects on the largest object.
Abstract
from arXiv · showhide
Analysis-by-synthesis has been a successful approach for many tasks in computer vision, such as 6D pose estimation of an object in an RGB-D image which is the topic of this work. The idea is to compare the observation with the output of a forward process, such as a rendered image of the object of interest in a particular pose. Due to occlusion or complicated sensor noise, it can be difficult to perform this comparison in a meaningful way. We propose an approach that "learns to compare", while taking these difficulties into account. This is done by describing the posterior density of a particular object pose with a convolutional neural network (CNN) that compares an observed and rendered image. The network is trained with the maximum likelihood paradigm. We observe empirically that the CNN does not specialize to the geometry or appearance of specific objects, and it can be used with objects of vastly different shapes and appearances, and in different backgrounds. Compared to state-of-the-art, we demonstrate a significant improvement on two different datasets which include a total of eleven objects, cluttered background, and heavy occlusion.
1 Introduction
The paper addresses 6D pose estimation in single RGB-D images by learning to compare observations with pose-specific renderings despite occlusion and sensor noise. It introduces a CNN-based comparison model and reports improvements over state-of-the-art methods, including generalization across varied objects and backgrounds.
- 6D pose estimation combines 3D rotation and 3D translation for object instances in a single RGB-D image.
- The proposed approach uses a CNN within a probabilistic framework to learn comparisons between rendered and observed images.
- Occlusion and missing or unreliable depth measurements make comparing observed RGB-D images with pose-specific renderings nontrivial.
- The method achieves considerable improvements over state-of-the-art methods for RGB-D pose estimation under heavy occlusion.
- The CNN does not specialize to particular object geometry or appearance and generalizes across objects with varied shapes, appearances, and backgrounds.
2 Related Work
The section reviews CNN- and analysis-by-synthesis-based approaches to object detection and pose estimation, then positions the paper as building on a random-forest framework while replacing its energy function with a learned CNN.
- Prior computer-vision work spans instance and category recognition, rigid and articulated objects, and both quantized and accurate 6D pose estimation.
- The paper models pose posteriors probabilistically as Gibbs distributions whose energy function is defined by a CNN.
- Brachmann et al. use a regression-classification random forest to predict object classes and 3D object coordinates for pixels, supporting pose estimation.
- Their analysis-by-synthesis energy compares rendered depth and object-coordinate images with recorded depth values and forest predictions.
- The same forest predictions support RANSAC pose estimation through object-class-probability sampling and 3D-3D correspondences.
- The proposed method retains the forest predictions and optimization scheme from but replaces its few-parameter energy with a roughly 600K-parameter CNN trained by maximum likelihood.
3 Method
The method estimates 6D object pose probabilistically by comparing observed RGB-D-derived inputs with renderings through a CNN-based energy function. It trains this model by maximum likelihood and uses Metropolis sampling to approximate posterior expectations during optimization.
- 3.1 The Pose Estimation Task: 6D pose H combines a 3 × 3 rotation matrix R with a 3D translation vector T in camera coordinates.The pose transforms coordinates from the rigid object's local system to the camera coordinate system.
- 3.1 The Pose Estimation Task: RGB-D observations comprise random-forest predictions and the recorded depth image, focusing the method on posterior modeling rather than prediction.The forest predictions include object probabilities and object coordinates.
- 3.2 Probabilistic Model: The posterior p(H|x; θ) is modeled as a Gibbs distribution whose CNN-based energy maps a pose and observations to a real value.The energy is computed by rendering the object at pose H, comparing the rendering with observed inputs, and evaluating the result with the CNN.
- 3.3 Convolutional Neural Network: The CNN assigns low energy to strong agreement and high energy to weak agreement between observed and rendered images.Rendered and observed images are supplied as separate input channels; the object is assumed to be present in the field of view.
- 3.4 Maximum Likelihood Training: Training uses maximum likelihood and stochastic gradient descent on labeled observations and ground-truth poses, with posterior expectations approximated by Metropolis samples.Metropolis proposes a new pose, then accepts it automatically when its posterior density is higher or probabilistically otherwise; rotational proposals account for SO(3).
- 3.5 Inference: At test time, the method seeks the MAP pose estimate by maximizing posterior density, using image-specific preprocessing computed once per image.The pipeline renders candidate poses, extracts matched cutouts, and feeds the resulting observed and rendered channels into the CNN.
4 Experiments
The experiments evaluate the method on two heavily occluded RGB-D datasets and compare it with prior approaches. Across the occlusion dataset, the method achieves higher average correct-pose accuracy than both and LineMOD.
- Training Procedure: Three CNNs are trained using single-object sequences from Toolbox 1, Cat 1, and Samurai 1, with separate validation sequences.The Samurai 1 sequence has its first 100 frames removed to increase the proportion of occluded frames.
- Comparison: The method improves by up to 9.23% over the re-implementation of and by 10.4% over the reported values in [5] on the occlusion dataset.The largest gain occurs for occlusion percentages between 50% and 60%.
- Comparison: On the dataset of Krull et al., the method improves by 10.97% over when averaged across objects.Evaluation uses the Toolbox 2, Cat 2, and Samurai 2 sequences because the other sequences were used for training and validation.
- Comparison: 72.98% of cases have the correct pose, compared with 63.24% for [5] and 48.84% for LineMOD [14].These are averages over all test and training objects on the occlusion dataset.
- Failure Cases: Failure cases arise from either the learned energy function or the optimization scheme, while Toolbox accuracy is lower than the competitor’s on the Krull et al. dataset.The authors attribute the Toolbox result to its larger size and stronger impact from downsampling.
5 Conclusion
The paper presents a CNN-based posterior model for 6D pose estimation that generalizes across objects and backgrounds, improving results on heavily occluded datasets. It also identifies broader applications and optimization-related failure cases.
- The method maps rendered and observed images to an energy value and trains this posterior model using maximum likelihood.
- Training on a single object is sufficient for the CNN to generalize across different objects and backgrounds.
- The method achieves considerable improvements over previously published pose-estimation results on two datasets featuring heavy occlusion.
- For occlusion levels between 50% and 60%, the method improves correctly estimated poses by over 20% relative to the re-implementation of [5].
- On the Krull et al. dataset, correctly estimated poses average 56.74% across all test and training objects.
- The CNN architecture could be investigated for pure-RGB pose estimation, object-class pose estimation, and comparisons between observed RGB and rendered depth images.
- Orange-frame failures can have lower ground-truth energy than estimated-pose energy, indicating that an improved optimization scheme may find a better pose.
- Learning a CNN to predict pose updates is proposed as future work that could replace refinement and might improve results.