Source-linked AI summary
Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation
Markus Oberweger, Mahdi Rad, Vincent Lepetit
TL;DR
Partial occlusions substantially reduce the accuracy of CNN-based 2D projection predictions, motivating a more robust pose-estimation approach. The paper predicts and accumulates heatmaps from many small patches, using least-squares learning to handle ambiguous appearances. The resulting pipeline outperforms more complex methods on challenging occluded-pose benchmarks.
Problem
Large occlusions and unknown occluders reduce pose-prediction accuracy, while similar-looking patches can correspond to different heatmaps.
Method
The method predicts heatmaps from many small patches, averages them after alignment, and uses least-squares training to represent ambiguous patch evidence.
Results
The pipeline outperformed existing methods on the Occluded LineMOD and YCB-Video datasets, with 17% average improvement in 2D Reprojection error below 5px from Feature Mapping.
Takeaways & Limitations
Aggregating patch-level heatmaps provides a simple pipeline for accurate 3D pose estimation under partial occlusion.
Takeaways & Limitations
The authors identify more robust heatmap merging, local-maxima pose estimation, object-mask patch sampling, and parallel prediction as extensions.
Abstract
from arXiv · showhide
We introduce a novel method for robust and accurate 3D object pose estimation from a single color image under large occlusions. Following recent approaches, we first predict the 2D projections of 3D points related to the target object and then compute the 3D pose from these correspondences using a geometric method. Unfortunately, as the results of our experiments show, predicting these 2D projections using a regular CNN or a Convolutional Pose Machine is highly sensitive to partial occlusions, even when these methods are trained with partially occluded examples. Our solution is to predict heatmaps from multiple small patches independently and to accumulate the results to obtain accurate and robust predictions. Training subsequently becomes challenging because patches with similar appearances but different positions on the object correspond to different heatmaps. However, we provide a simple yet effective solution to deal with such ambiguities. We show that our approach outperforms existing methods on two challenging datasets: The Occluded LineMOD dataset and the YCB-Video dataset, both exhibiting cluttered scenes with highly occluded objects. Project website: https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/robust-object-pose-estimation/
1 Introduction
The paper addresses the sensitivity of CNN-based 2D projection prediction to partial occlusions by aggregating heatmaps predicted independently from many small patches. A least-squares training strategy handles ambiguous patches, enabling pose estimation from accumulated predictions.
- Large occlusions and unknown occluders decrease the accuracy of predicted poses, even when networks are trained with occluded examples.
- The method predicts heatmaps for 2D projections from small image patches, then combines them to estimate pose despite patches covering occluders or background.
- Similar-looking patches from different object locations can correspond to different heatmaps, making individual patch predictions multimodal.
- Least-squares training makes ambiguous patches predict averaged heatmaps, which are then averaged across patches to resolve ambiguities.
- The method is evaluated against state-of-the-art methods on the Occluded LineMOD and YCB-Video datasets.
2 Related Work
Prior RGB-based pose methods use cascaded regression, segmentation, object parts, offset maps, or voting schemes to address pose estimation and occlusion. The proposed approach is positioned as a simpler deep-learning alternative to related multimodal and occlusion-handling methods.
- Some prior methods localize objects and regress surface coordinates, while others estimate 2D locations and rotations before applying pose refinement.
- Segmentation can improve robustness to occlusions, but one cited approach requires a highly complex model.
- Part-based methods predict 2D–3D correspondences from manually selected object parts, whose generality is unclear for the evaluated objects.
- A related offset-map method requires complex statistical measures to identify feature vectors supporting semantic parts, whereas this paper seeks a simpler solution.
- Other work modifies training to reduce large spatial filter supports or uses voting schemes for occlusion robustness, but these approaches do not fully eliminate occlusion influence.
- Hough Forests combine multimodal patch-based 2D localization with regression trees, while this method aims to combine multimodal prediction with deep-learning representation power.
3 Influence of Occlusions on Deep Networks
The occlusion study shows that partial occlusions alter activations throughout CNN and CPM layers, with deeper layers increasingly affected. These changes make projection predictions inaccurate, especially when projected points are occluded.
- Occlusion changes activations in all layers of both networks, and its influence increases with layer depth as receptive fields grow.
- The experiment compares a standard CNN and a three-stage Convolutional Pose Machine using object-centered images with and without artificial occlusions.
- For the standard CNN, more than 20% of the last feature map and over 45% of activations beyond the first fully connected layer change under occlusion.
- For the CPM, more than 29% of activations change in the last feature map, with alterations more localized to the occluded region.
- Standard CNN predictions are inaccurate for all projections, while CPM predictions are inaccurate when the corresponding 3D points are occluded.
4 Minimizing the Effect of Occlusions
The method trains a patch-based heatmap predictor to represent both unambiguous and ambiguous evidence, aligns and averages predictions from many patches, and uses their maxima for pose estimation. Background and occluder patches are trained to produce uniform heatmaps.
- Training: The predictor receives 32 × 32px patches and outputs 128 × 128px heatmaps for the 2D projections of the object’s 3D bounding-box corners.
- Training: Training targets are Gaussian heatmaps centered on expected projections for patches overlapping the object mask, with σ = 4px.
- Training: Patches outside the object mask and ImageNet patches use uniform heatmaps, reducing false positives from background and occluders at run time.
- Training: Least-squares loss makes similar-looking patches with different target heatmaps predict their average, preserving useful multimodal information.
- Run-Time Inference: At inference, predicted heatmaps are translated to align patch centers, averaged over sampled patches, and converted to 2D projections using global maxima.
- Run-Time Inference: The final pose is computed with PnP estimation using RANSAC on correspondences between 3D bounding-box corners and predicted 2D locations.
- Object Region Detection: A sliding-window procedure identifies the object-centered image region by accumulating heatmap scores over the full camera frame.
5 Evaluation
The evaluation uses color images on Occluded LineMOD and compares the method with recent approaches under severe occlusion. Results show strong performance, robustness without known occluders, and sensitivity to patch-size and patch-count choices.
- Evaluation setup: The evaluation covers Occluded LineMOD color images, using 2D Reprojection and AD{D|I} metrics alongside qualitative pose visualizations.Occluded LineMOD contains 1215 frames with eight severely occluded objects; the reported results use only color images.
- Occluded LineMOD results: Our method performs significantly more accurately than BB8, PoseCNN, Jafari et al., and Tekin et al. across all Occluded LineMOD sequences.The comparison uses the fraction of frames below 2D Reprojection-error thresholds; BB8 uses ground-truth detection, whereas ours does not.
- Occluded LineMOD results: The method significantly outperforms other methods on the ADD or ADI threshold and 2D Reprojection-error metrics.Table 1 reports the percentage of frames with ADD or ADI below 10% of object diameter and 2D Reprojection error below 5px.
- Occlusion-training analysis: Training with known dataset occluders benefits BB8 and CPM most, while the proposed method remains robust without advance occluder knowledge.Random geometric occlusions provide only limited, specific invariance for BB8 and CPM; dataset-object occluders give their best results but require knowing the occluders.
- Ablation analysis: 19% worse 2D Reprojection performance occurs with 8px patches, while 128px patches reduce the metric by 5%; accuracy flattens beyond 64 patches.Patch sizes from 25px to 40px perform similarly to the main result, and segmentation could reduce the required number of patches while retaining accuracy.
6 Discussion and Conclusion
The paper presents a simple pipeline that is inherently robust to partial occlusions by combining heatmaps predicted from small image patches. It outperforms more complex methods on two challenging datasets.
- Small image patches and merged heatmap contributions make the pose-estimation method inherently robust to partial occlusions.Heatmaps capture ambiguities from local patches, while their contributions are merged before pose computation.
- Heatmaps provide a convenient representation for capturing multiple possible predictions caused by ambiguous local patches.The paper relates heatmaps to density functions and uses them to represent ambiguities arising from small patches.
- The resulting simple pipeline outperformed much more complex methods on two challenging datasets.
- The approach could be extended through more robust heatmap merging, local-maxima-based pose estimation, object-mask-guided patch sampling, and parallel prediction.These extensions are presented as possible future improvements to the current pipeline.
Supplementary Material: Making Deep Heatmaps Robust to Partial
The supplementary material documents the evaluation metrics, visualizes occluded CNN feature maps, adds YCB-Video quantitative details, and provides further qualitative results.
- The supplementary material states the formulas used for evaluation metrics.
- It visualizes the effects of occlusions on feature maps.
- It reports additional quantitative evaluation details for the YCB-Video dataset.
- It provides additional qualitative results for the Occluded LineMOD and YCB-Video datasets.
1 Supplementary Material
The supplementary material defines the pose-evaluation metrics, examines occlusion effects and patch aggregation, and presents quantitative and qualitative results on both datasets.
- 1.1 Evaluation Metrics: The 2D Reprojection error measures distances between model points projected into 2D under the ground-truth and estimated poses.The projection uses the intrinsic camera calibration matrix K.
- 1.1 Evaluation Metrics: The ADD metric averages 3D distances between model points transformed by the ground-truth and estimated poses.
- 1.1 Evaluation Metrics: The ADI metric compares transformed model points with their closest corresponding 3D points for symmetric objects.
- 1.2 Visualization of Occluded Feature Maps: Occlusion influence increases with network depth because deeper layers have larger receptive fields, even with occlusion examples during training.The comparison covers a feedforward network and a three-stage Convolutional Pose Machine.
- 1.3 Number of Patches: Sampling more than 64 patches matches object-mask sampling performance, while accuracy begins to flatten beyond 64 randomly sampled patches.Aggregating more heatmaps produces distributions more robust to patches sampled from occluded regions.
- 1.5 Quantitative Results: On YCB-Video, the method performs better for almost all objects and significantly better on average across all metrics.The reported metrics include AUC, 2D Reprojection error below 5px, and ADD or ADI error below 10% of object diameter.