Source-linked AI summary
PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation
Sida Peng, Yuan Liu, Qixing Huang, Hujun Bao, Xiaowei Zhou
TL;DR
Single-image 6DoF pose estimation remains challenging when objects are occluded or truncated, because sparse keypoint localization can fail on unseen points. PVNet predicts pixel-wise vectors and uses RANSAC voting plus uncertainty-aware PnP to estimate poses. It reports state-of-the-art performance across three benchmarks and robustness on truncated objects, with real-time efficiency.
Problem
Two-stage pose methods that regress sparse keypoint coordinates or heatmaps are sensitive to occlusion and truncation, where keypoints may be unseen.
Method
PVNet predicts pixel-wise unit vectors toward keypoints, localizes them with RANSAC voting, and supplies uncertainty distributions to an uncertainty-driven PnP solver.
Results
PVNet achieves state-of-the-art performance on LINEMOD, Occlusion LINEMOD, and YCB-Video, with ADD of 86.3% vs. 79% on LINEMOD and 40.8% vs. 30.4% on OCCLUSION.
Takeaways & Limitations
The vector-field representation supports keypoint localization for occluded, truncated, and even out-of-image keypoints, while uncertainty-aware PnP further improves pose estimation.
Takeaways & Limitations
Pose recovery can fail when visible parts do not provide enough information, particularly for small objects such as duck and ape; Occlusion LINEMOD and Truncation LINEMOD are testing-only datasets.
Abstract
from arXiv · showhide
This paper addresses the challenge of 6DoF pose estimation from a single RGB image under severe occlusion or truncation. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise unit vectors pointing to the keypoints and use these vectors to vote for keypoint locations using RANSAC. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occlusion LINEMOD and YCB-Video datasets by a large margin, while being efficient for real-time pose estimation. We further create a Truncation LINEMOD dataset to validate the robustness of our approach against truncation. The code will be avaliable at https://zju-3dv.github.io/pvnet/.
1. Introduction
6DoF pose estimation from a single RGB image is difficult under occlusion, truncation, appearance variation, and clutter. PVNet addresses keypoint-localization weaknesses by predicting pixel-wise directions, voting for keypoints, and using uncertainty-aware PnP, achieving strong benchmark performance.
- Motivation: 6DoF pose estimation recovers an object's 3D rotation and translation from a single RGB image despite occlusion, lighting, appearance, and background clutter.Accurate pose estimation supports applications including augmented reality, autonomous driving, and robotic manipulation.
- Limitations of prior methods: Two-stage keypoint-based methods achieve state-of-the-art performance but struggle with occluded or truncated objects because some keypoints are unseen.Predicting unseen keypoints by memorizing similar patterns remains difficult to generalize.
- Proposed approach: PVNet predicts pixel-wise unit vectors from object pixels toward keypoints, then uses RANSAC-based voting to localize their 2D positions.This vector-field representation uses visible local parts and spatial relations to infer invisible or out-of-image keypoints.
- Proposed approach: Uncertainty-driven PnP uses voting-derived spatial distributions to handle inaccurate keypoint predictions and identify consistent correspondences.RANSAC prunes outlier predictions while the estimated distributions provide keypoint-location uncertainty to the PnP solver.
- Results: 25 fps on a GTX 1080 Ti supports the method's use for real-time pose estimation.The reported runtime includes data loading, network inference, RANSAC voting, and uncertainty-driven PnP.
- Results: 86.3% vs. 79% ADD on LINEMOD and 40.8% vs. 30.4% on OCCLUSION demonstrate significant performance improvements over the state of the art.The approach was evaluated on LINEMOD, Occlusion LINEMOD, and YCB-Video, and a Truncation LINEMOD dataset was created.
2. Related work
Prior work includes holistic pose regression, keypoint-based two-stage pipelines, and dense voting methods. These approaches differ in their representations and data requirements, while keypoint and dense methods address pose estimation through intermediate geometric predictions.
- Holistic methods: Holistic methods estimate an object's 3D location and orientation directly from an image, using template matching or CNN-based regression.Direct 3D localization is difficult because images lack depth information.
- Keypoint-based methods: Keypoint-based methods first predict 2D object keypoints and then compute pose from 2D-3D correspondences using PnP.The intermediate keypoint task is relatively easier than directly estimating 3D localization and rotation.
- PVNet overview: Figure 2 summarizes PVNet's localization pipeline from image and architecture through pixel-wise vectors, semantic labels, voting hypotheses, and keypoint distributions.Brighter hypotheses have higher voting scores; red stars mark distribution means and ellipses show covariance matrices.
- Keypoint-based methods: CNN keypoint detectors improve handling of texture-less objects but can lose accuracy under global distractions such as occlusions and low-resolution feature maps.The cited methods use segmentation, YOLO, or low-resolution feature maps to estimate semantic keypoints.
- Dense methods: Dense methods let every pixel or patch predict an output and vote for the final result in a generalized Hough voting scheme.Some dense approaches predict 3D object coordinates or use densely sampled patches, while several require RGB-D data.
3. Proposed approach
The proposed approach combines PVNet’s pixel-wise vector-field keypoint localization with uncertainty-aware PnP to estimate 6DoF poses robustly under occlusion and truncation.
- PVNet uses a two-stage pipeline that detects 2D keypoints and then computes 6D pose parameters with PnP.
- Voting-based keypoint localization: PVNet predicts object labels and pixel-wise unit vectors pointing from each object pixel toward every keypoint.
- Voting-based keypoint localization: The vector-field representation can infer invisible keypoints from visible object parts and represent keypoints outside the input image.
- Voting-based keypoint localization: RANSAC-based voting intersects randomly selected vector pairs, generates keypoint hypotheses, and scores them using votes from object pixels.
- Keypoint selection: Surface keypoints are preferred over bounding-box corners because their hypotheses have smaller localization variance and support more stable PnP.
- Uncertainty-driven PnP: Voting hypotheses define a spatial probability distribution whose mean and covariance are used by an uncertainty-driven PnP solver.The solver minimizes Mahalanobis distance and initializes pose estimation with four keypoints having the smallest covariance traces.
4. Implementation details
The implementation uses a fully convolutional ResNet-18-based network, CUDA-accelerated voting and density estimation, and synthetic data augmentation for training.
- PVNet processes an H × W × 3 image and outputs pixel-wise unit vectors and class-probability tensors.
- The backbone is a pretrained ResNet-18 modified to preserve a feature map at H/8 × W/8 resolution.
- Hypothesis generation, pixel-wise voting, and density estimation are implemented using CUDA, while EPnP initialization uses OpenCV and final optimization uses Ceres.
- PVNet learns unit vectors with a smooth ℓ1 loss and semantic labels with softmax cross-entropy.
- Training uses 10000 uniformly sampled renderings and 10000 Cut and Paste images per object, plus online cropping, resizing, rotation, and color-jitter augmentation.Models are trained for 200 epochs with a learning rate initially set to 0.001 and halved every 20 epochs.
5. Experiments
Experiments evaluate PVNet across standard 6D pose benchmarks, ablations, occlusion, truncation, and runtime. PVNet achieves strong accuracy, robustness to occlusion, and real-time efficiency, while failures remain when visible regions lack enough pose information.
- Datasets: LINEMOD, Occlusion LINEMOD, and YCB-Video provide evaluations spanning clutter, heavy occlusion, lighting variation, noise, and truncated objects.Truncation LINEMOD randomly crops LINEMOD images so only 40% to 60% of each target object remains; Occlusion LINEMOD and Truncation LINEMOD are testing-only datasets.
- Ablation studies: Pixel-wise voting is more robust to occlusion than direct keypoint-coordinate regression in the matched comparison with Tekin [39].The comparison uses the same pipeline and keypoints: eight bounding-box corners plus the object center.
- Ablation studies: FPS-selected surface keypoints improve pose estimation over bounding-box corners, while increasing the set from 8 to 12 yields negligible additional benefit.Accuracy increases across 4, 8, and 12 surface keypoints, but the gap between FPS 8 and FPS 12 is negligible; FPS 8 is selected for efficiency.
- Ablation studies: Considering keypoint uncertainty improves pose accuracy, making FPS 8 plus uncertainty-driven PnP the final configuration used in later experiments.Without uncertainty, pose estimation uses EPnP; the final configuration is denoted OURS.
- Comparison with the state-of-the-art methods: On LINEMOD ADD(-S), PVNet outperforms methods without refinement by at least 30.32% and SSD-6D with edge-alignment refinement by 7.27%.Symmetric objects eggbox and glue are evaluated with ADD-S.
- Comparison with the state-of-the-art methods: On Occlusion LINEMOD, PVNet outperforms other methods by 10.37% in ADD(-S), with the improvement attributed to recovering occluded keypoints from visible parts.Qualitative results are shown in Figure 4, where green boxes denote ground truth and blue boxes denote predictions.
6. Conclusion
The framework combines PVNet-based keypoint localization with uncertainty-driven PnP for 6DoF pose estimation, improving performance especially on occluded or truncated objects.
- PVNet predicts vector fields followed by RANSAC voting, outperforming direct keypoint-coordinate regression, especially for occluded or truncated objects.
- Uncertainty-aware PnP further improves pose estimation by incorporating uncertainties from predicted keypoint locations.
- The approach achieves state-of-the-art performance on three widely used benchmark datasets.
- A new truncated-object dataset demonstrates the method’s robustness to truncation.