Source-linked AI summary
PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation
Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, Jian Sun
TL;DR
6DoF pose estimation remains difficult under illumination changes, sensor noise, occlusion, and truncation, while direct regression can generalize poorly in nonlinear rotation space. PVN3D uses RGBD features, deep 3D keypoint Hough voting, instance segmentation, and least-squares fitting; it reports state-of-the-art results and large-margin improvements across benchmarks.
Problem
6DoF estimation is challenging under changing illumination, sensor noise, occlusion, and truncation, and direct pose regression can generalize poorly because rotation space is nonlinear.
Method
PVN3D predicts point-wise 3D offsets to object keypoints with a deep Hough voting network, jointly uses instance semantic segmentation, and fits pose parameters by least squares.
Results
PVN3D outperforms previous approaches by large margins across several datasets and achieves state-of-the-art performance on YCB and LineMOD.
Takeaways & Limitations
The analysis identifies 3D keypoints as a key factor for improving 6DoF pose estimation, with joint keypoint and semantic-segmentation training providing further gains.
Abstract
from arXiv · showhide
In this work, we present a novel data-driven method for robust 6DoF object pose estimation from a single RGBD image. Unlike previous methods that directly regressing pose parameters, we tackle this challenging task with a keypoint-based approach. Specifically, we propose a deep Hough voting network to detect 3D keypoints of objects and then estimate the 6D pose parameters within a least-squares fitting manner. Our method is a natural extension of 2D-keypoint approaches that successfully work on RGB based 6DoF estimation. It allows us to fully utilize the geometric constraint of rigid objects with the extra depth information and is easy for a network to learn and optimize. Extensive experiments were conducted to demonstrate the effectiveness of 3D-keypoint detection in the 6D pose estimation task. Experimental results also show our method outperforms the state-of-the-art methods by large margins on several benchmarks. Code and video are available at https://github.com/ethnhe/PVN3D.git.
1. INTRODUCTION
PVN3D addresses challenging single-image 6DoF pose estimation with deep 3D keypoint voting, instance segmentation, and least-squares pose fitting. The method uses RGBD geometry and reports state-of-the-art performance on YCB and LineMOD.
- 6DoF pose estimation recognizes an object's 3D location and orientation in a canonical frame for applications including robotic manipulation, autonomous driving, and augmented reality.
- Changing illumination, sensor noise, occlusion, and truncation make 6DoF estimation challenging, while direct pose regression can generalize poorly because rotation space is nonlinear.
- PVN3D predicts point-wise 3D offsets to selected keypoints with a deep Hough voting network and estimates pose through least-squares fitting.
- Instance semantic segmentation is jointly optimized with keypoint voting to handle multiple objects and mutually improve their predictions.Semantic information identifies object parts for offset learning, while offset size helps distinguish similarly appearing objects with different sizes.
- The method reports state-of-the-art 6DoF pose estimation performance on the YCB and LineMOD datasets.
- The analysis finds 3D keypoints boost pose estimation and that joint training with semantic segmentation can further improve performance.
2. Related Work
Related approaches estimate pose using templates, direct regression, 2D keypoints, or dense correspondences. PVN3D extends voting-based correspondence methods into 3D using RGBD information and combines voting with instance-aware processing.
- Template-based methods scan images for matched rigid templates but are not robust to clustered scenes.
- Direct DNN pose regression is difficult to learn and generalize because of rotation-space nonlinearity, motivating refinement and discretization strategies.
- Traditional and neural 2D keypoint methods estimate image keypoints before applying PnP, but classical methods cannot handle texture-less objects.
- Dense correspondence methods use per-pixel predictions and Hough voting to improve robustness to occlusion, although their output space is large.
- PVN3D uses RGBD features, keypoint, center, and semantic predictions, then clusters instance votes and fits 6DoF pose from predicted keypoints.
3. Proposed Method
PVN3D estimates 6DoF pose through a two-stage pipeline that detects object 3D keypoints with per-point voting and fits pose parameters by least squares. Instance semantic and center voting modules support multiple-object scenes, while joint training couples segmentation and keypoint learning.
- 3. Proposed Method: The pipeline extracts fused RGB appearance and point-cloud geometry features, predicts per-point offsets and labels, clusters votes by instance, and fits pose parameters from detected keypoints.A least-squares algorithm estimates rotation R and translation t from corresponding keypoints in camera and object coordinates.
- 3. Proposed Method: Instance semantic segmentation and 3D center voting distinguish objects with shared semantic labels in multi-object scenes.Center voting predicts offsets to object centers, while clustering separates instances with the same semantic label.
- 3. Proposed Method: The 3D keypoint module predicts Euclidean offsets from visible points to target keypoints, whose votes are clustered to obtain keypoint locations.The module uses seed-point coordinates and features, with an L1 loss supervising predicted offsets against ground truth.
- 3. Proposed Method: The network jointly trains keypoint detection, semantic segmentation, and center voting with a multi-task loss.The authors report that jointly training these tasks boosts the performance of each other.
- 3. Proposed Method: RGB and geometric information are fused using PSPNet with ResNet34, PointNet++, and a DenseFusion block before shared MLP modules process each point.The implementation samples N = 12288 points per RGBD frame and sets λ1 = λ2 = λ3 = 1.0.
- 3. Proposed Method: Keypoints are selected from 3D object models rather than using distant virtual bounding-box corners, which can increase localization errors.The method then matches detected camera-coordinate keypoints with corresponding object-coordinate points for rigid pose fitting.
4. Experiments
Experiments evaluate PVN3D on YCB-Video and LineMOD using ADD-based metrics and compare alternative pose formulations, keypoint selections, and multitask designs. PVN3D achieves strong benchmark performance, while 3D keypoints, joint segmentation training, and on-object FPS keypoints improve results and robustness.
- Datasets: YCB-Video contains 21 objects across challenging RGBD videos with varying lighting, image noise, and occlusions; LineMOD contains 13 low-textured objects in cluttered scenes.Both datasets use annotated 6D poses and instance masks, with synthetic images added during training.
- Evaluation metrics: ADD measures mean pair-wise distance between transformed object vertices, while ADD-S uses closest-point distances for symmetric objects; AUC summarizes accuracy across thresholds.The reported metrics include ADD-S AUC and ADD(S) AUC.
- Benchmark results: 6.4%: PVN3D exceeds PoseCNN+ICP on the ADD(S) metric, and 5.7%: it exceeds DenseFusion with iterative refinement on YCB-Video.PVN3D without iterative refinement surpasses competing single-view methods, while PVN3D+ICP performs better still; it also achieves the best reported LineMOD performance.
- Formulation comparisons: The 3D-keypoint formulation performs better than direct rotation-and-translation regression and dense correspondence prediction.The paper links this advantage to a smaller offset search space and easier keypoint detection in the camera system.
- Formulation comparisons: 13.7%: the 3D-keypoint formulation outperforms the 2D-keypoint formulation under ADD-S on YCB-Video.The paper attributes this to 3D modeling preserving object geometry that can overlap after 2D projection.
- Ablation studies: 9.2%: joint keypoint-offset and semantic-segmentation training improves extra-large-clamp mIoU, while center voting and clustering add 18.3% further improvement.Eight FPS-selected keypoints are reported as a good trade-off, and the 3D-keypoint method degrades more slowly under increasing occlusion than DenseFusion and PoseCNN+ICP.
5. Conclusion
The paper proposes a 3D keypoints voting network with instance semantic segmentation for 6DoF pose estimation. It reports large-margin improvements and argues that joint training is a promising direction.
- PVN3D combines deep 3D keypoint voting with instance semantic segmentation for 6DoF pose estimation.
- Jointly training 3D keypoint detection and semantic segmentation boosts the performance of both tasks.
- The method outperforms previous approaches by large margins on several datasets.
A.1. Architecture Details.
The feature extraction module combines RGB and point-cloud embeddings. It uses a ResNet-34 image encoder-decoder and a PointNet++ point-cloud network before feature fusion.
- The image embedding network uses an ImageNet-pretrained ResNet-34 encoder followed by four up-sampling decoder layers.
- The image network outputs 128-channel RGB features, while the PointNet++ point-cloud network also outputs 128-channel features.
- DenseFuion concatenates the RGB and point-cloud features for each sampled point.
A.2. Implementation: Models for LineMOD dataset.
Because LineMOD provides labels for only one target object per scene, the authors train separate models for each object. The segmentation module therefore predicts target-object and background labels.
- LineMOD scenes may contain multiple objects, but each scene provides a label for only one target object.
- The limited labeling makes training a single model for all LineMOD objects difficult.
- Following PVNet, the authors train models separately for each object.
- For each object, semantic segmentation predicts two labels: the target object and the background.
A.3. Implementation: PVN3D+ICP
PVN3D+ICP uses sampled scene points for prediction, then propagates semantic labels to unlabeled points before ICP. This addresses the sparse labeling available to ICP.
- PVN3D randomly samples 12288 points or pixels from the whole scene during training and inference.
- Only sampled points receive semantic labels from the instance segmentation module.
- For ICP, each unlabeled point is assigned the label of its closest labeled point.
A.4.1 Instance semantic segmentation
The section reports additional instance semantic segmentation results, visualizes detected keypoints, evaluates PVN3D on heavily occluded objects, and measures inference speed on LineMOD.
- Table 8 provides additional instance semantic segmentation results.
- Figure 6 visualizes some detected keypoints.
- On the heavily occluded Occlusion LineMOD dataset, PVN3D outperforms previous methods by a large margin.The dataset is additionally annotated from LineMOD and is harder for pose estimation because its objects are heavily occluded.
- 5 FPS is the overall runtime on the LineMOD dataset.Network forward propagation takes 0.17 seconds, and pose estimation takes 0.02 seconds per object during inference.