Source-linked AI summary
6-DoF Object Pose from Semantic Keypoints
Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G. Derpanis, Kostas Daniilidis
TL;DR
Single-image 6-DoF pose estimation must handle varied object appearance, object-instance or class variation, and uncertain keypoint detections. The paper combines convnet-predicted semantic keypoints with a deformable shape model and confidence-weighted pose optimization, achieving accurate recovery in instance- and class-based settings, including state-of-the-art PASCAL3D+ results.
Problem
Single-image continuous 6-DoF pose estimation must accommodate textured and textureless objects across instance-based and class-based settings.
Method
The method predicts semantic keypoints with a convnet, fits a deformable shape model to them, and incorporates heatmap responses to model detection certainty.
Results
The method accurately recovers 6-DoF pose in instance- and class-based scenarios and shows state-of-the-art viewpoint-estimation results on PASCAL3D+.
Takeaways & Limitations
The approach supports pose recovery across textured or textureless objects and both instance- and class-based scenarios from a single RGB image.
Abstract
from arXiv · showhide
This paper presents a novel approach to estimating the continuous six degree of freedom (6-DoF) pose (3D translation and rotation) of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior work, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training image data. Furthermore, the approach can be applied to instance- and class-based pose recovery. Empirically, we show that the proposed approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios with a cluttered background. For class-based object pose estimation, state-of-the-art accuracy is shown on the large-scale PASCAL3D+ dataset.
I. INTRODUCTION
The paper targets single-image continuous 6-DoF pose estimation across textured or textureless objects and instance- or class-based settings. It combines learned semantic keypoints with deformable shape modeling and reports accurate recovery, including state-of-the-art PASCAL3D+ results.
- Single-image 6-DoF pose estimation recovers an object's 3D translation and rotation, with applications including robotic manipulation.
- The approach combines convnet-predicted semantic keypoints with a deformable shape model to estimate continuous object pose.
- The same approach supports both instance-based and class-based pose recovery and does not require objects to be textured.
- The method addresses cluttered backgrounds without pose initialization and achieves state-of-the-art performance on PASCAL3D+.
III. TECHNICAL APPROACH
The technical pipeline assumes object localization is already available and focuses on keypoint localization followed by pose optimization.
- The pipeline includes object detection, keypoint localization, and pose optimization.
- Object detection is treated as a solved input stage rather than a focus of the method.
- The method assumes a bounding box supplied by an off-the-shelf detector such as Faster R-CNN.
A. Keypoint localization
Keypoint localization uses a stacked hourglass convnet to produce heatmaps for object keypoints, integrating information across spatial scales and refining predictions iteratively.
- The stacked hourglass network takes an RGB image and outputs one confidence heatmap per keypoint.
- Intermediate supervision follows the first module, while the second module's heatmaps provide the final keypoint-localization output.
- Two stacked hourglass modules perform bottom-up and top-down processing across resolutions to combine local and global appearance cues.
- Stacking the modules supplies iterative refinement of the network's keypoint estimates.
B. Pose optimization
Pose optimization fits a deformable 3D shape model to uncertain 2D keypoint detections while estimating camera pose and shape deformation.
- The method fits a deformable shape model because keypoint predictions may be imprecise and the exact test-instance model may be unavailable.
- Category-specific shape models use a mean 3D shape and PCA modes to represent possible intra-class variability.
- Optimization estimates rotation, translation, and shape-deformation coefficients from detected 2D keypoints.
- A Tikhonov regularizer penalizes large deviations from the mean shape.
- A diagonal weighting matrix incorporates keypoint-prediction uncertainty using localization confidence values.
1) Weak perspective model:
When camera intrinsics are unknown, the method uses a weak perspective model, optimizing pose and shape-related parameters from keypoint reprojection residuals.
- Weak perspective model: The weak perspective model is adopted when camera intrinsic parameters are unknown.It is usually a good approximation to full perspective when the camera is relatively far from the object.
- Weak perspective model: The parameter set is θ = {s, c, ¯R, ¯T}, where s is a scalar and ¯R and ¯T are the first two rows of rotation and translation.
- Weak perspective model: Block coordinate descent alternately updates variables, using closed-form least squares for s, c, and ¯T.The rotation update accounts for the SO(3) constraint and is optimized over the Stiefel manifold.
2) Full perspective model:
With known camera intrinsics, the method uses a full perspective camera model that minimizes ray-based reprojection residuals and alternates closed-form and rotation updates.
- Full perspective model: The full perspective camera model is used when camera intrinsic parameters are known.
- Full perspective model: Normalized homogeneous coordinates represent the 2D keypoints, while a diagonal depth matrix Z scales the corresponding 3D projections.
- Full perspective model: The full perspective objective minimizes distances from 3D points to rays crossing their corresponding 2D points.
- Full perspective model: Alternating optimization updates Z in closed form and rotation through orthogonal Procrustes analysis.Initialization by the weak perspective solution is used to avoid local minima.
A. Instance-based pose recovery: gas canister
The instance experiment evaluates pose recovery for a textureless gas canister using learned keypoints and a known 3D model. The full perspective solution outperforms EPnP and is more precise than the weak perspective solution.
- Experimental setup: The instance-based experiment targets pose recovery for a specific object, matching robotics settings where environmental objects are known.
- Experimental setup: A dataset of 175 RGB-D images of a textureless gas canister supported ground-truth pose and 2D keypoint annotation.Depth was used only for ground-truth generation, with 10 manually defined 3D keypoints projected into the images.
- Results: The hourglass network localized keypoints reliably across viewpoint variation and occlusion, including non-visible keypoints.The full-perspective solution aligned the projected 3D model more precisely than the weak-perspective solution.
- Results: Only 150 images were used to train the network from scratch, although overfitting remained a possible concern.The empirical results suggested that the model captured the object appearance well in this single-instance case.
- Results: The proposed full-perspective solution was much more precise than EPnP, while the weak-perspective solution was on average worse than EPnP.The reported RGB-only pose errors were described as suitable for a general grasping system.
B. Class-based pose recovery: PASCAL3D+
On PASCAL3D+, the approach establishes semantic keypoint correspondences across varied object instances and viewpoints, then estimates pose with improvements across most categories. Failures remain for boat and TV monitor, while two classes are excluded because of inconsistent annotations.
- Semantic correspondences: The network preserves semantic keypoint relations across class instances despite large appearance differences, extreme viewpoints, and intra-class variability.
- Pose estimation: The proposed method improves over the state of the art across most PASCAL3D+ categories in rotational error.The best results occur when the fine subclass and accurate CAD-model correspondence are known.
- Pose estimation: Confidence-weighted model fitting outperforms the uniform-weight baseline during pose estimation.
- Failure cases: Boat and TV monitor have higher errors than the state of the art, while dining table and motorbike are excluded because of inconsistent left-right annotations.TV monitor failures involve four coplanar keypoints; boat images often contain small objects with insufficient front-back cues.
C. Processing time
The pipeline is designed for near-real-time use, with keypoint localization and shape fitting forming the main processing stages illustrated by the qualitative results.
- Processing time: The full pipeline runs in under 0.3 seconds, including around 0.2 seconds for keypoint localization and less than 0.1 seconds for shape fitting.Timing was measured on an Intel i7 CPU, 8G RAM, and GeForce GTX Titan X GPU.
- Qualitative results: Figure 4 compares RGB inputs, convnet heatmaps, and 3D-model projections from weak-perspective and full-perspective pose estimates.The full-perspective model aligns better near the gas-canister handle.
- Semantic localization: Figure 5 visualizes eight images sampled from the top 50 responses for each diverse keypoint category.The figure examines semantic localization across class instances with varied viewpoints and intra-class appearance.
- PASCAL3D+ examples: Figure 6 presents PASCAL3D+ examples with RGB images, keypoint heatmaps, projected CAD models, and 3D CAD visualizations.
- Quantitative evaluation: Table II reports viewpoint-estimation median error in degrees.
V. SUMMARY
The paper proposes an efficient single-image method for continuous 6-DoF pose estimation by fitting a deformable shape model to convnet-predicted semantic keypoints. It investigates weak- and full-perspective settings and reports state-of-the-art viewpoint results on PASCAL3D+ with runtime under 0.3 seconds.
- Method: The method estimates continuous 6-DoF pose from a single RGB image using semantic keypoints predicted by a convnet.Its pose optimization fits a deformable shape model and uses heatmap responses to represent detection certainty.
- Camera models: Both weak-perspective and full-perspective camera models are investigated.
- Results: The method demonstrates state-of-the-art results for viewpoint estimation on PASCAL3D+ and supports instance-based evaluation.
- Efficiency: The implementation runs under 0.3 seconds, supporting near-real-time robotics applications.
- Limitation: Four coplanar keypoints can make TV-monitor pose estimation ill-posed in the weak-perspective case.