Source-linked AI summary
Single-Stage 6D Object Pose Estimation
Yinlin Hu, Pascal Fua, Wei Wang, Mathieu Salzmann
TL;DR
Existing 6D pose pipelines predict correspondences and then apply RANSAC-based PnP, leaving training misaligned with pose accuracy and preventing end-to-end optimization. This paper replaces that post-processing with a network that directly regresses pose from structured correspondence groups. Combined with existing correspondence extractors, the resulting single-stage frameworks systematically outperform their two-stage counterparts in accuracy and runtime.
Problem
Two-stage 6D pose estimation uses a surrogate correspondence loss rather than the final pose objective, is not end-to-end trainable, and relies on costly, poorly repeatable RANSAC.
Method
A network directly regresses 6D pose from candidate correspondence groups, ignoring within-group order while preserving the fixed order of 3D-keypoint groups.
Results
The resulting single-stage frameworks systematically outperform the original two-stage frameworks in both accuracy and runtime.
Takeaways & Limitations
Replacing RANSAC-based post-processing with the pose-regression network enables end-to-end trainable 6D pose frameworks that are accurate and repeatable.
Takeaways & Limitations
The pose-estimation network is less accurate than traditional PnP when very precise correspondences are available and is trained only for fixed 3D coordinate sets.
Abstract
from arXiv · showhide
Most recent 6D pose estimation frameworks first rely on a deep network to establish correspondences between 3D object keypoints and 2D image locations and then use a variant of a RANSAC-based Perspective-n-Point (PnP) algorithm. This two-stage process, however, is suboptimal: First, it is not end-to-end trainable. Second, training the deep network relies on a surrogate loss that does not directly reflect the final 6D pose estimation task. In this work, we introduce a deep architecture that directly regresses 6D poses from correspondences. It takes as input a group of candidate correspondences for each 3D keypoint and accounts for the fact that the order of the correspondences within each group is irrelevant, while the order of the groups, that is, of the 3D keypoints, is fixed. Our architecture is generic and can thus be exploited in conjunction with existing correspondence-extraction networks so as to yield single-stage 6D pose estimation frameworks. Our experiments demonstrate that these single-stage frameworks consistently outperform their two-stage counterparts in terms of both accuracy and speed.
1. Introduction
Existing 6D pose estimation commonly separates correspondence prediction from RANSAC-based PnP, but this two-stage design uses a pose-misaligned surrogate loss, is not end-to-end trainable, and can be slow or unstable. The paper introduces a correspondence-group network that directly regresses poses and enables single-stage systems.
- Two-stage methods first predict 3D-to-2D correspondences with a deep network, then compute pose using RANSAC-based Perspective-n-Point.
- The correspondence loss minimizes 2D projection error, whose relationship to pose accuracy is not one-to-one.
- Equal average 2D correspondence errors can produce substantially different pose estimates after RANSAC-based PnP.
- The two-stage process is not end-to-end trainable, while iterative RANSAC is time-consuming with many correspondences.
- Deep RANSAC is unsuitable for end-to-end pose estimation because outliers make it slow and correspondence ordering causes poor repeatability.
- The proposed network directly regresses 6D pose from correspondence groups, treating within-group order as irrelevant and keypoint-group order as fixed.
- Combining the network with two correspondence-extraction frameworks produces single-stage systems that systematically outperform the original methods in accuracy and runtime.
2. Related Work
Prior work largely follows correspondence detection followed by RANSAC-based PnP, but differentiable alternatives struggle with numerical stability, valid rotations, correspondence selection, or repeatability. The paper addresses the unresolved design problem with grouped feature aggregation tailored to unordered correspondence clusters.
- Traditional methods detect keypoints and apply RANSAC-based PnP, but often fail under severe occlusion and cluttered backgrounds.
- Deep RGB methods generally establish 3D-to-2D correspondences before RANSAC-based PnP because direct image-to-pose regression is less accurate.
- Making PnP differentiable is difficult because SVD can cause numerical instability, while DLT does not guarantee a true rotation and requires post-processing.
- Differentiable eigendecomposition does not address RANSAC correspondence selection, and deep RANSAC remains ill-suited because of poor repeatability.
- The architecture differs from PointNet by using grouped feature aggregation for clusters with ordered groups and orderless points.
3. Approach
The approach directly regresses 6D pose from correspondence clusters using an architecture that respects unordered correspondences within each fixed-order 3D-keypoint group. It combines shared local features, order-insensitive within-cluster aggregation, and global pose inference, and can be paired with correspondence-extraction networks.
- Single-stage framework: The framework can use either 3D-point-to-2D-point or 3D-point-to-2D-vector correspondences and can be combined with another deep regressor that extracts correspondence clusters from an RGB image.For point-to-vector correspondences, 2D locations are obtained from vector intersections.
- 6D pose from correspondence clusters: The method replaces classical PnP and RANSAC-based correspondence processing with a nonlinear deep network that directly predicts rotation and translation.The input consists of candidate 3D-to-2D correspondences for each 3D keypoint.
- Correspondence-set properties: The architecture treats correspondences within each cluster as unordered while preserving the fixed order of clusters corresponding to 3D keypoints.This design avoids discarding the known ordering of the 3D points.
- Network architecture: A global MLP processes the aggregated cluster representation and outputs the 6D pose as a quaternion and translation.The global module uses three fully connected layers.
- Network architecture: Each correspondence is processed by a shared three-layer MLP, after which max pooling aggregates features within each cluster and concatenation preserves cluster order.The aggregation is permutation-insensitive within clusters but not across them.
- Single-stage framework: The implementation uses a common cube to define 3D keypoints across objects and represents each input correspondence by image-grid coordinates and x/y offsets.This yields a 4D correspondence representation [x, y, dx, dy].
4. Experiments
Experiments evaluate the single-stage pose regressor on synthetic correspondences and challenging real datasets. The method is more robust to noisy correspondences, benefits from correspondence clustering, and improves accuracy and runtime over two-stage baselines.
- Experimental setup: The method is evaluated on synthetic data and real images from Occluded-LINEMOD and YCB-Video.Occluded-LINEMOD contains 8 objects, while YCB-Video contains 21 objects and about 130K real images from 92 video sequences.
- Synthetic data: Correspondences encode each grid-cell center and its offset to a projected 3D bounding-box corner.For each projected corner, the procedure creates correspondences from grid cells inside the object mask using [x, y, dx, dy].
- Synthetic data: When noise increases, the proposed method becomes more accurate and robust than RANSAC-based EPnP and P3P.EPnP and P3P show similar performance; the reported pose error is normalized by the target-object diameter.
- Synthetic data: Ignoring the fixed ordering of correspondence clusters significantly decreases accuracy.The comparison uses a simplified network with one max-pooling operation that does not model the order of keypoint-associated clusters.
- Synthetic data: With vector correspondences, the method is more robust to increasing noise than PVNet’s voting-based PnP.Voting-based PnP is more accurate at low noise, but the proposed method becomes more robust and accurate as noise increases.
- Real data: Replacing RANSAC post-processing with the proposed network improves both correspondence-extraction frameworks and outperforms state-of-the-art methods on real datasets.The method also runs nearly 10 times faster than PoseCNN and nearly 2 times faster than SegDriven and PVNet on YCB-Video.
- Limitations: The pose regressor is less accurate than geometry-based PnP when very precise correspondences are available and is trained only for fixed 3D coordinate sets.The authors identify generic PnP handling as future work.
5. Conclusion
The paper presents a single-stage approach that directly regresses 6D poses from candidate correspondences, with end-to-end training replacing the usual RANSAC-style procedure. Broader applicability and improved accuracy remain future-work goals.
- The approach uses a small network to convert candidate 3D-to-2D correspondences directly into a 6D pose.
- Combining the pose network with state-of-the-art correspondence methods enables end-to-end training and removes the cumbersome RANSAC-style procedure.
- Future work targets a more accurate and generic pose-estimation network usable in a broader context.