PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridPose: 6D Object Pose Estimation under Hybrid Representations

Chen Song, Jiaru Song, Qixing Huang

arXiv:2001.01869v4cs.CV

TL;DR

Keypoint-only 6D pose estimation can fail under occlusion and imperfect predictions. HybridPose combines keypoints, edge vectors, and symmetry correspondences with robust pose regression. It reports 47.5% ADD(-S) accuracy on Occlusion Linemod and 30 fps on a commodity workstation.

  • Problem

    Accurate 2D keypoint prediction and sufficient keypoint constraints for 6D pose regression often fail because of occlusion and prediction-network limitations.

  • Method

    HybridPose combines keypoints, edge vectors, and symmetry correspondences, then uses robust norms with pose initialization and refinement to filter outliers.

  • Results

    47.5% mean ADD(-S) accuracy is achieved on Occlusion Linemod, while the approach runs at 30 frames per second on a commodity workstation.

  • Takeaways & Limitations

    HybridPose provides robust-to-occlusion, real-time 6D pose estimation using multiple intermediate representations.

  • Takeaways & Limitations

    The authors corrected a previous experimental setup and dataset split after finding them inconsistent with baseline approaches.

Abstract

from arXiv · show

We introduce HybridPose, a novel 6D object pose estimation approach. HybridPose utilizes a hybrid intermediate representation to express different geometric information in the input image, including keypoints, edge vectors, and symmetry correspondences. Compared to a unitary representation, our hybrid representation allows pose regression to exploit more and diverse features when one type of predicted representation is inaccurate (e.g., because of occlusion). Different intermediate representations used by HybridPose can all be predicted by the same simple neural network, and outliers in predicted intermediate representations are filtered by a robust regression module. Compared to state-of-the-art pose estimation approaches, HybridPose is comparable in running time and accuracy. For example, on Occlusion Linemod dataset, our method achieves a prediction speed of 30 fps with a mean ADD(-S) accuracy of 47.5%, representing a state-of-the-art performance. The implementation of HybridPose is available at https://github.com/chensong1995/HybridPose.

1. Introduction

HybridPose addresses unreliable keypoint-only pose estimation by combining keypoints, edge vectors, and symmetry correspondences, then robustly regressing pose from them. It reports improved accuracy over keypoint-based methods while retaining real-time efficiency.

  • Keypoint-based pose estimation assumes accurate 2D predictions and sufficient constraints for 6D regression, but occlusion and network limitations can violate both assumptions.
  • HybridPose combines keypoints, edge vectors, and dense symmetry correspondences to represent diverse geometric information from an RGB image.Edge vectors encode spatial relations among object parts, while symmetry correspondences incorporate interior details.
  • Hybrid representations provide additional pose constraints and can support accurate prediction when some predicted elements are outliers, such as under occlusion.Symmetry correspondences also stabilize rotation, especially along the reflection-plane normal direction.
  • Trainable robust norms prune outliers during pose regression, combining pose initialization and refinement to improve the resulting object pose.The method separately trains the pose prediction and pose regression modules using training and validation sets.
  • 47.5% ADD(-S) accuracy is achieved on Occlusion Linemod, improving over DPOD while evaluating two benchmark datasets.The reported evaluations use Linemod and Occlusion Linemod.
  • 30 frames per second is achieved on a commodity workstation despite the accuracy gain, using a relatively simple network for hybrid representations.

2. Related Works

Prior pose-estimation methods use intermediate representations such as keypoints or dense object coordinates, each with distinct accuracy, sparsity, cost, or occlusion trade-offs. HybridPose combines multiple representations and uses robust regression to address prediction outliers.

  • Keypoint-based pipelines predict 2D keypoints and align them with 3D keypoints, but keypoints alone are a sparse representation whose accuracy gains are limited.
  • Dense object-coordinate representations provide 2D-3D correspondences and robustness under occlusion, but they are more costly and can be less accurate than keypoints.
  • HybridPose combines multiple intermediate representations to exploit their collaborative strength for pose estimation.
  • Its prediction networks output keypoints, edge vectors, and symmetry correspondences, followed by initialization and refinement in the pose regression module.Initialization solves a linear system; refinement uses a GM robust norm.
  • Intermediate-representation pose estimation is sensitive to outliers from occlusion and cluttered backgrounds, motivating weighting and robust-regression strategies.
  • Unlike recurrent refiners with fixed iteration counts, HybridPose considers critical points and the loss surface of a robust objective function.

3. Approach

HybridPose predicts keypoints, edge vectors, and symmetry correspondences, then combines them for robust 6D pose regression. Its regression module uses initialization, refinement, and outlier filtering to improve pose estimation when predictions are noisy or occluded.

  • Hybrid representation: HybridPose represents image geometry with 2D keypoints, edge vectors between adjacent keypoints, and dense pixel-wise symmetry correspondences.The three representations are predicted from an RGB image and collectively provide geometric constraints for pose estimation.
  • Prediction module: The prediction module uses a keypoint network, an edge-vector network, and a symmetry-correspondence network, with shared parameters except for the final convolutional layers.The keypoint branch uses an off-the-shelf architecture, while the other branches stabilize regression when keypoints are inaccurate or cluttered.
  • Hybrid representation: Edge vectors encode spatial relations among object parts, while symmetry correspondences capture interior details and stabilize rotation estimation.Symmetry correspondences are especially useful for estimating the reflection-symmetry plane despite many outliers.
  • Training: HybridPose trains the prediction and pose-regression modules using separate training and validation sets because end-to-end training generalized poorly under distribution shift.The predicted-element distribution differs between training and testing, making the fitted pose-regression model generalize poorly.
  • Pose regression: The pose-regression module combines all predicted representations in initialization and refinement, using robust norms to filter outliers.Initialization modifies EPnP with constraints from all three representations; refinement further models outliers and projection errors.
  • Representation complementarity: Stability analysis finds that keypoints significantly contribute to translation accuracy, whereas edge vectors and symmetry correspondences stabilize rotation regression.This supports the complementary role of the three intermediate representations.

4. Experimental Evaluation

HybridPose is evaluated on Linemod and Occlusion Linemod using ADD(-S), with experiments covering baselines, implementation, qualitative pose results, and representation ablations. It achieves strong accuracy and real-time inference, including under severe occlusion.

  • Experimental setup: HybridPose is evaluated on Linemod and Occlusion Linemod, with Occlusion Linemod containing more heavily occluded examples.The experiments use 15% of Linemod for training and the remaining data plus all Occlusion Linemod examples for testing.
  • Experimental caveat: The current version fixes a previous dataset split that was inconsistent with baseline approaches.The paper directs readers to the project’s issue discussions for related details.
  • Qualitative results: Figure 3 depicts accurate pose predictions for no, light, and severe occlusion, showing 50 randomly selected symmetry correspondences per example.The figure is qualitative and organizes examples by occlusion severity.
  • Extensions: The implementation can incorporate improved detectors and extend to multiple objects by extracting representations separately for each instance.The proposed extension uses instance-level rather than semantic-level segmentation masks.

2. In the ablation study, we compute and report the the an-

HybridPose is evaluated against established pose-estimation baselines and through intermediate-representation ablations on Linemod and Occlusion Linemod. Results show strong benchmark accuracy and gains from combining representations.

  • 91.3 and 47.5 are HybridPose’s average ADD(-S) accuracies on Linemod and Occlusion Linemod, respectively.
  • HybridPose outperforms all state-of-the-art approaches on Occlusion Linemod.
  • 47.5% improves on PVNet’s 40.8% ADD(-S) accuracy by 16.4% on Occlusion Linemod.
  • On Linemod, HybridPose outperforms PVNet consistently across all object classes and beats DPOD on six classes.
  • Adding symmetry correspondences to keypoints improves rotation while leaving translation error almost unchanged.
  • Adding edge vectors to keypoints and symmetry correspondences produces salient gains in both rotation and translation estimation.

5. Conclusions and Future Work

HybridPose combines multiple geometric intermediate representations for 6D pose estimation and achieves real-time prediction with improved accuracy. The authors identify robustness to occlusion and several extensions for future work.

  • HybridPose uses keypoints, edge vectors, and symmetry correspondences as intermediate representations for 6D pose estimation.
  • Experiments show real-time prediction and higher accuracy than current state-of-the-art pose-estimation approaches.
  • The authors characterize HybridPose as robust to occlusion.
  • Future directions include shape primitives, normals, planar faces, and consistency across representations as a self-supervision loss.

Supplemental Material: HybridPose: 6D Object Pose Estimation under Hybrid Representations

The supplemental material provides additional analysis of HybridPose’s pose-regression modules. It focuses on initialization, refinement stability, and sensitivity to noise in predicted representations.

  • The supplement gives detailed explanations of the initialization sub-module’s algorithm.
  • It analyzes the stability of the refinement sub-module.
  • It examines how the objective function’s optimal solution changes with noise in predicted representations.

1. Initial Solution for Pose Regression

HybridPose generalizes EPnP by converting keypoint, edge-vector, and symmetry-correspondence constraints into a linear affine pose system. Eigenvector-based initialization is followed by alternating optimization and projection onto valid rotations.

  • HybridPose generalizes EPnP by exploiting keypoints, edge vectors, and symmetry correspondences for pose estimation.
  • The edge-vector constraint uses an edge’s endpoint vertices and its predicted 2D vector to impose pose relations.
  • Keypoint, edge-vector, and symmetry constraints are linear in affine rotation and translation parameters.
  • Concatenating the constraints yields Ax = 0 with dimension (3|K| + 3|E| + |S|) × 12.
  • The method rescales edge and symmetry constraints using α_E and α_S to model their relative importance.
  • Following EPnP, HybridPose represents the solution using the smallest right singular vectors, choosing N = 4 under noisy predictions.
  • The refinement alternates SVD-based rotation optimization with linear optimization of latent coefficients.
  • The resulting matrix is projected onto SO(3), with det(R_init) > 0 enforced before computing translation.

2. Stability Analysis for Pose Refinement

The stability analysis relates prediction noise to pose variance under small-noise assumptions, then examines how keypoints, edge vectors, and symmetry correspondences constrain pose refinement. It shows that added representations can reduce variance, while edge and symmetry terms alone leave translation insufficiently constrained in the analyzed example.

  • Local stability framework: The analysis assumes small input noise and uses a low-order Taylor expansion to characterize local perturbations of the optimized pose.The pose is locally parameterized in 6D, while prediction noise includes keypoints, edge vectors, and symmetry correspondences.
  • Local stability framework: The framework approximates output-pose variance from the variance of prediction noise and the local derivatives of the optimization solution.It establishes local uniqueness under smoothness and positive-definite Hessian assumptions.
  • Pose-regression stability: Incorporating edge vectors and symmetry correspondences generally reduces the variance of the output pose.The analysis studies the interaction among objective terms associated with all three intermediate representations.
  • Pose-regression stability: Increasing the edge-vector weight from zero guarantees a positive variance reduction in both trace-norm and spectral-norm under the stated condition.The condition is satisfied when ΣKK = I and ΣKE = 0; uncorrelated keypoint and edge-vector noise is identified as a likely favorable case.
  • Example analysis: For the square-object example, edge vectors help reduce variance in the third dimension of the rotational component.The example analyzes an object whose surface normal aligns with the camera z-axis and selects an optimal edge-vector weight for one variance term.
  • Example analysis: Edge vectors constrain the full rotation, while symmetry correspondences constrain two rotational dimensions but do not constrain two translation dimensions in the simple model.This structure explains why using only edge vectors and symmetry correspondences produces poor object-translation results in that example.
Loading 2001.01869v4…