PaperlayerView exact revision on arXiv

Source-linked AI summary

EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

Hansheng Chen, Wei Tian, Pichao Wang, Fan Wang, Lu Xiong, Hao Li

arXiv:2303.12787v3cs.CV

TL;DR

Single-image object pose estimation requires learning 2D-3D correspondences, but complete end-to-end learning is difficult when PnP has ambiguous solutions and non-differentiable global optima. EPro-PnP replaces point-estimate PnP with a differentiable probabilistic pose layer trained by KL divergence, improving PnP-based systems across LineMOD and nuScenes while retaining stated limitations in inference and correspondence identifiability.

  • Problem

    Complete end-to-end learning of 2D-3D correspondences is difficult because ambiguous PnP problems have unstable local-minimum point estimates and non-differentiable global optima.

  • Method

    EPro-PnP represents PnP output as a differentiable pose distribution and learns weighted 2D-3D correspondences by minimizing KL divergence to a target pose distribution.

  • Results

    EPro-PnP reaches top-tier LineMOD 6DoF performance through CDPN integration and enables a deformable correspondence network with leading nuScenes single-frame image-based detection performance.

  • Takeaways & Limitations

    The probabilistic layer generalizes prior correspondence-learning approaches and supports attention-like weighting and new correspondence-network designs.

  • Takeaways & Limitations

    Inference still requires locating a posterior mode, and KL-only training leaves 2D-3D correspondences underdetermined and potentially difficult to learn without suitable architectural bias.

Abstract

from arXiv · show

Locating 3D objects from a single RGB image via Perspective-n-Point (PnP) is a long-standing problem in computer vision. Driven by end-to-end deep learning, recent studies suggest interpreting PnP as a differentiable layer, allowing for partial learning of 2D-3D point correspondences by backpropagating the gradients of pose loss. Yet, learning the entire correspondences from scratch is highly challenging, particularly for ambiguous pose solutions, where the globally optimal pose is theoretically non-differentiable w.r.t. the points. In this paper, we propose the EPro-PnP, a probabilistic PnP layer for general end-to-end pose estimation, which outputs a distribution of pose with differentiable probability density on the SE(3) manifold. The 2D-3D coordinates and corresponding weights are treated as intermediate variables learned by minimizing the KL divergence between the predicted and target pose distribution. The underlying principle generalizes previous approaches, and resembles the attention mechanism. EPro-PnP can enhance existing correspondence networks, closing the gap between PnP-based method and the task-specific leaders on the LineMOD 6DoF pose estimation benchmark. Furthermore, EPro-PnP helps to explore new possibilities of network design, as we demonstrate a novel deformable correspondence network with the state-of-the-art pose accuracy on the nuScenes 3D object detection benchmark. Our code is available at https://github.com/tjiiv-cprg/EPro-PnP-v2.

1 INTRODUCTION

EPro-PnP addresses the difficulty of learning complete 2D-3D correspondences for ambiguous poses by replacing point-estimate PnP with a differentiable pose distribution. It supports end-to-end learning and improves PnP-based performance across 6DoF pose estimation and 3D object detection.

  • Motivation: Differentiable PnP methods previously learned only 2D coordinates, 3D coordinates, or correspondence weights, leaving complete correspondence learning unresolved under pose ambiguity.Ambiguous PnP problems can contain multiple local minima, while differentiating a single pose estimate is unstable and the global optimum is not readily differentiable.
  • Proposed approach: EPro-PnP outputs a pose distribution instead of a single optimal pose, making its probability density differentiable with respect to learnable correspondences.The predicted and target pose distributions are trained with KL divergence, efficiently calculated using Adaptive Multiple Importance Sampling.
  • Results: EPro-PnP reaches top-tier 6DoF pose-estimation performance by inserting it into the CDPN framework.The cited contribution reports this result on the LineMOD benchmark.
  • Results: EPro-PnP supports deformable correspondence learning for accurate 3D object detection, where the entire correspondence set is learned from scratch.The extended experiments report that an enhanced EPro-PnP network leads single-frame image-based detectors on nuScenes.

2 RELATED WORK

Prior geometry-based methods represent objects with sparse keypoints or dense correspondences, while differentiable PnP methods backpropagate pose gradients to selected intermediate representations. EPro-PnP extends probabilistic learning to a complex continuous distribution derived from the PnP layer.

  • Geometry-based methods: Geometry-based pose methods use projection-constrained representations, including sparse keypoints such as box corners and dense correspondences.Examples include BB8 and RTM3D for 3D bounding-box corners, PVNet for sampled keypoints, and Deep MANTA for handcrafted templates.
  • Differentiable PnP: End-to-end differentiable PnP methods backpropagate pose-error gradients while learning 3D points, 2D keypoint locations, or correspondence weights.Brachmann and Rother learn 3D points, BPnP predicts 2D keypoints, and BlindPnP learns weights for unordered points.
  • Differentiable PnP: RePOSE learns feature-metric correspondences through a differentiable PnP solver but is insufficient under pose ambiguity.The paper uses it as a local regularization technique within its framework.
  • Probabilistic deep learning: Probabilistic deep-learning methods use distributions for continuous variables and mixtures to represent ambiguity, including ambiguous 6DoF pose.EPro-PnP extends this direction with a complicated continuous distribution derived from a nested PnP optimization layer and approximated by importance sampling.

3 GENERALIZED END-TO-END PROBABILISTIC PNP

EPro-PnP models PnP output as a differentiable pose distribution, enabling end-to-end learning of weighted 2D-3D correspondences under pose ambiguity. KL-based training is paired with Monte Carlo integration and derivative regularization to improve correspondence learning and pose optimization.

  • Probabilistic PnP: EPro-PnP predicts a pose distribution instead of a single PnP solution, providing differentiable probability density for ambiguous poses.The posterior is derived from the PnP likelihood, and multiple modes can represent pose ambiguity.
  • Probabilistic PnP: The method learns 2D coordinates, 3D coordinates, and correspondence weights jointly by minimizing KL divergence between target and predicted pose distributions.A narrow target distribution centered at the ground-truth pose yields a simplified loss involving reprojection errors at target and predicted poses.
  • KL Loss Function: The KL loss combines target-pose and predicted-pose reprojection terms, with the normalization term supplying the discriminative component needed to avoid point collapse.The target term alone can collapse all learned correspondences, whereas the second term penalizes incorrect predictions.
  • Monte Carlo Integration: AMIS improves Monte Carlo estimation by iteratively adapting proposal distributions to the loss integrand and reweighting samples from earlier iterations.Separate proposals are used for position and orientation because orientation is non-Euclidean.
  • Correspondence Learning: Learned correspondence weights balance inverse uncertainty with pose discrimination, emphasizing correspondences that are sensitive to pose variation.This discriminative component distinguishes EPro-PnP from uncertainty-only correspondence learning.
  • Derivative Regularization: Derivative regularization backpropagates through Gauss-Newton updates to direct the pose increment toward ground truth and mitigate KL-only training difficulties.It partially disentangles correspondences and is treated as secondary because it alone does not effectively address pose ambiguity.

4 IMPLEMENTATION DETAILS

The implementation stabilizes probabilistic PnP training with dynamic KL weighting, robust reprojection errors, and efficient initialization for ambiguous pose optimization. It uses adaptive sampling and GPU-batched hypotheses to seek strong starting poses before full-set refinement.

  • Dynamic KL weighting uses the reciprocal EMA of the summed-weight 1-norm to keep gradient magnitudes consistent across distribution entropies.The method compensates for the effect of correspondence-weight magnitude on gradients with respect to 2D coordinates.
  • Huber-robustified reprojection errors reduce sensitivity to outliers and improve expressiveness for multi-modal distributions representing pose ambiguity.An adaptive threshold depends on the 2D correspondence weights and coordinates, requiring rescaling of reprojection errors and the Jacobian.
  • A RANSAC-like random sampling algorithm searches for a global pose optimum because the LM solver only finds local solutions.It repeatedly samples weighted point subsets without replacement and solves pose hypotheses with LM.
  • GPU-batched subset hypotheses are refined efficiently, selecting the maximum-log-likelihood pose before subsequent LM iterations on the full correspondence set.Each subset pose can be solved in very few iterations, such as three, before full-set optimization.
  • During training, LM initialization is used by AMIS, since a poor local optimum can disrupt gradient-term balance and cause exploding gradients.The initial proposal location is especially important when the local optimum is far from the ground-truth pose.

5 6DOF POSE ESTIMATION BASED ON CDPN

EPro-PnP modifies CDPN to learn weighted 2D–3D correspondences with probabilistic pose supervision, including without geometry supervision. On LineMOD, combining KL and derivative-regularization losses reaches 67.36 accuracy, exceeding CDPN-Full’s 63.21 while using fewer parameters and faster inference.

  • 5.1 Network Architecture: The modified CDPN predicts dense 3D coordinates, two-channel correspondence weights, and a separately scaled global weight for PnP.Spatial Softmax focuses normalized weights on important image regions, while global scaling controls pose-distribution entropy.
  • 5.2 Training: EPro-PnP can train the entire correspondence network using KL and/or derivative-regularization losses, making geometry supervision optional.The KL-only setup uses randomly sampled dense points to reduce Monte Carlo overhead.
  • 5.4.1 KL Loss vs. Coordinate Regression: 61.87 versus 52.04: KL-only training outperforms the coordinate-regression baseline despite lacking ground-truth 3D-model supervision.This comparison is reported for B0 against A2.
  • 5.4.2 KL Loss and Derivative Regularization: 67.36 versus 63.21: combining KL and derivative-regularization losses surpasses CDPN-Full, while LineMOD shows limited pose ambiguity.Derivative regularization alone reaches 63.15 versus 61.87 for KL alone; pretrained fine-tuning later reaches 73.95 with coordinate regression retained.
  • 5.5 Comparison to Implicit Differentiation and Reprojection-Based Loss: Without object 3D models, EPro-PnP learns coordinates and weights from scratch, whereas implicit differentiation and reprojection loss fail to learn pose properly.The result supports using EPro-PnP as a general pose estimator without geometric priors.
  • 5.6 Efficiency and Visualization: Removing CDPN’s translation head reduces parameters from 113M to 27M and makes inference more than twice as fast.The comparison includes dataloading and uses batch size 32, despite introducing an iterative LM solver.

FORMABLE CORRESPONDENCE NETWORK

The deformable correspondence network extends FCOS3D with object queries, deformable attention, and learned 2D–3D points for probabilistic pose estimation. On nuScenes, it achieves leading single-frame monocular detection performance while modeling ambiguous orientations with multimodal distributions.

  • 6.1 Network Architecture: The network replaces FCOS3D’s direct 4DoF pose regression with object queries that generate correspondence predictions.Query appearance and position are disentangled in the adapted architecture.
  • 6.1 Network Architecture: Deformable attention samples interpolated dense feature maps to produce point-wise and object-level features.Point features predict NOC-space 3D points and Softmax-normalized weights, while object features predict confidence, scale, box size, velocity, and attributes.
  • 6.1.1 Implementation Details: The detector uses a ResNet-101-DCN backbone and predicts 128 2D–3D point pairs for each object.Training lasts 12 epochs with AdamW on nuScenes.
  • 6.2 Correspondence Losses: The deformable correspondences can be learned with KL divergence alone or together with derivative regularization and optional auxiliary losses.Auxiliary branches use reprojection and coordinate-regression losses implemented as Gaussian-mixture negative log-likelihoods.
  • 6.3 Dataset and Metrics: The nuScenes evaluation covers 1.4M annotated 3D boxes across 10 categories and reports NDS alongside translation, scale, orientation, velocity, and attribute errors.The dataset contains 1000 scenes split 700/150/150 for training, validation, and testing.
  • 6.4.1 Comparison Among Correspondence Loss Functions: 0.332 versus 0.607 orientation error: KL-only training substantially outperforms derivative-regularization-only training under nuScenes orientation ambiguity.Adding auxiliary losses does not make derivative regularization comparable to Monte Carlo KL loss.
  • 6.4.2 Comparison to the State of the Art: NDS 0.481 versus 0.470: EPro-PnP outperforms PolarFormer among ResNet-101 single-frame detectors without extra data.It also achieves mATE 0.559 versus 0.657 and mAOE 0.325 versus 0.405, and predicts multimodal orientation distributions for ambiguous objects.
  • 6.6 Inference Time: The batch PnP solver takes 26 ms/46 ms while processing 655.3 objects per frame before NMS.The measurement uses six surrounding 1600×672 images without test-time augmentation.

7 LIMITATIONS

The paper identifies training cost, dimensionality, and correspondence-network design as limitations of EPro-PnP. Its demonstrations also indicate that the evaluated LineMOD results largely reflect a setting without strong pose ambiguity.

  • 7 Limitations: Monte Carlo pose-loss training takes 241 seconds per epoch versus 143 seconds for coordinate-regression training, about 70% longer.The overhead can be adjusted through the number of Monte Carlo samples or corresponding points.
  • 7 Limitations: The Monte Carlo pose loss becomes impractical as the dimensionality of the nested optimization problem grows.The paper presents theoretical generalizability to other declarative networks but limits practical scalability.
  • 7 Limitations: Correspondence-network design remains important: removing 2D box size from the CDPN-based architecture notably decreases pose accuracy.The authors suggest feature-metric correspondence as a more expressive alternative to plain Euclidean reprojection error.

8 CONCLUSION

The conclusion frames EPro-PnP as a differentiable probabilistic replacement for non-differentiable PnP, supporting flexible end-to-end correspondence learning and practical integration into pose networks.

  • 8 CONCLUSION: EPro-PnP translates non-differentiable PnP into a differentiable probabilistic layer for end-to-end 2D-3D correspondence learning.The approach uses a pose distribution rather than a point estimate and discusses Monte Carlo KL and derivative regularization losses.
  • 8 CONCLUSION: The implementation provides a PyTorch batch Levenberg-Marquardt PnP solver for parallel GPU processing.The solver follows the Ceres implementation and supports Monte Carlo pose sampling and derivative regularization.
  • 8 CONCLUSION: Huber robustification rescales both weighted reprojection residuals and Jacobians before the LM step.The rescaling also affects derivative regularization and subsequent covariance estimation.
  • 8 CONCLUSION: The solver uses Gauss-Newton for fast inference because a large initialized trust-region radius makes LM effectively redundant in well-trained models.A small ε is used for numerical stability in the simplified implementation.
  • 8 CONCLUSION: During training, AMIS proposal concentration is determined from the local pose covariance around the LM solution.Translation proposals use a multivariate t-distribution with heavier tails, while the proposal location is determined by the LM solution and covariance.

B.1.2 Parameter Estimation from Weighted Samples

For yaw-only orientation, the proposal combines a von Mises distribution with a uniform component, then estimates its parameters from weighted samples.

  • B.1.2 Parameter Estimation from Weighted Samples: The proposal location is estimated as the circular mean of weighted samples.The weighted samples use importance weights vj and orientations θj.
  • B.1.2 Parameter Estimation from Weighted Samples: The yaw proposal is a mixture of von Mises and uniform distributions to represent orientation uncertainty.The uniform component captures potential additional modes under orientation ambiguity and has fixed mixture weight α = 1/4.
  • B.1.2 Parameter Estimation from Weighted Samples: The concentration is estimated from the weighted mean orientation vector and then divided by three for robust sampling.The weighted mean orientation vector norm provides the intermediate concentration estimate.
  • B.1.2 Parameter Estimation from Weighted Samples: The von Mises proposal is initialized at the PnP yaw estimate with concentration κ ← 1/(3σ²θ*).The initialization uses the estimated orientation variance.

B.2.2 Parameter Estimation from Weighted Samples

For quaternion orientation, EPro-PnP uses an angular central Gaussian proposal on the unit hypersphere and estimates its parameters from weighted samples.

  • B.2.2 Parameter Estimation from Weighted Samples: Quaternion proposals use a 4-dimensional angular central Gaussian distribution supported on the unit hypersphere.Its scale parameter is a 4×4 positive definite matrix Λ.
  • B.2.2 Parameter Estimation from Weighted Samples: Sampling from the angular central Gaussian is equivalent to normal sampling followed by normalization to unit radius.The underlying normal distribution has zero mean and covariance-related matrix Λ.
  • B.2.2 Parameter Estimation from Weighted Samples: The estimated inverse covariance is valid only in the rank-3 local tangent space at the predicted quaternion.The tangent-space constraint imposes l*ᵀΣ⁻¹l* = 0.
  • B.2.2 Parameter Estimation from Weighted Samples: Initial proposal parameters combine the local inverse covariance with identity regularization controlled by α = 0.001.The hyperparameter controls proposal dispersion for robust sampling.

B.3.2 Parameter Estimation from Weighted Samples

The deformable correspondence network combines weighted point sampling, attention-based feature processing, robust auxiliary losses, and object-level prediction heads for 3D detection.

  • B.3.2 Parameter Estimation from Weighted Samples: Derivative regularization comprises separate position and orientation losses.Position uses smooth L1 loss, while orientation uses cosine similarity based on angular distance for angle and quaternion parameterizations.
  • B.3.2 Parameter Estimation from Weighted Samples: The network samples N = nheadnhpts deformable points per object and predicts their 3D coordinates and 2D correspondence weights.Point features receive object-level context, positional encoding, self-attention, normalization, skip connections, and an FFN.
  • B.3.2 Parameter Estimation from Weighted Samples: A full Q-K dot-product attention layer aggregates sampled point features for object-level predictions.The outputs include localization score, weight scale, 3D bounding-box size, and optional velocity and attribute properties.
  • B.3.2 Parameter Estimation from Weighted Samples: The predicted 3D localization score represents positional uncertainty and is multiplied by the classification score to form the final detection score.It provides a faster alternative to Monte Carlo uncertainty evaluation during inference.
  • B.3.2 Parameter Estimation from Weighted Samples: An auxiliary branch densely predicts multi-head 3D coordinates and weights from RoI-aligned features inside ground-truth 2D boxes.Its attention weights come from Q-K dot-product attention and are normalized across heads and sampled points.
  • B.3.2 Parameter Estimation from Weighted Samples: The dense reprojection auxiliary loss is a Huber-robustified Gaussian-mixture negative log-likelihood.Per-head weights act as inverse square-root covariance terms, while head attention weights act as mixture-component weights.
  • B.3.2 Parameter Estimation from Weighted Samples: A second auxiliary loss regresses sampled 3D coordinates using sparse ground truth extracted from LiDAR point clouds.The coordinate loss is also formulated as a robustified Gaussian-mixture NLL.
  • B.3.2 Parameter Estimation from Weighted Samples: Projected LiDAR points additionally supervise a Gaussian-mixture scene-depth prediction produced by a two-layer MLP.Depth mixture weights are normalized with Softmax, and dynamic loss weights rescale the auxiliary objectives.

D.5 Experiments on the Uncertainty of Object Pose

The experiments assess whether the inferred pose-distribution entropy provides reliable localization confidence. Monte Carlo scoring matches the faster predicted-score implementation, supporting the reliability of the learned pose uncertainty.

  • Pose uncertainty is learned end-to-end through a KL-divergence-based pose loss rather than propagated from a surrogate reprojection uncertainty.The paper contrasts this with prior uncertainty propagation, which requires calibration and is described as unreliable.
  • Monte Carlo pose sampling provides a direct reference for evaluating localization confidence against the predicted 3D localization score.The comparison uses the PnP solution, sampled translations, and importance weights to compute the Monte Carlo score.
  • 0.393 vs. 0.392 mAP on the nuScenes validation set shows Monte Carlo scoring is on par with the standard predicted-score implementation.This parity indicates that the learned pose uncertainty is a reliable measure of detection confidence.
  • The notation defines target, proposal, prior, and posterior pose distributions, along with random pose samples, importance weights, and AMIS iterations.It also specifies symbols for point-pair counts, pose-sample counts, and the KL-divergence loss.
Loading 2303.12787v3…