PaperlayerView exact revision on arXiv

Source-linked AI summary

Ex-Sim(3)-Reg: 2D-3D Correspondence Pruning via Extended Sim(3) Registration

Pei An, Muyao Peng, Junfeng Ding, Jiaqi Yang, Liangliang Nan

arXiv:2608.28096v1cs.CV

TL;DR

Learning-based I2P registration needs robust correspondence pruning because outliers and noisy monocular depth priors undermine existing approaches. The paper introduces Ex-Sim(3)-Reg, which models depth noise as correspondence-specific scale perturbations within extended Sim(3) registration. Experiments across four datasets show that the approach significantly outperforms state-of-the-art methods, with up to 24.7% higher registration recall.

  • Problem

    Existing I2P registration methods struggle with severe outliers, while depth prior-based pruning relies on inaccurate monocular depth estimates.

  • Method

    Ex-Sim(3)-Reg reformulates correspondence pruning as extended Sim(3) registration, models depth noise with correspondence-specific scale perturbations, and searches maximum cliques in extended compatibility graphs.

  • Results

    24.7% improvement in registration recall is achieved over state-of-the-art baseline methods across four public datasets.

  • Takeaways & Limitations

    Modeling depth-prior errors within the registration formulation enables robust pruning and accurate, stable camera pose estimation.

  • Takeaways & Limitations

    The method can recover at most a subset of the true inlier set, motivating future refinement of extended compatibility graphs.

Abstract

from arXiv · show

Learning-based image-to-point-cloud (I2P) registration has garnered increasing attention in recent years. Nevertheless, existing methods still struggle with severe outliers under challenging scenarios with unseen, low-inlier, or distorted cases. A fast and robust 2D-3D correspondence pruning method is therefore highly desirable. Recently, a promising scheme lifts 2D-3D correspondences to 3D-3D correspondences using depth priors, casting correspondence pruning as a Sim(3) registration problem. However, depth priors estimated from monocular images are inherently noisy, which undermines the reliability of this scheme. In this paper, to explicitly model non-negligible depth noise, we reformulate correspondence pruning as an extended Sim(3) registration problem and propose a simple yet effective pruning algorithm termed Ex-Sim(3)-Reg. We further provide a theoretical analysis to justify the effectiveness of our method. Extensive experiments on the 7-Scenes, RGBD-V2, ScanNet, and TUM datasets demonstrate that Ex-Sim(3)-Reg achieves up to \textbf{24.7\% improvement} in registration recall over state-of-the-art baseline methods. Code is released at github.com/anpei96/ex-sim3-demo

1 Introduction

Learning-based I2P registration remains vulnerable to incorrect correspondences and inaccurate monocular depth priors. Ex-Sim(3)-Reg addresses this by modeling depth noise through extended Sim(3) registration and improves pruning across multiple datasets.

  • I2P methods often generate many incorrect 2D-3D correspondences, destabilizing downstream tasks in unseen and complex real-world scenarios.
  • Depth prior-based pruning is limited by monocular scale ambiguity, which introduces non-negligible errors and degrades registration results.
  • Ex-Sim(3)-Reg reformulates pruning as extended Sim(3) registration by modeling depth noise as correspondence-specific scale perturbations.
  • The method approximates the underdetermined problem with scale-parameterized compatibility graphs and combines with RANSAC-based P3P for practical pruning.

3 Pruning as an Extended Sim(3) Registration Problem

The paper first formulates correspondence pruning using lifted 3D-3D matches and regular Sim(3) registration. It then extends the formulation with correspondence-specific scale factors to explicitly optimize for depth noise and improve stability.

  • 3.1 Problem Formulation: Correspondence pruning filters outlier matches from predicted 2D-3D correspondences and outputs inliers together with the camera pose.
  • 3.2 Pruning as a Regular Sim(3) Registration: Regular Sim(3) registration lifts each 2D-3D correspondence into a 3D-3D correspondence using predicted metric depth and camera back-projection.
  • 3.2 Pruning as a Regular Sim(3) Registration: The regular formulation jointly selects inliers and estimates depth scale, rotation, and translation by minimizing weighted 3D alignment error.
  • 3.3 From Regular Sim(3) to Extended Sim(3) Registration: Extended Sim(3) replaces the shared scale with si = s · δi, allowing each inlier to follow a slightly different similarity transformation.
  • 3.3 From Regular Sim(3) to Extended Sim(3) Registration: By explicitly optimizing depth noise, extended Sim(3) provides more stable I2P registration than regular Sim(3).

4 Pruning with Extended Sim(3) Registration

Ex-Sim(3)-Reg models depth noise through correspondence-specific scale perturbations and approximates the resulting underdetermined registration with scale-parameterized compatibility graphs. Maximum-clique inlier candidates are scale-corrected into an SE(3) problem, followed by pose estimation and refinement with RANSAC-based P3P.

  • Extended Sim(3) formulation: Ex-Sim(3)-Reg models depth noise as correspondence-specific scale perturbations and solves the resulting underdetermined extended Sim(3) pruning problem heuristically.The method is supported by a theoretical analysis of extended compatibility graphs.
  • Extended compatibility graphs: An extended compatibility graph uses a scale parameter ζ to connect correspondences whose scale-adjusted distance difference is at most d_thr.The first-order graph uses d_ij(ζ)=|‖p_i−p_j‖_2−‖ζ·(q_i−q_j)‖_2|, while the second-order graph applies element-wise multiplication with its two-step adjacency product.
  • Pruning prototype: Maximum-clique search identifies potential inliers, whose correspondences are scale-corrected by e_i=φ(d_i,ζ) to form an SE(3) registration set.The scale-corrected candidates are then passed to an off-the-shelf SE(3) registration algorithm to estimate camera pose and inlier labels.
  • Pruning algorithm: The algorithm samples K scale hypotheses uniformly around a coarse scale estimate, searches inlier candidates across the corresponding graphs, and estimates pose and inliers.The pose is subsequently refined with RANSAC-based P3P, which pre-filters correspondences using a reprojection-error threshold.
  • Theoretical analysis: Theoretical analysis shows that practical and ideal maximum cliques overlap on a subset of ground-truth inliers, and experiments verify this subset is sufficient for robust pruning in most real-world scenarios.The analysis interprets the union of practical graph cliques as an approximation to the ideal graph and restricts the search to the corresponding inlier subset.

5 Experiments and Discussions

Experiments evaluate Ex-Sim(3)-Reg across four indoor datasets and disturbed, unseen, and low-inlier settings. Results indicate robust pruning under noisy depth priors, with accuracy–efficiency trade-offs governed by key design choices.

  • Experimental Setup: Experiments use 7-Scenes, RGBD-v2, ScanNet, and TUM, with correspondences generated under disturbed, unseen, and low-inlier conditions.MATR generates correspondences, while comparisons include non-learning pruning methods and depth priors predicted by Depth Anything v2.
  • Main Comparisons: 28.2% IR and 18.0% RR gains over baseline are achieved on disturbed 7-Scenes scenes.On TUM Freiburg_02, Ex-Sim(3)-Reg improves RR by 20%, while all baselines remain below 5%.
  • Main Comparisons: Ex-Sim(3)-Reg remains robust across unseen and low-inlier evaluations, where regular Sim(3) and RANSAC-based P3P methods can fail.Cross-dataset tables assess transfer from 7-Scenes to RGBD-v2 and ScanNet; RGBD-v2 also evaluates extremely sparse inlier conditions.
  • Main Comparisons: The method recovers denser, geometrically consistent inlier sets and improves integrated I2P registration methods on ScanNet.It is combined with FreeReg, Top-I2P, and MinCD, supporting the practical value of the pruning pipeline.
  • Results Analysis: Standard Sim(3) pruning ignores depth errors, whereas Ex-Sim(3)-Reg models depth noise and delivers more consistent performance across settings.The analysis attributes instability in competing methods to noisy depth maps and strict or minimal-clique criteria that do not adequately model this distortion.
  • Ablation Studies: K = 6 balances accuracy and efficiency because performance saturates when K > 6 while computational cost continues increasing.FastMAC is 2.78× faster than MAC with a favorable accuracy–efficiency trade-off, and scale correction substantially improves accuracy and robustness.

6 Conclusions

The paper reformulates depth-prior-based correspondence pruning as an extended Sim(3) problem and proposes Ex-Sim(3)-Reg to solve it. Experiments across four public datasets show that the approach outperforms state-of-the-art methods, while its recoverable inlier set remains bounded.

  • Extended Sim(3) accounts for depth-prior errors in 2D-3D correspondence pruning, unlike regular Sim(3).
  • Ex-Sim(3)-Reg detects and corrects maximum cliques from scale-parameterized compatibility graphs, reducing the problem to SE(3) registration with high-inlier sets.
  • Theoretical analysis and experiments on four public datasets support the method’s effectiveness against state-of-the-art pruning methods.
  • The recovered inlier set is at most a subset of the true inlier set, motivating future refinement of extended compatibility graphs.

A More Theoretical Discussions

The supplementary analysis motivates extended Sim(3) through depth-error statistics and explains the scale-parameterized compatibility-graph search. It also describes the main computational steps and the chosen accuracy-efficiency trade-offs.

  • Depth-error scale perturbations follow a long-tailed distribution in complex 7-Scenes scenes, supporting extended Sim(3) over regular Sim(3).
  • Adaptive voting estimates a concentrated average scale, after which extra search around that scale estimates inlier-specific perturbations and inliers.The search interval is centered at the estimated scale and uses an empirically selected Δs = 0.15s̃.
  • Maximum cliques from compatibility graphs generated across scale hypotheses are merged into a large inlier candidate set.
  • The pipeline consists of scale-hypothesis sampling, inlier-candidate search, and pose estimation.FastMAC reduces standard MAC search complexity from O(N^3) to O(N^2), yielding overall O(KN^2) complexity; its runtime is 4–10 milliseconds.

B More Experimental Details and Analysis

The supplementary experiments evaluate Ex-Sim(3)-Reg across noisy, unseen, and low-inlier settings using registration metrics, runtime comparisons, and qualitative examples. The method improves practical registration accuracy but is slower than Sim(3)-TurboReg.

  • The evaluation compares Ex-Sim(3)-Reg with state-of-the-art pruning methods across noisy, unseen, and low-inlier dataset settings.RR thresholds vary by dataset, including 5 cm for 7-Scenes and TUM, 2.5 cm for ScanNet, and 10 cm for RGBD-v2.
  • The qualitative comparisons cover 7-Scenes, RGBD-v2, ScanNet, and TUM, with figures reporting correct and incorrect correspondences using green and red lines.
  • Relative translation and rotation errors are reported on 7-Scenes, TUM, ScanNet, and RGBD-v2 alongside inlier ratio and registration recall.
  • Integrating the pruning method into FreeReg, Top-I2P, and MinCD improves I2P registration performance on the evaluated datasets.
  • Ex-Sim(3)-Reg is slower than Sim(3)-TurboReg but has a better accuracy-efficiency trade-off than Sim(3)-SC2-PCR++ and Sim(3)-MAC.The authors plan a parallel search module to improve computational efficiency.

C Qualitative Comparisons and Failure Analysis

Qualitative results show accurate inlier recovery across diverse indoor scenes and stable behavior across timestamps. The method can nevertheless fail when its hyperparameters are poorly tuned or outliers have small reprojection errors.

  • Ex-Sim(3)-Reg consistently produces accurate inliers across 7-Scenes, RGBD-v2, ScanNet, and TUM indoor scenes.
  • In one ScanNet example, the method achieves an 80.5% inlier ratio despite existing methods struggling with many outliers.
  • The method demonstrates stable performance across different timestamps in the TUM qualitative evaluation.
  • Improper tuning of K, the scale interval, or the reprojection-error threshold can prevent separation of outliers with small reprojection errors.Reducing sensitivity to hyperparameter settings is identified as future work.

D Verifications of Different MDE Models

The experiments examine pruning across monocular depth estimation models and model sizes on TUM Freiburg_01. The proposed method remains robust across evaluated models, while unsuitable hyperparameters can degrade performance.

  • The proposed method remains robust across all evaluated monocular depth estimation models and consistently achieves the highest registration accuracy.The evaluation uses the models listed in Table 18 and reports results in Table 19.
  • Improper hyperparameter settings can cause noticeable degradation in the proposed pruning method.Figure 12 identifies this as a failure case and refers to Sec. A3 for analysis.
  • Registration recall is compared across Depth Anything V2 model sizes on the TUM Freiburg_01 scene.Table 23 reports this model-size comparison for different pruning methods.

E Other Explanation of Our Method

Ex-Sim(3)-Reg addresses depth-noise-induced misalignment by introducing correspondence-wise scale hypotheses into the Sim(3) pruning formulation. It approximates the resulting optimization with scale-parameterized compatibility graphs and an algorithmic procedure.

  • Motivation: Regular Sim(3)-based pruning assumes predicted depths contain little noise, but noisy depths misalign 3D correspondences even under the ground-truth transformation.This misalignment makes maximum clique search unreliable and can produce suboptimal inlier selection.
  • Scale-aware strategy: The method generates candidate scales and constructs multiple scaled versions of the 3D correspondences.Each correspondence can then be evaluated under alternative scale hypotheses.
  • Scale-aware strategy: The optimal scale for each correspondence minimizes the squared residual between its scaled 3D point and the transformed reference point.The objective is min over ζ_i in the candidate scale set of ||ζ_i·q_i − (R p_i + t)||_2^2.
  • Scale-aware strategy: Correspondence-wise scale compensation produces new 3D correspondences with reduced scale distortion.Figure 13 contrasts global-scale misalignment with improved alignment from multiple correspondence-specific scales.

F Limitations

The method remains constrained by computational cost and hyperparameter sensitivity. The paper identifies acceleration and adaptive tuning as directions for future work.

  • Runtime exceeds 70 ms, indicating that the method still has relatively high computational cost.Future work will explore accelerating the MAC search procedure on multiple extended graphs.
  • Improper hyperparameter settings can cause noticeable performance degradation.Figure 12 illustrates this sensitivity as a failure case.
  • Adaptive hyperparameter tuning is identified as an important direction for future research.
Loading 2608.28096v1…