Source-linked AI summary
3D Local Features for Direct Pairwise Registration
Haowen Deng, Tolga Birdal, Slobodan Ilic
TL;DR
Pairwise 3D registration seeks the rigid transformation between point-cloud scans, but learned 3D descriptors lack local orientation cues. The paper learns pose-variant features and relative poses, then uses pose predictions for alignment. It reports improved pairwise registration and substantially lower runtime.
Problem
Learned 3D descriptors lack local orientation assignment, leaving pairwise registration dependent on nearest-neighbor queries and exhaustive RANSAC.
Source: §1 · p. 1Method
The method pairs invariant and pose-variant local descriptors with RelativeNet relative-pose predictions and a hypothesize-and-verify alignment scheme.
Source: §1 · pp. 1–2Results
6.83% improvement in pairwise registration over the state of the art accompanies a 20-fold runtime reduction.
Source: §1 · p. 2Takeaways & Limitations
Experiments on 3DMatch and Redwood report robust, faster registration and strong generalization to the challenging synthetic Redwood benchmark.
Source: §1 · p. 2, §5 · p. 8For non-invariant features, the method assumes pose action can be approximated by an additive linear operation in a sufficiently flat learned feature space.
Source: §3 · p. 4
Abstract
from arXiv · showhide
We present a novel, data driven approach for solving the problem of registration of two point cloud scans. Our approach is direct in the sense that a single pair of corresponding local patches already provides the necessary transformation cue for the global registration. To achieve that, we first endow the state of the art PPF-FoldNet auto-encoder (AE) with a pose-variant sibling, where the discrepancy between the two leads to pose-specific descriptors. Based upon this, we introduce RelativeNet, a relative pose estimation network to assign correspondence-specific orientations to the keypoints, eliminating any local reference frame computations. Finally, we devise a simple yet effective hypothesize-and-verify algorithm to quickly use the predictions and align two point sets. Our extensive quantitative and qualitative experiments suggests that our approach outperforms the state of the art in challenging real datasets of pairwise registration and that augmenting the keypoints with local pose information leads to better generalization and a dramatic speed-up.
1. Introduction
Learned 3D descriptors lack local orientations, leaving pairwise registration to nearest-neighbor queries and exhaustive RANSAC. The paper adds pose cues to local features and estimates relative transformations for faster alignment.
1. Introduction
- Current deeply learned 3D descriptors lack local orientation assignment, so pose estimation relies on nearest-neighbor queries and exhaustive RANSAC iterations.The authors describe that approach as neither reliable nor computationally efficient.Source: §1 · p. 1
- The method augments invariant PPF-FoldNet with a pose-variant sibling to learn pose-specific descriptors that decouple 3D structure from motion.The paper describes the resulting features as focused on pose variability up to a reasonable approximation.Source: §1 · pp. 1–2
- A hypothesize-and-verify scheme selects an alignment from a correspondence pool retrieved by mutually closest neighbors in latent space.The method uses direct network predictions to improve registration speed and accuracy.Source: §1 · p. 2
- RelativeNet estimates transformations between matching patches rather than seeking canonical local frames, which the authors describe as ambiguous and error-prone.They characterize relative motion estimation as more robust and easier to train than finding a local reference frame.Source: §1 · p. 2
- 6.83% improvement in pairwise registration over the state of the art accompanies a 20-fold runtime reduction.The authors attribute the gains to weak supervision that enables local features to provide rotation estimates, easing final transformation estimation.Source: §1 · p. 2
2. Related Work
Prior work spans handcrafted and learned local descriptors, global ICP alignment, and RANSAC-based correspondence rejection. This method sits between local feature learning and direct pairwise registration.
Local descriptors
- Handcrafted descriptors capture local geometry, while learned descriptors outperform them but do not attach local orientations to 3D patches.Noise, missing structures, occlusions, and clutter harm handcrafted descriptors; finding a unique, consistent 3D local frame is challenging.Source: §2 · p. 2
Pairwise registration
- ICP-based methods iteratively hypothesize correspondences and optimize rigid pose, but ICP is initialization-sensitive and tolerates only 15–30° misalignment.The passage also describes outlier-robust ICP as an open problem.Source: §2 · p. 2
- RANSAC-based methods reject erroneous keypoint matches before estimating pose from inliers, but may require many trials when inlier ratios are low.The passage notes that high expected confidence further increases the trial burden.Source: §2 · p. 2
- Direct registration alternatives include geometric hashing of point pair features and 4-point congruent sets, while this approach combines local feature learning with direct pairwise registration.Super4PCS is described as making 4-PCS efficient.Source: §2 · pp. 2–3
3. Method
The method separates local patch structure from pose, predicts relative motion without computing local reference frames, and uses correspondence-specific pose hypotheses to align point clouds. Its network pair, multi-task losses, and hypothesis-selection procedure support this direct registration approach.
3. Method
- A local patch is modeled as a transformed canonical patch, and the method targets patch-wise pose prediction rather than an absolute pose that requires an unreliable local reference frame.The relative-pose formulation uses paired scenes to avoid requiring a canonical local reference frame and to construct training labels.Source: §3 · pp. 3–4
3.1. Network Design
- The method predicts patch pose by separating structure from motion, using the difference between invariant and pose-varying latent features as a pose-focused representation.The authors assume the pose action on latent features can be approximated by an additive linear operation; the resulting difference represents motion up to a nonlinearity.Source: §3 · pp. 3–4, §3.1 · p. 4
- PPF-FoldNet provides rotation-invariant features, while PC-FoldNet uses point coordinates with the same architecture to produce features that vary with patch pose.The latent-feature difference is passed to RelativeNet to recover the relative rigid transformation.Source: §3.1 · p. 4
3.2. Multi-Task Training Scheme
- Training combines reconstruction, pose-prediction, and feature-consistency losses, with Chamfer distance training both auto-encoders and rotation loss supervising RelativeNet.Feature consistency draws on paired patches to bring features from corresponding local patches close in the embedding space; inputs are centralized and normalized, canceling translation for relative-rotation prediction.Source: §3.2 · pp. 4–5
3.3. Hypotheses Selection
- RelativeNet hypotheses are more centralized around the ground truth than standard RANSAC hypotheses for both rotation and translation.Figure 4 shows rotational components as 3D Rodrigues vectors in the first row and translational components in the second.Source: Figure 4 · p. 5, §3.3 · p. 5
- Each correspondence supplies a pose hypothesis, so the method can exhaustively verify a hypothesis set linear in the number of correspondences and refine the best-scoring inliers.Unlike standard RANSAC, which uses three correspondences to parameterize a pose, this approach can rely on a single correct match.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate local features and geometric registration on 3DMatch, then test transfer to Redwood using weights trained on real 3DMatch data. RelativeNet and the learned features improve registration results, with runtime and correspondence count depending on the matching strategy.
4.1. Evaluations on 3D Match Benchmark [56]
- The learned descriptors outperformed supervised PPFNet by approximately 12% and unsupervised PPF-FoldNet by approximately 6% in fragment-matching recall.The networks were architecturally identical to PPF-FoldNet; the authors attribute the improvement primarily to multi-task training and shape-pose decoupling.Source: §4.1 · p. 6
- 77.68% recall with the same correspondences was around 8% higher than RANSAC and 7% higher than PPFNet using RelativeNet’s direct pose predictions.This result was also around 3% higher than fragment-matching recall, including some pairs with less than 5% inlier ratio.Source: §4.1 · pp. 6–7
- RelativeNet achieved approximately 6.19% higher recall than the best USAC result and 2.61% better than standard RANSAC’s highest precision on 3DMatch.Table 2 compares learned features with RANSAC, our features with RANSAC variants, and direct pose prediction.Source: Table 2 · p. 7, §4.1 · p. 7
- RelativeNet took around 0.03 seconds per fragment pair—0.013 seconds for hypothesis generation and 0.016 seconds for verification—and was faster than Latent-RANSAC.The authors attribute the speed to generating and verifying fewer hypotheses than RANSAC methods.Source: §4.1 · p. 7, Table 3 · p. 7
- On Redwood, trained 3DMatch weights used without synthetic-data fine-tuning achieved about 8.5% higher recall than 3DMatch and 2.4% higher recall than CGF and CGF+CZK.The authors report that their method outperformed the other state-of-the-art methods on this benchmark.Source: Figure 6 · p. 8, §4.1 · p. 8
- Keeping more mutual neighbors generally increased registration recall, while Closest produced around 3,664 putative matches versus 334 for Mutual-1, increasing verification time.The authors frame this as a recall-versus-computation trade-off.Source: §4.1 · p. 8, Figure 5 · p. 8
5. Conclusion
The conclusion presents an end-to-end framework for local feature extraction and pose prediction. Experiments report improved correspondence quality, faster and more robust registration than RANSAC methods, and generalization to Redwood.
5. Conclusion
- On 3DMatch, multi-task training improved learned-feature quality and the correspondence sets used for registration.The framework combines local feature extraction and pose prediction end-to-end.Source: §5 · p. 8
- RelativeNet pose predictions made registration more robust and faster than several state-of-the-art RANSAC methods.Source: §5 · p. 8
- Performance on the challenging synthetic Redwood benchmark supported the authors’ conclusion that the method generalizes to unseen datasets.Source: §5 · p. 8
- The authors identify data-driven hypothesis verification as future work.Source: §5 · p. 8
A.1. Ablation Study
The ablations compare feature matching with individual training signals removed and with invariant versus equivariant features. Reconstruction is especially important for matching, while all training signals together yield the most robust features.
A.1. Ablation Study
- The ablation compared four otherwise identical networks trained for 10 epochs on the same data, varying which of reconstruction, feature-consistency, and pose-prediction losses were excluded.Source: §A.1 · p. 11
- Invariant PPF-FoldNet features yielded better fragment matches for registration than equivariant PC-FoldNet features.Invariant features produced many pairs with over 10% true matches, whereas equivariant features left many pairs below 5% inlier ratio; this finding was also verified by [19].Source: Figure 7 · p. 11, §A.1 · p. 11
- With all training signals, features were most robust and outperformed each variant missing a loss; without reconstruction loss, matching almost always failed.The absence of pose-prediction loss had the least negative influence on matching, but it was still needed for RelativeNet to predict relative pose.Source: §A.1 · p. 11, Figure 8 · p. 12
- Relaxing the correspondence criterion increased the number of putative matches and could improve recall, at the cost of more computation during registration.The average match count increased rapidly as more neighbors were retained.Source: Table 4 · p. 11, §A.1 · p. 12
A.2. Quantitative Results
RelativeNet’s pose hypotheses cluster nearer the ground-truth pose than RANSAC’s, and it registers challenging fragment pairs with very few correct correspondences. Pairwise estimates also produce rough but acceptable multi-scan alignments without final ICP refinement.
Multi-scan registration
- Pairwise estimates yield rough but acceptable Red Kitchen and Sun3D Hotel multi-scan alignments without final ICP refinement.The pairwise poses are fed to a global registration pipeline; the final global ICP step is deliberately omitted.Source: §A.2 · pp. 12, 15, 16, Figure 11 · p. 15, Figure 12 · p. 16
Distribution of hypotheses
- RelativeNet’s predicted pose hypotheses are centered more around the ground-truth pose than RANSAC’s in both rotation and translation.Figure 9 shows the distributions in 3D and in orthogonal 2D projections.Source: §A.2 · p. 12, Figure 9 · p. 13
Qualitative comparison against RANSAC
- RelativeNet succeeds on challenging fragment pairs where RANSAC fails, including a case with only two correct local correspondences.Those two correspondences do not meet RANSAC’s minimum inlier requirement.Source: §A.2 · p. 12, Figure 10 · p. 14