Source-linked AI summary
Deep Global Registration
Christopher Choy, Wei Dong, Vladlen Koltun
TL;DR
Real-world 3D scans contain partial overlap, multiple objects, self-occlusion, and noise, challenging existing registration approaches. Deep Global Registration combines correspondence confidence prediction, differentiable Weighted Procrustes pose estimation, and robust SE(3) refinement, and outperforms classical and learning-based methods. The authors present it as a ready-to-use replacement for registration methods in scene reconstruction pipelines.
Problem
Existing end-to-end registration approaches have limited accuracy or applicability on partially overlapping real-world scans.
Method
The framework uses a 6D convolutional network for inlier detection, differentiable Weighted Procrustes for scalable registration, and gradient-based SE(3) optimization for pose refinement.
Results
Experiments show that Deep Global Registration outperforms both classical and learning-based registration methods on real-world data.
Takeaways & Limitations
The framework can serve as a ready-to-use plugin replacing alternative registration methods in off-the-shelf scene reconstruction pipelines.
Takeaways & Limitations
Current end-to-end registration approaches work well on object-centric synthetic datasets but fail on real-world data with partial scans, self-occlusion, noise, and small overlap.
Abstract
from arXiv · showhide
We present Deep Global Registration, a differentiable framework for pairwise registration of real-world 3D scans. Deep global registration is based on three modules: a 6-dimensional convolutional network for correspondence confidence prediction, a differentiable Weighted Procrustes algorithm for closed-form pose estimation, and a robust gradient-based SE(3) optimizer for pose refinement. Experiments demonstrate that our approach outperforms state-of-the-art methods, both learning-based and classical, on real-world data.
1. Introduction
Deep Global Registration addresses weaknesses of end-to-end registration on partially overlapping real-world scans with three differentiable modules for confidence prediction, pose estimation, and refinement.
- Real-world registration supports 3D reconstruction, tracking, pose estimation, and object detection, motivating accurate and fast alignment.
- End-to-end methods can lose spatial acuity through global pooling or rely on point and correspondence distributions that fail for partially overlapping scans.
- The framework combines a 6-dimensional convolutional network, differentiable Weighted Procrustes registration, and robust SE(3) pose refinement.
- The 6-dimensional network analyzes correspondence geometry and estimates each correspondence’s accuracy or inlier likelihood.
- Weighted Procrustes uses predicted inlier probabilities, passes gradients through correspondence weights, and reduces complexity from O(N^2) to linear.This enables dense correspondence sets rather than sparse keypoints and can increase registration accuracy.
- Experiments on real-world pairwise registration and scene reconstruction datasets show robustness, accuracy, and speed against classical and end-to-end methods.Training and experiment scripts are publicly available.
2. Related Work
Related registration pipelines span correspondence matching, outlier filtering, and pose optimization; Deep Global Registration targets later stages while remaining compatible with varied feature descriptors.
- Feature-based correspondence matching: Feature-based pipelines extract local or global geometric structure to produce descriptors and establish correspondences.
- Feature-based correspondence matching: Learned feature methods include global-pooling models and convolutional networks, alongside traditional handcrafted histogram features.
- Feature-based correspondence matching: The proposed modules are agnostic to feature extraction and primarily address subsequent registration stages.They are compatible with a wide variety of feature descriptors.
- Outlier filtering: Outlier filtering commonly uses RANSAC, branch-and-bound, semidefinite programming, or maximal clique selection, with accurate methods often requiring longer iteration.
- Outlier filtering: The framework uses a convolutional network to identify inliers and outliers in one feed-forward pass without iterative optimization.
- Pose optimization: Pose optimization minimizes an alignment objective, with ICP and FGR among methods using second-order optimization.
- Pose optimization: Recent end-to-end frameworks combine learned features with pose optimization and backpropagate through that optimization.
- Pose optimization: Weighted Procrustes reduces optimization complexity from quadratic to linear and enables dense correspondences for accurate real-world registration.
3. Deep Global Registration
Deep Global Registration aligns partial 3D scans by learning correspondence reliability, solving a weighted rigid alignment, and refining the resulting pose. Its design uses the geometric structure of correspondences while keeping the solver differentiable and scalable to dense matches.
- Pipeline: Pairwise registration aligns partial scans, and its accuracy and robustness often determine the accuracy of the final reconstruction.The pipeline begins with pointwise feature extraction and matching, followed by correspondence scoring, weighted alignment, and robust refinement.
- Correspondence Confidence: A high-dimensional convolutional network analyzes correspondence geometry and estimates each correspondence's accuracy or inlier likelihood.The architecture uses translation-invariant convolutions and is trained with binary cross-entropy against ground-truth inlier correspondences.
- Correspondence Confidence: A 3D correspondence is represented as a point in 6D space, where inliers lie on a geometry-determined lower-dimensional surface and outliers scatter outside it.The network predicts the likelihood that each putative correspondence is an inlier.
- Weighted Procrustes: Weighted Procrustes minimizes weighted squared correspondence error and produces rotation and translation estimates from the learned correspondence weights.The normalized weights form a diagonal matrix, and the closed-form solver returns an initial pose for the robust registration module.
- Weighted Procrustes: Passing gradients through correspondence weights instead of point coordinates enables optimization to scale to dense correspondence sets.The method assumes partial scans of the same scene have the same scale, although scaling extensions are described.
4. Robust Registration
The framework refines registration with a robust gradient-based module, a continuous 6D rotation representation, and a failure-detection mechanism that supports fallback registration when alignments may be unstable.
- Fine-tuning: The refinement module initializes pose optimization from the Weighted Procrustes prediction and uses a continuous rotation representation to avoid discontinuities.The representation uses six parameters rather than Euler angles or quaternions.
- Failure detection: The method estimates whether an alignment may be unstable by approximating the fraction of valid correspondences from filtered weights.Weighted Procrustes can become numerically unstable when valid correspondences are insufficient because of small overlaps or noisy correspondences.
- Failure detection: When the valid-correspondence fraction is low, the system falls back to more accurate but slower methods such as RANSAC or branch-and-bound.This fallback is triggered before returning a potentially unstable result.
- SE(3) representation: The 6D rotation representation maps parameters a1, a2 ∈ R3 to an orthogonal rotation matrix using normalization and cross products.The final pose representation consists of a1, a2, and translation t, equivalent to R and t.
- Fine-tuning: A robust loss function fine-tunes registration using predicted inlier correspondences.The energy is parameterized by rotation and translation and optimized with gradient-based methods such as SGD or Adam.
5. Experiments
Experiments evaluate pairwise and multi-way registration across real-world 3D scan datasets, comparing the proposed system with classical and learning-based methods. The approach achieves strong accuracy, recall, speed, and cross-dataset performance, while several learning-based and ICP baselines struggle on challenging real-world scans.
- Evaluation Setup: Experiments cover pairwise registration on 3DMatch, multi-way registration on RGB-D reconstruction datasets, and outdoor pairwise registration on KITTI.The evaluation includes real-world and simulated-depth datasets, with models trained on 3DMatch and tested elsewhere to assess generalization.
- Pairwise Registration: 6.5× faster than the most accurate baseline while outperforming baselines across rotation- and translation-error thresholds.Figure 4 reports overall pairwise registration recall as thresholds vary.
- Evaluation Metrics: Recall is the ratio of successful registrations meeting predefined rotation-error and translation-error thresholds; average TE and RE are computed only on successful pairs.Failed registrations can produce poses far from ground truth, making their error metrics unreliable.
- Pairwise Registration: The method outperforms classical and learning-based baselines on 3DMatch recall while achieving consistently low translation and rotation errors across scenes.The comparison includes ICP, RANSAC, FGR, Go-ICP, Super4PCS, 3DRegNet, DCP, PRNet, and PointNetLK.
- Baseline Analysis: ICP variants mostly fail under small overlap and large viewpoint changes, while DCP, PRNet, PointNetLK, and 3DRegNet show limitations on real-world partial or complex scans.DCP relies on a correspondence assumption that fails with partial overlap; PRNet training produced crashes and high-variance losses, and 3DRegNet did not converge.
- Classical Baselines: RANSAC reaches recall as high as 70% and FGR reaches 40%, while the proposed method is about twice as fast as RANSAC with 2M iterations and achieves higher recall and accuracy.Doubling RANSAC iterations improves performance only marginally.
- Multi-way Registration: Replacing pairwise registration in a multi-way pipeline yields consistently low absolute trajectory error across Augmented ICL-NUIM scenes compared with online SLAM and offline reconstruction methods.The networks are trained on 3DMatch and evaluated on other multi-way registration datasets, demonstrating cross-dataset generalization.
6. Conclusion
Deep Global Registration robustly and accurately aligns real-world 3D scans through a learning-based framework. The method combines three modules and outperforms classical and learning-based registration methods, including on challenging fragment registrations.
- 6. Conclusion: The framework uses a 6D convolutional network, differentiable Weighted Procrustes, and a gradient-based optimizer for registration.These modules perform inlier detection, scalable pose estimation, and pose refinement, respectively.
- 6. Conclusion: Experiments show that the approach outperforms both classical and learning-based registration methods.The reported evaluation covers real-world 3D scan registration.
- 6. Conclusion: On fragment registrations, the method succeeds with small overlaps or ambiguous geometry where other methods fail.Combining Weighted Procrustes with gradient-based refinement also produces more accurate registrations in one pass and better aligned details.