Source-linked AI summary
Learning multiview 3D point cloud registration
Zan Gojcic, Caifa Zhou, Jan D. Wegner, Leonidas J. Guibas, Tolga Birdal
TL;DR
Multiview registration must reconcile ambiguous pairwise alignments caused by limited overlap and repetitive or symmetric structure. This paper jointly learns pairwise registration and globally consistent refinement in an end-to-end model, outperforming state-of-the-art methods while remaining computationally efficient.
Problem
Pairwise alignment can be ambiguous under low overlap, symmetries, and repetitive scene parts, while global refinement is typically separated from it.
Method
The method uses differentiable pairwise registration, confidence estimation, and iterative transformation synchronization to jointly refine pairwise and absolute transformations.
Results
The method outperforms current state-of-the-art on pairwise and multiview registration and exceeds it by more than 25 percentage points on average for rotation error statistics.
Takeaways & Limitations
End-to-end learning can combine pairwise alignment and globally consistent multiview refinement in one efficient registration algorithm.
Takeaways & Limitations
The edge-pruning threshold was determined on 3DMatch, and disconnected graphs are handled by reporting the last valid node values; more sophisticated handling is left for future work.
Abstract
from arXiv · showhide
We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmark datasets shows that our approach outperforms the state-of-the-art by a significant margin, while being end-to-end trainable and computationally less costly. Moreover, we present detailed analysis and an ablation study that validate the novel components of our approach. The source code and pretrained models are publicly available under https://github.com/zgojcic/3D_multiview_reg.
1. Introduction
Multiview registration is needed to fuse partial scans into a globally consistent scene, but pairwise alignment can be ambiguous and traditional pipelines separate alignment from global refinement. The paper introduces an end-to-end neural formulation that jointly learns these stages and improves registration efficiency and accuracy.
- Holistic scene representations support downstream 3D vision tasks, including semantic segmentation and object detection.
- Low overlap, repetitive structure, and reliance on local evidence can make pairwise registration inaccurate or ambiguous.Separate post-processing is then required to combine pairwise matches into a global representation.
- The method directly learns globally consistent registration for all input views instead of detaching pairwise alignment from global refinement.It accepts potentially overlapping point clouds and outputs one global transformation matrix per scan.
- The network differentiably solves pairwise Procrustes estimation and spectral transformation synchronization in its forward pass.
- A confidence block uses overlap pooling to estimate confidence in pairwise transformation parameters.
- The resulting algorithm is efficient, improves alignments through iterative residual feedback, and outperforms state-of-the-art pairwise and multiview methods.
2. Related Work
Prior multiview registration methods use multiple-view cues to address ambiguities in pairwise alignment, with approaches ranging from multiview ICP-like optimization to pose synchronization. The closest cited work learns weighting for synchronization, whereas this paper also learns relative transformations and avoids repeated depth-image conversion.
- Multiview registration incorporates cues from multiple views to resolve difficult or ambiguous cases arising in pairwise methods.
- Multiview ICP-like methods optimize camera poses and 3D point correspondences, but many incur increased correspondence-estimation complexity.
- Other methods optimize motion or poses using registration error and global cycle consistency, often starting from initial pairwise maps.
- Synchronization methods globally coordinate observed relative motions, while structure-from-motion can decompose rotation, translation, and scale.
- The closest cited work learns synchronization edge weights, but estimates relative transformations with FPFH and FGR rather than learning them.It also converts point clouds to depth images in each iteration to approximate its weighting function.
3. End-to-End Multiview 3D Registration
The method jointly learns pairwise registration and global transformation synchronization in one differentiable multiview pipeline. It iteratively re-estimates correspondences, transformations, weights, and residuals to produce globally consistent scan poses.
- Joint multiview registration: The network extends learned pairwise registration to multiple point clouds through a transformation synchronization layer.The resulting pipeline is fully differentiable and jointly couples pairwise registration with global consistency.
- Pairwise registration: Pairwise registration estimates transformation parameters from point correspondences using a differentiable weighted least-squares solution.Learned correspondence weights reduce the influence of outlier matches, whose dominance can otherwise produce incorrect transformations.
- Transformation synchronization: The method initializes graph edges from point-cloud pairs, then estimates global rotations and translations through separate synchronization problems with differentiable closed-form solutions.Rotation synchronization uses spectral relaxation, while translation synchronization solves a least-squares problem.
- Iterative refinement: Iterative refinement pre-aligns point-cloud pairs using synchronized transformations and reuses previous weights and residuals as correspondence-weighting cues.The pairwise registration and synchronization stages are repeated, allowing later estimates to incorporate information from earlier iterations.
- Iterative refinement: The proposed pipeline repeats the registration-refinement and transformation-synchronization blocks four times.The figure describes features, stochastic correspondences, initial registration outputs, and recurrent refinement inputs across these iterations.
4. Network Architecture
The architecture uses learned FCGF-based stochastic correspondences, neural weighting and confidence estimation, and differentiable synchronization. These components are connected into an end-to-end network trained with joint registration objectives and iterative refinement.
- Correspondence function: FCGF descriptors are extended with a soft assignment layer that produces differentiable stochastic point correspondences.The probabilistic nearest-neighbor rule approaches deterministic nearest-neighbor matching as the temperature tends to zero.
- Pairwise registration: The pairwise registration block predicts correspondence weights that feed a closed-form transformation solver, followed by an iterative registration block.The architecture learns weighting functions for putative correspondences and uses them to estimate pairwise transformations.
- Confidence estimation: Local confidence combines pooled registration features with the inlier ratio, while global confidence is derived from transformation-synchronization residuals.The two confidence sources are combined using a learned harmonic-mean weighting scheme.
- Evaluation: Registration recall is evaluated on 3DMatch by comparing 1-iteration pairwise output with the input to the fourth transformation-synchronization layer.The table caption distinguishes the pairwise result from the globally informed 4-iteration result.
- End-to-end training: The complete network is fine-tuned end-to-end on 3DMatch after pre-training its individual subnetworks.Training uses 2048 sampled feature vectors per point-cloud pair, four refinement iterations, and a joint multiview registration loss.
5. Experiments
Experiments evaluate pairwise registration, cross-domain generalization, multiview registration, computational efficiency, and ablations on established 3DMatch, Redwood, and ScanNet benchmarks. The method achieves strong accuracy and efficiency, while ablations identify benefits and failure modes of confidence-based pruning and iterative refinement.
- Evaluation setup: Experiments use the 3DMatch, Redwood, and ScanNet benchmark datasets to evaluate performance, efficiency, and generalization.The full pipeline is evaluated on ScanNet, while pairwise registration is evaluated on 3DMatch and Redwood.
- Pairwise registration performance: Our approach achieves the highest recall among the evaluated pairwise registration methods.Using the same FCGF features, it outperforms RANSAC-based transformation estimation with lower time complexity.
- Generalization: ≈4 percentage points higher recall than state-of-the-art is achieved on Redwood without training on synthetic data.Average precision is low across methods, but pruning can increase precision with little recall loss.
- Multiview registration performance: The multiview approach achieves a large improvement over baselines, including when all input edges are used.The analysis attributes further improvement to iterative refinement of pairwise relative transformation parameters and reports dominance over competitor scenarios.
- Computational complexity: >13 times faster computation is reported for a 60-fragment scene, with about 80s for the proposed approach versus >1100s for RANSAC-based estimation.Soft nearest-neighbor computation is approximately four times faster than traditional nearest-neighbor search, and model estimation gains about 23 times speedup over up to 50000 RANSAC iterations.
- Ablation study: The confidence-estimation pruning scheme improves ablation performance by more than 20 percentage points, while iterative refinement improves fourth-iteration inputs by approximately 2 percentage points.High outlier presence or inefficient pruning can instead make weights and residuals introduce a negative bias and worsen results.
6. Conclusions
The paper introduces an end-to-end learnable multiview registration system that jointly estimates pairwise transformations and globally synchronizes them. It reports improved rotation accuracy, speed, and cross-scene generalization, while providing differentiable closed-form synchronization components.
- The method replaces detached pairwise registration and global refinement with an end-to-end model that directly registers all views consistently.
- A pose graph uses confidence weights to refine relative transformations into globally consistent absolute transformations.
- The approach outperforms state-of-the-art methods by more than 25 percentage points on average for rotation error statistics.
- The method is over 13 times faster than RANSAC-based methods for 60 scans and achieves approximately 4 percentage points higher Redwood indoor recall than state of the art.
- The implementation is modular, supports varying numbers of point clouds with or without connectivity information, and releases code and pretrained models.
- The system uses differentiable closed-form solutions for pairwise registration, rotation synchronization, and translation synchronization.
B.3.1 FCGF local feature descriptor
The FCGF descriptor processes sparse voxelized point clouds with a fully convolutional UNet-style network. It produces per-point 32-dimensional features for correspondence estimation and is pretrained with rotational augmentation.
- FCGF represents a point cloud as unique coordinates with associated per-point features in sparse tensors.
- Its fully convolutional UNet architecture uses skip connections and ResNet blocks to extract 32-dimensional per-point descriptors.
- Voxel grid downsampling produces unique coordinates using a voxel size of 2.5 cm.
- The descriptor is pretrained on 3DMatch fragments for 100 epochs using stochastic gradient descent with batch size 4 and exponentially decayed learning rate.
- Random rotations of fragments are used as data augmentation to introduce rotation invariance.
B.3.2 Registration block
The registration block refines putative correspondences through PointNet-like processing, learned order-aware clustering, and differentiable unpooling. Its outputs include correspondence scores and confidence features used by the broader iterative registration system.
- The registration block combines PointNet-like ResNet processing with instance normalization and order-aware blocks.
- Learned soft pooling maps Nc correspondences to Mc clusters, whose features are processed at the next network level.
- Registration blocks are pretrained on 3DMatch using 5,000 sampled descriptors per fragment and ground-truth correspondence labels.
- Differentiable unpooling combines clustered features with features from the preceding level before further ResNet processing.
- The block outputs scores indicating whether putative correspondences are outliers or inliers, while intermediate 128-dimensional features feed confidence estimation.
- The complete system iterates four times over network and synchronization layers, with four synchronization executions per iteration.
B.5. Extended ablation study
The extended ablation study evaluates how local-confidence edge pruning and weighting affect angular and translation errors on ScanNet fragments.
- The ablation study measures the effects of local-confidence edge pruning and weighting schemes on angular and translation errors.
- Experiments are conducted on point cloud fragments from the ScanNet dataset.
B.5.1 Impact of the edge pruning threshold
The edge-pruning threshold has little effect on angular and translation errors when set above 0.2.
- Thresholds larger than 0.2 produce little change in angular error.
- Thresholds larger than 0.2 produce little change in translation error.
- The threshold analysis indicates stable performance across values above 0.2.
B.6. Impact of the harmonic mean weighting scheme
The harmonic-mean weighting scheme combines local and global confidence to improve synchronization under noisy graph connectivity, especially when paired with edge pruning.
- Combining local and global evidence with harmonic-mean weighting is essential for good performance under noisy graph connectivity.
- The combined confidence scheme reduces the impact of noisy graph connectivity and improves performance after transformation synchronization.
- Harmonic-mean weighting enables the method to outperform SE3 EIG.
- Edge pruning suppresses low-confidence outliers that corrupt the l2 solution and improves transformation synchronization.
B.7. Qualitative results
Qualitative evaluations compare the complete pipeline with SE3 EIG and intermediate pairwise results on 3DMatch and ScanNet, including both successes and failures.
- The qualitative comparisons include the complete synchronized pipeline, SE3 EIG, and pairwise results from the first and fourth iterations.
- Failures predominantly occur in low-structure planar areas and scenes with high symmetry or repetitive structures.
- Success cases are shown on Kitchen and Hotel 1 from 3DMatch and scene0057 01 and scene0309 00 from ScanNet.
- The reported failure cases still qualitatively outperform state-of-the-art results.