Source-linked AI summary
The Perfect Match: 3D Point Cloud Matching with Smoothed Densities
Zan Gojcic, Caifa Zhou, Jan D. Wegner, Andreas Wieser
TL;DR
3D point cloud matching requires reliable correspondences between partially overlapping scans despite viewpoint, sensor, scene, and rotation changes. 3DSmoothNet uses LRF-aligned SDV voxel grids with a Siamese fully convolutional network to learn compact descriptors. It reports strong benchmark performance, efficient correspondence search, and generalization across modalities and indoor-to-outdoor scenes.
Problem
3D point cloud matching must align overlapping fragments from different viewpoints, while existing learned descriptors can lack rotation invariance, require high dimensions, or generalize poorly.
Method
3DSmoothNet transforms local neighborhoods into LRF-aligned canonical representations, voxelizes them with SDV smoothing, and learns descriptors using a Siamese fully convolutional network.
Results
94.9% average recall on the 3DMatch benchmark accompanies more than 20 percentage points over state-of-the-art, with 32 output dimensions and indoor-trained models generalizing to outdoor laser scans.
Takeaways & Limitations
The compact rotation-invariant descriptor supports efficient correspondence search and transfer between sensor modalities and indoor and outdoor scenes.
Takeaways & Limitations
The formulation assumes static, rigid point clouds and neglects noise and differing point-cloud resolutions.
Abstract
from arXiv · showhide
We propose 3DSmoothNet, a full workflow to match 3D point clouds with a siamese deep learning architecture and fully convolutional layers using a voxelized smoothed density value (SDV) representation. The latter is computed per interest point and aligned to the local reference frame (LRF) to achieve rotation invariance. Our compact, learned, rotation invariant 3D point cloud descriptor achieves 94.9% average recall on the 3DMatch benchmark data set, outperforming the state-of-the-art by more than 20 percent points with only 32 output dimensions. This very low output dimension allows for near realtime correspondence search with 0.1 ms per feature point on a standard PC. Our approach is sensor- and sceneagnostic because of SDV, LRF and learning highly descriptive features with fully convolutional layers. We show that 3DSmoothNet trained only on RGB-D indoor scenes of buildings achieves 79.0% average recall on laser scans of outdoor vegetation, more than double the performance of our closest, learning-based competitors. Code, data and pre-trained models are available online at https://github.com/zgojcic/3DSmoothNet.
1. Introduction
3D point cloud matching must align partially overlapping scans captured from different viewpoints into a common coordinate system. 3DSmoothNet addresses limitations of learned descriptors with a compact, rotation-invariant approach designed to generalize across sensors and scenes.
- Overlapping point-cloud fragments require alignment before reconstruction or semantic segmentation can combine them into one complete scene.
- Many learned 3D descriptors are not rotation invariant, require high output dimensions, or generalize poorly to new domains.
- 3DSmoothNet combines low-dimensional output, high descriptiveness, rotation invariance, and cross-sensor and indoor-to-outdoor generalization.
- SDV voxelization reduces input sparsity and boundary effects while smoothing small local-reference-frame misalignments.
- A Siamese network with fully convolutional layers learns compact rotation-invariant descriptors that support efficient correspondence search and real-time applications.The descriptors have only 16 or 32 output dimensions.
2. Related Work
3D local descriptors address unordered, sparse point clouds through handcrafted geometric representations, projections, voxel grids, or learned raw-point processing. 3DSmoothNet combines local reference frames, smoothed density voxel grids, and a siamese CNN to retain local geometry while enabling convolutional learning.
- Hand-crafted 3D Local Descriptors: Handcrafted descriptors establish rotation invariance either through local reference frames or alternative strategies.SHOT, RoPS, USC, and TOLDI are examples using local reference frames typically estimated from neighborhood covariance.
- Learned 3D Local Descriptors: Learned descriptors represent point clouds using projected images, dense voxel grids, or direct processing of raw point sets.Examples include depth-map projections, multiview 2D CNNs, occupancy grids, and truncated distance functions.
- Learned 3D Local Descriptors: PointNet and PointNet++ use symmetric functions to achieve permutation invariance but do not satisfactorily capture local geometric information.The passage attributes this limitation largely to their inability to use convolutional operations effectively.
- Learned 3D Local Descriptors: 3DSmoothNet transforms neighborhoods into local reference frames, encodes them as smoothed density value grids, and learns descriptors with a siamese CNN.This hybrid strategy makes unstructured point clouds compatible with fully convolutional layers.
3. Method
3DSmoothNet converts aligned local point neighborhoods into smoothed density voxel grids and learns compact descriptors with a Siamese fully convolutional network. The workflow estimates local reference frames, trains with batch-hard negatives, and supports correspondence-based registration.
- Workflow: The workflow estimates an LRF for spherical neighborhoods, transforms them canonically, applies Gaussian-smoothed voxelization, and infers per-point descriptors for registration.The descriptors can be used in a RANSAC-based robust point-cloud registration pipeline.
- SDV voxelization: 3DSmoothNet represents each canonical neighborhood as an SDV voxel grid centered on the interest point and aligned with its LRF.The grid is normalized to reduce sensitivity to varying point-cloud densities.
- SDV voxelization: SDV smoothing reduces voxel-grid sparsity and boundary effects while mitigating small LRF misalignments and improving gradient flow during backpropagation.The representation is intended to model smoothing that deep networks commonly learn in their first layers.
- Network architecture: The network uses stacked convolutional layers with stride-based downsampling, batch normalization, ReLU activations, and final l2 normalization.The architecture is Siamese, with both branches sharing parameters.
- Training: Training uses overlapping 3DMatch fragments and samples hard negative examples on the fly from the mini-batch using a soft-margin batch-hard loss.Training fragments are drawn from 54 indoor scenes, while 8 scenes are reserved for testing.
4. Results
Experiments evaluate 3DSmoothNet on indoor RGB-D matching, rotated fragments, outdoor laser scans, and computational efficiency. The results show strong recall with compact descriptors, rotation robustness, cross-modality generalization, and efficient correspondence search.
- Output dimensionality: Performance saturates beyond 64 output dimensions, so subsequent experiments use 16- and 32-dimensional descriptors.The dimensionality study targets a compromise between matching performance, runtime, and memory use.
- 3DMatch results: 92.8% and 94.7% average recall are achieved by the 16- and 32-dimensional models, respectively, on 3DMatch.The 16-dimensional model improves average recall by almost 20 percentage points over PPF-FoldNet and SHOT while using fewer dimensions.
- Rotation invariance: 3DSmoothNet remains effective under rotation, while nonrotation-invariant descriptors drop to zero on the rotated benchmark.The rotated evaluation samples arbitrary rotations around all three axes.
- Cross-modality generalization: Models trained on indoor RGB-D data generalize to outdoor laser scans of vegetation and outperform competitors trained and tested on 3DMatch.The ETH evaluation covers Gazebo-Summer, Gazebo-Winter, Wood-Autumn, and Wood-Summer.
- Computation time: 3DSmoothNet takes 4.6 ms per feature point overall versus 5.0 ms for 3DMatch, with nearest-neighbor search taking 0.1 ms versus 0.8 ms.Input preparation is slower for 3DSmoothNet on CPU, but inference and correspondence search are faster.
5. Conclusions
3DSmoothNet is a compact, rotation-invariant deep-learning workflow for 3D point-cloud matching that outperforms prior methods and transfers across sensors and scene types.
- 3DSmoothNet outperforms existing methods by more than 20 percent points while using 16- or 32-dimensional descriptors.
- 94.9% average recall is achieved on the 3DMatch benchmark dataset.
- Low output dimensions enable efficient correspondence search.The descriptors have 16 or 32 dimensions.
- A model trained on indoor RGB-D scenes generalizes to terrestrial laser scans of outdoor vegetation.
- The method is rotation invariant and supports transferring trained models between modalities.
6. Supplementary Material
The supplementary material details the evaluation metric, comparison settings, datasets, and per-scene results. It reports strong performance across rotated and sparsified point clouds, while documenting dataset-specific preprocessing and evaluation thresholds.
- Evaluation Metric: The evaluation metric computes average recall from nearest-neighbor feature correspondences between fragment pairs with more than 30% ground-truth overlap.Correspondences use descriptor-space l2 nearest-neighbor search, with thresholds τ1 = 0.1m and τ2 = 0.05.
- Evaluation Metric: RANSAC theoretically needs approximately 55000 iterations at the selected inlier threshold, decreasing to 860 when τ2 = 0.2.The analysis links fewer iterations to increasing the inlier ratio threshold.
- Comparison Settings: Evaluation parameters are normalized relative to voxel-grid widths of 0.3m for 3DMatch and 1m for ETH.Spherical-support descriptors use equal-volume feature radii, while normal estimation uses the same radius.
- Datasets: 3DSparseMatch downsamples non-interest points to 50%, 25%, and 12.5%, evaluating mixed-density and both-sparse fragment pairs.The dataset tests robustness when one or both fragments have reduced point density.
- Datasets: ETH point clouds are voxel-downsampled to 0.02m, with overlap recomputed using τψ = 0.06m and pairs retained only when both directional overlaps exceed 0.3.Because interest-point indices are unavailable, 5000 interest points are randomly sampled.
- Results: On 3DMatch, the 32-dimensional descriptor exceeds 89% recall on every scene and consistently outperforms state-of-the-art descriptors.The supplementary results also report that performance differences can be masked by the low τ2 threshold.
- Results: On 3DRotatedMatch, 3DSmoothNet outperforms every competing descriptor across all scenes, while non-rotation-invariant descriptors drop to almost zero.Rotation-invariant descriptors remain unaffected by fragment rotations.
- Results: On 3DSparseMatch, 3DSmoothNet’s recall drops by at most 1 percentage point and remains more than 20 percentage points above competing methods.All descriptors perform better when only one fragment is sparsified than when both are downsampled.