Source-linked AI summary
Learning general and distinctive 3D local deep descriptors for point cloud registration
Fabio Poiesi, Davide Boscaini
TL;DR
Descriptors for point-cloud registration need to work across sensors, contexts, and geometric transformations, while existing dense and patch-based approaches have different generalisation and efficiency trade-offs. GeDi encodes canonicalised patches with a permutation-invariant network and learned quaternion refinement. The authors report stronger cross-domain generalisation than compared descriptors and strong same-domain performance.
Problem
Point-cloud descriptors must generalise across sensors and contexts, while dense descriptors can lack cross-domain generalisation and patch-based descriptors trade greater generalisation for reduced efficiency.
Source: §1 · p. 2Method
GeDi samples and canonicalises local patches, refines their orientation with a learned quaternion network, and encodes them using a multi-scale permutation-invariant network trained with Siamese contrastive learning.
Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4Results
Takeaways & Limitations
The authors conclude that GeDi’s patch-based design generalises better across domains while remaining effective in same-domain benchmarks.
Source: §5 · p. 9
Abstract
from arXiv · showhide
An effective 3D descriptor should be invariant to different geometric transformations, such as scale and rotation, robust to occlusions and clutter, and capable of generalising to different application domains. We present a simple yet effective method to learn general and distinctive 3D local descriptors that can be used to register point clouds that are captured in different domains. Point cloud patches are extracted, canonicalised with respect to their local reference frame, and encoded into scale and rotation-invariant compact descriptors by a deep neural network that is invariant to permutations of the input points. This design is what enables our descriptors to generalise across domains. We evaluate and compare our descriptors with alternative handcrafted and deep learning-based descriptors on several indoor and outdoor datasets that are reconstructed by using both RGBD sensors and laser scanners. Our descriptors outperform most recent descriptors by a large margin in terms of generalisation, and also become the state of the art in benchmarks where training and testing are performed in the same domain.
DOI: 10.1109/TPAMI.2022.3175371
The passage states restrictions on reuse of copyrighted material.
DOI: 10.1109/TPAMI.2022.3175371
- Reuse for reprinting, promotional use, collective works, resale, redistribution, or reuse in other works requires permission.The restriction applies to current and future media.Source: DOI: 10.1109/TPAMI.2022.3175371 · p. 1
1 INTRODUCTION
The paper addresses general and distinctive compact descriptors for point-cloud registration across sensors and contexts, where existing methods vary in their cross-domain generalisation. GeDi combines canonicalised patches, learned rotation refinement, and multi-scale point encoding; the authors report stronger generalisation than compared methods and same-domain effectiveness.
1 INTRODUCTION
- Patch-based descriptors generalise better than dense counterparts but are less efficient, while dense descriptors can lack cross-domain generalisation.The motivating domains vary by sensor, including RGBD and LiDAR, and context, including indoor and outdoor settings.Source: §1 · p. 2
- GeDi outperforms 19 compared methods by a large margin in generalisation while remaining as effective as other descriptors in same-domain training and testing.The evaluation covers ETH, KITTI, and 3DMatch.Source: §1 · p. 2
- GeDi canonicalises patches with an LRF, refines noisy canonicalisation with a learned quaternion network, and encodes points using a permutation-invariant network.QNet and the encoding network are trained concurrently through a Siamese contrastive-learning approach.Source: §1 · p. 2
- The paper extends earlier work by adding QNet and multi-scale processing, with QNet producing transformations in SO(3) by construction.The extension also replaces farthest-point patch sampling with random sampling at each iteration, which the authors describe as effective data augmentation.Source: §1 · pp. 2–3
2 RELATED WORK
Related work spans handcrafted and learned descriptors, with different approaches representing local geometry and handling rotation. The paper situates GeDi among methods that use local reference frames and methods that learn representations directly from point sets or rendered views.
2 RELATED WORK
- Handcrafted descriptors encode local geometry through coordinates, normals, histograms, covariance matrices, or point relationships, with different methods targeting registration or object pose estimation.MCOV uses geometric and photometric patch information; FPFH uses feature histograms, while PPF uses a hash-table representation.Source: §2 · p. 3
- LRF-based handcrafted methods canonicalise patch points before building compact geometric representations.ISS, SHOT, TOLDI, and RoPS use variants of covariance-based frames and histograms, projections, or moments to represent local structure.Source: §2 · p. 3
- Deep descriptors without LRF model rotation invariance through point-pair features, augmentation, or learned transformations, using voxel, point, sparse-convolution, or cylindrical representations.Examples include 3DMatch, PPFNet, FCGF, D3Feat, and SpinNet.Source: §2 · p. 3
- Deep descriptors with LRF process canonicalised points using voxel representations, PointNet, rendered views, or spherical CNNs.3DSmoothNet and DIP use TOLDI to canonicalise points, while LMVD aggregates viewpoint-specific image features.Source: §2 · pp. 3–4
3 OUR APPROACH
GeDi maps unordered point-cloud patches to compact descriptors by sampling and canonicalising points before deep encoding. Its design combines rotation- and scale-normalised inputs, multi-scale permutation-invariant encoding, quaternion-based transformation refinement, and Siamese contrastive training.
3 OUR APPROACH
- GeDi computes a compact descriptor from an unordered local patch by composing a sampling-and-canonicalisation function with a learnable deep network.The descriptor is d-dimensional, and the input patch is represented by 3D coordinates.Source: §3 · p. 4
3.1 Local sampling and canonicalisation
- Random sampling produces fixed-cardinality patches, whose points are transformed into an LRF and normalised relative to the patch centre and radius.When a patch contains fewer than m points, it is padded by sampling with replacement; n points are then selected, with n < m.Source: §3.1 · p. 4
3.2 Deep network design
- PointNet++ encodes permutation-invariant input using hierarchical receptive fields at multiple scales, while QNet outputs a unit-norm quaternion for an SO(3) transformation.The quaternion transformation is learned concurrently with the encoder without an additional regularisation loss or computational overhead.Source: §3.2 · p. 4
- Training uses a shared-weight Siamese network on corresponding patches and a hardest-contrastive loss with negatives mined outside a spherical region around each anchor.Training pairs come from point-cloud pairs with sufficient overlap, and patch centres are randomly sampled from the overlap region.Source: §3.2 · pp. 4–5, Figure 1 · p. 5
4 EXPERIMENTS
GeDi is evaluated across indoor and outdoor datasets, sensors, and training/testing domains, with ablations and qualitative registration examples. The experiments report strong cross-domain performance and examine how design and sampling choices affect descriptor quality.
4 EXPERIMENTS
- The experiments use 3DMatch, ETH, and KITTI to test cross-domain and same-domain performance, inference time, and ablations.3DMatch and ETH are evaluated with FMR; KITTI uses RANSAC registration and reports RTE, RRE, and success rate.Source: §4 · p. 5, §4.1 · p. 5
4.2 Parameters
- Training settings vary by dataset, and testing samples up to 50K points on KITTI because computing descriptors for every point is lengthy.Training uses 512 points per patch; testing uses 1024, with patch radii selected separately for 3DMatch, ETH, and KITTI.Source: §4.2 · pp. 5–6
4.3 Quantitative analysis
- GeDi outperforms competing descriptors across cross-domain tests, including 3DMatch→ETH and 3DMatch→KITTI, and leads on KITTI→3DMatch and same-domain 3DMatch benchmarks.On ETH, it improves FMR by 5.4% over DIP and SpinNet; on KITTI→3DMatch, it beats SpinNet by 7.7% on both 3DMatch and 3DMatchR.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6
- Patch-based descriptors generalise better than dense local descriptors in these experiments, while GeDi and DIP process 3D points directly.The authors associate dense processing with weaker rotation invariance and generalisation, and report GeDi’s multi-scale receptive fields as a way to aggregate local geometry.Source: §4.3 · p. 6
- On KITTI, GeDi achieves a 99.82% registration success rate, while D3Feat has lower RTE and RRE by .65cm and 9° respectively.The authors consider GeDi comparable because RANSAC’s estimated transformation may be refined, for example with ICP.Source: §4.3 · p. 7, Table 5 · p. 7
4.4 Ablation study
- GeDi’s FMR advantage over SpinNet averages .9% across sampled-point settings and reaches 3% when only 250 points are sampled.The paper reports that 97.3% of randomly selected points are, on average, mutual nearest neighbours in descriptor space and close in metric space.Source: §4.4 · p. 7, Table 6 · p. 8
- Ablations find that QNet and LRF canonicalisation improve results; increasing descriptor dimension or sampled points improves FMR, while larger patches can reduce distinctiveness.The .6-radius descriptor reaches better distinctiveness in fewer training iterations than the radius-3 descriptor; QNet is nearly equivalent in FMR to orthogonalised TNet alternatives and more computationally efficient than TNet with SVD.Source: §4.4 · pp. 7–8, Table 7 · p. 8, Table 8 · p. 8
4.5 Qualitative results
- Qualitative examples show correct registrations on ETH, KITTI, and 3DMatch, but small-overlap or weakly informative overlap regions can yield inaccurate or failed registration.The described difficult cases include partial structures with little geometric information and flat surfaces with partially captured objects.Source: §4.5 · p. 8
5 CONCLUSIONS
The paper presents GeDi, a method for learning distinctive 3D local descriptors from canonicalised patches with a permutation-invariant deep network. GeDi generalises better across domains than existing deep descriptors while remaining as effective when training and testing in the same domain.
5 CONCLUSIONS
- GeDi learns general and distinctive 3D local descriptors from canonicalised patches using a deep network invariant to input-point permutations.The network includes a transformation module trained concurrently with the encoding module to facilitate patch canonicalisation.Source: §5 · pp. 8–9
- Qualitative registrations using GeDi trained on 3DMatch include results on ETH, KITTI, and 3DMatch.Source: Figure 4 · p. 9
- GeDi generalises better across domains than existing deep learning-based 3D descriptors and is as effective when trained and tested in the same domain.Source: §5 · p. 9
- Future work includes end-to-end trainable LRF-based descriptors, learning patch radii, and improving computational efficiency.Source: §5 · p. 9