Source-linked AI summary
PRNet: Self-Supervised Learning for Partial-to-Partial Registration
Yue Wang, Justin M. Solomon
TL;DR
Partial-to-partial point-cloud registration requires alignment despite incomplete and only partially overlapping observations, which existing learning-based methods do not support. PRNet uses an iterative, self-supervised network that learns representations, shared keypoints, and correspondences, refining alignment through multiple passes. It outperforms PointNetLK, DCP, and non-learning methods on synthetic data, while its learned representations transfer to classification.
Problem
Existing learning-based registration methods cannot handle partial-to-partial point clouds, where only incomplete views are available for alignment.
Method
PRNet is an iterative, self-supervised framework that jointly learns geometric representations, shared keypoints, and keypoint correspondences for registration.
Results
PRNet achieves state-of-the-art partial-to-partial registration performance, transfers learned representations to shape classification, and remains reasonable with 75% of points missing.
Takeaways & Limitations
Registration serves as a useful proxy task for learning transferable 3D-shape representations across keypoint detection, correspondence prediction, and classification.
Takeaways & Limitations
Real-scan inference currently requires inference-time fine-tuning, making PRNet slow and limiting speed and generalizability.
Abstract
from arXiv · showhide
We present a simple, flexible, and general framework titled Partial Registration Network (PRNet), for partial-to-partial point cloud registration. Inspired by recently-proposed learning-based methods for registration, we use deep networks to tackle non-convexity of the alignment and partial correspondence problems. While previous learning-based methods assume the entire shape is visible, PRNet is suitable for partial-to-partial registration, outperforming PointNetLK, DCP, and non-learning methods on synthetic data. PRNet is self-supervised, jointly learning an appropriate geometric representation, a keypoint detector that finds points in common between partial views, and keypoint-to-keypoint correspondences. We show PRNet predicts keypoints and correspondences consistently across views and objects. Furthermore, the learned representation is transferable to classification.
1 Introduction
PRNet addresses partial-to-partial point-cloud registration with an iterative, self-supervised framework that detects and matches shared keypoints. It combines learned correspondence refinement with transferable geometric representations and strong registration performance.
- Motivation: Registration aligns one point cloud to another by predicting a rigid motion, but existing methods face steep efficiency–effectiveness trade-offs and tuning demands.Some methods are orders of magnitude slower than vanilla ICP, limiting practicality or generalizability.
- Motivation: PointNetLK and DCP improve speed and robustness but cannot handle partial-to-partial registration or refine one-shot alignments.Their one-shot constructions determine the output in a single pass.
- PRNet: PRNet iteratively refines registration by detecting shared keypoints, matching them, and solving the Procrustes problem.The sequential framework supports coarse-to-fine refinement of an initial registration estimate.
- PRNet: Gumbel–Softmax with a straight-through estimator modulates correspondence sharpness, using diffuse matches for coarse alignment and sharper maps for refinement.A sub-network predicts the correspondence temperature instead of introducing another hyperparameter.
- Evaluation: PRNet is evaluated on ModelNet40 and real data, and its learned representations transfer to shape classification with comparable state-of-the-art supervised performance.The study also visualizes keypoints and correspondences across same- and different-category shapes.
- Contributions: The paper presents PRNet as a deep-learning method for partial-to-partial registration with state-of-the-art performance and transferable representations.The contributions include correspondence estimation and transfer to keypoint detection, correspondence prediction, and shape classification.
2 Related Work
The related work spans classical registration, point-cloud representation learning, correspondence methods, self-supervision, and actor–critic approaches. PRNet is positioned as combining keypoint-based partial matching with self-supervised and actor–critic-inspired learning.
- Rigid Registration: Classical registration includes ICP variants, probabilistic models, and global optimization methods such as branch-and-bound and semidefinite programming.These approaches target rigid registration, uncertainty, partiality, or improved optimization.
- Learning on Point Clouds and 3D Shapes: Point-cloud and 3D-shape learning methods use shared MLPs, symmetric aggregation, local operators, graph convolutions, or grid-based representations.Examples include PointNet, PointNet++, DGCNN, PointCNN, PCNN, GCNs, and grid-based methods.
- Keypoints and Correspondence: Point-to-point correspondence is computationally difficult, with O(n^2) correspondence matrices and O(n!) permutations, so many methods use sparse correspondences and dense-map extensions.Bijectivity is often assumed or imposed as a regularizer.
- Keypoints and Correspondence: For partial-to-partial registration, bijective correspondences are unavailable, motivating sparse keypoint detection; PRNet applies this idea directly to point clouds.Prior work used registration and multiview consistency to supervise keypoint detection on 2D images.
- Self-supervised Learning: Self-supervised learning uses unlabeled information through prediction, proxy tasks, or consistency across views and modalities, and PRNet requires no labeled data.The broader motivation is transferring representations learned from unlabeled data to tasks with limited labels.
- Actor–Critic Methods: Actor–critic methods combine action selection with value estimation, a pattern related to PRNet’s value head for predicting correspondence granularity.The value estimate can guide the level at which two shapes are mapped.
3 Method
PRNet frames partial-to-partial registration as an iterative process that detects shared keypoints, learns their correspondences, and repeatedly refines rigid alignment. Its design combines approximately differentiable sharp matching with adaptive correspondence sharpness and multiple loss terms.
- Partial Registration Network: PRNet iteratively detects keypoints, predicts keypoint mappings, estimates a rigid transformation, transforms X, and repeats refinement.The iterative input becomes (R_XY X + t_XY, Y) for the next pass.
- Keypoint Detection: Keypoint detection selects important points from each partial cloud using feature L2 norms and removes irrelevant nonshared points before Procrustes alignment.The embeddings used for selection are learned by DGCNN and Transformer modules.
- Gumbel–Softmax Sampler: Gumbel–Softmax samples a sharp matching matrix while straight-through estimation makes the mapping approximately differentiable.This combines the sharp correspondences of ICP with the differentiability of soft correspondence methods.
- Actor-Critic Closest Point: PRNet adapts correspondence sharpness through λ: large λ smooths the map, whereas λ approaching zero makes it binary.A network Θ predicts λ from global features for each point-cloud pair.
- Loss Function: The loss sums rigid-motion, cycle-consistency, and global-feature-alignment terms across passes, with γ < 1 promoting alignment within earlier passes.Training passes each input pair through PRNet P times.
4 Experiments
PRNet is evaluated on synthetic and real partial-to-partial registration, including unseen objects, unseen categories, Gaussian noise, transfer learning, and visual consistency of keypoints and correspondences.
- Partial-to-Partial Registration on Unseen Objects: PRNet outperforms its counterparts across all metrics on unseen ModelNet40 objects.The evaluation compares registration performance on 9,843 training objects and 2,468 testing objects.
- Partial-to-Partial Registration on Unseen Categories: PRNet behaves more strongly than competing methods on held-out object categories.Training on ShapeNetCore produces substantially better held-out-category performance than training on ModelNet40.
- Partial-to-Partial Registration with Gaussian Noise: PRNet remains robust under Gaussian noise and is comparable to its noise-free performance.Noise is independently sampled from N(0, 0.01) and clipped to [-0.05, 0.05].
- Partial-to-Partial Registration on Real Data: On real Stanford Bunny and 3D Scanning Repository scans, PRNet is evaluated using inference-time fine-tuning.The real-data protocol fine-tunes the model on generated transformations of each scan.
- Keypoints and Correspondences: PRNet detects keypoints and correspondences on partially observable objects, with keypoints consistent across different views.Visualizations show inputs, predicted transformations, keypoints, and correspondences for object pairs.
- Transfer to Classification: The learned representations achieve comparable object-classification performance to supervised methods on ModelNet40.A linear SVM is trained on embeddings produced by the learned representation.
5 Conclusion
PRNet applies self-supervised learning to general partial-to-partial registration and connects registration with discrete optimization and reinforcement-learning-like frameworks. Future work targets non-bijective matching, inference efficiency, broader generalization, and large-scale real scans.
- Conclusion: PRNet learns geometric priors directly from data for general partial-to-partial registration.The conclusion frames this as evidence supporting learning for partial matching.
- Future Work: PRNet’s matchings are not bijective, motivating Gumbel–Sinkhorn as a possible extension.The issue is visible in examples involving cars and chairs.
- Future Work: Inference-time fine-tuning on real scans makes PRNet slow during inference.The authors identify universal representations as a route toward improved speed and generalizability.
- Applications: Future work includes integrating PRNet into SLAM or structure-from-motion and testing medical-imaging or particle-physics applications.These directions are proposed to assess robotics applications and robustness to realistic noise.
Supplementary
The supplementary material details PRNet’s co-contextual architecture, iterative training protocol, ablations, robustness tests, efficiency comparison, and additional visualizations of keypoints and correspondences.
- Architecture: DGCNN features are processed by an asymmetric Transformer that learns co-contextual representations for the two point clouds.The Transformer uses one encoder, one decoder, and 4-head self-attention.
- Architecture: An action head combines Gumbel–Softmax and SVD, while a value head predicts the correspondence temperature λ.The value head is a 4-layer MLP.
- Training Protocol: PRNet is trained for 100 epochs and applies each training pair through three iterative alignment updates before combining the local transformations.The initial learning rate is 0.001 and is divided by 10 at epochs 30, 60, and 80.
- Ablations and Robustness: Additional experiments examine keypoint selection, full-shape registration, discount-factor choice, keypoint count, missing data, and noise robustness.Performance remains reasonable with 75% of points missing and with noise σ = 0.1.
- Efficiency and Visualization: Learning-based methods run faster on GPUs than non-learning methods on CPUs; PRNet is comparable in speed to PointNetLK but slower than DCP.The supplementary material also visualizes keypoints and correspondences for different object pairs.