Source-linked AI summary
PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, Simon Lucey
TL;DR
PointNet’s successful use for classification and segmentation had not translated readily to point-cloud registration. The paper modifies LK for PointNet’s imaging function and unrolls both into PointNetLK, which achieves precision, robustness, efficiency, and generalization across challenging data settings. A practical limitation is that the Jacobian uses a fixed small perturbation rather than an infinitesimal one.
Problem
PointNet’s utility for point-cloud registration has remained elusive despite its success as a point-cloud representation for classification and segmentation.
Method
PointNetLK modifies LK to avoid convolution-based gradients on PointNet features, then unrolls the modified algorithm with PointNet into a recurrent neural network.
Results
PointNetLK achieves precision, robustness to initialization, computational efficiency, and generalization to noisy, partially visible, and substantially different shapes.
Takeaways & Limitations
Explicitly encoding alignment lets the network learn the PointNet representation rather than the alignment task, while avoiding costly point correspondences and supporting differentiable integration.
Takeaways & Limitations
The Jacobian perturbations use a small fixed value across iterations rather than the theoretically infinitesimal value.
Abstract
from arXiv · showhide
PointNet has revolutionized how we think about representing point clouds. For classification and segmentation tasks, the approach and its subsequent extensions are state-of-the-art. To date, the successful application of PointNet to point cloud registration has remained elusive. In this paper we argue that PointNet itself can be thought of as a learnable "imaging" function. As a consequence, classical vision algorithms for image alignment can be applied on the problem - namely the Lucas & Kanade (LK) algorithm. Our central innovations stem from: (i) how to modify the LK algorithm to accommodate the PointNet imaging function, and (ii) unrolling PointNet and the LK algorithm into a single trainable recurrent deep neural network. We describe the architecture, and compare its performance against state-of-the-art in common registration scenarios. The architecture offers some remarkable properties including: generalization across shape categories and computational efficiency - opening up new paths of exploration for the application of deep learning to point cloud registration. Code and videos are available at https://github.com/hmgoforth/PointNetLK.
1. Introduction
PointNet provides a learnable, fixed-dimensional representation for unstructured point clouds, but its use for registration remains elusive. PointNetLK adapts LK alignment to PointNet and unrolls both into a trainable recurrent network.
- PointNet as an imaging function: PointNet produces a fixed-dimensional representation regardless of point-cloud sample count or ordering.This treats PointNet as a learnable imaging function for unstructured point clouds.
- Motivation: PointNet’s utility for point-cloud registration has remained elusive.
- Motivation: The paper applies classical Lucas–Kanade image-alignment ideas by interpreting PointNet representations as an imaging function.
- Challenge: Naively applying LK fails because PointNet representations lack the local neighborhood dependencies needed for convolution-based gradient estimates.Grid-based images and volumetric representations provide local dependencies, whereas PointNet representations do not.
- PointNetLK: PointNetLK modifies LK without convolution, unrolls the resulting procedure as a recurrent neural network, and integrates it with PointNet.The architecture avoids costly point correspondences, supports differentiable integration, and is designed for efficient GPU execution.
- Generalization: PointNetLK generalizes to unseen object and shape variations by explicitly encoding alignment in the network architecture.The paper states that the network therefore learns the PointNet representation rather than the alignment task itself.
2. Related Work
Prior registration methods include PointNet-based representations, ICP and global optimization, interest-point descriptors, hand-crafted features, voxel grids, and depth or range images. Their limitations involve correspondence cost, initialization sensitivity, computational demands, limited generalization, and integration constraints.
- Point-cloud representations: PointNet introduced deep networks that process raw point clouds for classification and segmentation, with later variants aggregating local point features.
- ICP and variants: ICP registers point clouds by iteratively estimating point correspondences and solving a least-squares optimization.
- ICP and variants: ICP variants have quadratic point-count complexity, initialization sensitivity, and differentiability issues for deep-learning integration.
- Global registration: Globally optimal registration methods address local-optimum limitations but can require computation times unsuitable for real-time applications.The cited approaches include branch-and-bound, Riemannian, semidefinite, and mixed-integer optimization.
- Interest points: Interest-point methods may improve registration speed but do not generalize to all applications.
- Hand-crafted representations: Hand-crafted representations and learned maps can be object-specific, while ICDO has quadratic complexity in the number of points.
- Alternate representations: Voxel methods impose computation and memory costs, whereas depth or range images enable convolution over aggregated 2D views.
3. PointNetLK
PointNetLK adapts Lucas–Kanade alignment to PointNet descriptors by estimating a finite-difference Jacobian and unrolling the resulting inverse-compositional optimization into a recurrent network. The method computes the template Jacobian once, iteratively updates the source pose, and evaluates alignment using transform-based training loss.
- 3.1. Overview: PointNet maps each point cloud through a shared MLP and symmetric pooling to a fixed-dimensional global descriptor.The symmetric operator supports point-order permutation invariance; both maximum and average pooling are considered.
- 3.1. Overview: PointNetLK formulates registration as finding a rigid transform G such that φ(PT) = φ(G · PS).The transform is represented in SE(3) using an exponential map with six twist parameters.
- 3.2. Derivation: The inverse-compositional formulation computes the template Jacobian once, then repeatedly solves for twist parameters and updates the source point cloud.The final pose estimate composes all incremental transforms, and iterations stop when the transform update falls below a threshold.
- 3.2. Derivation: Because PointNet descriptors lack the local dependencies required for conventional convolutional gradients, the method estimates the LK Jacobian with finite differences.Each Jacobian column perturbs one twist parameter, avoiding an analytical PointNet gradient or image-style convolution.
- 3.3. Training: Training minimizes the difference between the estimated and ground-truth rigid transforms using a computationally efficient transform-based loss.The chosen formulation avoids a matrix logarithm during training.
- 3.3. Training: PointNetLK aligns point clouds in fewer iterations than ICP and retains accurate alignment on categories unseen during training.The reported comparisons use 10 iterations for both methods; average pooling is confirmed to outperform max pooling on noisy point clouds.
4. Experiments
The experiments evaluate PointNetLK against ICP across standard, noisy, unseen-category, partially visible, and same-category/different-object registration settings. Results show accurate generalization, robustness under suitable training, favorable convergence behavior, and lower stated computational complexity than ICP.
- Experimental setup: PointNetLK is evaluated against ICP on ModelNet40 using varied training and test data, symmetric pooling operators, and registration scenarios.The experiments use ModelNet40 and compare ICP at test time.
- Standard registration: PointNetLK typically converges to the correct solution in many fewer iterations than ICP under matched test point clouds and perturbations.Testing uses initial translations in [0, 0.3] and rotations in [0, 90] degrees, while both methods receive the same inputs and perturbations.
- Unseen categories: PointNetLK accurately aligns object categories unseen during training and generalizes to the substantially different Stanford bunny dataset.The model is trained on one 20-category subset of ModelNet40 and tested on the other 20 categories.
- Partially visible data: For partially visible data, training PointNetLK on partially visible inputs greatly improves performance and surpasses ICP on unseen test categories.A registration is successful when rotation error is below 5 degrees and translation error below 0.01; visibility is iteratively re-sampled during alignment.
5. Implementation Details
Implementation choices set numerical perturbation, convergence, training duration, and architecture behavior for different data conditions. Noisy and partially visible data require longer training, while 2.5D inputs require differentiability-preserving modifications.
- Numerical settings: PointNetLK uses a 1e−2 perturbation for Jacobian twist parameters and stops when each |∆ξ_i| is below 1e−7.The MLP dimensions are (3, 64, 64, 64, 128, K = 1024).
- Training settings: Fine-tuning uses 200 epochs normally, approximately 300 for noisy data, and approximately 400 for partial-visibility data.Training on 2.5D data also requires architecture modifications to maintain differentiability.
- Computational cost: PointNetLK’s computation cost grows as O(n) with point count, compared with O(n^2) for ICP.The implementation comparison omits a kd-tree from ICP for the reported timing experiment.
6. Conclusion
PointNetLK adapts PointNet for point cloud registration by modifying Lucas–Kanade and unrolling it into a recurrent network. It achieves precision, initialization robustness, efficiency, noisy and partial-data gains, and generalization beyond training shapes.
- PointNetLK modifies Lucas–Kanade to accommodate PointNet's inability to estimate gradients through convolution.
- The modified Lucas–Kanade framework is unrolled as a recurrent neural network and integrated with PointNet.
- PointNetLK achieves precision, robustness to initialization, and computational efficiency.
- Training on noisy or partially visible data produces large performance gains while preserving generalization to shapes far removed from the training set.
- The approach provides a differentiable, generalizable, and extendable strategy for point cloud registration.