PaperlayerView exact revision on arXiv

Source-linked AI summary

A comprehensive survey on point cloud registration

Xiaoshui Huang, Guofeng Mei, Jian Zhang, Rana Abbas

arXiv:2103.02690v2cs.CV

TL;DR

Point cloud registration is important across computer vision applications, but connections between optimization-based and deep learning methods remain unclear and cross-source alignment presents a newer challenge. This survey synthesizes registration methods and applications, introduces a cross-source benchmark, evaluates existing algorithms, and identifies future directions; current methods still struggle with cross-source data.

  • Problem

    Connections between optimization-based and deep learning registration methods remain unclear, while aligning cross-source point clouds has emerged as a challenging research direction.

  • Method

    The survey reviews same-source and cross-source registration methods, analyzes their connections, builds a cross-source benchmark, evaluates state-of-the-art algorithms, and summarizes applications and future directions.

  • Results

    Existing optimization-based, feature-metric, and correspondence-learning algorithms face difficulty aligning cross-source point clouds, with DGR achieving the best performance among them.

  • Takeaways & Limitations

    The benchmark provides a basis for choosing and developing registration approaches for cross-source point cloud applications.

  • Takeaways & Limitations

    Both accuracy and speed of current methods remain far behind the requirements of real-world applications.

Abstract

from arXiv · show

Registration is a transformation estimation problem between two point clouds, which has a unique and critical role in numerous computer vision applications. The developments of optimization-based methods and deep learning methods have improved registration robustness and efficiency. Recently, the combinations of optimization-based and deep learning methods have further improved performance. However, the connections between optimization-based and deep learning methods are still unclear. Moreover, with the recent development of 3D sensors and 3D reconstruction techniques, a new research direction emerges to align cross-source point clouds. This survey conducts a comprehensive survey, including both same-source and cross-source registration methods, and summarize the connections between optimization-based and deep learning methods, to provide further research insight. This survey also builds a new benchmark to evaluate the state-of-the-art registration algorithms in solving cross-source challenges. Besides, this survey summarizes the benchmark data sets and discusses point cloud registration applications across various domains. Finally, this survey proposes potential research directions in this rapidly growing field.

I. INTRODUCTION

Point cloud registration estimates transformations between partial scans to merge them into complete 3D scenes, supporting reconstruction, localization, and pose estimation. This survey reviews same-source and cross-source methods, clarifies optimization–deep-learning connections, and introduces a cross-source benchmark.

  • Motivation: Registration estimates transformation matrices between partial scans so they can be merged into complete 3D point clouds.The problem is central to constructing large 3D scenes from sensors with limited view ranges.
  • Applications: Registration supports 3D reconstruction, localization, and pose estimation in applications including autonomous driving, robotics, and underground mining.Examples include map construction, route planning, agent localization, and robotic-arm positioning.
  • Existing methods: Conventional registration alternates correspondence searching and transformation estimation to minimize geometric projection error.With accurate correspondences, transformation estimation has a close-form solution.
  • Existing methods: Deep learning methods learn distinctive 3D features and correspondences, while hybrid methods combine these strategies with conventional optimization.The survey identifies the connections between optimization-based and deep learning methods as unclear despite reported performance gains from hybrid frameworks.
  • Cross-source registration: Cross-source registration arises because sensors such as Kinect and LiDAR trade dense, short-range measurements for sparse, long-range measurements.Fusing these complementary sensor outputs creates a distinct registration problem.
  • Survey contributions: The survey reviews methods from 1992–2021, covers same-source and cross-source registration, and builds a benchmark for evaluating state-of-the-art cross-source algorithms.It also summarizes applications and proposes future research directions and open questions.

III. CHALLENGES

Point cloud registration faces noise, outliers, partial overlap, and sensor-dependent differences when scans are captured across times, views, or sources. Cross-source registration additionally involves density and scale variation, motivating a dedicated benchmark and continued method development.

  • Same-source challenges: Same-source scans captured at different times or viewpoints contain noise, outliers, and only partial overlap.These variations arise from changing environments and sensor conditions.
  • Cross-source challenges: Cross-source scans are more complicated because different sensor imaging mechanisms produce distinct noise, outlier, and overlap patterns.The paper distinguishes these challenges from same-source registration.
  • Cross-source challenges: Different imaging mechanisms and resolutions produce density differences between cross-source point clouds.Density variation is identified as a separate cross-source challenge.
  • Cross-source challenges: Different physical metrics can produce scale differences, so the survey builds a cross-source benchmark to evaluate algorithms against these challenges.The benchmark is intended to assess state-of-the-art registration methods on cross-source data.

IV. CATEGORIES

Point cloud registration is categorized into optimization-based, feature-learning, and end-to-end learning methods. These categories differ in how they estimate correspondences and transformations, and in their data, generalization, and modeling requirements.

  • Framework taxonomy: Figure 2 organizes same-source registration frameworks around distinct processing stages, including iterative optimization and neural-network-based feature or end-to-end pipelines.The framework descriptions distinguish optimization-based processing from feature-learning and end-to-end approaches.
  • Optimization-based registration: Optimization-based methods iteratively search correspondences and estimate the transformation matrix using optimization strategies.They require no training data and can generalize to unknown scenes, but handling noise, outliers, density variation, and partial overlap increases computation cost.
  • Feature learning methods: Feature-learning methods use deep networks to learn robust feature correspondences, followed by one-step transformation estimation such as RANSAC.Their limitations include large training-data requirements, reduced performance on distribution-shifted scenes, and separate feature-extraction training.
  • End-to-end learning-based registration: End-to-end methods take two point clouds as input and directly output a transformation matrix through a registration-specific neural network.Transformation estimation is embedded in network optimization rather than treated as a separate feature-learning stage.
  • End-to-end learning-based registration: End-to-end methods combine registration-specific neural optimization with conventional mathematical theories and deep neural networks.Regression methods remain sensitive to noise and density differences, while feature-metric methods account for local structure information important for registration.

D. Cross-source registration

ICP-based registration alternates correspondence estimation and transformation estimation, using geometric distance metrics and transformation solvers. The survey places these methods within broader optimization-based registration and notes practical limitations for cross-source alignment and 3D applicability.

  • ICP-based registration: ICP-based methods alternate correspondence estimation and transformation estimation, emphasizing robust correspondences and accurate transformation estimates.The two stages repeatedly update the registration solution.
  • Distance metrics: Point-point, point-plane, and plane-plane metrics provide alternative geometric criteria for finding correspondences or estimating transformations.Point-to-plane methods minimize orthogonal distance along surface normals, while plane-to-plane metrics use corresponding local planes.
  • Generalized ICP: Generalized ICP incorporates covariance matrices for both point-to-point and point-to-plane variants, with standard ICP forms obtained as special cases.The transformation parameters consist of rotation and translation, and the distance metric is part of the objective.
  • Transformation estimation: SVD-based, Lucas-Kanade, and Procrustes methods estimate transformations after correspondences are available.SVD-based methods decompose correspondence differences, Lucas-Kanade uses Jacobians and approximation methods, and Procrustes converts estimation into linear least squares.
  • Limitations: Procrustes registration depends strongly on correspondence accuracy and is reported here with experiments limited to 2D, leaving 3D research for future work.ProcrustesDTW automatically establishes correspondences but does not remove this stated 3D scope boundary.

B. Graph-based registration

Graph-based registration models point clouds as graphs and searches for correspondences using vertex and edge relationships. Its methods differ by matching order and by approximation strategies for the underlying hard optimization problem.

  • Graph formulation: Graph-based registration finds point correspondences between graphs by considering both vertices and edges.The correspondence search is formulated as an optimization problem, and more accurate correspondences support better transformation estimation.
  • Matching order: Graph matching methods are divided into second-order and high-order approaches according to the relationships they compare.Second-order methods compare vertex-to-vertex and edge-to-edge similarities, whereas high-order methods compare structures involving more than two points.
  • Optimization: Graph matching is a quadratic assignment problem that is NP-hard, so approximation strategies are central to solving it.Second-order methods use doubly stochastic, spectral, or semidefinite programming relaxation categories.
  • High-order matching: High-order methods compare hyper-edges or hyper-nodes and can remain invariant to affine variations such as scale difference.Examples include probabilistic approaches and triangle-similarity tensor optimization.

C. GMM-based registration

GMM-based registration reformulates point cloud alignment as likelihood maximization, estimating both the transformation and Gaussian-mixture parameters while improving robustness to noise and outliers. Related relaxation methods seek globally optimal correspondences but face substantial scalability limits.

  • GMM formulation: GMM-based methods formulate registration as likelihood maximization over the input data.Optimization estimates both the transformation matrix and Gaussian mixture model parameters, aligning distributions rather than individual points.
  • GMM variants: CPD adds motion-drift constraints, CH-GMM uses convex hulls to reduce computation, JRMPC recasts registration as clustering, and DeepGMR learns point-to-component correspondences.These variants extend the GMM framework through transformation constraints, geometric simplification, clustering, or learned matching.
  • Semi-definite relaxation: Semi-definite registration approximates the NP-hard quadratic correspondence assignment problem through alternative projection formulations.The resulting correspondence optimization can be expressed as a semi-definite optimization problem.
  • Symmetric matrix: A symmetric matching-potential matrix encodes pairwise correspondence compatibility under binary, row-sum, column-sum, and cardinality constraints.Solving the maximization yields correspondences, after which the transformation is calculated in closed form.
  • Laplacian matrix: Graph-Laplacian formulations exploit positive semidefiniteness for relaxation, while later methods target multiple sets, rotation subproblems, and computational efficiency.PSR-SDP addresses multiple point sets, Teaser certifies rotation optimality efficiently, and OPRASANC continues this direction.
  • Limitations: Semi-definite relaxation can achieve the global minimum, but is typically tractable only for small or medium point clouds, leaving efficiency unresolved.The cited discussion reports tractability up to 15 points for some formulations and identifies scalability as a remaining research problem.

VI. FEATURE-LEARNING METHODS FOR REGISTRATION

Feature-learning registration uses deep networks to learn distinctive correspondences before a one-step transformation estimate, with methods divided between volumetric and point-cloud representations. Volumetric approaches capture local structure but incur memory and rotation-sensitivity costs.

  • Framework: Feature-learning methods use deep features to estimate accurate correspondences, then compute transformation with one-step optimization such as SVD or RANSAC.This removes the iterative alternation between correspondence estimation and transformation estimation used by classical methods.
  • Volumetric data: 3DMatch converts point-cloud patches into volumetric data and extracts 512-dimensional local features from RGBD-trained networks.The feature incorporates local structure around each interest point.
  • Volumetric data: 3DMatch requires large GPU memory and is sensitive to rotation variations.These drawbacks arise from representing sparse point clouds as volumetric data.
  • Volumetric data: 3DSmoothNet aligns 3D patches using a local reference frame before smoothing voxelized inputs to obtain rotation-invariant features.The local reference frame is estimated from the eigendecomposition of the points’ covariance matrix.
  • Volumetric data: Octree-based methods reduce volumetric memory costs by removing or hierarchically representing empty voxels.OctNet, octree decoding, and O-CNN store or process features through sparse hierarchical structures.

B. Learning on point cloud

Point-cloud learning methods operate directly on geometry, while end-to-end methods predict transformations through regression or combine neural networks with conventional optimization. The latter combination reports better accuracy than previous methods.

  • Learning on point cloud: PPFNet learns rotation-invariant local descriptors directly from geometry using point-pair features and PointNet.It combines local and max-pooled global features in an MLP to produce correspondence-search features.
  • Learning on point cloud: 3DFeatNet uses weak supervision from GPS/INS-tagged triplets to learn feature correspondences without manually annotated matching clusters.Training uses anchor, positive, and negative point clouds with triplet loss.
  • Learning on point cloud: RPMNet learns soft point correspondences and handles partial visibility to reduce ICP sensitivity to initialization and noise or outliers.ICP’s hard closest-point assignments can otherwise lead to incorrect local minima.
  • End-to-end registration: End-to-end registration takes two point clouds as input and outputs the transformation matrix, either through regression or neural-network-plus-optimization frameworks.Regression directly fits transformation parameters, whereas hybrid methods combine learned components with registration optimization.
  • Registration by regression: Regression methods estimate pose, detect keypoints from matching probabilities, or classify inliers and regress transformations from them.RelativeNet, DeepVCP, and the inlier/outlier classification-regression framework exemplify these designs.
  • Registration by optimization and neural network: Hybrid methods pair learned features or correspondences with inverse-compositional, GMM, weighted-Procrustes, SVD, or related optimization modules.PointNetLK, DeepGMR, and DGR illustrate optimization embedded alongside neural feature extraction or correspondence prediction.
  • Registration by optimization and neural network: Hybrid optimization and deep-learning methods obtain better accuracy than previous registration methods.DeepGMR learns pose-invariant point-to-distribution correspondences before GMM transformation estimation, while DGR uses weighted Procrustes.

VIII. CROSS-SOURCE POINT CLOUD REGISTRATION

Cross-source registration aligns point clouds from different sensors and requires methods that address severe source differences. The survey reviews optimization and learning approaches, builds a benchmark, and frames hybrid optimization–deep-learning methods as a promising direction.

  • Cross-source registration: The survey reviews cross-source registration by separating existing methods into optimization-based and learning-based categories.It presents this as the first comprehensive review of cross-source point cloud registration.
  • Optimization-based methods: Optimization-based cross-source methods extend same-source strategies with more complicated formulations for severe cross-source challenges.Examples include high-order correspondence constraints, tensor optimization, and scale normalization before RANSAC.
  • Optimization-based methods: Optimization-based methods require no training data and can generalize to unknown scenes, but sophisticated strategies incur high computation costs and dataset-dependent performance.These methods retain mathematical convergence or performance guarantees within the cited scope.
  • Learning-based methods: FMR combines optimization with a deep neural network and estimates transformation by minimizing global feature difference under noise, outliers, and density difference.The survey identifies FMR as the first learning-based method for cross-source registration.
  • Benchmark: The survey builds a new cross-source benchmark to evaluate state-of-the-art registration algorithms because cross-source performance is less reported.The comparison is intended to provide insights for future research.
  • Connections: Deep learning can replace original point coordinates with learned features, while conventional optimization contributes theoretical convergence guarantees.The survey describes hybrid loss and transformation-estimation strategies as combining complementary strengths.
  • Connections: Existing hybrid approaches remain initial trials whose accuracy, robustness, and efficiency require further improvement.The survey presents advanced loss strategies combining optimization theory and neural networks as a research direction.
  • Benchmark: The evaluation section summarizes existing metrics, reports same-source results, introduces a cross-source dataset, and compares registration methods.Together, these components form a benchmark spanning same-source and cross-source registration.

A. Evaluation metrics

The survey reviews registration metrics, datasets, and cross-source benchmark construction. The proposed benchmark combines multiple sensor sources and explicitly captures cross-source noise, outliers, density differences, and partial overlap.

  • rmseP measures mean point-point projection error after transformation, while rmseT measures transformation error against ground truth.
  • The cross-source benchmark contains 202 point-cloud pairs captured using LiDAR, Kinect, and camera sensors, with manually aligned ground truth.It includes Kinect–RGB camera and LiDAR–Kinect acquisitions in indoor workspace scenes.
  • Cross-source pairs combine noise, outliers, density differences, and partial overlap because different sensors have distinct imaging mechanisms and noise.Figure 10 illustrates these mixed variations in a cross-source pair.

2) Evaluation:

The evaluation benchmarks same-source and cross-source registration methods using established baselines and a recall criterion. Existing state-of-the-art methods remain difficult to apply to cross-source data, with DGR performing best among the compared methods.

  • 2) Evaluation:: Registration recall counts point-cloud pairs with RE < 15° and TE < 0.3m divided by the total number of pairs.
  • 2) Evaluation:: The same-source baselines include FGR, FMR, and DGR, representing optimization-based, feature-metric, and correspondence-learning approaches.FMR and DGR are trained on 3DMatch and evaluated on the proposed cross-source benchmark.
  • 2) Evaluation:: The cross-source comparisons include registration components based on ICP, Gaussian mixture model alignment, RANSAC, and GCTR.Sampling is used because Gaussian mixture model alignment and GCTR have substantial memory or computation costs.
  • 2) Evaluation:: The compared optimization-based, feature-metric, and correspondence-learning methods still face difficulty aligning cross-source point clouds.DGR obtains the best performance among the existing methods evaluated.
  • 2) Evaluation:: Robust keypoint extraction is proposed as a promising direction because it can identify critical information in noisy point clouds for transformation estimation.DGR generates high probabilities for critical correspondences and uses them with weighted Procrustes estimation.

A. Construction

The survey presents point-cloud registration as a foundation for connecting digital models with physical environments and supporting spatial monitoring. Across construction, mining, autonomous driving, and robotics, accuracy and efficiency remain central requirements.

  • A. Construction: BIM uses 3D models and building properties, while point clouds align digital models with physical space for evaluation, visualization, and renovation.
  • A. Construction: Construction applications are limited by costly 3D sensors and slow capture and registration processes.A Leica RTC360 Laser Scanner Kit is cited as costing about $100,000, with 360° capture taking more than 20 minutes.
  • A. Construction: Mining point clouds support terrain visualization, tunnel monitoring, pit-wall instability detection, and development-progress assessment.
  • A. Construction: Registration merges multiple mining scans into larger scans, and its accuracy affects downstream applications such as coal-mine volume estimation.
  • A. Construction: Autonomous driving uses registration to create larger 3D scans and estimate vehicle pose, requiring high accuracy and real-time efficiency.The survey identifies registration with prior road information as a research direction.
  • A. Construction: Robotic registration generates 3D maps for search and rescue, inspection, shoreline monitoring, and environmental surveys.Accuracy and efficiency are identified as key requirements for robotics registration algorithms.

E. Other applications

The survey identifies accuracy, robustness, speed, and generalization as continuing challenges across geomechanical applications and future registration research. It emphasizes difficult real-world variation, partial overlap, computational cost, and the need to combine learning with geometric methods.

  • E. Other applications: Geomechanical applications use registration to compare point-cloud models over time for safety and stability monitoring, making accuracy essential.
  • E. Other applications: The survey frames future work around robust and accurate registration for same-source and cross-source data, together with fast execution.
  • E. Other applications: Real-world noise and outliers arise from sensor variation and environmental changes, while current methods remain behind application requirements in accuracy and speed.
  • E. Other applications: Deep learning can achieve 3cm registration accuracy on KITTI, but robustness and generalization to other datasets remain less reported.
  • E. Other applications: Registration refinement with ICP can be computationally expensive because modern point clouds often contain millions of points.Removing refinement can substantially reduce accuracy in methods that depend on ICP.
  • E. Other applications: Partial overlap can fall below 20%, and keypoint-based solutions depend strongly on keypoint-detection quality.The survey calls for robust methods for low-overlap registration.
  • E. Other applications: The survey reviews same-source and cross-source methods, connects optimization with deep learning, evaluates a cross-source benchmark, and proposes applications and future directions.
Loading 2103.02690v2…