PaperlayerView exact revision on arXiv

Source-linked AI summary

REGTR: End-to-end Point Cloud Correspondences with Transformers

Zi Jian Yew, Gim Hee Lee

arXiv:2203.14517v1cs.CV

TL;DR

Point cloud registration methods often depend on nearest-neighbor matching and RANSAC after feature learning. REGTR uses transformer self- and cross-attention to predict final correspondences and overlap scores directly, then estimates the rigid transformation from them. It achieves state-of-the-art performance on scene and object benchmarks, with a limitation from quadratic transformer complexity on large point sets.

  • Problem

    Many learned registration methods still require nearest-neighbor matching and RANSAC after feature learning, and post-processing choices can affect performance.

  • Method

    REGTR uses transformer self- and cross-attention to predict clean final point correspondences and overlap scores, enabling direct rigid-transformation estimation.

  • Results

    REGTR achieves state-of-the-art performance on both scene and object point cloud datasets, including 3DMatch and ModelNet.

  • Takeaways & Limitations

    Direct correspondence prediction removes additional nearest-neighbor matching and RANSAC steps while retaining precise alignments with a small number of correspondences.

  • Takeaways & Limitations

    Quadratic-complexity transformer layers restrict REGTR to downsampled point clouds and prevent use on large numbers of points.

Abstract

from arXiv · show

Despite recent success in incorporating learning into point cloud registration, many works focus on learning feature descriptors and continue to rely on nearest-neighbor feature matching and outlier filtering through RANSAC to obtain the final set of correspondences for pose estimation. In this work, we conjecture that attention mechanisms can replace the role of explicit feature matching and RANSAC, and thus propose an end-to-end framework to directly predict the final set of correspondences. We use a network architecture consisting primarily of transformer layers containing self and cross attentions, and train it to predict the probability each point lies in the overlapping region and its corresponding position in the other point cloud. The required rigid transformation can then be estimated directly from the predicted correspondences without further post-processing. Despite its simplicity, our approach achieves state-of-the-art performance on 3DMatch and ModelNet benchmarks. Our source code can be found at https://github.com/yewzijian/RegTR .

1. Introduction

REGTR addresses the reliance on nearest-neighbor matching and RANSAC by using transformer attention to predict final correspondences directly. Its predicted correspondences support direct pose estimation, while experiments report state-of-the-art performance on 3DMatch and ModelNet.

  • Motivation: Learned registration methods commonly retain nearest-neighbor matching and RANSAC after feature learning, making performance sensitive to post-processing choices.The conventional pipeline detects keypoints, computes descriptors, matches features, and robustly estimates the transformation.
  • Motivation: REGTR uses self and cross attention to predict a consistent set of final point correspondences instead of relying on explicit matching or RANSAC.The approach targets the correspondence output directly rather than only learning more discriminative features.
  • Approach: REGTR directly computes the rigid transformation from its predicted correspondences without additional nearest-neighbor matching or RANSAC.The overview describes the network as outputting final correspondences and overlap scores for direct transformation estimation.
  • Approach: The network extracts point features, aggregates global information with transformer layers and positional encodings, and predicts corresponding transformed locations.Positional encodings let the network use point positions and rigidity constraints when correcting correspondences.
  • Results: REGTR achieves state-of-the-art performance on the 3DMatch and ModelNet datasets.The reported contribution emphasizes precise alignments despite using a small number of correspondences.

2. Related Work

Prior correspondence-based methods often learn descriptors and then use nearest-neighbor matching and RANSAC, while related attention methods mainly improve descriptors. REGTR instead uses attention to predict point correspondences directly.

  • Correspondence-based registration: Correspondence-based registration typically learns or designs keypoint descriptors before robustly estimating the rigid transformation.Many learned methods optimize descriptor similarity between matching and non-matching points.
  • Correspondence-based registration: Most descriptor-learning approaches still rely on subsequent nearest-neighbor matching and RANSAC to select correct correspondences.These post-processing steps remain separate from descriptor training.
  • Attention-based methods: REGTR applies transformer attention to directly output a consistent set of final point correspondences.This distinguishes its use of attention from approaches that use attention primarily for feature aggregation.

3. Problem Definition

Point cloud registration seeks the rotation and translation that align a source point cloud with a target point cloud.

  • Problem definition: The registration objective is to recover a rigid transformation consisting of rotation R ∈ SO(3) and translation t ∈ R3.The source and target point clouds are denoted X ∈ R^M×3 and Y ∈ R^N×3, respectively.

4. Our Approach

REGTR converts downsampled point-cloud features into bidirectional transformed keypoint locations and overlap scores, then estimates the rigid transformation directly from predicted correspondences. Transformer self- and cross-attention supplies within-cloud context and cross-cloud comparison, while end-to-end losses supervise overlap, correspondence, and feature learning.

  • 4.1. Feature Extraction: The framework downsamples both point clouds into keypoints with associated features before transformer processing.KPConv produces reduced keypoint sets and features; REGTR predicts transformed keypoint locations directly at this resolution.
  • 4.3. Correspondence Prediction: REGTR predicts transformed locations in both directions and overlap probabilities for every downsampled keypoint.Rows of the original and predicted locations define correspondences, while overlap scores identify points in the shared region.
  • 4.2. Transformer Cross-Encoder: Each cross-encoder layer combines separate self-attention, cross-attention, and a position-wise feed-forward network.Cross-attention compares the two point clouds, while self-attention lets points interact within one cloud when predicting transformed positions.
  • 4.4. Estimation of Rigid Transformation: The rigid transformation is estimated from concatenated predicted correspondences using overlap scores as weights.The paper solves the weighted objective in closed form with a weighted Kabsch-Umeyama algorithm.
  • 4.5. Loss Functions: Training uses overlap, correspondence, and feature losses supervised with ground-truth poses and overlap labels.Overlap scores use binary cross entropy, transformed locations use an L1 loss in overlapping regions, and conditioned features use InfoNCE.
  • 4.5. Loss Functions: Downsampled overlap labels are obtained by average pooling dense ground-truth labels using the KPConv downsampling indices.Dense labels compare transformed points with spatial nearest neighbors under a predefined overlap threshold.

5. Experiments

REGTR is evaluated across real and synthetic point-cloud benchmarks, with analyses of accuracy, runtime, correspondence quality, and architectural choices. It achieves strong registration performance while directly predicting correspondences and avoiding RANSAC in the main pipeline.

  • 3DMatch and 3DLoMatch: REGTR achieves the highest average registration recall across both 3DMatch and 3DLoMatch, while also obtaining the lowest RTE and RRE.These results hold despite using only a small number of points for pose estimation.
  • ModelNet40: REGTR substantially outperforms all baselines across every metric on both ModelNet and ModelLoNet overlap regimes.ModelNet averages 73.5% pairwise overlap, whereas ModelLoNet averages 53.6% and is used for direct generalization.
  • Runtime: Under 100ms, REGTR runs faster than correspondence-based pipelines whose pose estimation includes feature matching and RANSAC.Predator takes 234ms when sampling 1,000 points, excluding KPConv preprocessing.
  • Correspondence accuracy: 0.028m is REGTR’s median predicted-correspondence error, below the 0.112m median keypoint distance and the 0.071m oracle-matcher error.The authors attribute the precise registration to overcoming resolution issues caused by downsampling.
  • Correspondence accuracy: REGTR predicts accurate correspondences within overlap regions and projects non-overlapping points near overlap boundaries, suggesting use of rigidity constraints.Visualization shows short correspondence-error lines even in non-informative regions such as floors.
  • Ablations: Performance generally improves with more cross-encoder layers but saturates around L = 6, the configuration used in all experiments.The network cannot function without cross-encoder layers.
  • Ablations: Applying RANSAC to REGTR’s predicted correspondences produces marginally worse registration recall, indicating no further benefit from RANSAC.The predicted correspondences are already consistent with a rigid transformation.
  • Ablations: Weighted-sum coordinate decoding slightly improves RTE and RRE but lowers registration recall relative to MLP coordinate regression.The comparison is reported in rows 2 and 6 of Table 4.

6. Limitations

REGTR’s quadratic-complexity transformer layers prevent operation on large numbers of points, so the method is applied only to downsampled point clouds.

  • Quadratic-complexity transformer layers prevent REGTR from operating on large numbers of points.The authors therefore apply the transformer layers only to downsampled point clouds.
  • REGTR’s direct correspondence prediction alleviates, but does not eliminate, the resolution issue caused by downsampling.
  • Linear-complexity transformer layers produced subpar performance, while sparse attention and coarse-to-fine registration are proposed as alternative workarounds.

7. Conclusions

The paper proposes REGTR, which uses multiple transformer layers to directly predict clean correspondences for rigid point cloud registration and estimates the rigid transformation without nearest-neighbor matching or RANSAC.

  • REGTR directly predicts clean point correspondences using multiple transformer layers.
  • The rigid transformation is estimated from the predicted correspondences without further nearest-neighbor feature matching or RANSAC.
  • Direct correspondence prediction overcomes resolution issues from downsampled features.
  • REGTR achieves state-of-the-art performance on both scene and object point cloud datasets.

Supplementary Material

The supplementary material adds dataset and preprocessing details, explains rigid-transformation recovery and positional encodings, expands the network description, and reports additional results.

  • Section A provides additional details on datasets and their preprocessing.
  • Section B describes recovering the rigid transformation from predicted correspondences.
  • Sections C and D cover sinusoidal position encodings and additional network-architecture information.
  • Sections E and F provide detailed ScanNet results and additional qualitative results.

A. Dataset Details

The supplementary material describes the 3DMatch and ModelNet40 datasets, rigid-transformation recovery, positional encodings, and the REGTR architecture, including an ablation showing the importance of positional encoding.

  • Dataset Details: ModelNet40 provides CAD models from 40 object categories, with 2,048 sampled points scaled to a unit sphere and partial scans generated by random half-space sampling.
  • Dataset Details: The supplementary material documents raw 3DMatch data sources and their licenses in Table 5.
  • Rigid Transformation Estimation: Rigid transformation recovery uses weighted centroids, centered correspondences, a 3 × 3 weighted covariance matrix, Kabsch rotation recovery, and translation estimation.
  • Position Encodings: Coordinates are encoded with generalized 3D sinusoidal positional encodings of dimension d = 256, padding four elements because 256 is not divisible by 6.
  • Experimental Comparisons: Table 6 compares weighted-coordinate and regression correspondence decoding across different positional-encoding choices.
  • Position Encodings: Removing positional encodings causes a significant drop in registration recall and accuracy, supporting their role in exploiting rigidity constraints.
  • Network Architecture: The transformer cross-encoder projects KPConv features to d = 256 and processes them through L = 6 self- and cross-attention layers before decoding correspondences and overlap scores.

E. Detailed Registration Results for 3DMatch

REGTR shows strong scene-level registration results on 3DMatch and 3DLoMatch, including high recall and low rotation and translation errors despite downsampled features.

  • REGTR obtains the highest registration recall in three 3DMatch scenes and four 3DLoMatch scenes.
  • REGTR achieves the lowest rotation and translation errors for a majority of scenes across both settings.
  • Figure 9 presents REGTR’s transformer cross-encoder layers.

F. Additional Qualitative Results

Additional qualitative results cover 3DMatch, 3DLoMatch, ModelNet, and ModelLoNet, while also showing failure cases and the KPConv backbone configuration.

  • The KPConv backbone is shown for the 3DMatch and ModelNet datasets.
  • The detailed backbone figure includes a residual block labeled ResBlockB (1024, 1024, /2).
  • Figure 11 presents qualitative results across 3DMatch, 3DLoMatch, ModelNet, and ModelLoNet benchmarks.Keypoints are colored by predicted overlap scores, with red indicating high overlap.
  • The final two rows of Figure 11 show example 3DMatch failure cases.During failures, both overlap and correspondences are usually predicted incorrectly.
  • Table 7 reports detailed results on the 3DMatch and 3DLoMatch datasets.Results for 3DSN, FCGF, D3Feat, and Predator-5k are taken from reference [23].
Loading 2203.14517v1…