PaperlayerView exact revision on arXiv

Source-linked AI summary

Rotation-Invariant Transformer for Point Cloud Matching

Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, Slobodan Ilic

arXiv:2303.08231v3cs.CV

TL;DR

Point cloud matching requires descriptors that remain consistent across pose changes, while augmentation-based rotation handling cannot cover continuous rotations reliably. RoITr introduces intrinsically rotation-invariant local and global transformers, and experiments show strong performance across rigid and non-rigid benchmarks, especially under enlarged rotations and low overlap.

  • Problem

    Point cloud matchers need rotation-invariant descriptors, but finite rotation augmentation cannot span continuous SO(3) and can be unstable for rarely seen rotations.

  • Method

    RoITr uses PPF-based attention and an attention-based encoder-decoder for local geometry, plus a global transformer with rotation-invariant cross-frame position awareness.

  • Results

    RoITr outperforms state-of-the-art models on rigid and non-rigid benchmarks, especially in low-overlap scenarios and under enlarged rotations.

  • Takeaways & Limitations

    Intrinsic rotation-invariant local geometry and global cross-frame spatial awareness make RoITr robust to arbitrary pose variations within the evaluated matching scenarios.

  • Takeaways & Limitations

    RoITr is slower than convolution-based methods, struggles with symmetric structures and extremely limited overlap, and larger models require more training data.

Abstract

from arXiv · show

The intrinsic rotation invariance lies at the core of matching point clouds with handcrafted descriptors. However, it is widely despised by recent deep matchers that obtain the rotation invariance extrinsically via data augmentation. As the finite number of augmented rotations can never span the continuous SO(3) space, these methods usually show instability when facing rotations that are rarely seen. To this end, we introduce RoITr, a Rotation-Invariant Transformer to cope with the pose variations in the point cloud matching task. We contribute both on the local and global levels. Starting from the local level, we introduce an attention mechanism embedded with Point Pair Feature (PPF)-based coordinates to describe the pose-invariant geometry, upon which a novel attention-based encoder-decoder architecture is constructed. We further propose a global transformer with rotation-invariant cross-frame spatial awareness learned by the self-attention mechanism, which significantly improves the feature distinctiveness and makes the model robust with respect to the low overlap. Experiments are conducted on both the rigid and non-rigid public benchmarks, where RoITr outperforms all the state-of-the-art models by a considerable margin in the low-overlapping scenarios. Especially when the rotations are enlarged on the challenging 3DLoMatch benchmark, RoITr surpasses the existing methods by at least 13 and 5 percentage points in terms of Inlier Ratio and Registration Recall, respectively.

1. Introduction

Point cloud correspondence estimation depends on pose-invariant geometry, but recent deep matchers often obtain rotation invariance through augmentation and can become unstable under rarely seen rotations. RoITr addresses this with rotation-invariant local geometry encoding and global cross-frame position awareness.

  • Motivation: Correspondence estimation matches descriptors between partially overlapping point clouds observed from different views.The task supports applications including tracking, reconstruction, pose estimation, and 3D representation learning.
  • Results: RoITr ranks first on both 3DLoMatch and Rotated 3DLoMatch while showing the best robustness to enlarged rotations.
  • Motivation: Pose invariance is central because the same geometry must be described identically under different transformations.
  • Motivation: Recent deep models learn descriptors from raw points, while intrinsic rotation-invariant methods can lose global context and feature distinctiveness.For example, symmetric human halves may receive nearly identical descriptions without contextual cues.
  • RoITr: RoITr uses PPF-based local coordinates and attention-based encoder-decoder layers to learn discriminative, rotation-invariant geometry.
  • RoITr: RoITr adds a global transformer with rotation-invariant cross-frame position awareness to enhance feature distinctiveness.

2. Related Work

Related work spans rotation-sensitive deep matchers, intrinsically rotation-invariant descriptors, and methods that add global context or geometric fusion. These approaches leave challenges in rotation robustness, cross-frame spatial awareness, or local geometry representation.

  • Rotation-sensitive methods: Many deep point cloud matching methods use rotation-sensitive inputs and become unstable under additional rotations.
  • Rotation-sensitive methods: Existing approaches include fully convolutional, keypoint-based, global-context, coarse-to-fine, non-rigid, and end-to-end coordinate-regression methods.
  • Intrinsic rotation invariance: Intrinsic rotation-invariant methods address pose changes but can lose global context and feature distinctiveness.RIGA adds global context, yet relies on PointNet for local geometry and uses a simple position representation for cross-frame relationships.
  • Geometric context: GeoTrans incorporates geometric information in global intraframe fusion, but cross-frame aggregation lacks spatial-position awareness.

3. Method

RoITr combines PPF-based pose-agnostic local geometry encoding with rotation-invariant global context aggregation for point-cloud matching. Its PPFTrans encoder-decoder builds local descriptors, while global transformers add cross-frame spatial awareness before coarse-to-fine correspondence estimation.

  • Method overview: RoITr matches partially overlapping point clouds by estimating correspondences between point sets under rigid or non-rigid mappings.
  • Correspondence estimation: After global enhancement, RoITr applies coarse-to-fine matching to obtain superpoint correspondences and refine them into point correspondences.
  • PPF Attention Mechanism: PAM projects geometric coordinates and learned context into attention components, then aggregates pose-agnostic messages into updated anchor features.The mechanism uses separate projections for geometric cues, positional encoding, queries, keys, and values.
  • PPF Attention Mechanism: PPF Attention Mechanism uses local PPF coordinates to encode geometry independently of varying poses.For each anchor point, neighboring support points are represented in a PPF-based local coordinate system before attention aggregation.
  • PPFTrans architecture: PPFTrans stacks encoder blocks with AAL downsampling and PAL feature updates, then uses decoder blocks with TUL upsampling, skip connections, and PAL refinement.The encoder produces coarser superpoint triplets, while the decoder reconstructs point-level triplets from these representations.
  • Global transformer: The global transformer combines GSM and PCM to aggregate rotation-invariant geometry and perform position-aware cross-frame context aggregation.GSM learns a rotation-invariant position representation, and PCM applies directional aggregation in both frame directions.

4. Experiment

RoITr is evaluated on rigid and non-rigid matching benchmarks, including rotated and low-overlap settings, with comparisons, ablations, runtime analysis, and correspondence-count tests. The results support strong performance and rotation robustness, while the method retains efficiency and training-data limitations.

  • Benchmark evaluation: RoITr is evaluated on rigid 3DMatch/3DLoMatch and non-rigid 4DMatch/4DLoMatch benchmarks, including rotated rigid data.Rigid evaluations also assess registration with RANSAC, while 4DMatch uses Inlier Ratio and Non-rigid Feature Matching Recall.
  • Rigid matching results: RoITr outperforms competing methods in Inlier Ratio on 3DMatch and 3DLoMatch and leads 3DLoMatch registration recall with a substantial advantage.On 3DMatch, its registration performance is comparable to GeoTrans and Lepard; on 3DLoMatch, it leads with GeoTrans.
  • Rigid matching results: RoITr remains robust under additional rotations, outperforming the other evaluated methods on the rotated benchmarks.The evaluation uses full-range rotations independently added to the two frames of each point-cloud pair.
  • Robustness analysis: RoITr maintains strong performance across varying numbers of correspondences, indicating stability when only a few correspondences are available.The corresponding rotated-data test is reported in the Appendix.
  • Non-rigid matching results: RoITr achieves outstanding non-rigid matching performance despite its rotation-invariance design being primarily intended for rigid scenarios.The comparison includes PWC, FLOT, Predator, Lepard, and GeoTrans on 4DMatch and 4DLoMatch.
  • Ablation study: PPF-based local coordinates produce the best ablation performance, while pooling-based abstraction degrades performance relative to the Attentional Abstraction Layer.Embedding PPF coordinates into PointTransformer improves its matching performance and makes it rotation-invariant, but remains behind RoITr.
  • Runtime analysis: RoITr’s local attention is faster than PointTransformer’s but remains slower than convolutional alternatives.Its efficiency advantage over PointTransformer increases with channel count and becomes more significant after parameter normalization.
  • Limitations: The appendix notes that larger models require more training data, limiting the practical benefit of scaling model size.This limitation is stated in the context of comparing models with different parameter counts.

5. Conclusion

The paper concludes that RoITr provides intrinsic rotation invariance for point-cloud matching through PPF-based local geometry and a global transformer. Experiments on rigid and non-rigid benchmarks demonstrate strong performance and robustness to arbitrary rotations.

  • Method: RoITr combines PPF-based local attention and a global transformer to encode representative, pose-agnostic geometry with rotation-invariant cross-frame spatial awareness.The local design composes PPFTrans through AAL, PAL, and TUL components.
  • Conclusion: Experiments on rigid and non-rigid benchmarks demonstrate RoITr’s superiority, especially its robustness against arbitrary rotations.The conclusion describes the experiments as extensive and covering both scenario types.

6. Appendix

The appendix supplies implementation details, architecture descriptions, geometric embeddings, losses, metrics, additional results, runtime analysis, and limitations. It details both the PPFTrans encoder-decoder and the global transformer used by RoITr.

  • Appendix scope: The appendix covers implementation, network architecture, geometric embedding, loss functions, evaluation metrics, additional results, runtime, and limitations.These topics are organized across Sections 6.1–6.8 and include quantitative and qualitative results.
  • PPFTrans encoder-decoder: PPFTrans uses four encoder blocks and four decoder blocks, with AAL aggregating local neighborhoods and PAL enhancing features with pose-agnostic geometry and learned context.Decoder blocks use TUL to subsample points and incorporate information from earlier stages.
  • Global transformer: The global transformer contains three blocks, each pairing geometry-aware self-attention with position-aware cross-attention.Self-attention aggregates global context within each cloud, while cross-attention transfers context between frames in both directions.
  • Geometric embedding: The geometric embedding represents pairwise superpoint relationships using rotation-invariant geometric information, including Euclidean distances and angles.Euclidean distance is projected into an even-dimensional sinusoidal embedding, while angle-based embeddings are computed from local neighbors.

6.4. Loss Function

The appendix defines the superpoint and point matching losses used by RoITr. These losses rely on overlap-weighted positive and negative superpoint samples and normalized point-similarity matrices.

  • Superpoint matching loss: Superpoint matching uses Circle Loss with overlap ratios weighting ground-truth superpoint correspondences.Positive samples are selected using an overlap threshold, and negative superpoints are sampled separately.
  • Architecture context: The appendix describes the PPFTrans and global-transformer architectures alongside the loss formulation.These architecture references contextualize where the matching objectives are used in the full model.
  • Superpoint matching loss: The overlap ratio is computed over superpoint vicinities and uses τr=0.1 by default to select positive samples.The loss applies these sample relationships to superpoints in both point clouds.
  • Point matching loss: Point matching operates on superpoint correspondences using a normalized similarity matrix with an appended learnable slack row and column.The Sinkhorn algorithm is applied iteratively, after which mutual top-k entries form point correspondences.

6.5. Detailed Metrics

The paper defines matching and registration metrics for rigid and non-rigid point-cloud benchmarks, using correspondence accuracy, pair-level recall, and deformation-flow recovery.

  • Rigid matching metrics: Inlier Ratio counts putative correspondences whose transformed Euclidean distance falls below dataset-specific thresholds.The thresholds are 0.1m on 3DMatch/3DLoMatch and 0.04m on 4DMatch/4DLoMatch.
  • Rigid matching metrics: Feature Matching Recall counts point-cloud pairs whose Inlier Ratio exceeds τ2 = 0.05.
  • Registration metric: Registration Recall measures the fraction of pairs registered correctly according to RMSE computed from estimated and ground-truth transformations.For original benchmarks, RMSE is defined using the ground-truth correspondence set, estimated transformation, and ground-truth transformation as specified.
  • Non-rigid matching metric: Non-Rigid Feature Matching Recall measures the fraction of ground-truth correspondences recovered by putative correspondences.
  • Non-rigid matching metric: For non-rigid evaluation, each putative correspondence defines a deformation flow as target point minus source point.Flows for ground-truth correspondences are then computed through interpolation using neighboring putative matches, with k = 3 in practice.

6.6. More Quantitative Results

Additional experiments examine RoITr under varying correspondence counts and through ablations on rotated and unrotated 3DMatch, where the default model achieves the best performance.

  • Varying correspondence number: RoITr maintains outstanding performance on both datasets across varying numbers of correspondences, demonstrating stability when few correspondences are available.The corresponding rotated-benchmark test is reported in the appendix.
  • Varying correspondence number: The rotated 3DMatch and 3DLoMatch experiments quantify performance across different numbers of sampled points or correspondences.
  • Ablation study: The default RoITr model achieves the best performance in the (Rotated) 3DMatch ablation study.This result further confirms the significance of each individual RoITr design.
  • Ablation study: Table 8 compares ablated models on (Rotated) 3DMatch using 5,000 points or correspondences and marks intrinsically rotation-invariant methods.

6.7. More Qualitative Results

The paper provides additional qualitative results for indoor scenes on 3DLoMatch and partially scanned deformable objects on 4DLoMatch.

  • Indoor scenes: Figure 10 presents more qualitative matching results on the challenging 3DLoMatch indoor-scene benchmark.
  • Deformable objects: Figure 11 presents qualitative results on 4DLoMatch, which consists of partially scanned deformable objects.

6.8. Runtime

The runtime comparison evaluates RoITr, Lepard, and GeoTrans under a fixed hardware, batching, and preprocessing protocol on the 3DMatch test set.

  • Evaluation setup: Runtime is compared with Lepard and GeoTrans on a single Nvidia RTX 3090 GPU and AMD Ryzen 5800X 3.80GHz CPU.
  • Evaluation setup: All models are tested without CPU parallelism and with batch size 1.
  • Evaluation setup: Reported times are averaged over the 1,623 point-cloud pairs in the 3DMatch testing set.
  • Runtime accounting: The runtime table separates data-preparation time from model time, including GPU nearest-neighbor search in the latter.

6.9. Limitations

RoITr has several limitations despite strong rigid and non-rigid performance: attention remains slower than convolutions, intrinsic rotation invariance struggles with symmetric structures, and extremely limited overlap can cause failures.

  • Attention runs faster than Point Transformer but remains slower than convolutions.The efficiency limitation is reported in the runtime comparison of Table 9.
  • Intrinsic rotation invariance can reduce RoITr’s ability to match symmetric structures.This drawback is observed in the 4DLoMatch data.
  • RoITr relies mainly on feature distinctiveness to filter occluded areas, which fails under extremely limited overlap.The paper identifies this reliance as a cause of failures on 3DLoMatch.
  • Extremely limited overlap can cause failures when flattened regions dominate the overlap.Reported 3DLoMatch failures include 17.6% overlap mainly on a floor and 10.7% overlap mainly on a wall.
  • Symmetric left-right structures create ambiguity in low-overlap 4DLoMatch cases.Examples include horse and pig point clouds with 18.1% and 17.9% overlap, respectively.
Loading 2303.08231v3…