Source-linked AI summary
CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registration
Dongxu Zhang, Yingsen Wang, Yiding Sun, Haoran Xu, Peilin Fan, Jihua Zhu
TL;DR
Robust point cloud registration remains difficult under sensor imperfections, occlusions, and low overlap. CMHANet fuses 2D visual context with 3D geometry through hybrid attention and a geometric-semantic optimization objective. It reports leading Registration Recall on 3DMatch and 3DLoMatch, while retaining limitations in extreme low-overlap and featureless settings.
Problem
Real-world noise, sparsity, occlusions, non-uniform sampling, and low overlap can degrade learned features and hinder reliable point correspondences.
Method
CMHANet fuses 2D texture with 3D geometry using hybrid attention and an optimization objective promoting geometric fidelity and semantic coherence.
Results
CMHANet achieves 92.4% Registration Recall on 3DMatch and 75.5% on 3DLoMatch, with leading performance across overlap conditions.
Takeaways & Limitations
Cross-modal fusion improves reported registration accuracy and robustness relative to single-modal approaches, at a modest increase in inference time and memory.
Takeaways & Limitations
The model may exhibit registration deviations below 10% overlap or on strictly featureless, flat surfaces where geometric and textural cues are scarce.
Abstract
from arXiv · showhide
Robust point cloud registration is a fundamental task in 3D computer vision and geometric deep learning, essential for applications such as large-scale 3D reconstruction, augmented reality, and scene understanding. However, the performance of established learning-based methods often degrades in complex, real world scenarios characterized by incomplete data, sensor noise, and low overlap regions. To address these limitations, we propose CMHANet, a novel Cross-Modal Hybrid Attention Network. Our method integrates the fusion of rich contextual information from 2D images with the geometric detail of 3D point clouds, yielding a comprehensive and resilient feature representation. Furthermore, we introduce an innovative optimization function based on contrastive learning, which enforces geometric consistency and significantly improves the model's robustness to noise and partial observations. We evaluated CMHANet on the 3DMatch and the challenging 3DLoMatch datasets. \rev{Additionally, zero-shot evaluations on the TUM RGB-D SLAM dataset verify the model's generalization capability to unseen domains.} The experimental results demonstrate that our method achieves substantial improvements in both registration accuracy and overall robustness, outperforming current techniques. We also release our code in \href{https://github.com/DongXu-Zhang/CMHANet}{https://github.com/DongXu-Zhang/CMHANet}.
1. INTRODUCTION
Point cloud registration is important but difficult on noisy, sparse, and irregular real-world data because geometric methods can miss 2D contextual cues. CMHANet addresses this gap by fusing 3D geometry and 2D information through cross-modal hybrid attention, achieving leading benchmark recall.
- Point cloud registration aligns multiple 3D point sets and supports applications including scene reconstruction, augmented reality, and object shape analysis.
- Real-world noise, sparsity, irregular sampling, and omitted image context make robust registration challenging for geometry-focused methods.
- Fusing complementary 3D geometry with dense, texture-rich 2D context can improve correspondence matching and registration robustness.
- CMHANet uses separate point-cloud and image encoders followed by a multi-stage cross-modal attention pipeline to refine geometric-visual interactions.
- Its hybrid attention mechanism models the interplay between 2D and 3D features for precise, adaptive multimodal correspondence matching.
2. RELATED WORK
Prior registration methods span optimization-centric and correspondence-based approaches, but real-world imperfections and unimodal geometric representations remain major challenges. Multimodal fusion and attention motivate CMHANet’s more adaptive geometric-visual correspondence framework.
- 2.1. Correspondence-based Point Cloud Registration Methods: Point cloud registration estimates a rigid transformation aligning multiple 3D point sets in a common coordinate system.
- 2.1. Correspondence-based Point Cloud Registration Methods: ICP-based optimization methods are sensitive to initialization, can converge to local minima, and remain limited under large misalignments or low overlap.
- 2.1. Correspondence-based Point Cloud Registration Methods: Deep correspondence methods increasingly use attention to capture global context and fine-grained local geometric details in point clouds.
- 2.1. Correspondence-based Point Cloud Registration Methods: Many deep learning methods remain geometric-only, limiting feature discriminability when texture is informative or geometric cues are ambiguous.
- 2.2. Multimodal Fusion and Attention Mechanisms: Multimodal fusion combines sparse 3D geometry with dense 2D texture and semantic context, as explored by methods including IMFNet and CMIGNet.
- 2.2. Multimodal Fusion and Attention Mechanisms: Attention in Transformer architectures models long-range dependencies and supports robust inter- and intra-modal correspondences.
- 2.2. Multimodal Fusion and Attention Mechanisms: Generic fusion mechanisms leave adaptive geometric-visual interaction as an open design challenge, which CMHANet targets with hybrid attention.
- 2.3. Robustness Challenges in Point Cloud Registration: Noise, sparsity, occlusion, non-uniform sampling, low overlap, and repetitive structures degrade features and hinder reliable correspondence establishment.
3. METHODS
CMHANet performs multimodal point cloud registration by fusing 3D geometry and 2D visual information through hybrid attention, then refining matches into dense correspondences and estimating rigid alignment. Its coarse-to-fine design combines geometric and cross-modal feature processing with global verification for robust transformation estimation.
- Feature Extraction and Downsampling: CMHANet uses parallel point-cloud and image encoders to extract geometric and visual features while downsampling the point clouds into representative superpoints.The point-cloud encoder produces sparse grouped keypoints, while the image encoder extracts corresponding 2D features.
- Superpoint Matching Module with Hybrid Attention: The matching module generates robust superpoint associations that guide dense point-to-point correspondence estimation across the original full-resolution point clouds.Dense refinement uses coarse matches to resolve local ambiguities and construct a detailed alignment relationship map.
- Transformation Estimation: The transformation stage computes a rigid rotation and translation from dense correspondences to align the source and target point clouds.The registration objective represents the transformation as T = (R, t), with R ∈SO(3) and t ∈R3.
- Superpoint Matching Module with Hybrid Attention: Hybrid Attention combines self-attention, aggregation-attention, and cross-attention to capture global geometry, fuse image context, and establish source-target consistency.The three modules alternate for N iterations to progressively refine hybrid features for superpoint matching.
- Estimator Computation: Local-to-global verification filters incorrect local estimates caused by symmetric structures, producing a globally consistent transformation without RANSAC's nondifferentiability.CMHANet's high-inlier correspondences support accurate registration while reducing computational costs.
- Overall Objective: The overall loss combines global alignment, local precision, and cross-modal feature consistency in a coarse-to-fine optimization strategy.The formulation is expressed as L = Lc + Lf + λLcmc, where λ is a hyperparameter.
4. EXPERIMENT
CMHANet is evaluated on multimodal 3DMatch and 3DLoMatch benchmarks, using recall, error, estimator, generalization, qualitative, and ablation analyses. It achieves strong registration performance and robustness, while remaining limited in extreme low-overlap or featureless settings.
- Benchmark results: CMHANet achieves Registration Recall of 92.4% on 3DMatch and 75.5% on 3DLoMatch.The results indicate robust correspondence identification under both standard and low-overlap conditions.
- Multimodal comparison: CMHANet outperforms PCR-CG by +9.2% RR on 3DLoMatch in comparison with multimodal registration methods.The comparison includes PCR-CG and IMFNet.
- Robustness and efficiency: With LGR, CMHANet reaches 91.9% RR on 3DMatch and 74.2% RR on 3DLoMatch, while pose estimation is over 100 times faster than RANSAC.The model remains competitive across RANSAC, Weighted SVD, and LGR estimators.
- Cross-domain generalization: A zero-shot model trained only on 3DMatch achieves mean RMSE of 0.76 (10^-2) on unseen TUM RGB-D SLAM data.The evaluation uses the TUM dataset without fine-tuning and reports low errors on several sequences.
- Limitations: CMHANet may exhibit registration deviations below 10% overlap or on strictly featureless, flat surfaces.These conditions provide the stated boundary for future robustness improvements.
- Ablation analysis: Ablations show that removing the loss, Hybrid Attention, or Image Module reduces performance, while the full model improves alignment in challenging scenes.The Hybrid Attention and cross-modal feature transformation are identified as important for correspondence quality and visual alignment.
5. CONCLUSION
CMHANet combines multimodal features with a two-stage coarse-to-fine correspondence process for point cloud registration. This fusion improves registration accuracy and robustness, while modestly increasing inference time and memory use.
- CMHANet uses a two-stage correspondence process that progresses from coarse matching to fine alignment.
- Cross-modal fusion augments 3D geometric details with contextual features from 2D images through enhanced attention.
- The method improves registration accuracy and robustness compared with single-modal approaches, with a modest increase in inference time and memory footprint.
- Future work targets low-overlap conditions, including decoupling rotation and translation computations to improve alignment.
6. Future Work
The cross-modal hybrid attention architecture may extend beyond spatial alignment to applications in other interdisciplinary domains.
- The cross-modal hybrid attention architecture has potential for broader interdisciplinary applications beyond spatial alignment.