PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration

Zhixin Cheng, Jiacheng Deng, Xiaotian Yin, Baoqun Yin, Richang Hong, Tianzhu Zhang

arXiv:2609.18088v1cs.CVcs.AI

TL;DR

Image-to-point-cloud registration must bridge substantial differences between image and point-cloud representations, while random MAE masking may miss registration-critical regions. M23D combines cross-modal reconstruction in a dual-MAE framework with similarity-initialized, reinforcement-learning-refined masking. It achieves state-of-the-art performance on RGB-D Scenes v2 and 7-Scenes.

  • Problem

    Image-to-point-cloud registration must bridge the modality gap between dense 2D image grids and sparse, unordered 3D points, while random masking may overlook critical regions.

    Source: §I · pp. 1–2

  • Method

    M23D combines a dual-MAE framework that uses point-cloud features to assist image reconstruction with similarity-initialized, reinforcement-learning-refined masking.

    Source: §I · p. 2

  • Results

    M23D achieves state-of-the-art performance on RGB-D Scenes v2 and 7-Scenes.

    Source: §I · p. 2, §V · p. 13

  • Takeaways & Limitations

    The method aligns modalities and establishes correspondences through dual-MAE learning, while adaptive masking focuses on key matching regions.

    Source: §I · p. 2

Abstract

from arXiv · show

Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.

I. INTRODUCTION

Image-to-point-cloud registration must bridge dense 2D images and sparse, unordered 3D points, whose modality gap complicates cross-modal matching. M23D combines a dual-MAE framework with adaptive masking, and the paper reports state-of-the-art results on two benchmarks.

I. INTRODUCTION

  • Image-to-point-cloud registration estimates a rigid transformation, but dense image grids and sparse, unordered point clouds create a significant modality gap.Source: §I · p. 1
  • M23D combines the Intermodal Dual-MAE Framework (ID-MAE) and Similarity-based RL Masking Strategy (SRLM) for image-to-point-cloud registration.Source: §I · p. 2
  • ID-MAE uses point-cloud features to assist image reconstruction, with low-resolution point-cloud constraints and original-image supervision to align modalities and establish correspondences.Source: §I · p. 2
  • SRLM initializes masks in overlapping regions using cross-modal similarity, then applies reinforcement-learning refinement to select high-value regions.Source: §I · p. 2
  • The authors position M23D as an early-representation approach that complements methods refining feature interactions or manifold alignment after extraction.Source: §I · p. 2
  • M23D achieves state-of-the-art performance on RGB-D Scenes v2 and 7-Scenes.Source: §I · p. 2

II. RELATED WORK

Prior 2D–3D methods use varied forms of feature interaction and masking, but the cited review identifies no method that optimizes masks for cross-modal overlap, correspondence reliability, or registration. M23D combines cross-modal reconstruction with adaptive masking for registration-oriented correspondence learning.

II. RELATED WORK

  • The Fig. 3 pipeline processes dual-MAE features with attention, uses cosine similarity and maximum operations for coarse matching and fine correspondence refinement, then applies PnP+RANSAC.Source: Figure 3 · p. 3
  • PiMAE, I2P-MAE, CrossNet, Inter-MAE, and Joint-MAE use distinct cross-modal representation or reconstruction designs, but their approaches include projection-based, fixed-semantic, or random masking.Source: §II · p. 3
  • The reviewed approaches do not explicitly optimize mask generation for cross-modal overlap, correspondence reliability, or the registration objective.Source: §II · p. 3
  • M23D instead uses a dual-MAE framework with cross-modal reconstruction and SRLM, which initializes masks from bidirectional similarity and refines them with reinforcement learning.Source: §II · p. 3

A. Overview

M2-3D constructs modality-aligned image and point-cloud features through an intermodal dual-MAE framework. Attention and similarity support patch matching, followed by dense correspondence refinement and rigid-transform estimation.

A. Overview

  • Image-to-point-cloud registration estimates the rigid transformation from the point-cloud coordinate system to the camera coordinate system.Source: A. Overview · p. 4
  • M2-3D uses an intermodal dual-MAE framework to construct modality-aligned image and point-cloud features for detection-free registration.Source: A. Overview · p. 4
  • Attention and similarity generate patch-level matches, high-resolution features refine dense correspondences, and PnP+RANSAC regresses the rigid transformation.Source: A. Overview · p. 4

B. Intermodal Dual-MAE Framework

ID-MAE uses unsupervised masked reconstruction during normal training to enhance feature extraction across images and point clouds. Its point-cloud features guide image reconstruction, while a separate point-cloud branch reconstructs masked points.

Framework design

  • ID-MAE applies unsupervised MAE during normal training to enhance encoder features and aggregate modalities; image and point-cloud features are extracted with ResNet/FPN and KPFCNN, respectively.The extracted lowest-resolution features receive positional encoding.Source: B. Intermodal Dual-MAE Framework · pp. 4–5

Dual-modal MAE Enhancement

  • The point-cloud MAE masks features, fills masked positions with learnable tokens, and reconstructs point clouds with L2 supervision.Its encoder uses local self-attention, and the decoder uses point-cloud cross-attention and self-attention blocks.Source: 1) Dual-modal MAE Enhancement · p. 5

Training and inference

  • The low-resolution point cloud is a training-only reconstruction target and is removed at inference, while finer-scale point features remain for matching and registration.The backbone's hierarchical point pyramid determines the low-resolution set used by the point MAE.Source: 1) Dual-modal MAE Enhancement · p. 5

Cross-Modal Guided Reconstruction

  • Point-cloud features serve as keys and values to reconstruct masked image features, aggregating relevant features by similarity to support cross-modal alignment.The image feature map is completed with learnable mask tokens before cross-attention decoding.Source: 2) Cross-Modal Guided Reconstruction · p. 5

Design rationale

  • The framework reconstructs image features from point-cloud features because point clouds provide a broader scene view, while the reverse direction would require the unknown ground-truth pose.The paper also motivates point-cloud reconstruction supervision by the sparsity of point clouds and use of their richer perspective to assist image reconstruction.Source: 1) Dual-modal MAE Enhancement · p. 5, 2) Cross-Modal Guided Reconstruction · p. 6

C. Similarity-based RL Masking Strategy

SRLM initializes masks from cross-modal similarity, then uses reinforcement learning to refine which image and point-cloud regions are masked. Its task-aligned reward and constrained policy are designed to favor informative regions while stabilizing training.

C. Similarity-based RL Masking Strategy

  • SRLM uses image-to-point and point-to-image similarity maps with thresholds and top-k selection to initialize masks in both modalities.The selected regions are masked to form the initial state for refinement.Source: 1) Similarity-based Masking Selection · p. 6
  • Similarity-based selection alone may be insufficient because repetitive textures and image–point-cloud domain gaps can compromise its initialization.The strategy uses similarity during representation learning to prioritize regions for masked modeling rather than directly establish correspondences.Source: 1) Similarity-based Masking Selection · p. 6
  • A policy network adaptively refines the initial masks, using a reward derived from matching loss to guide selection toward better matching accuracy.The RL component is an auxiliary mechanism applied during training, not a standalone decision module.Source: 2) RL-driven Refinement · p. 6
  • The combined design is intended to reduce mask-selection randomness, stabilize MAE optimization, and improve matching reliability without adding inference overhead.The RL masking strategy is applied only during training.Source: 2) RL-driven Refinement · pp. 6–7
  • The RL policy is warmed up after 20 epochs of MAE reconstruction training and uses a dense inverse-task-loss reward and lightweight policy-loss weighting.Factorized token-level Bernoulli sampling and similarity-based priors are also used to reduce policy-gradient variance and constrain exploration.Source: 2) RL-driven Refinement · pp. 7–8

D. Model Training & Inference

Training proceeds in two stages, first optimizing the point MAE and then the image MAE. The total objective combines reconstruction, matching, and SRLM losses; ID-MAE is not used at inference.

D. Model Training & Inference

A. Datasets and Implementation Details

The evaluation covers RGB-D Scenes v2, 7-Scenes, and KITTI, with feature matching and registration assessed using defined correspondence metrics. Implementation details include similarity-map thresholds and top-k settings.

A. Datasets and Implementation Details

B. Evaluations on Dataset

M23D outperforms the 2D3D-MATR baseline across the reported RGB-D Scenes v2 and 7-Scenes metrics, and also reports lower registration errors than competing methods on KITTI.

B. Evaluations on Dataset

  • On RGB-D Scenes v2, M23D gains 11.1 percentage points in inlier ratio (32.4 to 43.5), 3.3 in feature matching recall (90.8 to 94.1), and 21.6 in registration recall (56.4 to 78.0).The passage reports a 7.8-percentage-point registration-recall improvement over the prior state of the art.Source: B. Evaluations on Dataset · p. 9
  • On 7-Scenes, M23D reaches 53.4% inlier ratio, 92.8% feature matching recall, and 81.3% registration recall, exceeding 2D3D-MATR by 3.3, 0.7, and 5.5 percentage points, respectively.B2-3Dnet performs slightly better in feature matching recall, while M23D has the greater overall advantage across the reported results.Source: B. Evaluations on Dataset · pp. 9–10
  • The M23D visualization uses a strict 50-pixel error threshold.Source: Figure 6 · p. 10
  • On KITTI, M23D reports 1.58 ± 2.33 meters RTE and 2.34 ± 3.29 degrees RRE, lower than the listed competing image-to-point methods.The comparisons include RetrI2P and VP2P.Source: B. Evaluations on Dataset · p. 10

C. Ablation Studies

The ablations favor the complete M23D design and SRLM, while hyperparameter tests indicate stable performance near the defaults. Efficiency comparisons report inference speed comparable to 2D3D-MATR, and visualizations show aligned projections and selected cross-modal regions.

C. Ablation Studies

  • The projected point cloud aligns with the image without large-scale misalignment.The visualization describes precise rigid transformation and alignment without significant discrepancies.Source: Figure 8 · p. 11
  • The full model achieves the best 7-Scenes ablation results after adding cross-modal alignment to Dual MAE.The individual Image-MAE and Point-MAE variants improve slightly over baseline, while Dual MAE improves further.Source: C. Ablation Studies · pp. 10–11
  • SRLM achieves the highest masking-strategy performance while balancing accuracy and computational cost.Similarity-based masking improves on random masking; direct reinforcement learning performs strongly but increases computation substantially.Source: C. Ablation Studies · p. 11
  • Moderate hyperparameter settings perform best, with results relatively stable around γ1 = γ2 = γ3 = 1.Further increases yield marginal fluctuations or slight degradation; extreme settings in another parameter produce noticeable drops.Source: Figure 7 · p. 10, C. Ablation Studies · p. 11
  • RL-based masking selects image and point-cloud regions, highlighted in yellow and blue, respectively.The selected regions are visualized as corresponding regions chosen during training.Source: Figure 9 · p. 12
  • M23D runs at 7.684 FPS, comparable to 2D3D-MATR at 7.852 FPS, with 535.1 GFLOPs and 28.6M parameters.The added computation is attributed mainly to similarity computation; MAE and RL-based masking are applied only during training.Source: C. Ablation Studies · pp. 11–12, Table VI · p. 11

D. Visualization

Visualizations show improved matching with ID-MAE and SRLM, while cross-dataset tests report gains for SRLM on RGB-D Scenes v2 and ScanNet. The analyses also identify limits under low overlap, noisy initial similarity, and larger-scale settings.

D. Visualization

  • Fig. 6 visualizations show improved matching with ID-MAE and further improvement with SRLM across seven scene pairs.Matches below 50 pixels of projection distance are shown in green; others are red.Source: D. Visualization · p. 12
  • SRLM-selected regions tend to concentrate on high-texture image areas and prominent point-cloud structures.The image and point-cloud selections need not correspond one-to-one because of viewpoint, modality, and partial-visibility differences.Source: D. Visualization · p. 12
  • Low overlap leaves few reliable cross-modal cues, while noisy initial similarity or large viewpoint changes may weaken region selection.Under minimal overlap, selected regions can concentrate in limited areas, reducing the masking strategy’s overall discriminative capability.Source: D. Visualization · p. 12
  • Extending the framework to large-scale environments may require hierarchical processing or more memory-efficient similarity computation.The current implementation is primarily evaluated on indoor and moderate-scale outdoor datasets.Source: D. Visualization · p. 13
  • SRLM improves Inlier Ratio and Registration Recall on both target datasets when trained on 7-Scenes.The comparison covers RGB-D Scenes v2 and ScanNet, as shown in Table VII.Source: Table VII · p. 12, D. Visualization · p. 13

V. CONCLUSION

M23D combines dual-MAE feature learning with similarity-based reinforcement-learning masking for image-to-point cloud registration. The paper reports state-of-the-art performance on RGB-D Scenes v2 and 7-Scenes.

V. CONCLUSION

  • M23D uses a dual MAE to enhance feature extraction by aligning features and establishing correspondences.Source: §V · p. 13
  • Its similarity-based RL masking strategy reduces unnecessary masking and focuses on key matching regions.Source: §V · p. 13
  • M23D achieves state-of-the-art image-to-point cloud registration performance on RGB-D Scenes v2 and 7-Scenes.Source: §V · p. 13
Loading 2609.18088v1…