PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Remote Sensing Image Registration: A Comprehensive Review, Challenges and Prospects

Zhiqiang Han, Yuanxin Ye, Qiuyun Wu, Jinhao Chen, Bai Zhu, Siyuan Hao

arXiv:2610.11176v1cs.CV

TL;DR

Multimodal image differences challenge high-precision registration, a prerequisite for downstream remote-sensing applications. This survey organizes registration methods into three categories and synthesizes algorithms, datasets, bottlenecks, and future directions.

  • Problem

    Radiometric, geometric, scale, viewpoint, and temporal differences make high-precision multimodal registration challenging, despite its role as a prerequisite for downstream applications.

    Source: §1 · pp. 2–3

  • Method

    The survey classifies methods as region-based, feature-based, and deep learning-based, tracing their evolution and reviewing representative algorithms.

    Source: §1 · p. 4

  • Results

    The review analyzes publicly available multimodal datasets and identifies bottlenecks in high-precision registration while outlining future directions.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The survey provides a reference spanning multimodal registration methods, datasets, and research prospects.

    Source: §1 · p. 4

    Performance across varying geographic scenes and sensor types remains a critical concern.

    Source: (iii) Modality Unification Methods… · p. 23

Abstract

from arXiv · show

Multimodal remote sensing image registration is a crucial prerequisite for the collaborative processing and downstream application of remote sensing data, such as image fusion, change detection, and target recognition. However, significant variations in radiometry, geometry, scale, viewpoint, and time often exist between multimodal images. These differences, driven by varying sensor geometries, physical radiation mechanisms, imaging platforms, and environmental disturbances, pose severe challenges to achieving high-precision, robust registration. This paper systematically reviews the progress of mainstream multimodal remote sensing image registration methods. Based on their registration pipelines, existing approaches are categorized into three main types: region-based, feature-based, and deep learning-based methods. We detail the core principles, representative algorithms, advantages, and limitations of each category. Additionally, we summarize publicly available multimodal image datasets in the remote sensing domain, analyzing their specific characteristics and applicable scenarios. Finally, we highlight current bottlenecks in high-precision registration research and outline future development trends. This review aims to provide a comprehensive reference and valuable insights for researchers in related fields.

1. Introduction

Multimodal registration aligns corresponding points despite substantial differences in radiometry, geometry, scale, viewpoint, and time, which complicate high-precision matching. The survey organizes existing approaches by registration pipeline and reviews their development, available datasets, current bottlenecks, and future directions.

1. Introduction

  • Radiometric, geometric, scale, perspective, and temporal discrepancies arise from sensor mechanisms, platforms, and environmental conditions, challenging high-precision registration.Noise, cloud occlusion, and blur are among the cited interferences.Source: §1 · p. 3
  • Registration identifies corresponding point pairs to estimate a geometric transformation, supporting downstream applications such as image fusion and change detection.The paper describes registration as a prerequisite for collaborative processing of remote-sensing data.Source: §1 · pp. 2–3
  • Traditional methods struggle with multimodal distortions because they lack robust descriptors and similarity measures for shared characteristics.The introduction also says commercial tools relying on traditional techniques do not achieve high-precision multimodal registration.Source: §1 · pp. 3–4
  • The survey classifies registration approaches by pipeline as region-based, feature-based, and deep learning-based methods.It presents deep learning as a third category alongside region-based and feature-based approaches.Source: §1 · p. 4
  • The review traces the three categories from single-modal to multimodal applications and discusses representative algorithms and open-source implementations.The paper points to its overall framework in Figure 3.Source: §1 · p. 4, Figure 3 · p. 4
  • The survey analyzes publicly available multimodal datasets and closes by identifying high-precision registration bottlenecks and outlining future development trends.Source: §1 · p. 4

2. Region-based registration method

This section surveys region-based registration methods that estimate geometric transformations by optimizing similarity measures, sometimes using georeferencing information to constrain template matching. It covers information-theoretic and structure-feature-based approaches, including tailored descriptors and dense matching frameworks, and discusses their advantages, limitations, and sparse–dense tradeoffs for multimodal imagery.

2. Region-based registration method

  • Georeferencing data such as POS or RPC can reduce scale and rotation discrepancies and constrain template matching to a coarse search range after initial errors of a few dozen pixels.Region-based methods estimate geometric transformations by optimizing similarity metrics.Source: §2 · p. 5

2.1. Region registration method based on information theory

  • Information-theoretic metrics accommodate nonlinear radiometric variation and are more robust than intensity-based SSD, NCC, and PC for multimodal registration.MI became a widely adopted measure for multimodal remote sensing registration.Source: §2 · p. 5, §2.1 · p. 6
  • For MI joint-histogram estimation, nearest-neighbor interpolation was most effective; GPVE was proposed to mitigate interpolation-induced artifacts, SPSA achieved successful alignment, and Suri constructed an automated hierarchical coarse-to-fine MI pipeline for high resolutions.Source: §2.1 · pp. 6–7
  • NCMI improved LiDAR-optical registration precision, Jeffrey divergence outperformed earlier modified MI techniques, and TOMI combined gradient distributions with structural orientations for resilience to noise and radiometric shifts.TOMI is described as resilient to heavy noise and extreme nonlinear radiometric shifts.Source: §2.1 · p. 7
  • GNMI with variable-size block segmentation avoids conventional feature extraction and improves speed and accuracy for heterogeneous SAR-optical pairs, while computational demands have limited information-theoretic methods in large-scale applications (Wang et al. 2024a).High computational complexity has traditionally made these methods time-consuming, although block-based strategies have begun mitigating the bottleneck.Source: §2.1 · p. 7

2.2. Structure-feature-based region registration methods

  • Sparse structural descriptors include HOG, DLSS, and HOPC, but sparse sampling can discard structural information and limit registration accuracy and robustness.HOPC with NCC is reported to work across optical, LiDAR, and SAR imagery with severe nonlinear radiometric differences.Source: §2.2 · pp. 8–9
  • Dense structure-based methods use pixel-wise descriptors to map intensities into high-dimensional feature spaces, including the CFOG template-matching framework.CFOG is presented as a fast and robust framework using pixel-wise channel features of oriented gradients.Source: §2.2 · p. 9
  • Fan et al. (2021) angle-weighted CFOG gradients, while Xiang et al. (2022b) proposed DCF-EPW for large georeferencing errors, using dilated convolutions and epipolar-oriented rectangular templates.The larger receptive fields maintain positional invariance, and rectangular templates capture more overlapping regions to establish higher-precision optical-SAR correspondences.Source: §2.2 · p. 9
  • Ye et al. (2022b) developed SFOC, combining first- and second-order gradients into discriminative structural features and an open-source coarse-to-fine framework that accelerates NCC with FFT and integral images.SFOC addresses gaps in prior dense methods involving local pixel relationships, multi-scale contexts, and second-order gradient details.Source: §2.2 · p. 9
  • Xiang et al. (2023) paired MAGD with a geometry-invariant mask in confidence-guided local matching to handle geometric distortions and released an evaluation dataset of hundreds of multimodal image pairs.Source: §2.2 · pp. 9–10
  • Huang et al. 2026 integrated multi-channel oriented gradient templates with self-supervised super-resolution for complex underwater terrain navigation in Arctic waters.Source: §2.2 · p. 10
  • Dense structure-based methods use spatial- and frequency-domain similarity metrics, while sparse methods are limited to spatial-domain evaluation on predefined grids; dense approaches balance robustness and efficiency against severe nonlinear radiometric variation.The review identifies dense methods as advantageous for modern multimodal remote sensing applications.Source: §2.2 · p. 10

3. Feature-based registration method

This section reviews feature-based registration methods that match salient features and descriptors to address multimodal radiometric and geometric challenges, covering gradient-optimization, local-self-similarity, and phase-consistency approaches. It considers their tradeoffs in sensor-specific noise reduction, geometric-layout resilience, radiometric robustness, and computational overhead.

3. Feature-based registration method

  • Feature-based methods match salient points, lines, or regions using descriptors without requiring initial georeferencing, but intensity- or gradient-dependent classical operators struggle with severe geometric and radiometric differences.These methods are presented as an alternative when region-based methods are sensitive to rotation, scale variation, or inaccurate georeferencing.Source: §3 · pp. 10–11

3.1. Feature registration method based on gradient optimization

  • SAR-SIFT uses ROEWA-derived gradients robust to speckle for multi-view SAR registration, while CoFSM uses co-occurrence and Butterworth filters to reduce nonlinear radiometric interference.OS-SIFT combines multiscale Sobel and ROEWA gradients across optical and SAR imagery; its Harris scale space and GLOH-like descriptor establish correspondences.Source: §3.1 · p. 12

3.2. Feature registration method based on local self-similarity

  • Local self-similarity descriptors replace gradients with correlation-based structural cues; DOBSS registered multi-sensor satellite imagery, while later variants target rotation invariance, speckle suppression, and richer structure.HOSS uses a Rotation Index of the Maximal Correlation with adaptive log-polar structure; ASS suppresses coherent speckle, and ASTC adds 3D Gaussian convolution to amplify structural saliency and mitigate noise.Source: §3.2 · p. 12

3.3. Feature registration method based on phase consistency

  • Phase-consistency methods use illumination- and contrast-invariant features: MMPC-Lap and LHOPC combine scale-space keypoints and spatial descriptors, while RIFT uses phase-congruency maps and log-Gabor sequences.RIFT constructs a Maximum Index Map to improve radiation invariance; HAPCG adds a log-polar descriptor, and MABPC adaptively bins multiscale phase-congruency features.Source: §3.3 · p. 14
  • MALG detects highly repeatable features, while RMIM in a DAISY configuration provides radial and rotational invariance.Source: §3.3 · p. 14
  • Li et al. (2023a) proposed RIFT2, replacing the convolution sequence ring with dominant-index values to reduce memory consumption and processing time.Jia et al. (2023) developed MSPCO, combining nonlinear diffusion scale space to preserve edge information with a scale-ratio strategy to improve computational efficiency.Source: §3.3 · pp. 14–15
  • Xie et al. (2023) introduced a multiscale PC-Harris keypoint extractor with a polar-coordinate descriptor to address large-scale and rotation discrepancies.Source: §3.3 · p. 15
  • GLIFT combines log-Gabor filtering, FAST keypoint detection, global MIM search, and local pixel-level cosine similarity, while EPC-SRC uses weighted moments and multiscale MIM to improve descriptor distinguishability.GLIFT achieves superior match count and precision; EPC-SRC uses a GLOH-like structure.Source: §3.3 · p. 15
  • Gradient-based methods such as SAR-SIFT and OS-SIFT mitigate targeted interference like SAR speckle, while their sensor-specific design limits generalizability; LSS handles moderate shifts but struggles with severe nonlinear radiometric variation.LSS captures internal geometric layouts and tolerates structural distortions, but its descriptors have lower discriminative power under severe radiometric change.Source: §3.3 · p. 15
  • Phase-consistency methods are the most reliable choice for highly heterogeneous imagery and severe nonlinear radiometric variation, despite higher computational overhead.Their overhead is being reduced by memory-optimized RIFT2.Source: §3.3 · p. 15

4. Deep Learning-based Registration Methods

Deep learning-based registration uses learned representations to address multimodal heterogeneity and is organized into modular and end-to-end frameworks. Modular methods insert learned components into traditional pipelines, while end-to-end methods directly produce registration results from raw images.

4. Deep Learning-based Registration Methods

  • Severe nonlinear radiometric discrepancies limit the generalizability of traditional handcrafted representations, motivating the use of deep learning.The review describes deep learning as a learning-feedback-adjustment approach that extracts high-level semantic information from datasets.Source: §4 · pp. 15–16
  • Deep architectures extract high-level semantic and common features that better mitigate multimodal radiometric and geometric discrepancies than handcrafted descriptors.The review notes that traditional feature-based methods improve robustness to scale and rotation variations but remain limited by their reliance on handcrafted descriptors.Source: §4 · pp. 15–16
  • The review divides deep learning-based registration into modular methods and end-to-end methods, which replace pipeline stages or directly output registration results from raw images.Modular approaches use learned components within region- or feature-based workflows; end-to-end approaches use unified neural architectures.Source: §4 · p. 16

4.1. Modular registration methods

  • Modular registration embeds deep networks in traditional feature- or region-based pipelines to learn detectors, descriptors, or similarity metrics.This integration is described as leveraging deep learning’s representation capabilities to enhance overall registration robustness.Source: §4.1 · p. 16

(i) Learning-based Template Matching Methods

Learning-based template matching extracts deep representations from image pairs and assesses similarity between their feature maps. Methods progress from convolutional designs toward semantic and multiscale representations, but remain limited in adapting to major scale and rotation differences.

(i) Learning-based Template Matching Methods

  • Template matching methods extract deep features from the images and compare the resulting feature maps using neural operations or classical similarity metrics.Common comparisons use neural layers, dot products, convolutions, or classical metrics.Source: (i) Learning-based Template Matching… · pp. 16–17
  • Semantic and multiscale designs address radiometric variation through high-level features, gradient features, feature fusion, or structural similarity.Examples include DDFN, MCGFs, OSMNet, PCNet, and MSF SiamUNet-7.Source: (i) Learning-based Template Matching… · p. 18
  • Despite stronger deep representations, learning-based template matching remains constrained by limited adaptability to substantial scale and rotation variations.The review identifies scale and rotation differences as persisting challenges for these methods.Source: (i) Learning-based Template Matching… · p. 18

(ii) Learning-based Feature Matching Methods

Learning-based feature matching replaces or augments handcrafted features with deep detectors, descriptors, and cross-modal aggregation. The reviewed approaches report strong results, including zero-shot cross-modal generalization, while research remains more focused on feature description than specialized detection.

(ii) Learning-based Feature Matching Methods

  • Coarse-to-fine and rotation-aware methods refine alignment using feature pyramids, adaptive constraints, outlier filtering, or ordinal-regression-based rotation rectification.Quan et al. (2023) uses deep ordinal regression for rotation rectification followed by deep descriptor learning for fine registration.Source: (ii) Learning-based Feature Matching… · pp. 19–20
  • Methods combine learned keypoints and descriptors, context aggregation, or deep-shallow features to improve matching across modality and geometric variation.Examples include CMM-Net’s joint keypoint-descriptor extraction and MAP-Net’s global-local context pooling with attention.Source: (ii) Learning-based Feature Matching… · pp. 19–20
  • Transformer and hybrid approaches aggregate global and local features, with geographic-location attention and phase consistency reported to improve generalization and matching accuracy for complex multi-source SAR imagery.The review also describes CNN-Transformer hybrids and modality-independent matching approaches.Source: (ii) Learning-based Feature Matching… · p. 20
  • MINIMA achieves exceptional zero-shot cross-modal generalization by training on randomly selected modality pairs from a synthetic multimodal dataset.Its data engine transforms RGB data into MD-syn rather than relying on modality-specific modules.Source: (ii) Learning-based Feature Matching… · p. 20
  • SFA-Net achieves superior performance across multiple metrics by processing SuperPoint features augmented with SAM-derived edge structures through regional self-attention.The passage reports the performance outcome without giving metric values.Source: (ii) Learning-based Feature Matching… · p. 21
  • Deep feature matching improves detector repeatability and descriptor robustness, but current work focuses relatively less on specialized feature detection for heterogeneous images.The review describes the field as predominantly focused on feature description.Source: (ii) Learning-based Feature Matching… · p. 21

(iii) Modality Unification Methods Based on Style Transfer

Style-transfer methods translate one modality before applying established matching algorithms, whereas end-to-end methods jointly learn feature extraction and spatial alignment. The approaches address cross-modal differences but face distinct constraints, including translation variability and computational or interpretability costs.

(iii) Modality Unification Methods Based on Style Transfer

  • Reported examples include GAN-based pseudo-SAR synthesis and style-transfer matching, with studies reporting improved registration accuracy and robustness or superior matching accuracy.Merkle et al. (2018) report superior matching accuracy using NCC and SIFT; Zhao et al. (2022) report enhanced registration accuracy and robustness.Source: (iii) Modality Unification Methods… · p. 21
  • Style-transfer registration uses a two-step pipeline: generate a radiometrically unified image, then apply traditional matching to estimate geometric transformation.The matching stage may use SIFT, CFOG, or NCC; the generative stage may use a GAN or diffusion model.Source: (iii) Modality Unification Methods… · p. 22
  • Sequential generative translation and iterative matching incur substantial computational cost and latency, limiting suitability for real-time or resource-constrained applications.The review contrasts this sequence with a single optimized neural forward pass.Source: (iii) Modality Unification Methods… · p. 22
  • Style transfer can mitigate nonlinear radiometric differences, but translation quality depends on training-data scale and diversity and may degrade across geographic scenes and sensor types.Inconsistent translation can compromise the robustness and precision of subsequent registration.Source: (iii) Modality Unification Methods… · pp. 22–23

4.2. End-to-end registration methods

  • End-to-end methods directly regress geometric parameters or displacement fields, combining feature extraction and matching within a unified network.Examples include D-DenseNet’s corner-displacement regression and fusion-based matching with point-displacement regression.Source: §4.2 · pp. 23–24
  • Recent end-to-end methods address large geometric distortions with dense correspondences and geometric constraints, or use modality prompts and detail enhancement in a deformation architecture.GDROS refines a multiscale correlation volume and constrains optical flow with least-squares regression; PGMR uses a shared encoder, deformation decoder, and detail enhancement module.Source: §4.2 · pp. 24–25
  • End-to-end training can facilitate global optimization and mitigate local optima, while modular methods allow specialized components but can accumulate errors across stages.The review also notes end-to-end methods’ high computational costs, complex architectures, and limited interpretability.Source: §4.2 · p. 25

5. Multimodal image registration datasets

The reviewed datasets vary in size, resolution, geography, modality, and intended use, spanning matched optical-SAR collections, cross-perspective imagery, and broader multimodal sets. Some provide large and diverse training pairs, while OSEval is designed for quantitative evaluation with corner-point ground truth.

5. Multimodal image registration datasets

  • SARptical provides 10,108 strictly aligned optical-SAR image patches captured over Berlin, Germany; SEN1-2 offers 282,384 SAR-optical pairs from Sentinel-series satellites in 2017, covering five continents in four seasons.SEN1-2 optical and SAR images have 10-meter resolution.Source: §5 · p. 25
  • University-1652 links ground, drone, and satellite perspectives across 1,652 buildings at 72 universities worldwide.Each building has an average of one satellite image, one drone image, and 3.38 real street-view images.Source: §5 · p. 27
  • OS-dataset contains 10,692 high-resolution optical-SAR pairs from multiple cities, whereas QXS-SAROPT contains 20,000 pairs focused on port areas.QXS-SAROPT covers Shanghai, Qingdao, and San Diego maritime environments.Source: §5 · pp. 27–28
  • The Multimodal Image Matching Datasets collection includes 164 image pairs spanning 18 modality types, with examples from remote sensing.Its source pairs come from medicine, remote sensing, and computer vision datasets.Source: §5 · p. 28
  • SOPatch contains over 650,000 optical and SAR pairs across varied satellite sources and scenes, while OSEval provides hundreds of sub-meter pairs for evaluation.OSEval uses clearly structured corner points as ground truth to quantitatively assess registration and support unbiased comparisons.Source: §5 · p. 29
  • 3MOS contains 113,000 SAR-optical pairs from five SAR satellites and Google Earth, spanning eight scene types at 3.5-to-12.5-meter resolution.The listed scene types include urban, rural, plains, hills, mountains, water, desert, and frozen earth.Source: §5 · pp. 29–30
  • RMSE measures residual geometric deviation between point pairs, while Correct Match Rate or inlier ratio reflects identification of true correspondences and rejection of outliers.The review presents the two metrics as complementary measures of precision and robustness.Source: §5 · p. 30

6. Summary and prospect of multimodal remote sensing image registration

The review traces multimodal registration’s evolution from addressing linear radiometric differences to complex nonlinear and geometric distortions, and finds inherent constraints across all three method families. It identifies paradigm-specific research priorities, including improved efficiency, greater data availability, and integration of remote-sensing knowledge with computer vision.

Summary and prospect

  • Registration supports image fusion, change detection, and large-scale stitching, while research has shifted from simple linear radiometric differences toward nonlinear distortions and severe geometric deformations.The review identifies post-disaster assessment, cross-season ecological monitoring, and alignment in extreme environments as emerging application scenarios.Source: §6 · pp. 31–32
  • Region-based methods depend on prior information to constrain the search space, and their performance is sensitive to overlap, with insufficient coverage frequently leading to failure.The cited prior information includes RPC or POS data, coarse mapping models, and manual tie points.Source: §6 · p. 31
  • Feature-based methods incur substantial computation and remain vulnerable to outliers, especially amid co-occurring scale, rotation, and radiometric variation.Such mismatches can make them less robust than region-based methods; handcrafted descriptors are also vulnerable to severe nonlinear radiometric discrepancies.Source: §6 · pp. 31–32
  • Deep learning methods extract abstract semantic information and bridge radiometric gaps, but their progress in remote sensing is limited by scarce representative datasets and substantial data and compute demands.The review links dataset scarcity to limits on the generalization and evolutionary potential of learning-based frameworks.Source: §6 · pp. 31–32
  • Region-based research should develop deformation-invariant similarity metrics combining local appearance with structural features.The review identifies this as a priority for meeting demands across methodological paradigms.Source: §6 · p. 32
  • Feature-based research should improve computational efficiency and outlier rejection to support consistent sub-pixel precision.The review names Fast Fourier Transform acceleration strategies as one efficiency direction.Source: §6 · p. 32
  • Learning-based research should expand comprehensive public datasets and explore unsupervised learning and domain adaptation to address data scarcity.The review proposes open-source platforms for curating datasets.Source: §6 · p. 32
  • Integrating geoscientific remote-sensing knowledge with computer-vision architectures is proposed as a route to task-specific semantic extraction models tailored to multimodal sensor physics.The review presents these models as a path toward resilient, broadly applicable registration solutions.Source: §6 · p. 32
Loading 2610.11176v1…