PaperlayerView exact revision on arXiv

Source-linked AI summary

A Robust Multimodal Remote Sensing Image Registration Method and System Using Steerable Filters with First- and Second-order Gradients

Yuanxin Ye, Bai Zhu, Tengfeng Tang, Chao Yang, Qizhi Xu, Guo Zhang

arXiv:2202.13347v1cs.CV

TL;DR

Fine registration of multimodal images faces a central challenge from nonlinear differences. The paper pairs SFOC with Fast-NCCSFOC and reports better matching than five similarity measures, while its registration system outperforms ENVI, ERDAS, and PCI.

  • Problem

    Fine registration of multimodal images faces a central challenge from nonlinear differences.

    Source: §1 · p. 3

  • Method

    The method pairs the SFOC descriptor with Fast-NCCSFOC, a similarity measure combining NCC and SFOC.

    Source: §6 · p. 40

  • Results

    SFOC achieved the best matching among MI, MTM, PCSD, CFOG, and SDFG, and the system outperformed ENVI, ERDAS, and PCI.

    Source: §6 · p. 40

  • Takeaways & Limitations

    The reported comparisons support the proposed registration system as an alternative to ENVI, ERDAS, and PCI.

    Source: §6 · p. 40

    SFOC is sensitive to global geometric distortions and cannot be adapted to matching images with large scale or rotation changes.

    Source: §6 · p. 40

Abstract

from arXiv · show

Co-registration of multimodal remote sensing images is still an ongoing challenge because of nonlinear radiometric differences (NRD) and significant geometric distortions (e.g., scale and rotation changes) between these images. In this paper, a robust matching method based on the Steerable filters is proposed consisting of two critical steps. First, to address severe NRD, a novel structural descriptor named the Steerable Filters of first- and second-Order Channels (SFOC) is constructed, which combines the first- and second-order gradient information by using the steerable filters with a multi-scale strategy to depict more discriminative structure features of images. Then, a fast similarity measure is established called Fast Normalized Cross-Correlation (Fast-NCCSFOC), which employs the Fast Fourier Transform technique and the integral image to improve the matching efficiency. Furthermore, to achieve reliable registration performance, a coarse-to-fine multimodal registration system is designed consisting of two pivotal modules. The local coarse registration is first conducted by involving both detection of interest points (IPs) and local geometric correction, which effectively utilizes the prior georeferencing information of RS images to address global geometric distortions. In the fine registration stage, the proposed SFOC is used to resist significant NRD, and to detect control points between multimodal images by a template matching scheme. The performance of the proposed matching method has been evaluated with many different kinds of multimodal RS images. The results show its superior matching performance compared with the state-of-the-art methods. Moreover, the designed registration system also outperforms the popular commercial software in both registration accuracy and computational efficiency. Our system is available at https://github.com/yeyuanxin110.

1. INTRODUCTION

The introduction frames multimodal remote-sensing registration as challenging because of geometric and radiometric differences and motivates a robust, coarse-to-fine approach. It outlines using prior georeferencing for coarse alignment and steerable-filter features with fast similarity matching to address remaining differences.

Introduction

  • Even pixel-range misregistration can affect subsequent applications, making evenly distributed, high-precision control points important for estimating the geometric transformation.The introduction notes that registration accuracy affects applications including image fusion, change detection, and environmental monitoring.Source: §1 · p. 2
  • The proposed approach uses georeferencing to handle global geometric differences, then robust matching to resist nonlinear radiometric differences and noise for fast, robust multimodal registration.Georeferencing can address obvious rotation and scale changes, but nonlinear radiometric differences remain a major challenge; speckle noise is also serious in SAR images.Source: §1 · p. 3
  • Intensity-based SSD and NCC are sensitive to nonlinear radiometric differences, while MI resists them but is time-consuming to compute.The introduction also says phase correlation is not sufficiently robust to nonlinear radiometric differences and noise in multimodal images, especially optical and SAR imagery.Source: §1 · p. 4
  • Feature-based methods can resist geometric distortion, but significant radiometric differences make it difficult to extract many uniform, stable features; phase-congruency methods also require complicated, time-consuming calculations.The introduction identifies the required phase-congruency amplitude and orientation calculations as a source of this complexity.Source: §1 · pp. 4–5

1. INTRODUCTION

  • Learning-based methods can require substantial training and labeled data, and their training efficiency depends on computing infrastructure, limiting practical use.Their advantages are said to emerge in multimodal registration on high-performance computer infrastructure.Source: §1 · p. 6
  • Traditional measures such as NCC and MI in commercial registration software struggle to achieve high precision when multimodal images have significant radiometric differences and geometric distortions.The introduction identifies a need for robust, automatic registration for engineering practice.Source: §1 · p. 6
  • SFOC combines first- and second-order gradients with multiscale steerable filters, while Fast-NCCSFOC uses FFT and integral images to improve matching efficiency.The descriptor is designed to depict multidirectional and multiscale structure characteristics and to address significant nonlinear radiometric differences.Source: §1 · pp. 6–7
  • The coarse-to-fine registration system uses prior georeferencing information and an RFM-based coarse stage to provide spatial geometry constraints for fine registration.In the fine stage, the proposed matching method detects control points and RANSAC removes larger outliers.Source: §1 · pp. 6–7
  • This work extends Ye et al. (2021a) with the SFOC descriptor, Fast-NCCSFOC for control-point detection, and a more thorough evaluation on multimodal remote-sensing images.The descriptor uses first- and second-order gradients, and the similarity measure uses FFT and integral images.Source: §1 · p. 7

2. PROPOSED MATCHING METHOD

This section presents a matching method that combines a multiscale SFOC descriptor, built from first- and second-order steerable channels, with Fast-NCCSFOC for template matching. The similarity measure uses FFT and integral images to reduce matching cost relative to NCC.

2. PROPOSED MATCHING METHOD

  • The proposed matching method comprises a structural feature descriptor and a fast similarity measure.Source: §2 · p. 7

2.1 Construction of structural feature descriptor

  • CFOG is noise-sensitive; CFOG and AWOG lack multiscale local inter-pixel modeling, while CFOG, AWOG, and SDFG use only first-order gradients and omit curvature information.Source: §2.1 · p. 8
  • SFOC combines first- and second-order steerable-filter channels with multiscale processing to describe multimodal images’ directional and scale-varying structure.Dilated Gaussian convolution is used to mitigate lost local spatial information and noise interference.Source: §2.1 · pp. 8, 10
  • SFOC’s first-order channels use six orientations and multiple Gaussian scales, while its second-order channels use three directional derivatives.The construction also includes dilated Gaussian convolution and normalization.Source: §2.1 · pp. 10–11
  • SFOC combines first- and second-order steerable channels across scales, then applies parallel dilated Gaussian convolutions to enhance context and smooth noise without increasing computational complexity.The resulting normalized channels are stacked to form the descriptor.Source: §2.1 · p. 12
  • The SFOC heatmap with parallel dilated Gaussian convolution shows a smoother, more discriminative matching peak than general Gaussian convolution, resisting noise interference.The paper attributes this to noise smoothing and enhanced multiscale context structure features.Source: §2.1 · p. 12, Figure 3 · p. 13

2.2 Establishment of fast similarity measure

  • Fast-NCCSFOC combines SFOC with normalized cross-correlation, using FFT for the correlation operation and integral images to accelerate sliding-window sums.The integral-image approach replaces summation with a limited number of additions and subtractions.Source: §2.2 · pp. 14–15
  • Fast-NCCSFOC’s computational complexity is independent of template size, whereas NCC’s is approximately proportional to the product of template and search sizes, especially when the template is small relative to the search window.Source: §2.2 · p. 17
  • For a template window m=100 and search window M=200, Fast-NCCSFOC takes about 0.965% of NCC’s time; as template and search sizes increase, its computational advantage grows.Source: §2.2 · p. 17

3. REGISTRATION SYSTEM BASED ON THE PROPOSED MATCHING METHOD

The system combines local coarse registration with fine matching and rectification, using georeferencing information to constrain matching and SFOC features to address severe radiometric differences. Its stages detect interest points, locally correct geometry, match control points, reject outliers, and rectify the sensed image.

3. REGISTRATION SYSTEM BASED ON THE PROPOSED MATCHING METHOD

  • The coarse-to-fine system combines interest-point detection and local geometric correction with SFOC feature matching, outlier rejection, and image rectification.Fast-NCCSFOC is used for matching; the system also targets sufficient, uniformly distributed control points and removal of mismatches from occlusions.Source: §3 · p. 18

3.1 Local coarse registration

  • Georeferencing information guides coarse registration: RPC-equipped sensed images use local RFM correction, while corrected image pairs use geographic coordinates to predict the search area.When image resolutions differ, the higher-resolution image is uniformly sampled to the lower-resolution image.Source: §3.1 · pp. 19–20

3.1.1 Detection of interest points (IPs)

  • Block-based FAST addresses uneven interest-point distribution, which can impair geometric-model fitting and reduce registration accuracy.The partitioning strategy detects a sufficient number of uniformly distributed interest points.Source: §3.1.1 · p. 20

3.1.2 Local geometric correction

  • For RPC-equipped images, local RFM correction removes obvious scale and rotation differences and narrows the matching area while reducing memory use.The authors report these benefits compared with correcting the entire sensed image; bilinear interpolation is used for resampling.Source: §3.1.2 · pp. 21–22, §3.2.3 · p. 25

3.2 Fine registration

  • Fine registration assumes georeferencing constrains image offsets to a few dozen pixels, then detects control points with Fast-NCCSFOC and rejects large-error points with RANSAC..Source: §3.2 · p. 22

3.2.1 Features extraction using the SFOC descriptor

  • After local geometric correction reduces scale and rotation differences, SFOC supplies structural features for matching multimodal patches despite severe radiometric differences.The paper reports similar-looking SFOC descriptors for corresponding sensed and reference patches.Source: §3.2.1 · p. 23

3.2.2 CP detection using Fast-NCCSFOC

  • Fast-NCCSFOC detects control points by matching SFOC features extracted from a corrected sensed-image template and its predicted reference-image search window.The most similar SFOC feature location is selected as the control point.Source: §3.2.2 · p. 24

3.2.3 Outlier detection and image rectification

  • RANSAC removes mismatches; RPC-equipped images use iterative affine compensation with the RFM model, while other cases use a projection transformation.The corrected sensed image is resampled by bilinear interpolation for alignment.Source: §3.2.3 · pp. 24–25

4. EXPERIMENTAL EVALUATION: PERFORMANCE OF THE PROPOSED MATCHING METHOD

The matching-method evaluation uses ten multimodal image-pair cases spanning several sensor pairings and spatial resolutions. It compares SFOC with five similarity measures and also examines robustness to noise.

4. EXPERIMENTAL EVALUATION: PERFORMANCE OF THE PROPOSED MATCHING METHOD

  • SFOC was evaluated against five similarity measures and tested for robustness to Gaussian white noise and speckle noise.The comparators were MI, MTM, PCSD, CFOG, and SDFG.Source: §4 · p. 25

4.1 Experimental settings

  • Ten multimodal image-pair cases with significant NRD span low (10m), medium (2m-3m), and high (0.5-1.5m) resolutions.The cases include two Optical-to-Infrared, three LiDAR-to-Optical, three Optical-to-SAR, and two Optical-to-Map pairs.Source: §4.1 · p. 25
  • The paired images were pre-registered to the same ground sample distance to remove obvious rotation and scale differences..Source: §4.1 · p. 25

LiDAR

The analysis evaluates SFOC’s matching performance, robustness to nonlinear radiometric differences and noise, and computational efficiency across multimodal images, including its parameter settings and second-order-gradient contribution.

4.2 Comparison and analysis of matching performance

  • SFOC outperformed all compared methods on NCM, CMR, RMSE, and matching time across all ten multimodal test cases.The comparison used a template size of 80 × 80 pixels.Source: Figure 12 · p. 28, §4.2 · p. 28
  • F-SFOC performed below SFOC, supporting the effectiveness of adding second-order gradients to the descriptor.F-SFOC was comparable to SDFG and had lower RMSE than CFOG, especially for LiDAR-to-Optical and Optical-to-SAR cases.Source: §4.2 · p. 29
  • SFOC produced evenly distributed, relatively precise control points despite significant radiometric differences and noise across the test pairs.The displayed matches used 80 × 80-pixel templates.Source: Figure 13 · p. 30, §4.2 · p. 30

4.3 Comparison and analysis of noise sensitivity

  • SFOC and F-SFOC showed superior capacity as Gaussian and speckle noise increased, followed by SDFG and CFOG.The results support the effectiveness of dilated Gaussian convolution for noise resistance; MI was relatively stable but had lower average CMR than SFOC, F-SFOC, and CFOG.Source: §4.3 · p. 31
  • SFOC demonstrates effectiveness against significant nonlinear radiometric differences and noise between multimodal images, alongside high computational efficiency.Source: §4.3 · p. 32
  • Multiscale first-order filters capture multidirectional structure, while second-order filters and three parallel dilated Gaussian kernels emphasize detail and improve discrimination and anti-noise capability.Source: §4.3 · p. 32
  • Fast-NCCSFOC uses FFT and integral-image techniques to ensure fast computation.Source: §4.3 · p. 32

5. EXPERIMENTAL EVALUATION: PERFORMANCE OF THE DESIGNED REGISTRATION SYSTEM

Across the first-category tests, the designed system led on registration accuracy and control-point metrics, with the fastest matching time except against PCIFFTP. It also performed best in the two RPC-image tests, where several comparison methods had low performance or failed.

5.1 Description of experimental datasets

  • Experiments applied three existing systems and the designed system to multimodal images acquired across different geographic districts and times; datasets had large image sizes, severe geometric distortions, significant NRD, and large differences in acquisition dates and spatial resolutions.Source: §5.1 · p. 33

5.2 Implementation details and evaluation criteria

  • Matching performance was assessed using NCM, CMR, RMSE, and matching time, with qualitative inspection of control-point pairs and image overlays.The systems shared matching parameters as far as possible, and rectification used a second-order polynomial model.Source: §5.2 · pp. 34–35

5.3 Comparison and analysis of registration performance

  • In the first-category tests, the designed system had the smallest RMSE, fastest MT except PCIFFTP, highest CMR, and most NCM across all test cases.ENVIMI ranked second in matching accuracy; the reported overall ordering placed ERDASNCC third, PCIFFTP fourth, and NCC-based systems last.Source: §5.3 · pp. 35–36, Table 5 · p. 36
  • The system achieved the best matching performance in two tested cases; its CMR was 84.75% with higher accuracy than three comparators, while its test 4 CMR was 64.5%.ENVINCC, PCINCC, and PCIFFTP matching comprehensively failed on the Optical-to-SAR test 4 case.Source: §5.3 · p. 36
  • The authors attribute robustness to SFOC structural features and matching efficiency to Fast-NCCSFOC.They describe SFOC as depicting multi-scale and multi-directional structural information.Source: §5.3 · p. 37
  • Detected control points were uniformly distributed and judged satisfactory in location precision despite geometric distortion, NRD, noise, and spatial-resolution differences.Visual inspection of the overlays indicated that all sensed images had been aligned correctly.Source: §5.3 · pp. 37, 39, Figure 17 · p. 39

6. CONCLUSION

The conclusion presents SFOC and Fast-NCCSFOC within a coarse-to-fine registration system and reports strong matching and system performance against comparison methods. It also identifies sensitivity to global geometric differences and dependence on georeferenced information.

6. CONCLUSION

  • SFOC combines first- and second-order Gaussian steerable-filter information to capture distinctive structures against NRD, while Fast-NCCSFOC uses FFT and integral images to speed matching.Source: §6 · p. 40
  • The coarse-to-fine system uses block-based FAST interest points and local geometric correction before fine registration with SFOC template matching.Source: §6 · p. 40
  • Across ten varied multimodal images, experiments demonstrated SFOC's robustness and effectiveness.Source: §6 · p. 40
  • SFOC achieved the best matching performance among MI, MTM, PCSD, CFOG, and SDFG, while the designed system outperformed ENVI, ERDAS, and PCI in registration performance.Source: §6 · p. 40
  • SFOC remains sensitive to global geometric distortions and cannot handle multimodal matching with large scale or rotation differences.Source: §6 · p. 40
  • The system depends on prior georeferenced information or RPCs to remove obvious geometric distortions and is inapplicable without them.Source: §6 · p. 40
Loading 2202.13347v1…