PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity

Yuanxin Ye, Jie Shan, Lorenzo Bruzzone, Li Shen

arXiv:2103.16871v1cs.CV

TL;DR

Multimodal remote sensing registration must handle substantial nonlinear radiometric differences that challenge intensity-based matching. The paper proposes the structural descriptor HOPC, the HOPCncc similarity metric, and fast template matching for control-point detection. Experiments across multimodal datasets report stronger matching and registration performance than NCC and MI, while scale and rotation invariance remain limitations.

  • Problem

    Nonlinear radiometric differences make automatic registration of multimodal remote sensing images difficult and often necessitate manual control-point selection.

  • Method

    The paper extends phase congruency with orientation information, constructs HOPC descriptors and their NCC-based HOPCncc similarity metric, and uses fast template matching for control-point detection.

  • Results

    HOPCncc outperforms NCC and MI in matching performance across varied multimodal datasets, including a 92% CMR versus MI's 54.5% on one high-resolution case.

  • Takeaways & Limitations

    Structural descriptors based on phase congruency provide a robust basis for multimodal remote sensing registration under nonlinear radiometric differences.

  • Takeaways & Limitations

    HOPCncc is not invariant to scale and rotation changes, so substantial such deformations need elimination through direct georeferencing or other methods.

Abstract

from arXiv · show

Automatic registration of multimodal remote sensing data (e.g., optical, LiDAR, SAR) is a challenging task due to the significant non-linear radiometric differences between these data. To address this problem, this paper proposes a novel feature descriptor named the Histogram of Orientated Phase Congruency (HOPC), which is based on the structural properties of images. Furthermore, a similarity metric named HOPCncc is defined, which uses the normalized correlation coefficient (NCC) of the HOPC descriptors for multimodal registration. In the definition of the proposed similarity metric, we first extend the phase congruency model to generate its orientation representation, and use the extended model to build HOPCncc. Then a fast template matching scheme for this metric is designed to detect the control points between images. The proposed HOPCncc aims to capture the structural similarity between images, and has been tested with a variety of optical, LiDAR, SAR and map data. The results show that HOPCncc is robust against complex non-linear radiometric differences and outperforms the state-of-the-art similarities metrics (i.e., NCC and mutual information) in matching performance. Moreover, a robust registration method is also proposed in this paper based on HOPCncc, which is evaluated using six pairs of multimodal remote sensing images. The experimental results demonstrate the effectiveness of the proposed method for multimodal image registration.

I. INTRODUCTION

Multimodal remote sensing registration is hindered mainly by nonlinear radiometric differences, motivating structural similarity methods. The paper introduces HOPC and HOPCncc, together with fast template matching and a registration method.

  • Motivation: Multimodal remote sensing images support Earth observation but often require manual control-point selection because of geometric and nonlinear radiometric differences.Registration accuracy affects image fusion, change detection, and mosaicking.
  • Motivation: The central difficulty is nonlinear radiometric variation between modalities after direct georeferencing largely removes global geometric distortions.Optical and SAR images can retain different intensity and texture characteristics despite depicting the same scene.
  • Existing Metrics: NCC is invariant to linear intensity changes but vulnerable to nonlinear differences, whereas MI is more robust yet computationally expensive and sensitive to template size.SSD is also sensitive to radiometric changes despite its efficiency.
  • Structural Similarity: Structural and shape properties can remain similar across modalities despite different intensity patterns, motivating their use for multimodal registration.The paper specifically observes similar contour structures and geometric shapes in optical and SAR images.
  • Proposed Approach: HOPC extends phase congruency with orientation information, while HOPCncc compares dense HOPC descriptors using NCC to detect control points.The method also includes fast template matching and an automatic registration procedure.
  • Contributions: The study extends earlier work with a principled HOPCncc derivation, parameter analysis, a registration method, and broader multimodal evaluation.The paper evaluates visible, infrared, LiDAR, SAR, and map data.

A. Analysis of Importance of Phase

Image phase carries key contour and structural information, making it useful for feature detection and comparatively robust to illumination and contrast changes. The paper uses phase congruency with multiscale, multiorientation log Gabor wavelets to capture these properties.

  • Phase importance: Phase information contributes more to image structure and contour representation than amplitude information.The paper illustrates this by combining phases and amplitudes from different images and observing that phase preserves the structural appearance.
  • Phase congruency: Phase congruency detects corners and edges where Fourier components are maximally in phase.It is based on local phase and is independent of signal amplitude, supporting invariance to illumination and contrast changes.
  • Phase congruency: The improved PC2 model uses log Gabor wavelets across multiple scales and orientations to improve feature localization in noisy and blurred images.The wavelets have even-symmetric and odd-symmetric components that produce image responses for phase analysis.
  • Wavelet responses: Log Gabor wavelet responses provide amplitude and phase values for each scale and orientation, forming the inputs to phase congruency computation.The response vector consists of convolutions with the even- and odd-symmetric wavelets.

C. Orientation of Phase Congruency

The conventional phase congruency model describes feature strength but not feature direction. The paper therefore extends it with an orientation representation derived from directional odd-symmetric wavelet responses.

  • Motivation: Traditional phase congruency uses feature amplitude but cannot represent the significant directions of local feature variation.This limits its ability to describe feature distributions in local image regions and motivates an orientation extension.
  • Orientation computation: The orientation is computed by projecting directional odd-symmetric wavelet responses onto horizontal and vertical image directions.These projections yield x-direction and y-direction derivatives, which are used to calculate phase congruency orientation.
  • Orientation representation: Phase congruency orientation is defined over the angular range [0°, 360°).The orientation uses convolution results from the log Gabor odd-symmetric wavelet at each direction.

D. Structural Feature Descriptor

HOPC is a structural descriptor that combines phase congruency amplitude and orientation in a HOG-like histogram framework. Its normalized local histograms are designed to remain more stable under illumination changes than HOG.

  • Descriptor design: HOPC uses both phase congruency amplitude and orientation to describe structural properties that are relatively independent of image intensity patterns.The descriptor is inspired by HOG but replaces gradient information with phase congruency information.
  • Descriptor extraction: HOPC is computed densely from normalized local histograms of phase congruency orientations over a selected template window.The processing chain evaluates a dense grid of local histograms and combines block descriptors into a feature vector for template matching.
  • Descriptor extraction: The extraction process computes phase congruency amplitude and orientation, divides the window into overlapping blocks and cells, and accumulates orientation histograms.Gaussian spatial weighting, phase-congruency amplitude weighting, trilinear interpolation, and L2 normalization are applied during histogram construction.
  • Robustness: HOPC descriptors are more robust to illumination changes than HOG descriptors in the illustrated comparison.The example reports greater similarity between HOPC descriptors from images with significant illumination variation.

E. Similarity Metric Based on Structural Properties

HOPCncc compares image regions through the NCC of HOPC descriptors, which represent structural properties that are relatively independent of intensity distributions.

  • HOPC descriptors capture image structure and can be directly compared across multimodal images using NCC.The descriptor is extracted separately for each image before similarity computation.
  • HOPCncc is defined as the normalized correlation coefficient between the HOPC descriptors of two image regions.The descriptors are denoted for regions A and B, with their respective means used in the NCC calculation.
  • The method relies on structural similarity because structural properties are relatively independent of multimodal intensity-distribution patterns.This design targets images with significant non-linear radiometric differences when their shapes remain similar.

F. Fast Matching Scheme

The fast HOPCncc matching scheme avoids repeated descriptor calculations between adjacent template windows by precomputing block-HOPC descriptors across the search region. It substantially reduces runtime as template or search-region sizes increase, within a broader registration pipeline that includes coarse rectification and bidirectional matching.

  • Fast descriptor computation: Adjacent template windows share many pixels, so the traditional HOPCncc matcher performs repetitive computations.This overlap motivates a dedicated fast matching scheme.
  • Fast descriptor computation: Block-HOPC descriptors are computed around each image pixel, sampled at intervals, and assembled into dense HOPC descriptors for template windows.The scheme uses a block-HOPC image as an intermediate representation.
  • Computational efficiency: The proposed scheme requires much less time than traditional matching, with the advantage increasing for larger template windows and search regions.The comparison uses 200 matched interest points and varies template and search-region sizes.
  • Registration pipeline: The registration pipeline first coarsely rectifies images and equalizes their ground sample distance before detecting distributed Harris interest points.These steps remove obvious translation, rotation, and resolution differences before matching.
  • Registration pipeline: HOPCncc uses bidirectional template matching, retains mutually consistent point pairs, removes large-error correspondences, and refines the transformation iteratively.A projective model supports global consistency checking, while residual-based refinement removes mismatched control points.
  • Registration pipeline: A piecewise linear transformation based on Delaunay triangulation addresses local distortions caused by terrain relief.Affine transformations are applied within corresponding triangular regions.

IV. EXPERIMENTAL RESULTS: HOPCNCC MATCHING PERFORMANCE

HOPCncc matching performance is evaluated using similarity curves, correct match ratios, and computational efficiency on synthetic and real multimodal image pairs. Synthetic experiments model non-uniform illumination, contrast changes, and nonlinear radiometric distortion.

  • HOPCncc is evaluated using similarity curves, correct match ratio, and computational efficiency, with NCC and MI as comparison metrics.The experiments test parameter effects and compare HOPCncc with state-of-the-art similarity measures.
  • The evaluation uses synthetic image pairs with nonlinear radiometric differences and real multimodal image pairs.These two categories are used to assess HOPCncc effectiveness.
  • Synthetic datasets: Synthetic images are generated from a 1382 ×1382 urban image using spatially varying intensity warping, piecewise linear mapping, and Gaussian noise.The master and slave images are simulated with different radiometric transformations.
  • Synthetic datasets: The spatially varying model introduces non-uniform illumination and contrast changes, while piecewise linear mapping introduces nonlinear radiometric distortion.The two models jointly create significant radiometric differences between synthetic image pairs.

Real Datasets

Real-data experiments cover diverse optical, infrared, LiDAR, SAR, and map combinations across resolutions, terrains, and distortion conditions. Parameter studies identify settings that balance CMR and computational efficiency, while matching uses CMR with dataset-specific checking procedures.

  • Datasets: Ten real multimodal image pairs span Visible-to-Infrared, LiDAR-to-Visible, Visible-to-SAR, and Image-to-Map categories.The datasets include medium-resolution 30 m imagery and high-resolution 0.5 m to 3 m imagery from urban and suburban areas.
  • Datasets: The real datasets include local geometric distortions, LiDAR noise, temporal changes, and nearly complete radiometric differences between visible images and maps.These conditions make several image pairs difficult to match visually.
  • Evaluation: CMR is the ratio of correctly matched point pairs to total matched point pairs, with correctness determined using localization-error thresholds or check points.Synthetic data use a 0.5-pixel threshold, while real-data evaluation uses selected check points and dataset-specific procedures.
  • Evaluation: For difficult LiDAR and SAR data, HOPCncc detects 200 evenly distributed control points with 200×200-pixel templates before global consistency filtering.The 40–60 points with the least residuals are then selected as check points.
  • Parameter tuning: The orientation-bin study finds that average CMR generally increases with bin number and reaches its maximum at β=8.β=8 is selected for HOPCncc.
  • Parameter tuning: Increasing block overlap raises average CMR, but the difference between one-half and three-quarters overlap is small, so one-half overlap is chosen for efficiency.The larger overlap is more time-consuming.
  • Parameter tuning: Three-by-three cell blocks of three-by-three-pixel cells achieve the highest CMR, but three-by-three blocks of four-by-four-pixel cells differ by only 0.2% and are more computationally efficient.The four-pixel cell setting is therefore used in the experiments.

D. Analysis of Similarity Curve

Similarity-curve analysis shows that HOPCncc most reliably identifies the correct control point under strong nonlinear radiometric differences, with a smoother and more distinguishable peak than competing metrics.

  • Experimental setup: A 68×68 visible-image template and 20×20 SAR search window were used to compare NCC, MI, HOGncc, and HOPCncc.The similarity curves were evaluated on a high-resolution visible–SAR image pair.
  • Similarity-curve comparison: NCC and MI fail to detect the correct control point when radiometric differences are significant.Their similarity curves do not peak at the correct matching position in the visible–SAR example.
  • Similarity-curve comparison: HOGncc reaches the correct control point at its maximum, but its curve peak is not very significant.Its peak is less distinguishable than the HOPCncc peak.
  • Similarity-curve comparison: HOPCncc detects the correct control point and produces a smoother, more distinguishable similarity peak than the other metrics.The result indicates greater robustness to nonlinear radiometric differences in this example.

Results on Synthetic Datasets

Across synthetic and real multimodal datasets, structural descriptors outperform intensity-based metrics in correct match ratio, with HOPCncc generally ranking first and remaining comparatively stable across template sizes.

  • Synthetic datasets: HOPCncc, HOGncc, and MI improve as template size increases, whereas NCC shows no similar regularity.HOPCncc is slightly superior to HOGncc because phase congruency better handles non-uniform illumination and contrast changes than gradients.
  • Real multimodal datasets: In almost all real-image tests, HOPCncc outperforms the other metrics for every template size, with HOGncc second, MI third, and NCC lowest.The real datasets include Visible–Infrared, LiDAR–Visible, Visible–SAR, and Image–Map pairs.
  • Interpretation: Structural metrics outperform intensity-based metrics because phase congruency and gradients are more robust to nonlinear radiometric differences.HOPCncc improves over HOGncc because phase congruency is more robust to illumination and contrast changes than gradients.

F. Analysis of Computational Efficiency

HOPCncc uses a fast matching scheme but requires more computation than HOGncc and NCC because extracting its phase-congruency descriptor is time-consuming.

  • Computational setup: Run time was compared for NCC, MI, HOGncc, and HOPCncc across template sizes on an Intel Core i7-4710MQ 2.50GHz PC.HOPCncc and HOGncc used the proposed fast matching scheme.
  • Runtime comparison: NCC requires the least run time among the compared similarity metrics.Its lower computational requirement is attributed to its lower computation complexity.
  • Runtime comparison: HOPCncc requires more run time than HOGncc because phase-congruency extraction is more time-consuming than gradient calculation.The fast matching scheme reduces matching cost but does not remove the descriptor-extraction overhead.

V. EXPERIMENTAL RESULTS: MULTIMODAL REGISTRATION

The HOPCncc-based registration method successfully registers six multimodal image pairs and achieves the highest reported accuracy, while performance varies with image resolution and local geometric distortion.

  • Experimental design: The evaluation uses six multimodal image sets spanning Visible–Infrared, LiDAR–Visible, SAR–Visible, and Image–Map data.Images were captured by different sensors and spectral systems, producing significant nonlinear radiometric differences.
  • Registration accuracy: The proposed method successfully registers all six image pairs and achieves the highest registration accuracy.It is compared with manual registration and a SIFT-based method.
  • Registration accuracy: For SAR–Visible and LiDAR–Visible tests, HOPCncc outperforms manual registration by detecting more control points for fitting complex deformations.The SAR and LiDAR data also contain significant noise, making visual control-point localization difficult.
  • Comparison with SIFT: SIFT-based registration fails in most tests except Visible–Infrared 1 because significant radiometric differences prevent highly repeatable common-feature extraction.This limitation is attributed to SIFT’s difficulty with multimodal radiometric differences.
  • Accuracy variation: Visib-Infra 1 and SAR-Visib 1 achieve sub-pixel accuracy, whereas the other test sets have RMSE larger than 1 pixel.Higher-resolution urban pairs contain relief-displacement distortions that image-to-image registration cannot resolve without true orthorectification.
Loading 2103.16871v1…