Source-linked AI summary
Identifying Corresponding Patches in SAR and Optical Images with a Pseudo-Siamese CNN
Lloyd H. Hughes, Michael Schmitt, Lichao Mou, Yuanyuan Wang, Xiao Xiang Zhu
TL;DR
The paper addresses the difficult problem of matching corresponding patches across very-high-resolution SAR and optical imagery, whose sensing modalities produce strongly different appearances. It proposes a pseudo-Siamese CNN with separate modality-specific streams and later feature fusion, trained with labeled patch pairs. On a challenging urban test setting, the approach achieves accuracy exceeding 77% at a 5% false positive rate and shows potential for multi-sensor key-point matching.
Problem
Matching corresponding SAR and optical patches remains difficult because the modalities produce strongly different appearances, while handcrafted descriptors have limitations in dense VHR urban scenes.
Method
A pseudo-Siamese CNN processes SAR and optical patches in separate convolutional streams, fuses their feature maps later, and trains with one-hot correspondence labels.
Results
Accuracy exceeds 77% on a separate test dataset when the false positive rate is fixed at 5%.
Takeaways & Limitations
The first evaluation indicates promising potential for developing generalized multi-sensor key-point matching procedures.
Abstract
from arXiv · showhide
In this letter, we propose a pseudo-siamese convolutional neural network (CNN) architecture that enables to solve the task of identifying corresponding patches in very-high-resolution (VHR) optical and synthetic aperture radar (SAR) remote sensing imagery. Using eight convolutional layers each in two parallel network streams, a fully connected layer for the fusion of the features learned in each stream, and a loss function based on binary cross-entropy, we achieve a one-hot indication if two patches correspond or not. The network is trained and tested on an automatically generated dataset that is based on a deterministic alignment of SAR and optical imagery via previously reconstructed and subsequently co-registered 3D point clouds. The satellite images, from which the patches comprising our dataset are extracted, show a complex urban scene containing many elevated objects (i.e. buildings), thus providing one of the most difficult experimental environments. The achieved results show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development towards a generalized multi-sensor key-point matching procedure. Index Terms-synthetic aperture radar (SAR), optical imagery, data fusion, deep learning, convolutional neural networks (CNN), image matching, deep matching
I. INTRODUCTION
Matching corresponding patches across SAR and optical imagery remains difficult because the sensors produce strongly different appearances, especially for elevated structures in VHR urban scenes. The paper addresses this challenge with a pseudo-Siamese CNN that processes both modalities separately.
- Cross-sensor image matching remains widely unsolved, particularly for jointly exploiting SAR and optical imagery.
- SAR uses range-based imaging, whereas optical imagery reflects chemical characteristics through perspective imaging, producing different appearances for buildings and trees.
- Handcrafted descriptors reach their limits on highly resolving, densely built-up urban scenes that can be difficult to interpret even for trained SAR experts.
- The work learns a correspondence predictor for SAR and optical VHR patches using a convolutional neural network.
- Its pseudo-Siamese design uses separate but identical convolutional streams to process SAR and optical patches in parallel rather than sharing weights.
II. THE NETWORK ARCHITECTURE
The proposed network uses separate feature-extraction streams for SAR and optical patches, then fuses their representations at a later decision stage. Its fusion layers are designed to accommodate spatial mismatches between the modalities.
- A. Pseudo-Siamese Convolutional Network: SAR and optical patches are processed in separate yet identical convolutional streams because their geometric and radiometric appearances differ strongly.
- A. Pseudo-Siamese Convolutional Network: The two streams learn modality-specific representations before combining information at a higher decision level.
- A. Pseudo-Siamese Convolutional Network: The streams use 3×3 convolutional filters, which capture directional patterns while increasing the network’s nonlinearity through stacked layers.
- A. Pseudo-Siamese Convolutional Network: Seven max-pooling layers reduce feature-map dimensionality using 2 × 2 windows with stride 2.
- A. Pseudo-Siamese Convolutional Network: Fusion applies two convolutional layers to concatenated feature maps, followed by two fully connected layers.
- A. Pseudo-Siamese Convolutional Network: The fusion design omits pooling after its first convolution and uses stride 2 to preserve spatial information while learning tolerance to spatial mismatches.
B. Loss Function
The network is trained on SAR-optical patch pairs with one-hot correspondence labels using binary cross-entropy. Its softmax output represents the predicted class for each input pair.
- B. Loss Function: The training set consists of SAR-optical patch pairs represented as X = (x_sar_i, x_opt_i).
- B. Loss Function: The network minimizes binary cross-entropy loss during training.
- B. Loss Function: Each pair receives a one-hot label y_i distinguishing dissimilar pairs [1, 0] from similar pairs [0, 1].
- B. Loss Function: The softmax output vector gives the network’s prediction for the correspondence class of each SAR-optical input pair.
C. Configuration Details
Training uses normalization and regularization choices intended to accelerate optimization and reduce over-fitting.
- Batch normalization follows each convolutional activation to increase training speed and reduce internal covariate shift.
- L2 regularization uses λ = 0.001 on convolution kernels in both streams, while the first fully connected layer uses dropout at rate 0.7.
III. AUTOMATIC PATCH POOL GENERATION
The paper introduces a fully automatic procedure for generating corresponding and non-corresponding SAR–optical patch pools for training and testing.
- Automatic patch-pool generation addresses the difficulty of annotating corresponding VHR SAR and optical patches in complex urban scenes.The paper describes this annotation task as highly nontrivial even for experienced human experts.
A. The “SARptical” Framework
The SARptical framework matches SAR and optical imagery in 3D object space, enabling automated correspondence generation from reconstructed point clouds and aligned image patches.
- The “SARptical” Framework: SARptical matches SAR and optical images in 3D space to handle geometric distortions between the sensing modalities.The procedure uses object-space matching through 3D point clouds derived from SAR tomography.
- The “SARptical” Framework: The dataset uses 109 TerraSAR-X images and 9 UltraCAM images covering Berlin, producing 10,108 corresponding optical pixels from 8,840 sampled SAR pixels.The TerraSAR-X imagery has about 1 m resolution, while the optical imagery has 20 cm ground spacing.
- The “SARptical” Framework: SAR patches are centered on SAR pixels, while resampled and rotated optical patches are centered on corresponding optical pixels for approximate alignment.The fixed SAR patch size is 112 × 112 pixels with approximately 1 m pixel spacing.
- The “SARptical” Framework: Each SAR patch receives one randomly selected correct optical correspondence and one randomly assigned incorrect correspondence, yielding 17,680 pairs.Optical patches are converted to grayscale and normalized to [0; 1] after mean subtraction.
C. Patch Pool Partitioning
The patch pool is split deterministically into training, validation, and test sets to reduce positively biased evaluation while testing patch-size variants under controlled randomness.
- C. Patch Pool Partitioning: 9724 (55%) patch pairs form the training set, 2652 (15%) the validation set, and 5304 (30%) the test set.
- C. Patch Pool Partitioning: Deterministic image partitioning prevents related samples from being assigned through purely random splitting and helps avoid positively biased test results.Positive and negative samples are generated separately within the deterministic partitions.
- C. Patch Pool Partitioning: Five network versions are trained at different patch sizes to evaluate their effect on classification accuracy.The random number generator is seeded with 0 for each patch-size experiment.
B. Evaluation Results
Evaluation compares patch sizes through accuracy–false-positive-rate curves and confusion matrices, then tests the model in a localized multi-candidate key-point matching scenario.
- B. Evaluation Results: Accuracy–false-positive-rate curves and confusion matrices are reported for each evaluated input patch size.The confusion matrix represents the network at the point of highest overall performance for each patch size.
- B. Evaluation Results: The key-point experiment compares every SAR patch with every optical patch among 100 neighboring TomoSAR key-points.The localized selection reproduces conditions encountered in a real-world key-point matching application.
- B. Evaluation Results: Similarity-score matrices place corresponding SAR and optical patches at matching index numbers, while sorted non-similar scores expose incorrect-match strength.A threshold converts similarity scores into binary correspondence decisions.
- B. Evaluation Results: Patch-size comparisons are presented in Fig. 4, while Table I reports the corresponding confusion-matrix values.
V. DISCUSSION
The proposed network shows promising discriminative power, but the results require careful interpretation.
- The authors identify major considerations that should guide interpretation of the reported network results.The discussion introduces these considerations before detailing specific effects and limitations.
A. Influence of the Patch Size
Patch size strongly affects the network’s discriminative power: smaller crops can remove defining cross-modal features, while larger context may preserve them. The approach also shows useful correspondence performance on separate testing and key-point matching tasks.
- Patch size strongly affects the network’s discriminative power, as indicated by Tab. I and Fig. 4.
- Fig. 6 presents exemplary patch correspondence results.
- Smaller patches may crop out defining features needed to match SAR and optical imagery.SAR distortions arise from range-based imaging geometry, including layover and multi-path reflections.
- 77% accuracy was exceeded on a separate test dataset at a fixed 5% false-positive rate.This was in the same order of magnitude as the HOPC descriptor with an L2-norm cost function.
- 43% top-1 and 74% top-3 matching accuracy were achieved, while 8% of points had no valid key-point matches.The results were associated with a clear diagonal pattern in Fig. 5a.
C. Possible Reasons for False Predictions
False predictions frequently occur when SAR and optical patches have extreme viewing-angle differences, although larger patches may provide useful context. Strongly distorted image parts may need exclusion, and broader training and testing are planned for transfer beyond key-point matching.
- Extreme viewing-angle differences produce many false positives and false negatives in patch correspondence predictions.
- Larger patch sizes may partially reduce these errors by providing valuable contextual information.
- Image parts with very strong distortions may need to be excluded from further processing.
- The pseudo-siamese CNN showed promising potential for multi-sensor key-point matching procedures.
- Additional training patches and data from completely different sources are planned to assess transferability beyond key-point applications.The stated target includes applications such as dense matching.