Source-linked AI summary
Remote Sensing Image Fusion Based on Two-stream Fusion Network
Xiangyu Liu, Qingjie Liu, Yunhong Wang
TL;DR
Pan-sharpening must combine high-spatial-resolution PAN imagery with lower-spatial-resolution multispectral imagery. This paper proposes TFNet, which extracts and fuses PAN and MS features before reconstructing the output, and reports promising spatial and spectral quality on QuickBird and GaoFen-1 images, further improved by ℓ1 loss and residual learning.
Problem
Pan-sharpening must combine PAN spatial detail with MS spectral information, while previous CNN methods performed the task at pixel level.
Method
TFNet uses two feature-extraction streams, fuses PAN and MS features, and reconstructs the pan-sharpened image from the fused representation.
Results
Experiments on QuickBird and GaoFen-1 images produced pan-sharpened images with promising spatial and spectral qualities.
Takeaways & Limitations
ℓ1 loss and residual learning further improve the proposed network's pan-sharpening results.
Takeaways & Limitations
Future work includes developing loss functions more suitable for pan-sharpening and studying unsupervised methods.
Abstract
from arXiv · showhide
Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel level, the proposed TFNet aims to fuse PAN and MS images in feature level and reconstruct the pan-sharpened image from the fused features. The TFNet mainly consists of three parts. The first part is comprised of two networks extracting features from PAN and MS images, respectively. The subsequent network fuses them together to form compact features that represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features through an image reconstruction network. Experiments on Quickbird and \mbox{GaoFen-1} satellite images demonstrate that the proposed TFNet can fuse PAN and MS images, effectively, and produce pan-sharpened images competitive with even superior to state of the arts.
1. Introduction
Pan-sharpening combines complementary PAN and MS modalities to obtain high spatial and spectral resolution. TFNet performs this fusion in feature space, with architectural and training refinements evaluated against prior methods.
- Motivation: PAN sensors provide high spatial resolution in one band, while MS sensors provide lower spatial resolution across multiple bands.Pan-sharpening seeks to combine these complementary modalities into imagery with both PAN spatial and MS spectral resolution.
- Related work: Earlier CNN-based pan-sharpening methods commonly adapted three-layer super-resolution networks or combined CNN enhancement with GS fusion.These approaches treated pan-sharpening primarily as a super-resolution or pixel-level fusion problem.
- Contribution: TFNet introduces a two-stream CNN architecture that addresses pan-sharpening through feature-level fusion.The paper positions this design as an alternative to previous pixel-level CNN approaches.
- Improvements: Residual learning is incorporated to further improve TFNet performance in the pan-sharpening problem.The authors explicitly demonstrate an improvement associated with residual learning.
- Scope: The study extends earlier work with expanded background, design motivation, two improvements, and more extensive experiments.The paper reports experiments on QuickBird and GaoFen-1 satellite imagery.
2. Background and Related Work
CNNs use hierarchical feature extraction through locally connected, weight-shared filters and have become important in image processing. In pan-sharpening, prior CNN methods largely framed fusion as super-resolution or residual prediction.
- CNN fundamentals: CNNs generate feature maps by applying multiple learned filters, with pooling commonly down-sampling their spatial dimensions.Local connectivity and weight sharing reduce parameter counts and support deeper architectures.
- Residual learning: Residual learning addresses degradation in deep networks by separating identity mappings from learned residual mappings through shortcut connections.This framework made it possible to build extremely deep residual networks.
- CNN pan-sharpening: Early CNN pan-sharpening methods treated the task as a special form of image super-resolution using three-layer CNN architectures.Because pan-sharpening has PAN and MS inputs, one method stacked an upsampled MS image with PAN data.
- CNN pan-sharpening: Later pan-sharpening studies used CNNs to learn residuals between upsampled MS images and ground-truth MS images.The reviewed approaches included residual prediction and deeper networks, while PanNet addressed the task with a deep network.
3.1. Motivation
PAN and MS images carry different spatial and spectral information, but these components are difficult to define and represent independently. TFNet therefore fuses their hierarchical features before reconstructing the pan-sharpened image.
- Information sources: PAN images primarily carry geometric detail, whereas MS images preserve spectral information needed for pan-sharpening.The target combines PAN spatial details with MS spectral information in a high-resolution MS image.
- Prior assumptions: Traditional ARSIS and CS methods rely on explicit high-frequency or transformed-component assumptions about where spatial information resides.ARSIS methods associate spatial information with PAN high frequencies, while CS methods substitute a transformed MS component with PAN information.
- Feature-level fusion: TFNet fuses PAN and MS in feature space and reconstructs the pan-sharpened image from fused features.The motivation is that pan-sharpened-image features should contain information from both input modalities.
3.2. Problem formulation
The paper formulates pan-sharpening as feature-level fusion: CNNs extract representations from PAN and MS images, which are concatenated, compacted, and reconstructed end-to-end.
- A complete feature representation should permit recovering the input image without information loss.
- CNNs extract hierarchical features from PAN and MS images using separate networks denoted ΦP and ΦM.
- The method avoids pooling-based fusion because pooling can lose information, instead concatenating the channel-wise feature maps.
- A fusion network compacts the concatenated features, and a reconstruction network recovers the pan-sharpened image in one end-to-end two-stream architecture.
3.3. TFNet
TFNet uses an encoder-decoder architecture with separate PAN and MS feature extractors, feature fusion, and image reconstruction. Downsampled fused representations are progressively upsampled, with skip connections restoring details before producing a 4-band MS image.
- Feature extraction network: MS is up-sampled to the PAN image size before entering the MS feature extraction network.
- 3.3. TFNet: TFNet comprises feature extraction, feature fusion, and image reconstruction, with the first two forming an encoder and the last forming a decoder.
- Feature extraction network: The PAN and MS sub-networks have similar architectures but different weights, accepting single-band PAN and 4-band MS inputs respectively.
- Feature fusion network: The extracted PAN and MS feature maps are concatenated and passed through a three-layer fusion network to form compact representations containing spatial and spectral information.
- Image reconstruction network: Because encoded feature maps occupy 1/4 of the input width and height, transposed convolutions progressively upsample them in a structure symmetrical to the encoder.
- Image reconstruction network: Skip connections concatenate encoder features with corresponding decoder features to inject details lost during downsampling, and the last layer outputs a 4-band high-resolution MS image.
3.4. Improved with residual blocks
The paper improves TFNet by replacing plain CNN units with residual units. Residual learning uses identity mappings and requires added convolutional layers where concatenation changes the feature pathway.
- 3.4. Improved with residual blocks: Residual learning is introduced to further improve TFNet performance for pan-sharpening.
- Skip connections: The network uses skip connections between encoder and decoder without applying the crop operation.
- Residual units: The paper distinguishes plain and residual CNN units as alternative building blocks for the network.
- Residual units: A residual unit maps xl to xl+1 through a residual function, activation function, and identity mapping h(xl), typically h(xl) = xl.
- Residual TFNet: ResTFNet replaces every two successive convolutional layers with a residual unit and adds a convolutional layer after each concatenation layer to match input and output sizes.
3.5. Loss function
TFNet parameters are optimized against corresponding high-resolution MS ground truth images. The paper chooses ℓ1 loss because ℓ2-based restoration can produce blur and evaluates both losses under the same architecture.
- 3.5. Loss function: The network parameters Θ are optimized by minimizing loss between pan-sharpened outputs and corresponding ground truth images.
- Loss choice: The ℓ1 loss is selected because prior restoration studies suggest it is preferable to ℓ2 loss, which can produce blurred images.
- Training objective: The training samples contain PAN and low-resolution MS inputs paired with corresponding high-resolution MS images.
- Loss definition: The loss definition uses N, the number of training samples in a mini-batch.
4. Experiments and analysis
Experiments evaluate TFNet variants and competing methods on Quickbird and GaoFen-1 imagery using six quality indicators and Wald’s protocol. Results show benefits from ℓ1 loss, residual learning, and feature compensation, with TFNets achieving the strongest quantitative performance.
- Data sets: Quickbird provides 0.6 m PAN and 2.4 m four-band MS imagery, while the experiments use 2 m PAN and 8 m MS data from GaoFen-1.The GaoFen-1 spectral wavelength features are similar to Quickbird, but Quickbird has higher spatial resolution.
- Evaluation protocol: Wald’s protocol downsamples PAN and MS inputs by four in width and height, using original MS images as references for evaluation.Input MS images are also bicubically upsampled to match PAN resolution.
- Evaluation indexes: The study quantitatively evaluates the proposed and competing methods with six widely used indicators, including SAM, CC, sCC, UIQI, Q4, and ERGAS.SAM measures spectral distortion, while sCC evaluates similarity of high-frequency spatial details.
- Impacts of detail compensation: Skip connections that concatenate low-level PAN and MS features accelerate convergence and reduce training errors.The connections provide a path for information propagation between low- and high-level features during backward propagation.
- Loss functions and residual learning: ℓ1-trained TFNet improves spatial, spectral, and global quality over ℓ2-trained TFNet, producing finer details, sharper edges, and better SAM, ERGAS, and UIQI.Residual units further improve ResTFNet-ℓ1 across spatial, spectral, and global quantitative metrics.
- Comparison with other methods: TFNet-ℓ1 and ResTFNet-ℓ1 achieve the best performance across all reported indicators on the two test images.Residual units particularly reduce spectral distortions, while visual comparisons show richer spatial details and stronger spectral preservation than competing methods.
5. Conclusion
The proposed TFNet performs pan-sharpening by fusing PAN and MS images in the feature domain, producing promising spatial and spectral qualities on Quickbird and GaoFen-1 images. Results can be further improved through ℓ1 loss and residual learning.
- TFNet addresses pan-sharpening by fusing PAN and MS images in the feature domain rather than the pixel domain.The network extracts features from both inputs, fuses them, and reconstructs the pan-sharpened image.
- Experiments on Quickbird and GaoFen-1 images demonstrate promising spatial and spectral qualities.
- ℓ1 loss and residual learning further improve the results.The paper identifies more suitable loss functions and unsupervised methods as future work.