Source-linked AI summary
PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening
Qingjie Liu, Huanyu Zhou, Qizhi Xu, Xiangyu Liu, Yunhong Wang
TL;DR
Remote-sensing pan-sharpening must combine complementary PAN and MS measurements to produce HR MS images while preserving spatial and spectral fidelity. PSGAN addresses this through a conditional generative adversarial network, and experiments across three satellite datasets report high-quality results, strong comparisons with existing methods, and full-scale generalization. The paper also identifies two-stream input as generally preferable to stacking and notes that full-scale generalization remains underdeveloped.
Problem
Pan-sharpening seeks HR MS images from complementary PAN and MS inputs, but a gap remains between synthetic and real HR MS images and achieving high spectral and spatial fidelity is challenging.
Method
PSGAN uses a conditional GAN with PAN and LR MS inputs, a generator for HR MS images, adversarial discrimination, and evaluated architectural designs including two-stream input, stacking, batch normalization, and attention.
Results
Experiments on QuickBird, GaoFen-2, and WorldView-2 images report high-quality pan-sharpened images with fine spatial details and high-fidelity spectral information under low-scale and full-scale settings.
Takeaways & Limitations
The study reports PSGAN superiority to many popular pan-sharpening approaches, with two-stream architecture generally better than stacking and the models generalizing well to full-scale images.
Takeaways & Limitations
The authors state that PSGAN generalization to full-scale images remains underdeveloped and identify unsupervised learning without preprocessing as future work.
Abstract
from arXiv · showhide
This paper addresses the problem of remote sensing image pan-sharpening from the perspective of generative adversarial learning. We propose a novel deep neural network based method named PSGAN. To the best of our knowledge, this is one of the first attempts at producing high-quality pan-sharpened images with GANs. The PSGAN consists of two components: a generative network (i.e., generator) and a discriminative network (i.e., discriminator). The generator is designed to accept panchromatic (PAN) and multispectral (MS) images as inputs and maps them to the desired high-resolution (HR) MS images and the discriminator implements the adversarial training strategy for generating higher fidelity pan-sharpened images. In this paper, we evaluate several architectures and designs, namely two-stream input, stacking input, batch normalization layer, and attention mechanism to find the optimal solution for pan-sharpening. Extensive experiments on QuickBird, GaoFen-2, and WorldView-2 satellite images demonstrate that the proposed PSGANs not only are effective in generating high-quality HR MS images and superior to state-of-the-art methods and also generalize well to full-scale images.
I. INTRODUCTION
Pan-sharpening fuses high-resolution PAN and low-resolution MS imagery to generate high-resolution MS images, but achieving both spatial and spectral fidelity remains challenging. PSGAN reformulates the task as conditional image generation using adversarial learning and evaluates architectural alternatives for this goal.
- Motivation: Pan-sharpening combines spatial information from high-resolution PAN images with spectral information from low-resolution MS images to generate high-resolution MS imagery.
- Prior approaches: Traditional approaches include component substitution, multi-resolution analysis, inverse-problem optimization, and machine-learning formulations.
- Motivation: A persistent challenge is the gap between synthetic and real HR MS images, particularly in obtaining high spectral and spatial fidelity.
- Proposed approach: PSGAN reformulates pan-sharpening as image generation and uses a GAN conditioned on PAN and LR MS inputs to produce high-quality pan-sharpened images.
- Evaluation: The study evaluates PSGAN configurations, including input strategy, batch normalization, and attention mechanisms, to identify factors affecting pan-sharpening performance.
- Proposed approach: Its generator uses a two-stream CNN architecture, while a fully convolutional discriminator learns an adaptive loss function to improve pan-sharpened image quality.
- Evaluation: The authors report that PSGAN produces strong pan-sharpening results, illustrated through generated HR MS images compared with ground-truth HR MS images.
II. GENERATIVE ADVERSARIAL NETWORKS
GANs learn data distributions through an adversarial game between a generator and discriminator. The generator produces samples intended to resemble real data, while the discriminator distinguishes real from generated samples through iterative optimization.
- GAN framework: GANs provide a framework for learning generative models from unlabeled data by estimating underlying data distributions.
- GAN framework: The generator maps random samples to data-space outputs and is trained to produce samples that cannot be distinguished from real data.
- Optimization: The adversarial objective trains the discriminator to classify real and generated samples correctly while the generator seeks outputs that appear real.
- Optimization: Training alternates optimization of the discriminator and generator; with sufficient capacity and training time, the learned generator distribution can converge to the real data distribution.
III. PSGAN
PSGAN reformulates pan-sharpening as conditional image generation: a generator maps LR MS and HR PAN inputs to HR MS images, while a discriminator promotes outputs indistinguishable from reference images.
- Pan-sharpening estimates an HR MS image ˆP from LR MS X and HR PAN Y, with r = 4 in this paper.The target ˆP and reference P have dimensions rw×rh×b.
- The model expresses pan-sharpening as ˆP = f(X, Y; Θ), where f maps the two inputs to the desired HR MS output.Θ denotes the collection of model parameters.
- PSGAN treats this mapping as conditional image generation, with generator G producing ˆP from (X, Y) and discriminator D distinguishing it from reference P.The formulation is trained as an adversarial mini-max game.
- Adversarial learning is intended to generate faithful HR MS images for pan-sharpening.
B. Architectures of the generator
The generator exploits the complementary spatial and spectral information in PAN and MS imagery through either two-stream processing or alternative up-scaling arrangements.
- Unlike one-to-one image-generation tasks, pan-sharpening combines PAN and MS images from different sensors with distinct characteristics.
- Two-stream generator: The two-stream generator uses separate subnetworks to extract hierarchical PAN and MS features before an encoder fuses them and a decoder reconstructs HR MS images.The design targets complementary spatial and spectral information from the two modalities.
- Because MS resolution is one-quarter of the desired pan-sharpened resolution, the generator can up-scale MS before processing or up-scale within the network.
- PSGAN: PSGAN first up-samples MS, then feeds it and PAN into two feature-extraction subnetworks whose fused features are decoded into the desired output.
- FU-PSGAN: FU-PSGAN instead accepts original-sized MS and performs feature up-scaling through an up convolution after its first convolution layer.
2) PAN & MS Stacked generator:
The stacked generator concatenates PAN with upsampled MS into one input, while the discriminator remains conditional and fully convolutional across PSGAN variants.
- PAN & MS Stacked generator: Stacking concatenates PAN and upsampled MS along the channel dimension to form a single inflated image input.This design can inherit models developed for related tasks such as single-image super-resolution.
- PAN & MS Stacked generator: ST-PSGAN uses a deeper residual stacked generator, removing one stream and its skip connection to match the concatenated PAN–MS input.
- Fully convolutional discriminator: The conditional discriminator distinguishes reference MS images from generated pan-sharpened images using a fully convolutional five-layer network.Its first three convolutional layers use stride 2, and the last two use stride 1.
- The architectures limit depth and kernel size to balance model complexity and performance, excluding much deeper networks and 5×5 or 7×7 kernels.
- Figures 3 and 4 provide detailed parameters and shared architectural legends for the PSGAN generator and its variants.
D. Loss function
PSGAN training alternates generator and discriminator optimization using pixel-wise and adversarial objectives, with experiments conducted on three satellite-image datasets under Wald’s protocol.
- Loss function: The generator is optimized with pixel-wise and adversarial losses, contrasting with prior methods that use ℓ2 mean-squared-error loss.
- Loss function: The discriminator is trained alternately with the generator as part of PSGAN’s adversarial optimization.
- Loss function: The loss hyperparameters α and β are set to 1 and 100, respectively, in the experiments.
- Experiments: Experiments evaluate PSGANs on QuickBird, GaoFen-2, and WorldView-2 imagery using Wald’s protocol with down-sampling factor r = 4.Original MS images serve as references because desired HR MS images are unavailable.
- Implementation: Models are trained in PyTorch with Adam on a single NVIDIA Titan 2080Ti GPU, using minibatches of 8 and approximately 8 hours per model.
B. Evaluation indexes
The paper evaluates pan-sharpening with reference and no-reference indexes covering spectral fidelity, spatial detail, and global fused-image quality.
- Reference-based indexes: Five reference-based metrics evaluate spectral and spatial quality: SAM, CC, sCC, ERGAS, and Q4.SAM measures spectral distortion; CC measures spectral quality; sCC evaluates spatial-detail similarity; ERGAS is a global quality index; Q4 extends the Q index to four bands.
- Reference-based indexes: SAM compares spectral vectors pixelwise, with 0 representing ideal spectral fidelity.The metric is averaged across images to produce a global spectral-distortion measurement.
- Reference-based indexes: CC measures correlation with the reference image, ranging from -1 to +1, with +1 ideal.The cited definition uses image means and spatial dimensions in calculating the correlation coefficient.
- Reference-based indexes: sCC applies a high-pass filter before correlating pan-sharpened and reference details; higher values indicate more injected PAN spatial information.The final sCC averages correlations across all multispectral bands.
- No-reference indexes: Dλ, DS, and QNR provide full-resolution assessment without a reference image.Dλ measures spectral quality, DS is its complementary spatial metric, and QNR combines both; lower Dλ and DS are better, while ideal QNR is 1.
C. Impact of patch size
The experiments examine how training patch size affects PSGAN performance, finding that larger patches improve reconstructed image quality despite reducing batch size.
- Impact of patch size: Larger training patches lead to higher image quality for reconstruction, although the corresponding batch size decreases.The study tests a much larger patch size than its previous work and reports the improvement experimentally.
D. Impact of number of feature maps and kernel size
PSGAN’s feature-map and kernel-size ablations expose a trade-off between model efficiency, trainability, and dataset-dependent performance.
- Feature maps: Halving feature maps creates PSGAN-f16 with about one-quarter the parameters and faster execution, but unsatisfactory performance.PSGAN-f16 outperforms PSGAN+BN and PSGAN+SA but remains weaker than PSGAN and PSGAN-k5×5.
- Kernel size: PSGAN-k5×5 uses more than twice PSGAN’s parameters and about ten times PSGAN-f16’s, making training difficult.The larger-kernel design performs better than PSGAN on GF-2 but worse on QB and WV-2.
- Kernel size: Kernel enlargement is therefore dataset-dependent: PSGAN-k5×5 works especially well on GF-2 but not consistently across QB and WV-2.The comparison is reported relative to the original PSGAN.
E. Batch normalization is harmful
The ablations report that batch normalization substantially harms PSGAN performance, while self-attention improves over batch normalization but remains worse than the original model.
- Batch normalization: Adding batch-normalization layers severely decreases performance, especially on QB and WV-2 images.The authors consequently remove all batch-normalization layers from their models.
- Batch normalization: Batch normalization adds storage and computational burdens and removes scale information useful for scale-sensitive vision tasks.The paper identifies pan-sharpening and image super-resolution as examples of such tasks.
- Self-attention: Self-attention improves over PSGAN+BN on QB and WV-2 but still performs worse than the original PSGAN.Some CC, sCC, and Q4 results are satisfactory, yet the authors conclude that self-attention is not suitable for their models.
G. Two-stream is better than stacking
The two-stream PSGAN designs generally outperform stacking, with dataset-dependent differences among PSGAN, FU-PSGAN, and ST-PSGAN. PSGAN variants also compare favorably with traditional and deep pan-sharpening methods across datasets.
- Two-stream versus stacking: ST-PSGAN is worst among the three variants in almost all cases, except on GF-2 where it achieves the second-best results.The comparison uses quantitative metrics from Tables IV–VI, including full-scale non-reference measurements.
- Two-stream versus stacking: PSGAN is strongest on QB, with the lowest Dλ and DS and the highest QNR among the three variants.FU-PSGAN remains better than ST-PSGAN on QB for all metrics except sCC.
- Full-scale generalization: The PSGAN variants generalize well to full-scale images and achieve competitive performance across the three datasets.FU-PSGAN is particularly strong on full-scale WV-2 images.
- Comparison with other methods: Deep models generally outperform traditional methods, and the proposed PSGAN obtains the best QB metrics except SAM while its variants outperform other methods on GF-2 and WV-2.On WV-2, FU-PSGAN has the highest SAM, CC, ERGAS, and Q4, with slightly worse sCC than PSGAN.
I. Visual comparisons
Visual evaluations show that the PSGAN variants preserve faithful colors and spatial details, while full-scale results remain competitive without target images for training. The authors conclude that PSGANs produce high-quality outputs across low-scale and full-scale settings.
- Visual comparisons: LMVM and PNN tend to blur images, whereas several traditional methods inject spatial information efficiently and produce clean high-frequency details.The visual comparison covers QuickBird and GF-2 test samples displayed in true color.
- Full-scale evaluation: Full-scale evaluation applies optimized networks directly to original PAN and MS images because no target images are available for training.Generalization across scales is the main concern in this experiment.
- Full-scale evaluation: The proposed PSGANs generalize well to full-scale images and achieve competitive performance on QuickBird, GaoFen-2, and WorldView-2.FU-PSGAN achieves the best full-scale results on WV-2 images.
- Conclusion: The authors report high-quality pan-sharpened images with fine spatial details and high-fidelity spectral information under low-scale and full-scale settings.The conclusion is based on experiments across QuickBird, GaoFen-2, and WorldView-2 images.