Source-linked AI summary
Deep Gradient Projection Networks for Pan-sharpening
Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, Chunxia Zhang
TL;DR
Pan-sharpening must recover high-resolution multispectral images from complementary PAN and LRMS observations, while many deep networks require substantial training data and do not model the observation process. The paper formulates separate PAN and LRMS optimization problems with deep priors, solves them by gradient projection, and unrolls the updates into alternating GPPNN blocks. Experiments on three satellites show that GPPNN outperforms state-of-the-art methods visually and quantitatively, while its trained parameters do not generalize across satellites.
Problem
Existing convolutional pan-sharpening networks require many training samples and do not account for the PAN and LRMS observation process, limiting interpretability.
Method
GPPNN uses separate PAN and LRMS generative observation models with deep-prior regularized optimization, then unrolls gradient-projection iterations into alternately stacked neural blocks.
Results
GPPNN outperforms 13 state-of-the-art and classic pan-sharpening methods quantitatively and visually on Landsat8, QuickBird, and GF2.
Takeaways & Limitations
The work provides a model-driven pan-sharpening paradigm combining deep unrolling with PAN and LRMS observation models.
Takeaways & Limitations
GPPNN trained on one satellite cannot be generalized to another satellite, motivating future work on improving its generalization.
Abstract
from arXiv · showhide
Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multispectral images. Recently, deep learning has become the most popular tool for pan-sharpening. This paper develops a model-based deep pan-sharpening approach. Specifically, two optimization problems regularized by the deep prior are formulated, and they are separately responsible for the generative models for panchromatic images and low resolution multispectral images. Then, the two problems are solved by a gradient projection algorithm, and the iterative steps are generalized into two network blocks. By alternatively stacking the two blocks, a novel network, called gradient projection based pan-sharpening neural network, is constructed. The experimental results on different kinds of satellite datasets demonstrate that the new network outperforms state-of-the-art methods both visually and quantitatively. The codes are available at https://github.com/xsxjtu/GPPNN.
1. Introduction
Pan-sharpening combines low-resolution multispectral and high-resolution panchromatic observations to recover high-resolution multispectral images. The paper proposes GPPNN, a model-based network that unrolls gradient-projection updates derived from separate PAN and LRMS observation models.
- Motivation: Satellites measure complementary images: low-spatial-resolution multispectral data and high-spatial-resolution, single-band panchromatic data.These observations are used together in pan-sharpening to obtain high-resolution multispectral imagery.
- Motivation: Conventional deep pan-sharpening networks often treat PAN as an input channel, overlooking differences between PAN and LRMS observations.The paper identifies this design choice as a motivation for two-branch or model-based alternatives.
- Approach: The paper formulates separate generative models for PAN and LRMS images using observation models and deep priors, producing two optimization problems.PAN images are modeled as linear combinations of HRMS bands, while LRMS images arise from blurring and downsampling HRMS images.
- Approach: Gradient projection solves the formulated optimization problems, and their iterative steps become two interpretable neural blocks that are alternately stacked in GPPNN.The MS block reconstructs the next HRMS estimate from LRMS residuals; the PAN block is interpreted similarly.
- Results: Experiments on Landsat8, QuickBird, and GF2 compare GPPNN with 13 state-of-the-art and classic methods, showing quantitative and visual improvements.The reported evaluation spans three popular satellite datasets.
2. Related work
Related work spans classic fusion algorithms, increasingly deep convolutional networks, two-branch architectures, and model-based neural networks. Model-based methods formulate task-specific observation models or optimization problems and translate algorithm iterations into network layers.
- Classic methods: Classic pan-sharpening methods include component substitution, multiresolution analysis, and other decomposition-based algorithms.Component-substitution methods combine PAN spatial information with LRMS spectral information to reconstruct HRMS images.
- Deep learning methods: Deep methods evolved from the three-unit PNN to networks with dozens of layers and two-branch designs that separately extract PAN and multispectral features.The two branches later fuse their features to reconstruct the HRMS image.
- Model-based methods: Model-based neural networks integrate task-specific observation models or optimization problems with prior knowledge, translating algorithm iterations into network layers.Stacked layers therefore correspond to executing the underlying algorithm for a selected number of iterations.
3. GPPNN
GPPNN combines observation models for LRMS and PAN images with learned deep priors, then unrolls gradient-projection updates into alternating MS and PAN blocks. The network initializes HRMS from bicubic-upsampled LRMS and reconstructs the final HRMS through stacked blocks.
- 3.1. Model formulation: The model assumes LRMS images arise by blurring and downsampling HRMS images, while PAN images are spectral responses of HRMS images.The observation model is L = DKH and P = HS, with separate LRMS and PAN data-fidelity terms.
- 3.1. Model formulation: Two deep priors separately account for the LRMS and PAN observations, replacing hand-crafted regularizers in the optimization formulation.The paper denotes these priors as h_l(·) and h_p(·).
- 3.2. MS Block: Gradient projection updates are translated into interpretable neural blocks whose computational flows follow the corresponding observation models.The MS Block and PAN Block are derived from separate optimization problems and can be embedded into a neural network.
- 3.2. MS Block: The MS Block generates an LRMS estimate from the current HRMS estimate, computes the LRMS residual, upsamples it, and applies a learned proximal update.The low-pass filter is replaced by cascaded convolutions, while downsampling uses bicubic interpolation.
- 3.3. PAN Block: The PAN Block uses a PAN estimate and residual to update the HRMS image, with 1 × 1 convolutional units reflecting the band-reduction assumption.The PAN image has one channel, and S/S^T is treated as a band reduction/expansion operator.
- 3. GPPNN: GPPNN initializes HRMS by bicubic interpolation of LRMS, then alternates MS and PAN blocks across K layers to produce the final reconstructed HRMS.Training uses an ℓ1 loss, and paired samples are generated with the Wald protocol; the reported balanced settings are C = 64 and K = 8.
4. Experiments
Experiments evaluate GPPNN against deep-learning and classic pan-sharpening methods across Landsat8, QuickBird, and GaoFen2 using quantitative metrics, visual inspection, and ablations. GPPNN achieves the strongest reported overall performance, while ablations support the roles of deep priors, separate blocks, and independently parameterized kernels.
- Datasets and metrics: Experiments compare GPPNN with six deep-learning methods and seven classic methods on Landsat8, QuickBird, and GaoFen2 datasets.The datasets are split into training, validation, and test sets, with patch sizes specified for multispectral and panchromatic inputs.
- Datasets and metrics: PSNR, SSIM, ERGAS, and SAM evaluate spatial and spectral distortion, with higher PSNR and SSIM but lower ERGAS and SAM indicating better images.PSNR, SSIM, and ERGAS measure spatial distortion, while SAM measures spectral distortion.
- Network configuration: K = 8 and C = 64 balance validation PSNR and model size, because increasing depth or width does not necessarily improve performance.The study varies K from 2 to 14 with C fixed at 64, then varies C from 8 to 128 with K fixed at 8.
- Comparison with SOTA methods: GPPNN outperforms other methods on all metrics across the three satellites and shows the smallest reported spatial and spectral distortions in visual comparisons.On QuickBird, competing methods show artifacts, noise, blur, or spectral distortion; GPPNN is reported without these issues.
- Ablation experiments: Removing proximal modules makes all metrics dramatically worse, supporting the significance of the network’s deep prior.The proximal modules are the convolutional units in Eqs. (9) and (12d).
- Ablation experiments: Weight sharing weakens performance, a single block corresponding to the unsplit problem performs worse than GPPNN, and transposed kernels slightly worsen metrics.The ablations support separate LRMS- and PAN-aware deep priors and retain greater freedom in the convolutional kernels.
5. Conclusion and Future Work
GPPNN combines deep unrolling with pan-sharpening observation models by alternating MS and PAN blocks derived from optimization problems. Experiments across three satellites support its performance, while satellite-specific imaging parameters limit cross-satellite generalization.
- Conclusion: GPPNN combines deep unrolling and observation models by alternatively stacking MS and PAN blocks inspired by two optimization problems.Its network structure is model-driven rather than purely intuitive.
- Conclusion: Experiments on three satellites show that GPPNN outperforms state-of-the-art methods, while ablation experiments support its network structure.The satellites are Landsat8, QuickBird, and GaoFen2.
- Future Work: GPPNN trained on one satellite cannot be generalized to another because each satellite has unique imaging parameters.Improving cross-satellite generalization is identified as future work.