PaperlayerView exact revision on arXiv

Source-linked AI summary

Pansharpening via Detail Injection Based Convolutional Neural Networks

Lin He, Yizhou Rao, Jun Li, Antonio Plaza, Jiawei Zhu

arXiv:1806.08898v1eess.IV

TL;DR

Pansharpening must combine low-resolution MS imagery with PAN detail to obtain high-resolution MS imagery, while existing CNN methods can lack domain-specific physical structure. The paper proposes DiCNN1 and DiCNN2 within the DiPAN detail-injection framework, directly learning MS details end-to-end. Experiments report excellent pansharpening performance, with DiCNN2 also supporting transfer learning when image types change.

  • Problem

    Pansharpening requires sharpening low-resolution MS images with PAN detail, but existing CNN approaches lack the physical interpretability and domain-specific structure of detail injection.

  • Method

    The paper develops DiCNN1 and DiCNN2 within DiPAN to learn MS details end-to-end from LRMS-plus-PAN inputs or PAN-only inputs, respectively.

  • Results

    The proposed methods achieve excellent pansharpening performance, while DiCNN1 and DiCNN2 obtain the highest reported Q8, SAM, ERGAS, and SCC scores in one reduced-resolution assessment.

  • Takeaways & Limitations

    DiCNNs provide explicit physical interpretations and fast convergence, and DiCNN2 additionally supports transfer learning when the MS or PAN image type changes.

Abstract

from arXiv · show

Pansharpening aims to fuse a multispectral (MS) image with an associated panchromatic (PAN) image, producing a composite image with the spectral resolution of the former and the spatial resolution of the latter. Traditional pansharpening methods can be ascribed to a unified detail injection context, which views the injected MS details as the integration of PAN details and band-wise injection gains. In this work, we design a detail injection based CNN (DiCNN) framework for pansharpening, with the MS details being directly formulated in end-to-end manners, where the first detail injection based CNN (DiCNN1) mines MS details through the PAN image and the MS image, and the second one (DiCNN2) utilizes only the PAN image. The main advantage of the proposed DiCNNs is that they provide explicit physical interpretations and can achieve fast convergence while achieving high pansharpening quality. Furthermore, the effectiveness of the proposed approaches is also analyzed from a relatively theoretical point of view. Our methods are evaluated via experiments on real-world MS image datasets, achieving excellent performance when compared to other state-of-the-art methods.

I. INTRODUCTION

Pansharpening fuses low-spatial-resolution multispectral imagery with registered PAN imagery to produce high-spatial-resolution MS imagery. Existing CS/MRA methods motivate a unified detail-injection framework, while CNN approaches seek learned alternatives with clearer physical structure and efficiency.

  • Motivation: Pansharpening combines a low-spatial-resolution MS image with a registered PAN image to sharpen spatial detail while retaining MS information.The task supports applications including classification, target detection, scene interpretation, and spectral unmixing.
  • Existing methods: CS methods substitute image components, whereas MRA methods extract and integrate spatial details across multiple scales.Representative examples include PCA, Brovey, and Gram–Schmidt transforms for CS, and wavelet or Laplacian-pyramid methods for MRA.
  • Detail injection: Traditional CS/MRA approaches differ mainly in how PAN details are extracted and injected into the pre-interpolated LRMS image.Their unified formulation represents injected MS details as PAN details combined with band-wise injection gains.
  • CNN limitations: CNN pansharpening methods learn mappings from image inputs to HRMS outputs, but existing approaches can require long training, lack physical interpretability, or add dimension-reduction computation.PNN stacks the pre-interpolated LRMS and PAN images, while another approach learns a residual whose pansharpening meaning is not explicit.
  • Proposed approach: The proposed DiPAN framework guides DiCNNs to learn MS details directly in end-to-end manners within a physically interpretable detail-injection structure.DiCNN1 uses pre-interpolated LRMS and PAN inputs, while DiCNN2 uses only PAN input under the assumption that ideal MS detail depends only on PAN imagery.

III. SUPER-RESOLUTION AND PANSHARPENING USING CNN STRATEGY

CNN-based pansharpening learns mappings from low-resolution MS and PAN inputs to high-resolution MS outputs, but existing approaches can lack domain-specific structure and physical interpretability. DiPAN addresses these weaknesses through detail-injection-oriented learning.

  • CNN strategy: CNN layers use local receptive fields, shared spatially invariant weights, convolutional parameters, biases, and activation functions to transform image inputs.ReLU is commonly used because it mitigates gradient vanishing and is computationally simple.
  • CNN strategy: Pansharpening is a multiple-input single-output resolution-enhancement task using low-resolution MS and PAN images to recover HRMS imagery.Super-resolution usually has one input, whereas pansharpening combines two distinct data sources.
  • CNN strategy: PNN stacks pre-interpolated LRMS and PAN images and learns their mapping to pansharpened HRMS images with a CNN.This follows the basic SRCNN strategy of directly learning a low-resolution-to-high-resolution mapping.
  • CNN strategy: The pansharpening objective minimizes the Frobenius-norm discrepancy between the predicted HRMS image and the ideal HRMS target over training examples.The formulation uses the concatenated LRMS-PAN input and matrix-unfolded image tensors.
  • Limitations of existing CNN methods: PNN treats pansharpening as a black-box procedure, leading to long training and limited learning ability, while DRPNN adds dimension-reduction computation without explicit physical meaning for its residual.These limitations motivate a domain-specific detail-injection structure.

IV. PROPOSED METHODS

The proposed DiCNNs incorporate traditional CS/MRA detail structure while directly learning MS details end to end. This design reduces model uncertainty and supplies physically meaningful detail-injection interpretations alongside strong learning performance.

  • Proposed methods: DiCNNs directly learn MS details without separately estimating PAN details and band-wise injection gains.Avoiding the intermediate estimation of these two information types reduces model uncertainty.
  • Proposed methods: The methods retain the detail structure of traditional CS/MRA pansharpening while using CNN-based learning.This combines domain-specific structure with end-to-end detail learning.
  • Proposed methods: DiCNNs provide clear physical interpretations in the detail-injection context while achieving excellent learning performance.The framework is presented as a general detail-injection pansharpening approach called DiPAN.

A. DiCNN1

DiCNN1 reconstructs MS details from jointly provided LRMS and PAN inputs, then injects those details into the pre-interpolated LRMS image through a shortcut-based network. Its design gives the learned pathway a direct detail-injection interpretation within an ill-posed pansharpening problem.

  • A. DiCNN1: The stacked convolutional pathway reconstructs MS details, and adding them to the shortcut output yields the predicted HRMS image.The reconstructed details are parameterized by the concatenated LRMS-PAN input and network parameters.
  • A. DiCNN1: Pansharpening is ill-posed because a given low-resolution input can correspond to multiple HRMS solutions.The framework therefore addresses an underdetermined inverse problem whose solution is not unique.
  • A. DiCNN1: DiCNN1 feeds concatenated pre-interpolated LRMS and PAN images into stacked layers while sending only LRMS through the shortcut connection.The stacked pathway therefore predicts details that supplement the LRMS shortcut to form HRMS output.
  • A. DiCNN1: DiCNN1 differs from PNN and DRPNN by explicitly aligning its stacked-layer output with the MS details required by the detail-injection framework.This design provides a direct physical interpretation for the learned residual-like pathway.

B. DiCNN2

DiCNN2 is designed to improve robustness when test MS images differ from training images by extracting learned details from PAN alone. It also supports transfer learning by fine-tuning only the final layer for new image types.

  • B. DiCNN2: DiCNN2 is motivated by transfer across different image types, including cases where test MS images contain bad bands.The paper frames this transfer as possible because pansharpening details mainly arise from PAN imagery and filtering rules are shared across image types.
  • B. DiCNN2: DiCNN2 removes LRMS from the convolutional detail-extraction pathway and connects only the PAN image to it.This avoids the strong influence of changing MS-image types on detail extraction in the stacked layers.
  • B. DiCNN2: For a new kind of image, only the final layer needs fine-tuning after the convolutional pathway parameters are fixed.This enables pre-training transfer while preserving the learned convolutional detail-extraction pathway.
  • B. DiCNN2: DiCNN2 can serve both pre-training transfer and usual same-sensor pansharpening tasks.The paper attributes to it meaningful detail-injection interpretation and high computational efficiency alongside DiCNN1-like benefits.

C. Analysis of Effectiveness

The analysis argues that DiCNNs gain an optimization advantage by explicitly learning multispectral details within a detail-injection structure. Their initialization and training losses are lower than those of PNN and DRPNN, supporting faster and more effective optimization.

  • Network structure: In DiCNN1 and DiCNN2, the stacked convolutional pathway outputs MS details, whereas PNN outputs the pansharpened HRMS image and DRPNN outputs residuals.This difference gives the DiCNN outputs a direct role in detail injection.
  • Initialization and optimization: The trace inequality used in the analysis is verified on three datasets, supporting the theoretical comparison of initialization losses.The reported trace values verify the inequality in the derivation.
  • Initialization and optimization: DiCNN1 has lower initial loss than PNN, and both DiCNN1 and DiCNN2 have lower initial losses than PNN and DRPNN during gradient descent.The paper interprets these lower initial losses as better initialization.
  • Initialization and optimization: PNN remains above DiCNN1 and DiCNN2 in training loss, while DRPNN’s faster loss decrease does not overcome its inferior initialization.The analysis concludes that gradient-based iteration does not compensate for inappropriate initialization in these comparisons.

V. EXPERIMENTAL RESULTS

The experiments evaluate the proposed pansharpening methods on three real-world sensor datasets using reduced-resolution, full-resolution, and transfer-learning settings. Reduced-resolution tests follow Wald’s protocol, with consistent CNN architecture and training settings for fair comparison.

  • Experimental setup: The evaluation uses WorldView-2, IKONOS, and Quickbird datasets across reduced-resolution, full-resolution, and transfer-learning experiments.These datasets represent real remotely sensed imagery acquired by three sensors.
  • Experimental setup: Reduced-resolution assessments follow Wald’s protocol by Gaussian-degrading MS and PAN images by a factor of 4 before pre-interpolating the MS image.The pre-interpolation uses a polynomial EXP kernel.
  • Experimental setup: The assessment criteria include Qx, SCC, SAM, and ERGAS for evaluating pansharpening quality.The passage introduces these criteria as the evaluation measures.
  • Experimental setup: All compared CNNs use three convolutional layers in the convolution pathway and 3.0 × 10^5 training iterations.This controls for hidden-layer depth when comparing the basic network structures.
  • Compared methods: The comparison includes DiCNN1, DiCNN2, PNN, DRPNN, and representative CS/MRA methods such as GSA, PRACS, ATWT, BDSD, and GLP-CBD.The methods are evaluated under consistent experimental conditions.

A. Experiment 1: WorldView-2 Washington Dataset

On the WorldView-2 Washington dataset, DiCNN1 and DiCNN2 achieve the strongest reported reduced-resolution quality, while DiCNN2 is the fastest CNN method. Visual results also favor CNN-based outputs, with DiCNN1 showing stronger spectral preservation in small objects.

  • Reduced-resolution assessment: DiCNN1 and DiCNN2 achieve the highest Q8, SAM, ERGAS, and SCC scores among all compared methods on the reduced-resolution WorldView-2 assessment.DiCNN2 is also the fastest among the CNN-based methods.
  • Reduced-resolution assessment: CNN-based methods produce images more similar to ground truth and without noticeable artifacts or spectral distortions than the CS/MRA methods.The corresponding detail images support this visual comparison.
  • Reduced-resolution assessment: DiCNN1 impedes spectral distortion more efficiently when preserving small ground objects than the other CNN-based methods.The passage identifies this difference in the bottom-left portion of the WorldView-2 results.
  • Full-resolution assessment: In full-resolution results, CNN-based methods produce sharper outputs than the other tested methods, especially in vegetation areas.DiCNN1, PNN, and DiCNN2 slightly outperform DRPNN in artifact reduction.

B. Experiment 2: IKONOS Hobart Dataset

On the IKONOS Hobart dataset, DiCNN1 obtains the strongest reported quality-index results, while DiCNN2 is the fastest CNN-based method. Visual comparisons likewise show CNN-based methods outperforming CS/MRA methods, with DiCNN outputs closest to ground truth spectrally.

  • Dataset: The IKONOS MS image has four bands, the PAN channel spans 450nm to 900nm, and their spatial resolutions are 4m and 1m, respectively.The dataset uses 11-bit radiometric resolution and 256 × 256 pixel areas for experiments.
  • Reduced-resolution assessment: DiCNN1 achieves the highest Q4, SAM, ERGAS, and SCC scores on the reduced-resolution IKONOS Hobart dataset.PNN is the most time-consuming method, while DiCNN2 has the least computational time among CNN-based methods.
  • Visual assessment: CNN-based methods show better visual pansharpening results than CS/MRA methods, particularly around roof edges.DiCNN1 and DiCNN2 look most similar to ground truth in spectral fidelity over vegetation.

C. Experiment 3: Quickbird Sundarbans Dataset

On the QuickBird Sundarbans dataset, CNN-based methods outperform CS- and MRA-based methods overall. DiCNN1 leads several quality indexes, while DiCNN2 offers the lowest CNN training time but trails DRPNN on quality.

  • CNN-based methods achieve better pansharpening quality than CS-based and MRA-based methods on the Chilika Lake dataset.
  • DiCNN1 overpasses other methods in Q4, SAM, ERGAS and SCC scores.
  • DiCNN2 has the shortest training time among CNN-based methods but lags behind DRPNN.
  • In reduced-resolution results, DiCNN2 exhibits fewer ringing artifacts than DiCNN1 and DRPNN, especially around lake edges.PNN shows this phenomenon more frequently.
  • In full-resolution results, DiCNN2 introduces more spatial blurring than PNN, while DiCNN1 exhibits fewer artifacts than DRPNN and PNN.

D. Experiment 4: Transfer Learning

The transfer-learning experiments test DiCNN2 when MS bands are removed, using fine-tuning to accommodate altered band counts. DiCNN2 achieves the strongest reported results on WorldView-2 and outperforms other CNN methods on most IKONOS quality indexes with substantially lower training time than DiCNN1.

  • DiCNN2 is pretrained on the original dataset and fine-tuned after MS bands are removed to accommodate the test image’s band count.WorldView-2 removes four of eight bands, while IKONOS removes one of four.
  • DiCNN2 yields the best scores in all evaluation metrics on the WorldView-2 Washington dataset.
  • DiCNN2’s WorldView-2 training time is less than half that of the longest method because only its final convolutional layer is fine-tuned.
  • On the three-band IKONOS experiment, DiCNN2 outperforms other CNN-based methods in most quality indexes.
  • Although DiCNN1 attains comparative results with DiCNN2, DiCNN2 requires far less training time.

VI. CONCLUSIONS AND FUTURE LINES

The paper concludes that DiCNN1 and DiCNN2 learn multispectral details end-to-end within the DiPAN detail-injection framework. The methods provide explicit physical meaning, faster convergence, strong pansharpening performance, and transfer learning for DiCNN2 when image types change.

  • DiCNN1 and DiCNN2 learn MS details end-to-end within the DiPAN detail injection framework.
  • The methods have explicit physical meaning and avoid separately handling injection gains and PAN details as in traditional CS and MRA methods.
  • Low initial loss tends to yield faster convergence, while both methods exhibit excellent pansharpening performance.
  • DiCNN2 can realize transfer learning when the MS image type or PAN image type changes.
  • Future work will explore pansharpening CNNs with more hidden layers and more complex interconnections among convolution layers.
Loading 1806.08898v1…