PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening

Wele Gedara Chaminda Bandara, Vishal M. Patel

arXiv:2203.02503v3cs.CVeess.IV

TL;DR

HS pansharpening must add PAN spatial detail to LR-HSI while preserving spectral information, yet existing fusion methods can cause distortions. HyperTransformer uses multi-head attention to transfer relevant PAN texture through cross-feature dependencies and multi-scale fusion. Experiments on three widely used datasets report significant improvements over classical and ConvNet-based state-of-the-art methods on spatial and spectral measures.

  • Problem

    Existing HS pansharpening methods use limited or straightforward PAN–LR-HSI fusion, which can produce spatial and spectral distortions.

  • Method

    HyperTransformer uses separate PAN and LR-HSI feature extractors, multi-head feature soft attention, textural-spectral fusion, and multi-scale backbone integration.

  • Results

    HyperTransformer significantly outperforms classical and ConvNet-based state-of-the-art pansharpening methods on quantitative evaluations.

  • Takeaways & Limitations

    The approach provides a transformer-based strategy for transferring PAN texture while learning cross-feature dependencies between PAN and LR-HSI representations.

  • Takeaways & Limitations

    Relatively high MAE remains in UV bands 1–10 and IR bands 90–104, possibly because PAN lacks corresponding UV and IR features.

Abstract

from arXiv · show

Pansharpening aims to fuse a registered high-resolution panchromatic image (PAN) with a low-resolution hyperspectral image (LR-HSI) to generate an enhanced HSI with high spectral and spatial resolution. Existing pansharpening approaches neglect using an attention mechanism to transfer HR texture features from PAN to LR-HSI features, resulting in spatial and spectral distortions. In this paper, we present a novel attention mechanism for pansharpening called HyperTransformer, in which features of LR-HSI and PAN are formulated as queries and keys in a transformer, respectively. HyperTransformer consists of three main modules, namely two separate feature extractors for PAN and HSI, a multi-head feature soft attention module, and a spatial-spectral feature fusion module. Such a network improves both spatial and spectral quality measures of the pansharpened HSI by learning cross-feature space dependencies and long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be utilized across multiple spatial scales at the backbone for obtaining improved performance. Extensive experiments conducted on three widely used datasets demonstrate that HyperTransformer achieves significant improvement over the state-of-the-art methods on both spatial and spectral quality measures. Implementation code and pre-trained weights can be accessed at https://github.com/wgcban/HyperTransformer.

1. Introduction

HS pansharpening transfers PAN spatial details to LR-HSI while preserving spectral characteristics, but existing approaches can distort both. HyperTransformer addresses this with attention-based cross-feature fusion and multi-scale processing.

  • 1. Introduction: HS pansharpening supports remote-sensing tasks including unmixing, change detection, object recognition, scene interpretation, and classification.It is used as preprocessing to improve spatial and spectral detail for identifying phenomena that are difficult to see in LR-HSIs.
  • 1. Introduction: Traditional methods use CS, MRA, Bayesian, and variational formulations, but suffer from mismatched assumptions, inaccessible sensor characteristics, and limited hand-crafted representations.These limitations include linear spectral-mixture and sparsity assumptions that can produce spatial and spectral distortions.
  • 1. Introduction: HyperTransformer addresses spatial and spectral distortions by transferring PAN texture to LR-HSI through cross-feature attention rather than straightforward concatenation.The method extracts cross-feature dependencies and selects texturally advanced, spectrally similar features before fusion.
  • 1. Introduction: HyperTransformer combines separate PAN and LR-HSI feature extractors, multi-head feature soft attention, and textural-spectral feature fusion.The architecture learns dependencies between PAN and LR-HSI features before effectively fusing texture and spectral information.
  • 1. Introduction: The network applies HyperTransformers at different backbone scales to capture multi-scale long-range PAN details and cross-feature dependencies.The proposed strategy injects textural information across multiple spatial scales rather than only at the high-resolution scale.

2. Related Work

Classical pansharpening methods comprise component substitution, multi-resolution analysis, hybrid, and Bayesian approaches, each using a distinct fusion strategy. Representative methods include GS, GSA, PCA, SFIM, MTF-GLP, MGH, and GFPCA.

  • Classical approaches: Classical pansharpening approaches are categorized into component substitution, multi-resolution analysis, hybrid, and Bayesian methods.These categories organize prior fusion strategies by how spatial and spectral information are combined.
  • Component substitution: Component substitution methods decompose LR-HSI into spectral and spatial components, replace the spatial component with PAN, and transform the result back.Gram–Schmidt, GS-adaptive, and PCA are representative component substitution methods.
  • Multi-resolution analysis and hybrid methods: Multi-resolution analysis methods inject PAN-derived spatial features into LR-HSI using spatial filtering, with SFIM, MTF-GLP, and MGH as examples.Hybrid methods such as GFPCA were proposed considering limitations of component substitution and multi-resolution analysis, while Bayesian methods formulate fusion as Bayesian inference.

3. Methodology

HyperTransformer transfers PAN texture to LR-HSI features through attention-based cross-feature modeling, then fuses the transferred representation with backbone features. The design uses separate PAN and HSI extractors, multi-head soft attention, textural-spectral fusion, and injection at three spatial scales.

  • HyperTransformer Overview: HyperTransformer uses separate PAN and HSI feature extractors, a multi-head feature soft-attention mechanism, and a textural-spectral fusion module.The extractors produce the attention mechanism’s query, key, and value features before fusion.
  • Feature Extraction: LR-HSI features form queries, while downsampled-upsampled PAN features form keys and PAN features form values for attention-based texture transfer.The transformed PAN representation is made domain-consistent with upsampled LR-HSI inputs before attention.
  • Multi-Head Feature Soft-Attention: Multi-head attention creates global descriptors, computes query-key cross-correlations, normalizes them with Softmax, and applies the resulting attention to PAN value descriptors.This process extracts cross-feature dependencies between LR-HSI and PAN features and captures long-range PAN details.
  • Textural-Spectral Feature Fusion: The transferred representation T is concatenated with backbone spectral features F, then a 3 × 3 convolution and Batch Normalization generate the fused output.The resulting representation is used by the backbone to generate the pansharpened HSI.
  • Multi-Scale Feature Fusion: HyperTransformer injects textural details at ×1 ↑, ×2 ↑, and ×4 ↑ spatial scales to capture multi-scale long-range details and cross-feature dependencies.The three scales correspond to the LR-HSI, intermediate two-times-upsampled, and desired HR spatial resolutions.

4. Experiments

Experiments on three HSI datasets compare HyperTransformer with classical and ConvNet-based methods and evaluate its components, scales, and losses.

  • Experimental Setup: The evaluation uses Pavia Center, Botswana, and Chikusei datasets with CC, SAM, RMSE, RSNR, ERGAS, and PSNR as spatial and spectral quality measures.Reference HSIs are converted into PAN and LR-HSI inputs using Wald’s protocol.
  • Results and Discussion: Approximately 0.9/26.9/32.6/29.4/13.3%, 0.3/19.2/11.9/14.0/3.2%, and 0.6/12.7/13.6/13.8/4.1% improvements in CC/SAM/RMSE/ERGAS/PSNR are reported for Pavia Center, Botswana, and Chikusei, respectively.HyperTransformer significantly outperforms classical and ConvNet-based state-of-the-art methods on all three datasets.
  • Ablation Studies: Using the feature-attention mechanism improves Pavia Center CC/SAM/RMSE/ERGAS/PSNR over the baseline by approximately 1.5/21/35/29/13% when N = 16.The ablation also reports reduced MAE across spectral bands, especially in the infrared region.
  • Results and Discussion: Table 1 reports average quantitative results, while Table 2 presents visual comparisons and normalized spectral-band MAE across competing algorithms and ground truth.The experiments compare classical methods and six ConvNet-based state-of-the-art approaches.
  • Ablation Studies: Using HyperTransformers at all three scales improves CC/SAM/RMSE/ERGAS/PSNR by approximately 0.3/3.7/11.2/9.0/2.1% over using them only at the ×4 scale.The evaluated scales are ×1, ×2, and ×4 relative to LR-HSI resolution; the best performance occurs when all three are used.

5. Limitations and Future Work

The reported limitation is elevated error in the ultraviolet and infrared spectral regions, attributed to missing corresponding PAN features.

  • Limitations and Future Work: Additional research is needed because MAE remains relatively high around UV bands 1–10 and IR bands 90–104.The paper suggests this may result from a lack of UV and IR features in the PAN image.

6. Conclusion

The conclusion presents HyperTransformer as a textural-spectral fusion network that transfers PAN texture to LR-HSI through attention and performs strongly across three datasets.

  • Conclusion: HyperTransformer transfers high-resolution PAN texture to LR-HSI spectral features through multi-head feature soft attention.Its components include separate PAN and LR-HSI feature extractors, attention for long-range cross-feature dependencies, and a textural-spectral fusion module.
  • Conclusion: Extensive experiments on three widely used HSI datasets demonstrate superiority over state-of-the-art methods in quantitative and qualitative evaluations.The network can also operate at multiple spatial scales to learn stronger texture representations.

Supplementary Material for HyperTransformer: A Textural and Spectral

Supplementary analyses explain HyperTransformer’s attention design, long-range context, and multi-scale feature behavior in pansharpening.

  • Attention Mechanism: Unlike prior attention methods that re-weight PAN and LR-HSI maps, HyperTransformer computes cross-correlation and uses LR-HSI, PAN↓↑, and PAN as queries, keys, and values.Multi-head soft attention identifies texturally advanced and spectrally similar PAN representations before fusion.
  • Feature Visualization: At each spatial scale, HyperTransformer adds missing texture details to LR-HSI queries while maintaining their spectral characteristics through cross-correlation.The visualization compares input and output feature maps across scales.
  • Long-Range Details: A larger receptive field lets the network enhance a pixel using texture and spectral details from both adjacent and distant pixels.Adding HyperTransformer is associated with a significant reduction in MAE across spectral bands.
Loading 2203.02503v3…