Source-linked AI summary
Wavelet-Assisted Multi-Frequency Attention Network for Pansharpening
Jie Huang, Rui Huang, Jinghao Xu, Siran Pen, Yule Duan, Liangjian Deng
TL;DR
Pansharpening must fuse LRMS and PAN imagery into HRMS imagery while better preserving frequency information than many existing approaches. The paper proposes WFANet, whose MFFA uses wavelets and physically meaningful attention roles within a progressive multi-scale framework. Experiments report state-of-the-art performance on three diverse pansharpening datasets.
Problem
Existing pansharpening methods often operate spatially or do not fully exploit frequency-domain processing, limiting precise preservation and fusion of frequency features.
Method
WFANet uses wavelet-based MFFA with Frequency-Query, Spatial-Key, and Fusion-Value, frequency-preserving operations, and progressive multi-scale fusion.
Results
WFANet achieves state-of-the-art performance on three diverse pansharpening datasets with quantitative and qualitative support.
Takeaways & Limitations
Wavelet-separated, physically guided attention and multi-scale fusion provide the paper's supported route to more precise frequency-domain pansharpening.
Abstract
from arXiv · showhide
Pansharpening aims to combine a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to produce a high-resolution multispectral (HRMS) image. Although pansharpening in the frequency domain offers clear advantages, most existing methods either continue to operate solely in the spatial domain or fail to fully exploit the benefits of the frequency domain. To address this issue, we innovatively propose Multi-Frequency Fusion Attention (MFFA), which leverages wavelet transforms to cleanly separate frequencies and enable lossless reconstruction across different frequency domains. Then, we generate Frequency-Query, Spatial-Key, and Fusion-Value based on the physical meanings represented by different features, which enables a more effective capture of specific information in the frequency domain. Additionally, we focus on the preservation of frequency features across different operations. On a broader level, our network employs a wavelet pyramid to progressively fuse information across multiple scales. Compared to previous frequency domain approaches, our network better prevents confusion and loss of different frequency features during the fusion process. Quantitative and qualitative experiments on multiple datasets demonstrate that our method outperforms existing approaches and shows significant generalization capabilities for real-world scenarios.
Introduction
Pansharpening fuses LRMS and PAN images to produce HRMS imagery, but spatial-domain methods can blur details and existing frequency-domain methods may incompletely preserve frequency information. WFANet addresses this with wavelet-based frequency separation, physically meaningful attention, frequency-preserving operations, and progressive multi-scale fusion.
- Motivation: Pansharpening combines LRMS and PAN images to produce HRMS images with enhanced spatial and spectral resolution.HRMS imagery supports applications including environmental monitoring and urban planning.
- Motivation: Spatial-domain fusion can lose or blur details because frequency information is separated imprecisely.Frequency-based processing instead targets separate frequencies and helps preserve high-frequency information while reducing interference.
- Prior approaches: Existing frequency-domain approaches include high-frequency injection and expert-mixture fusion, but their designs can lose low-frequency information or incompletely address frequency integration.AFM-DIN may lose low-frequency information, while FAMENet balances high- and low-frequency information through an expert mixture model.
- Proposed approach: WFANet uses wavelet transforms and MFFA to separate frequencies, assign physically meaningful attention roles, and reconstruct fused features without loss.Frequency-Query represents frequency features, Spatial-Key encodes spatial information, and Fusion-Value represents preliminary spatial-spectral fusion.
- Proposed approach: The network preserves frequency features across operations and progressively fuses information through a multi-scale wavelet pyramid.WFANet combines MFFA with SDEM in a multi-scale workflow for frequency-feature processing and spatial-detail enhancement.
- Results: WFANet achieves state-of-the-art performance on three diverse pansharpening datasets, supported by quantitative and qualitative evidence.The paper presents these experiments as validating the proposed frequency-fusion strategies.
Proposed Method
WFANet combines Multi-Frequency Fusion Attention (MFFA), Spatial Detail Enhancement Modules (SDEM), and a wavelet pyramid to fuse and reconstruct pansharpening features across frequencies and scales. MFFA generates frequency-aware attention components, while SDEM extracts spatial detail separately before lossless reconstruction and progressive multi-scale fusion.
- Multi-Frequency Fusion Attention (MFFA): MFFA consists of Frequency Attention Triplet Generation and Attention-Driven Frequency Reconstruction phases for cross-frequency information fusion.The first phase generates the triplet, and the second reconstructs features at different frequencies.
- Multi-Frequency Fusion Attention (MFFA): DWT separates PAN features into low- and high-frequency components, which provide the frequency-aware representations used by MFFA.The low-frequency component represents overall spatial appearance, while high-frequency components capture edge and texture details.
- Multi-Frequency Fusion Attention (MFFA): The Frequency Attention Triplet uses frequency features as Query, low-frequency spatial features as Key, and fused spectral-spatial information as Value.Frequency attention maps are computed from Query-Key correlations and applied to the Fusion-Value before MLP and residual processing.
- Spatial Detail Enhancement Module (SDEM): SDEM extracts spatial information independently from four frequency features using Frequency Adaptation Blocks and reconstructs complete spatial details with lossless IDWT.Each Frequency Adaptation Block consists of a linear layer and sigmoid activation; the design favors linear layers over convolution blocks for frequency adaptation.
- Network Framework and Loss: A wavelet pyramid organizes the network into multiple scales, progressively fusing MFFA and SDEM outputs from the smallest scale toward the final HRMS reconstruction.The paper uses two scales, with each layer's output becoming the input to the next scale before final convolution.
- Network Framework and Loss: Training uses an ℓ1 loss between predicted and ground-truth images over K training samples.The loss uses the ℓ1 norm of the difference between each reconstructed image and its corresponding ground truth.
Experiments
Experiments evaluate WFANet on reduced- and full-resolution datasets using standard pansharpening metrics, comparisons with state-of-the-art methods, and ablations of its components and strategies. The method achieves leading results across datasets, with ablations supporting the contributions of its frequency-attention, spatial-detail, multi-scale, and frequency-adaptation designs.
- Evaluation setup: WFANet is evaluated on WV3, GF2, and QB datasets using reduced- and full-resolution protocols and standard pansharpening metrics.Reduced-resolution evaluation uses SAM, ERGAS, and Q4/Q8; full-resolution evaluation uses HQNR, Ds, and Dλ.
- State-of-the-art comparison: WFANet achieves the best results across all reduced-resolution metrics, improving PSNR over the second-best method by 0.228dB on WV3, 0.334dB on QB, and 0.417dB on GF2.Residual maps are also reported as darkest for the proposed method on WV3 and GF2 visual comparisons.
- State-of-the-art comparison: WFANet achieves the best HQNR on WV3 full-resolution samples, reflecting its reported ability to balance spectral and spatial distortions.The full-resolution experiments are presented as evidence of generalization ability.
- Ablation studies: Ablation experiments on WV3 evaluate the Frequency Attention Triplet and key WFANet components and strategies.The triplet ablations separately examine Frequency-Query, Spatial-Key, and Fusion-Value, while further tests examine MFFA, SDEM, multi-scale training, and frequency-adaptation blocks.
- Ablation studies: Removing or replacing the triplet, MFFA, SDEM, multi-scale network, or frequency-adaptation blocks produces inferior reported outcomes, supporting each design choice.The reported ablations associate the triplet with component effectiveness, MFFA with attention-based fusion, SDEM with spatial-detail enhancement, multi-scale training with significance, and FABs with superior cross-frequency feature extraction.
Conclusion
The paper proposes MFFA for precise frequency-domain fusion and combines it with frequency-adapted operations and a multiscale fusion strategy. Extensive experiments on three satellite datasets show that the model outperforms state-of-the-art methods.
- Conclusion: MFFA uses frequency decomposition and reconstruction with Frequency-Query, Spatial-Key, and Fusion-Value to improve frequency-domain fusion precision.The triplet is designed with distinct physical meanings for the represented features.
- Conclusion: The approach also adapts operations to the frequency domain and applies a comprehensive multiscale fusion strategy.Ablation experiments are reported as confirming the effectiveness of the overall approach.
- Conclusion: Experiments on three satellite datasets demonstrate that the model outperforms state-of-the-art methods.
Supplemental Material
The supplementary materials extend the paper with methodological explanations, additional experiments, and more detailed ablation information.
- Supplementary methodology: The supplement explains the discrete wavelet transform and the multi-scale strategy used by the method.
- Additional experiments: Additional material compares parameter numbers and further validates the Frequency Attention Triplet.
- Ablation details: The supplement provides broader ablation settings and additional discussions of the proposed method.
Method Supplementary
The method uses DWT to separate image information into low- and high-frequency components, reconstructs images with IDWT, and builds a wavelet pyramid by recursively decomposing low-frequency features. Multi-scale fusion proceeds from the smallest scale toward the LRMS scale.
- Wavelet decomposition: DWT decomposes an image into a low-frequency LL component and horizontal, vertical, and diagonal high-frequency components.The high-frequency components are denoted LH, HL, and HH, respectively.
- Wavelet reconstruction: IDWT combines LL, LH, HL, and HH to reconstruct the original image accurately and losslessly.
- Wavelet pyramid: Applying DWT recursively to the LL component constructs a wavelet pyramid with progressively smaller scales.
- Multi-scale fusion: The multi-scale network repeatedly applies DWT to PAN features until their scale matches the LRMS scale, then begins fusion at the smallest scale.The architecture uses two scales in this paper, and adjacent scales differ by a factor of 2.
Experimental Supplementary
The experiments examine WFANet’s parameter–performance trade-off, Frequency Attention Triplet variants, ablations, and qualitative results across multiple datasets. WFANet and its lightweight version achieve strong performance with relatively low complexity, while the full method shows the strongest qualitative results among compared approaches.
- Parameter–Performance Trade-off: WFANet and WFANet-L achieve strong PSNR performance while maintaining relatively low parameter counts.WFANet is heavyweight, whereas WFANet-L reduces the common channel size from 32 to 24 and simplifies several MLP layers.
- Frequency Attention Triplet Validation: The Frequency Attention Triplet is evaluated through six configurations that permute Frequency-Query, Spatial-Key, and Fusion-Value as Query, Key, and Value.The original configuration is labeled Ours, while five alternative permutations are labeled V1 through V5.
- Ablation Experiments: The ablation experiments separately replace the original Query, Key, or Value component while keeping the other two components unchanged.The ablations correspond to separate settings for the three components, and the MFFA ablation concatenates convolved M with features from different frequency domains before convolutional feature extraction.
- Qualitative Results: The qualitative comparisons cover QB reduced-resolution, WV3 full-resolution, and GF2 full-resolution datasets using visual results, residuals, or HQNR maps.The visualizations include residual maps for QB and HQNR maps for WV3 and GF2.
- Qualitative Results: Ours shows the largest and deepest red areas in the comparison maps, indicating the best performance among the compared methods.The maps use redder areas for better performance and bluer areas for poorer performance.