Source-linked AI summary
Pansharpening for Thin-Cloud Contaminated Remote Sensing Images: A Unified Framework and Benchmark Dataset
Songcheng Du, Yang Zou, Jiaxin Li, Mingxuan Liu, Ying Li, Changjing Shang, Qiang Shen
TL;DR
Thin-cloud pansharpening must address spatial-resolution degradation and cloud-induced spectral distortion, but existing approaches often separate cloud removal from fusion. Pan-TCR jointly restores both degradations using frequency-domain prompts and introduces PanTCR-GF2 for realistic evaluation, with experiments reporting superior and robust performance.
Problem
Thin-cloud contamination jointly degrades spatial resolution and spectral fidelity, while existing methods often use sequential cloud removal and pansharpening without joint degradation modeling.
Method
Pan-TCR is an end-to-end framework using NIR amplitude and PAN phase prompts for component-wise frequency restoration, with IFC enforcing inter-frequency consistency.
Results
Pan-TCR achieves state-of-the-art performance across multiple benchmarks while using fewer parameters, FLOPs, and inference time.
Takeaways & Limitations
Pan-TCR provides a unified and computationally efficient solution for thin-cloud pansharpening and PanTCR-GF2 enables benchmarking under realistic atmospheric interference.
Takeaways & Limitations
Frequency-based restoration and low-rank channel gating inherently limit context-aware spectral refinement and fine-grained inter-band modeling.
Abstract
from arXiv · showhide
Pansharpening under thin cloudy conditions is a practically significant yet rarely addressed task, challenged by simultaneous spatial resolution degradation and cloud-induced spectral distortions. Existing methods often address cloud removal and pansharpening sequentially, leading to cumulative errors and suboptimal performance due to the lack of joint degradation modeling. To address these challenges, we propose a Unified Pansharpening Model with Thin Cloud Removal (Pan-TCR), an end-to-end framework that integrates physical priors. Motivated by theoretical analysis in the frequency domain, we design a frequency-decoupled restoration (FDR) block that disentangles the restoration of multispectral image (MSI) features into amplitude and phase components, each guided by complementary degradation-robust prompts: the near-infrared (NIR) band amplitude for cloud-resilient restoration, and the panchromatic (PAN) phase for high-resolution structural enhancement. To ensure coherence between the two components, we further introduce an interactive inter-frequency consistency (IFC) module, enabling cross-modal refinement that enforces consistency and robustness across frequency cues. Furthermore, we introduce the first real-world thin-cloud contaminated pansharpening dataset (PanTCR-GF2), comprising paired clean and cloudy PAN-MSI images, to enable robust benchmarking under realistic conditions. Extensive experiments on real-world and synthetic datasets demonstrate the superiority and robustness of Pan-TCR, establishing a new benchmark for pansharpening under realistic atmospheric degradations.
Introduction
Thin clouds jointly degrade spatial textures and spectral reflectance, while existing sequential pipelines accumulate errors and computational costs. Pan-TCR unifies cloud removal and pansharpening through frequency-guided restoration and introduces PanTCR-GF2 for realistic benchmarking.
- Motivation: Cloud contamination degrades spatial textures and distorts spectral reflectance non-uniformly in optical satellite imagery.
- Motivation: Existing methods commonly assume clear skies or sequentially perform cloud removal and pansharpening, limiting reliability under thin-cloud conditions.
- Challenges: Cascaded pipelines propagate cloud-removal artifacts and increase inference time and memory consumption.
- Proposed framework: Pan-TCR jointly models resolution degradation and thin-cloud interference in an end-to-end frequency-prompt formulation.
- Proposed framework: NIR amplitude guides cloud-resilient restoration, while PAN phase guides structure-preserving enhancement.
- Proposed framework: The FDR block restores amplitude and phase components, while IFC performs interactive cross-modal refinement for inter-frequency consistency.
- Dataset and contributions: PanTCR-GF2 contains 15,603 aligned cloudy/clean PAN-MSI pairs across five land-cover categories for realistic benchmarking.
- Dataset and contributions: Pan-TCR reports state-of-the-art performance across multiple benchmarks with lower model size, FLOPs, and inference time.
Related Work
Pansharpening research has progressed from traditional fusion methods to deep learning, while thin-cloud removal has developed through physical priors and learning-based frequency or attention mechanisms. Thin clouds remain partially transmissive, making restoration possible but challenging for complex patterns.
- Pansharpening: Traditional pansharpening methods include component substitution, multi-resolution analysis, and variational optimization, but can produce spectral distortions.
- Pansharpening: Deep CNN-based methods such as PNN, PanNet, and BDPN improved fusion performance through stronger learned representations.
- Thin-cloud removal: Thin clouds permit partial signal transmission, unlike thick clouds that completely obscure the surface, enabling algorithmic restoration.
- Thin-cloud removal: Thin-cloud removal methods use spectral correlation, dark-channel, frequency-filtering, residual-learning, attention, wavelet, and Fourier-based priors.
Proposed Method
Pan-TCR jointly performs thin-cloud removal and pansharpening through frequency-decoupled restoration, spectral refinement, and cross-frequency consistency. It uses PAN and NIR cues to reconstruct cloud-free high-resolution MSI and introduces PanTCR-GF2 for realistic evaluation.
- Framework Architecture: Pan-TCR takes upsampled LR-MSI and PAN inputs and progressively reconstructs cloud-free HR-MSI through a three-stage encoder–decoder pipeline.Each stage integrates frequency-decoupled restoration and spectral enhancement modules.
- Frequency-Decoupled Restoration: The FDR block parallelizes amplitude and phase restoration to address cloud-induced amplitude degradation and resolution-related phase degradation.NIR amplitude guides cloud-resilient restoration, while PAN-derived phase guides structural enhancement.
- Frequency-Decoupled Restoration: The MAFG module suppresses modality-inconsistent amplitude features through channel-wise spectral gating before amplitude refinement.It uses concatenated NIR and input amplitude features to generate adaptive gating weights.
- Interactive Inter-Frequency Consistency: The IFC module uses bidirectional cross-modulation so amplitude and phase branches guide each other, producing inter-frequency-consistent representations before IFFT reconstruction.Cross-frequency residuals are added to the respective components, and the refined components are combined into a spatially consistent cloud-free representation.
- Spectral Enhancement: A lightweight spectral enhancement module complements frequency restoration by modeling long-range spectral dependencies and recalibrating spectral responses in the image domain.This addresses the limited fine-grained spectral and semantic modeling capacity of frequency-based processing.
Experiment
Experiments evaluate Pan-TCR against deep learning and traditional methods on real and synthetic data under two-stage and end-to-end settings. Pan-TCR consistently performs best across major metrics, while ablations support its degradation-aware, frequency-domain, spectral, and unified design.
- Comparison with State-of-the-Art Methods: Pan-TCR is evaluated on real PanTCR-GF2 and synthetic WV3 data under reduced- and full-resolution settings.The comparison includes two-stage pipelines and end-to-end processing, with reference-based and no-reference metrics.
- Comparison with State-of-the-Art Methods: Pan-TCR consistently achieves superior results across major metrics in both reduced- and full-resolution settings.WFANet and ARConv perform well on structure-focused end-to-end metrics, whereas methods without explicit cloud removal are weaker on spectral and quality-aware metrics.
- Comparison with State-of-the-Art Methods: Figure 5 residual maps show Pan-TCR producing the darkest residuals, indicating superior fidelity over the baselines.The qualitative comparisons include restored images and residual error maps.
- Ablation Experiments: Removing degradation-aware modules or either NIR or PAN prompt causes performance drops, especially in SAM and ERGAS.The ablations support complementary roles for NIR’s cloud robustness and PAN’s spatial richness.
- Ablation Experiments: Removing frequency-specific branches harms either structural fidelity or spectral consistency and cloud suppression, confirming the need for decoupled processing.Removing the phase branch decreases PSNR and SSIM, while removing the amplitude branch increases SAM and BRISQUE.
- Ablation Experiments: Replacing or removing FDR, IFC, MAFG, or the parallel branch design degrades fusion quality, frequency alignment, or efficiency.The results favor frequency-based modeling, interactive inter-frequency consistency, modality-adaptive gating, and parallel processing.
- Ablation Experiments: Removing or replacing spectral enhancement degrades performance, particularly on SAM and ERGAS, while localized spectral correction improves spectral coherence.The spectral enhancement module complements frequency restoration through a spectral-wise transformer.
- Ablation Experiments: Converting the unified model into a cascaded two-stage variant decreases performance, especially on cloud-relevant and perceptual metrics.This result supports joint modeling of thin-cloud removal and pansharpening despite the cascaded variant’s increased complexity.
Conclusion
Pan-TCR is presented as a unified, lightweight end-to-end framework for simultaneous thin-cloud removal and pansharpening. Its frequency-domain restoration and PanTCR-GF2 benchmark are positioned as a practical basis for accurate evaluation under realistic atmospheric degradation.
- Conclusion: Pan-TCR simultaneously addresses thin-cloud removal and pansharpening within a single end-to-end framework.The framework targets cloud contamination and spatial-resolution limitations jointly.
- Conclusion: Frequency-domain priors from PAN and NIR decouple amplitude and phase restoration for cloud suppression and structural detail enhancement.The approach is described as computationally efficient relative to conventional two-stage pipelines.
- Conclusion: PanTCR-GF2 is introduced as the first real-world dataset tailored to thin-cloud pansharpening.The dataset addresses an underexplored benchmark setting involving cloud-related degradations.
- Conclusion: Experiments demonstrate superior reconstruction quality and efficiency for the unified approach over existing algorithms.The reported evaluation covers both accuracy and efficiency.