PaperlayerView exact revision on arXiv

Source-linked AI summary

GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion

Fang Xu, Yilei Shi, Patrick Ebel, Lei Yu, Gui-Song Xia, Wen Yang, Xiao Xiang Zhu

arXiv:2206.02850v3cs.CVeess.IV

TL;DR

Cloud removal must reconstruct optical information erased by clouds, while SAR assistance is complicated by cross-modal domain gaps and speckle noise. GLF-CR uses global-local fusion to guide global optical context interactions and transfer SAR details locally. The method achieves state-of-the-art performance across cloud-cover levels, while performance in fully cloudy scenes depends entirely on the information available in SAR.

  • Problem

    Cloud removal must recover missing optical information, but SAR-optical transfer is limited by domain gaps, speckle noise, and insufficient global-context modeling.

  • Method

    GLF-CR uses SAR-guided global context interaction and complementary local fusion to maintain recovered-region structure and generate reliable texture details.

  • Results

    GLF-CR achieves state-of-the-art performance across different cloud-cover levels.

  • Takeaways & Limitations

    The approach leverages neighboring cloud-free regions and corresponding SAR data for high-quality cloud-free image reconstruction.

  • Takeaways & Limitations

    When images are completely cloudy, reconstruction quality depends entirely on SAR information, which may fail to represent different agricultural landscapes.

Abstract

from arXiv · show

The challenge of the cloud removal task can be alleviated with the aid of Synthetic Aperture Radar (SAR) images that can penetrate cloud cover. However, the large domain gap between optical and SAR images as well as the severe speckle noise of SAR images may cause significant interference in SAR-based cloud removal, resulting in performance degeneration. In this paper, we propose a novel global-local fusion based cloud removal (GLF-CR) algorithm to leverage the complementary information embedded in SAR images. Exploiting the power of SAR information to promote cloud removal entails two aspects. The first, global fusion, guides the relationship among all local optical windows to maintain the structure of the recovered region consistent with the remaining cloud-free regions. The second, local fusion, transfers complementary information embedded in the SAR image that corresponds to cloudy areas to generate reliable texture details of the missing regions, and uses dynamic filtering to alleviate the performance degradation caused by speckle noise. Extensive evaluation demonstrates that the proposed algorithm can yield high quality cloud-free images and outperform state-of-the-art cloud removal algorithms with a gain about 1.7dB in terms of PSNR on SEN12MS-CR dataset.

1. Introduction

Cloud removal is difficult because clouds erase optical textures, while SAR assistance introduces domain-gap and speckle-noise challenges. GLF-CR addresses these issues through global context interaction and local feature compensation to reconstruct structurally consistent regions with reliable details.

  • Motivation: About 67% of global cloudiness and 55% of land cloud fraction make cloud removal important for continuous satellite monitoring.Clouds obstruct surface observations, and cloud-covered regions lose texture information.
  • Challenges: SAR can provide information beneath clouds, but transferring complementary information between SAR and optical images is difficult because their imaging mechanisms create a large domain gap.The paper identifies cross-modal transfer as non-trivial.
  • Challenges: Speckle noise in SAR images can overlap surface information and degrade cloud-removal reconstruction.The noise appears as uneven bright and dark pixels, including in homogeneous regions.
  • Prior limitations: Existing concatenation-based methods overlook long-range context, incompletely exploit cross-modal interactions, and remain susceptible to speckle noise.These limitations can produce texture and structure discrepancies and hinder cloud-removal performance.
  • Proposed method: GLF-CR uses SAR to restore texture details and maintain global consistency through complementary global and local fusion mechanisms.The method uses two parallel backbones for optical and SAR representation learning, with hierarchical SAR-feature compensation.
  • Proposed method: The SGCI block guides global interactions among optical contexts, while the SLFC block transfers SAR information and mitigates speckle-noise influence during texture reconstruction.Together, these blocks target structural consistency and reliable local detail generation.

2. Related Work

Related cloud-removal work includes inpainting, SAR-to-optical translation, SAR-optical fusion, and extensions of CNN architectures. The paper motivates global-context integration because thick clouds remove local optical information and conventional architectures or channel stacking incompletely exploit cross-modal relationships.

  • Cloud removal: Early cloud-removal methods treat corrupted regions as an inpainting problem, using surrounding information to predict missing data.These methods assume corrupted and remaining regions share statistical and geometrical structures.
  • SAR-assisted methods: SAR-to-optical translation exploits cloud-penetrating SAR, but SAR lacks spectrally resolved measurements, making the quality of generated optical images difficult to guarantee.The paper describes domain-specific potentials and peculiarities that translation cannot fully compensate.
  • SAR-assisted methods: SAR-optical fusion methods combine both modalities to guide cloud removal, including direct concatenation of SAR and optical inputs.The cited approaches include DSen2-CR and a two-step SAR translation strategy.
  • CNN-based methods: Many cloud-removal methods extend CNN architectures by adding input channels for multispectral or multimodal data.Examples include McGAN and DSen2CR.
  • CNN-based methods: Thick clouds erase all local information in covered regions, limiting architectures designed for tasks where local information is only partially lost.The paper therefore emphasizes integrating global context between cloudy and neighboring cloud-free regions.

3. Problem Statement

SAR-enhanced cloud removal must reconstruct cloud-covered regions despite missing optical information, while preserving global structure and transferring reliable SAR-derived texture. Existing pixel-by-pixel translation is limited by weak cross-modal interaction, spatial inconsistency, and SAR speckle noise.

  • Cloud removal infers the cloud-covered region from the cloud-free optical region, making it an ill-posed inpainting task.The missing information caused by clouds makes reconstruction difficult when only the cloudy optical image is used.
  • Existing SAR-based methods translate stacked SAR and optical channels pixel-by-pixel, but only partially exploit their interactions and correlations.This straightforward concatenation limits the transfer of complementary information between modalities.
  • Pixel-by-pixel translation ignores spatial consistency between cloudy and neighboring cloud-free regions, causing failures in global structural consistency.
  • SAR-enhanced removal must transfer complementary SAR information while overcoming speckle noise to generate reliable texture details.
  • The recovered region must use cloud-free optical surface information to remain structurally consistent with the remaining cloud-free regions.
  • Global-local fusion conditions removal on both SAR-optical relationships and whole-image latent structures, using non-local SAR-guided context and corresponding SAR regions.The proposed fusion strategy targets global structural consistency and more reliable texture details.

4. Method

GLF-CR is a two-stream network that hierarchically fuses SAR features into optical features through global and local fusion to recover cloud-free images. Global fusion guides cross-window optical relationships, while local fusion transfers SAR information from cloudy areas and mitigates speckle noise.

  • Overview: GLF-CR uses separate optical and SAR streams, hierarchically fusing SAR features into optical features to compensate for information loss in cloudy regions.The network processes optical and SAR representations in parallel before combining their features through repeated SGCI and SLFC blocks.
  • Local fusion: The architecture updates both optical and SAR features through dual information propagation before reconstructing the final cloud-free image.This mechanism transfers complementary SAR information to optical features while also updating the SAR representation.
  • Global fusion: SGCI combines local feature extraction with cross-window interaction, using dense-connected streams for optical and SAR features and STL-based window attention.STL partitions features into non-overlapping windows and computes self-attention within each window.
  • Global fusion: Global fusion uses SGCI to guide relationships among local optical windows with SAR-derived context, preserving consistency between recovered and remaining cloud-free regions.The SAR attention map refines the optical attention map because SAR interactions remain estimable in cloudy areas.
  • Local fusion: Local fusion uses SLFC to transfer complementary SAR information from corresponding cloudy areas into optical features.The enhanced optical and SAR features are passed to subsequent SGCI blocks for further representation learning.
  • Local fusion: Dynamic filters are generated at each SAR-feature position from concatenated optical and SAR features to address spatially uneven speckle noise before information transfer.Unlike shared standard convolution filters, spatially varying dynamic filters adapt to local conditions.

5. Experiments

Experiments on SEN12MS-CR compare GLF-CR with state-of-the-art methods across qualitative, cloud-cover, and ablation settings. GLF-CR consistently produces stronger reconstructions by combining SAR information with global context interactions and local feature compensation.

  • Experimental setup: The SEN12MS-CR experiments use Sentinel-1 SAR, cloud-free Sentinel-2, and temporally close cloud-covered Sentinel-2 triplets from 122,218 samples across 169 regions.Images are 256 × 256 pixels and cover all inhabited continents and meteorological seasons.
  • Comparisons with state-of-the-art methods: GLF-CR achieves the best qualitative results across three 128 × 128 scenes, recovering more details and producing fewer artifacts than competing methods.The comparison includes SpA GAN, SAR2OPT, SAR-Opt-cGAN, Simulation-Fusion GAN, DSen2-CR, and Concat.
  • Comparisons with state-of-the-art methods: SAR2OPT reconstructs prominent geometric features but suffers content vanishing and color-distribution differences, while SAR-Opt-cGAN performs worse than SAR2OPT.SAR2OPT uses only SAR, whereas SAR-Opt-cGAN uses both cloudy optical and SAR inputs.
  • Comparisons with state-of-the-art methods: GLF-CR outperforms DSen2-CR and Concat by better transferring SAR complementarity while preserving globally consistent structure and restoring texture details.Concat improves over SAR-Opt-cGAN and DSen2-CR but can leave clouds or blur textures because concatenation only partially exploits cross-modal interactions.
  • Ablation study: Ablations show that SAR improves texture recovery at higher cloud cover, global interactions improve structural consistency, SAR guidance reduces artifacts, and dynamic filtering mainly protects PSNR and MAE.Without SAR, reconstructions become over-smoothed; without dynamic filtering, degradation is more severe for PSNR and MAE than for SSIM and SAM.

6. Discussion

GLF-CR can recover approximate ground-object information under complete cloud cover, but texture and spectral fidelity remain constrained by SAR information. The discussion also examines speckle, registration, temporal intervals, nuisance variability, and real-versus-synthetic evaluation trade-offs.

  • Performance on Challenging Conditions: Complete cloud cover allows GLF-CR to recover approximate ground-object information, but texture details remain poor and spectral fidelity degrades.With no cloud-free optical region available, reconstruction depends entirely on SAR information; missing spectral information also reduces fidelity.
  • Performance on Challenging Conditions: SAR’s inability to distinguish different agricultural landscapes can cause corresponding details to be lost in reconstructed images.This limitation is illustrated for a scene in Fig. 9.
  • Speckle Noise in SAR Data: The SEN12MS-CR SAR data are multi-looked to reduce speckle, but the data still exhibit substantial interference between speckle and ground-object information.The dataset uses Sentinel-1 SAR at 10 m resolution, which is described as avoiding excessive distortion.
  • Registration error between the optical and SAR Data: The method addresses small optical–SAR registration deviations by using a 5-sized dynamic filter with a larger receptive field.The stated co-registration accuracy is within 2 pixels.
  • Nuisances between cloudy and cloud-free images: Nuisances are considered negligible for the large, globally and seasonally sampled test split, allowing fair benchmarking on the considered dataset.The sampling is described as unbiased toward specific sunlight conditions.
  • Temporal interval: Using cloud-free references with larger temporal intervals provides less reference value for assessing reconstruction of the cloudy input.The experiment compares cloud-free images dated July 30, August 11, and September 28 with a July 17 cloudy image and July 18 SAR input.
  • Real versus synthetic observations: Real observations are used despite shortcomings because synthetic cloud simulations may poorly approximate real data and synthetic performance may not transfer equally to real data.Strict correspondence is easier to guarantee with synthetic clouds, but the cited evidence identifies limitations in common simulation techniques.

7. Conclusion

GLF-CR uses global-local fusion to exploit SAR for cloud removal by preserving spatial consistency and compensating for missing information while reducing speckle-related degradation. Extensive experiments report state-of-the-art performance across cloud-cover levels.

  • Global fusion: GLF-CR guides relationships among local optical windows to exploit spatial consistency between cloudy and neighboring cloud-free regions.This global fusion component targets consistency across the recovered and visible regions.
  • Local fusion: GLF-CR enhances SAR utilization to compensate for missing information while alleviating performance degradation caused by speckle noise.This local fusion component complements the global consistency objective.
  • Experimental results: GLF-CR achieves state-of-the-art performance across different cloud-cover levels in extensive experiments.The conclusion attributes the reported performance to information from neighboring cloud-free regions and corresponding SAR data.
Loading 2206.02850v3…