PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretation-Oriented Cloud Removal via Observation-Anchored Residual Flow with Geo-Contextual Alignment

Ziyao Wang, Maonan Wang, Yucheng He, Xianping Ma, Ziyi Wang, Hongyang Zhang, Yirong Cheng, Man-on Pun

arXiv:2607.02471v1cs.CV

TL;DR

Cloud removal methods can produce visually plausible images while failing to preserve the semantic information needed for reliable downstream interpretation. GACR addresses this with observation-anchored residual flow and geo-contextual semantic alignment, consistently improving reconstruction quality and downstream task accuracy across six datasets and twelve tasks.

  • Problem

    Existing cloud-removal methods emphasize pixel-level visual fidelity without explicitly preserving task-relevant geographical, structural, and categorical information.

  • Method

    GACR combines observation-anchored residual flow with VFM-induced geo-contextual prior alignment to reconstruct cloud-free imagery while preserving semantic structures.

  • Results

    GACR consistently improves reconstruction quality and downstream interpretation accuracy across six datasets and twelve downstream tasks.

  • Takeaways & Limitations

    Coupling observation-anchored physical priors with semantic constraints supports cloud removal that preserves interpretation-relevant information beyond visual enhancement.

Abstract

from arXiv · show

Cloud removal (CR) is essential for optical remote sensing, serving as a prerequisite for reliable downstream interpretation, such as semantic segmentation and change detection. However, existing CR approaches often prioritize visual realism while overlooking their impact on subsequent analytical tasks, leading to semantic drift and degraded downstream performance. To address this issue, we propose Geo-Anchored Cloud Removal (GACR), a unified framework that jointly ensures faithful reconstruction and robust interpretability. At its core, GACR incorporates Observation-Anchored Residual Flow (OAR-Flow), which reformulates CR as a physically grounded residual inversion process. By anchoring the generative trajectory to the cloudy observation rather than pure noise, OAR-Flow enables fast, stable, and faithful reconstruction. To further preserve semantic structures critical for downstream interpretation, GACR integrates Geo-Contextual Prior Alignment (GCPA) to constrain the reconstruction within a semantic manifold induced by a Vision Foundation Model (VFM). Consequently, GACR strictly maintains the spatial-semantic integrity of complex landscapes. Extensive experiments across six CR datasets and twelve downstream tasks demonstrate that GACR produces superior reconstruction quality while consistently improving downstream task accuracy. The code is available at https://github.com/wzy6055/GACR.

1 Introduction

Clouds obscure surface information and expose a gap between visually optimized cloud removal and semantic reliability for downstream interpretation. GACR addresses this gap with observation-anchored residual inversion and VFM-based semantic alignment, improving reconstruction and downstream accuracy across six benchmarks and twelve tasks.

  • Motivation: Clouds obscure surface information and introduce uncertainty, making cloud removal a critical component of the remote sensing interpretation pipeline.Cloud removal has evolved beyond simple preprocessing because optical imagery usability depends on accurate surface representation and semantic consistency.
  • Limitations: Existing cloud removal methods often optimize pixel-level fidelity, encouraging artifact removal for PSNR and SSIM without explicitly ensuring semantic correctness.This visual-fidelity-oriented paradigm can equate conventional image quality with semantic correctness while overlooking downstream interpretation needs.
  • GACR: OAR-Flow formulates cloud removal as physically grounded residual inversion anchored to the cloudy observation rather than pure noise.It adapts to cloud opacity, using structured perturbations for stable reconstruction and controlled semantic completion under heavy occlusion.
  • GACR: GCPA uses a pretrained Vision Foundation Model to constrain reconstruction within a geo-contextually consistent semantic manifold.The alignment preserves task-relevant structural patterns and category-specific information beyond pixel-level similarity.
  • Results: Experiments across six benchmarks and twelve downstream tasks show that GACR improves reconstruction quality while yielding higher downstream interpretation accuracy.The framework jointly targets faithful reconstruction and robust interpretability.

2 Related Work

Cloud-removal methods have advanced from residual prediction to generation-based approaches, but diffusion-based CR increasingly emphasizes efficient, observation-conditioned trajectory learning. GACR’s OAR-Flow distinguishes itself by grounding deterministic generative dynamics in cloudy observations rather than unconditional noise.

  • Cloud-removal methods: Cloud removal is a fundamental Earth-observation preprocessing step for recovering cloud-free imagery for reliable surface analysis.
  • Cloud-removal methods: Deep-learning CR methods broadly comprise residual-prediction-based and generation-based approaches.Residual prediction estimates cloud residuals or clean reflectance using CNNs, Transformers, or Mamba architectures.
  • Diffusion-based methods: Diffusion models evolved from DDPM and DDIM to score-based SDE/ODE formulations, achieving higher fidelity and diversity than GAN- and VAE-based paradigms.Early diffusion studies mainly used U-Net architectures, while recent DiT models improve scalability.
  • Diffusion-based methods: OAR-Flow differs from existing flow-based CR methods by using an observation-conditioned residual formulation grounded in cloudy input rather than unconditional noise.It reformulates diffusion as deterministic probability-flow ODEs for more efficient trajectory learning.

3 Methodology

GACR combines the observation-anchored OAR-Flow reconstruction process with GCPA’s VFM-based semantic constraint, then evaluates whether cloud removal preserves task-relevant information. OAR-Flow uses the cloudy observation as a trajectory anchor, while the unified objective jointly optimizes velocity matching and geo-contextual consistency.

  • OAR-Flow: OAR-Flow transports samples between the clean surface x∗ and cloudy observation xc, explicitly modeling cloud coverage as structured atmospheric degradation rather than random noise.Its forward interpolant is x_t = α_t x_* + β_t x_c + σ_t ε, with the cloudy observation serving as an anchor.
  • OAR-Flow: The observation anchor preserves structural cues in thin clouds, while stochastic perturbations provide semantic completion flexibility in thick-cloud regions.The anchor strength is controlled by ρ under the linear schedule α_t = 1 - t, β_t = ρt, and σ_t = t.
  • OAR-Flow: OAR-Flow parameterizes the velocity with a network conditioned on x_t, t, and xc, then integrates the deterministic flow from t = T to t = 0 to obtain the reconstruction.Because the trajectory remains anchored to xc throughout integration, reconstructed surface structures remain spatially consistent with the original observation.
  • Training and evaluation: The unified objective combines velocity matching loss L_vel and geo-contextual consistency loss L_GCI as L = L_vel + λL_GCI, balancing reconstruction fidelity and geo-contextual integrity.Evaluation covers land-cover classification, building extraction, semantic segmentation, and height estimation to assess preservation of task-relevant semantic and structural information.
  • GCPA: GCPA aligns HDiT bottleneck features with a pretrained VFM’s clear-image representation using an Adaptive Projector and patch-wise cosine similarity.This alignment constrains generation to a VFM-induced semantic manifold and reduces semantic drift in heavily occluded regions.

4 Experiments and Discussion

Experiments across six real and synthetic cloud-removal datasets assess reconstruction fidelity, downstream interpretation, and efficiency. GACR achieves strong reconstruction and downstream performance, while OAR-Flow and GCPA improve convergence, reconstruction metrics, and computational efficiency.

  • Experimental Setup: GACR is evaluated on six datasets spanning publicly available real-cloud data and self-constructed synthetic thin- and thick-cloud scenarios.The datasets are CUHKCR-EXT-GZ, CUHKCR-EXT-CS, Potsdam-CR-thin, Potsdam-CR-thick, Vaihingen-CR-thin, and Vaihingen-CR-thick.
  • Reconstruction Fidelity: 33.642 dB and 36.918 dB are the highest PSNR values on Potsdam-CR-thin and Vaihingen-CR-thin, respectively, achieved by GACR-SAT/1.The observation-anchored formulation preserves fine-grained textures, low-frequency spatial continuity, and reliable regions under thin clouds.
  • Downstream Performance: 0.833 and 0.781 are GACR-SAT/2’s highest classification accuracies on CLS-1 and CLS-2, exceeding AST by 2.0% and 1.4%, respectively.For height estimation, GACR-SAT/2 also achieves RMSE values of 1.891 on HE-1 and 1.482 on HE-3.
  • Downstream Performance: GACR yields greater gains than competing methods on dense prediction tasks, producing more distinct and semantically coherent object boundaries.The reported downstream evaluation uses a DINOv3-based encoder initialized with ViT-L/16-LVD-1689M weights, with classification, building extraction, segmentation, and height estimation tasks.
  • Convergence and Efficiency: OAR-Flow converges substantially faster than EMRDM, while replacing diffusion dynamics with OAR-Flow improves reconstruction metrics and GCPA further enhances reconstruction quality and efficiency.DiT-based modeling is reported to incur GFLOPs = 697.20.

5 Conclusion

GACR jointly improves cloud-removal reconstruction fidelity and downstream reliability through observation-anchored residual inversion and VFM-based semantic alignment. Evaluations across six datasets and twelve downstream tasks show consistent gains in reconstruction quality and task accuracy.

  • 5 Conclusion: GACR is an interpretation-oriented framework that jointly improves reconstruction fidelity and downstream reliability.Its design targets both faithful cloud removal and dependable subsequent analysis.
  • 5 Conclusion: OAR-Flow reformulates cloud removal as a physically grounded residual inversion process anchored to cloudy observations, reducing geographically implausible artifacts.The cloudy observation anchors the generative process instead of relying solely on an unconstrained reconstruction trajectory.
  • 5 Conclusion: GCPA constrains reconstruction within a VFM-induced semantic manifold to preserve task-relevant structures and category-specific information.This semantic constraint is intended to protect structures needed for downstream interpretation.
  • 5 Conclusion: Evaluations on six datasets and twelve downstream tasks show that GACR consistently enhances reconstruction quality and task accuracy across diverse cloud conditions and remote sensing scenarios.The reported evaluation spans both cloud-removal reconstruction and downstream interpretation under varied conditions.

Supplementary Material · A Proof of the probability flow ODE with the velocity.

The supplementary proof derives the backward probability-flow formulation by differentiating the characteristic function of p_t(x), identifying the velocity field, and using Fourier-transform identities to obtain the transport equation. It thereby connects the marginal dynamics of x_t to the velocity expression in Eq. (8).

  • Supplementary Material: The proof starts from the time-dependent marginal density p_t(x) of x_t and its characteristic function, defined as an expectation over x∗, x_c, and ϵ.The characteristic function is written as ˆp_t(k) = E[e^{i k·x_t}].
  • A Proof of the probability flow ODE with the velocity.: Differentiating the characteristic function in time and applying conditional expectation expresses its derivative through the conditional expectation of ẋ_t e^{i k·x_t}.The derivation uses the tower property of conditional expectation before conditioning on x_t = x.
  • A Proof of the probability flow ODE with the velocity.: Substituting ẋ_t = α̇_t x∗ + β̇_t x_c + σ̇_t ϵ yields the conditional components used to define the velocity field v_t(x).The resulting velocity includes terms such as α̇_t E[x∗ | x_t = x], β̇_t E[x_c | x_t = x], and σ̇_t E[ϵ | x_t = x].
  • A Proof of the probability flow ODE with the velocity.: The proof then rewrites the time derivative of the Fourier transform of p_t using v_t(x), obtaining an integral involving i k · v_t(x) e^{i k·x} p_t(x).This is the Fourier-domain identity stated in Eq. (22).
  • A Proof of the probability flow ODE with the velocity.: Integration by parts transforms the Fourier-domain expression into the negative divergence of v_t(x)p_t(x) multiplied by e^{i k·x}.The proof applies the identity v_t(x) · ∇_x[e^{i k·x}] = -∇_x · [v_t(x)p_t(x)] in the integrated form shown in Eq. (24).
  • A Proof of the probability flow ODE with the velocity.: By the properties of the Fourier transform, the resulting identity implies that p_t(x) satisfies the transport equation in Eq. (7).The divergence operator is identified through ∂/∂x_i [v_i p_t].

B Downstream Networks · C Datasets Details

The downstream networks use frozen DINOv3 features with lightweight task-specific decoders, applying the class token to classification and reshaped patch tokens to dense prediction tasks. The designs include linear classification and cascaded spatial reconstruction with task-specific prediction heads and losses.

  • B Downstream Networks: DINOv3’s frozen weights serve as the downstream backbone because of their transferability and generalization capability.Only lightweight task-specific decoders are trained.
  • B Downstream Networks: The class token supports CLS, while reshaped patch tokens support BLD, SEG, and HE dense prediction tasks.These four downstream task types use different token representations from DINOv3.
  • B Downstream Networks: CLS applies a linear classifier to the class token and optimizes category probabilities with standard cross-entropy loss.The classifier transforms the global representation into category probabilities.
  • B Downstream Networks: BLD, SEG, and HE use a unified lightweight cascaded decoder that progressively restores spatial resolution through interpolation, convolution, activation, and normalization.Hierarchical refinement continues until the original image scale is restored.
  • B Downstream Networks: Task-specific prediction heads generate the final output maps after cascaded spatial refinement.The decoder is shared across the three dense prediction tasks, while the final head is task-specific.
  • B Downstream Networks: BLD predicts a single-channel probability map with sigmoid activation, binary cross-entropy training, and a fixed 0.5 inference threshold.The threshold converts predictions into building and non-building regions.

C.1 CUHKCR-EXT Dataset · C.2 Synthetic Cloud Dataset · D Detailed Experiment Implements

The paper evaluates cloud removal on real and synthetically constructed high-resolution datasets, including downstream semantic annotations and controlled cloud opacity. CUHKCR-EXT provides paired cloudy and cloud-free satellite imagery across regions with differing cloud thickness.

  • C.1 CUHKCR-EXT Dataset: CUHKCR-EXT contains 0.5 m ultra-high-resolution paired cloudy and cloud-free images captured by the Jilin-1 satellite.The dataset covers two regions in Guangzhou and Changsha, China.
  • C.1 CUHKCR-EXT Dataset: CUHKCR-EXT-GZ is primarily covered by thin clouds, whereas CUHKCR-EXT-CS has a higher proportion of thick clouds.
  • C.1 CUHKCR-EXT Dataset: CUHKCR-EXT is divided into a cloud-removal part and a downstream-tasks part.
  • C.2 Synthetic Cloud Dataset: Synthetic datasets were constructed with both thin- and thick-cloud types using an atmospheric scattering model and a cloud-generation approach.
  • C.2 Synthetic Cloud Dataset: The synthetic datasets include SEG and HE annotations for high-resolution cloud-removal evaluation.
  • C.2 Synthetic Cloud Dataset: A scalar thickness parameter was added to the cloud generator to jointly scale mask intensity and systematically control cloud opacity.

D.1 Training Configurations … E.3 Downstream performance with DINOv3-SAT backbone

The paper evaluates GACR under standardized training and comparison protocols, using established restoration and downstream metrics. Additional analyses examine computational efficiency, feature separability, and performance with DINOv3-SAT weights.

  • D.1 Training Configurations: GACR is trained for 200k steps with batch size 4, validation every 10k steps, AdamW, learning rate 1 × 10−4, and 256 × 256 inputs.The optimizer uses β1 = 0.9, β2 = 0.999, and ϵ = 1 × 10−8.
  • D.2 Compared Methods: Six baselines span CNN, Transformer, Mamba, dynamic-filter, and diffusion-based cloud-removal methods, with retraining used to ensure fair comparison.The baselines are MPRNet, Restormer, AST, MambaIR, DFCFormer, and EMRDM.
  • D.3 Evaluation Metrics: Cloud-removal quality is assessed with PSNR, SSIM, LPIPS, and RMSE, while downstream tasks use Acc, IoU, mIoU, and RMSE for classification, building detection, segmentation, and height estimation.LPIPS uses perceptual feature distance, and higher IoU indicates better localization consistency.
  • E.1 Model Complexity: With p = 2, GACR requires only 56.05G FLOPs while maintaining competitive performance, whereas p = 1 improves visual quality but increases cost to 223.60G FLOPs.The p = 2 configuration is presented as a favorable trade-off between efficiency and capacity.
  • E.2 Visualization of t-SNE: t-SNE features from cloudy images already form well-separated class clusters, and cloud-removal preprocessing does not further improve their separability.The analysis uses DINOv3 ViT-L/16-SAT-300M class tokens from CUHKCR-EXT-GZ and attributes the result to classification relying mainly on global representations.
  • E.3 Downstream performance with DINOv3-SAT backbone: Under DINOv3 ViT-L/16-SAT-300M weights, GACR consistently outperforms other methods across the reported downstream setting.Most models degrade when switching from LVD to SAT weights, indicating different downstream behavior across pretrained backbones.
  • E.3 Downstream performance with DINOv3-SAT backbone: SAT weights provide more domain-specific representations beneficial for low-level restoration, whereas LVD weights offer stronger semantic understanding suited to high-level downstream tasks.Differences in pretraining input resolution may also contribute to the observed domain discrepancy.

E.4 Evaluation on Heterogeneous Downstream Architectures. · E.5 CKNNA with Cloud-free Data · E.6 Additional Visual Results

Across heterogeneous segmentation backbones, GACR remains highly competitive and improves semantic recognition, while CKNNA analysis indicates stronger feature alignment after cloud removal. Additional experiments also provide process visualizations for OAR-Flow and comparisons across four datasets.

  • E.4 Evaluation on Heterogeneous Downstream Architectures.: GACR achieves the best or second-best performance across all evaluated configurations using the ResNet-based A2-FPN and CNN–Transformer hybrid UNetFormer.Both downstream networks are trained on cloud-free data under the same evaluation protocol, with quantitative results reported on Vaihingen-CR.
  • E.4 Evaluation on Heterogeneous Downstream Architectures.: On Vaihingen-CR-thin, GACR-SAT obtains mIoU scores of 0.739 with A2-FPN and 0.678 with UNetFormer.These results are reported for the two heterogeneous downstream segmentation architectures.
  • E.4 Evaluation on Heterogeneous Downstream Architectures.: On Vaihingen-CR-thick, GACR-SAT reaches mIoU scores of 0.721 with A2-FPN and 0.662 with UNetFormer.The results support performance gains across different downstream models rather than dependence on one specific backbone.
  • E.4 Evaluation on Heterogeneous Downstream Architectures.: Geo-contextual alignment helps restore semantically meaningful structures that benefit recognition across different downstream models.This behavior is presented as the mechanism underlying consistent improvements across heterogeneous architectures.
  • E.5 CKNNA with Cloud-free Data: CKNNA values for most cloud-removal results are generally higher than those of the original cloudy images across six datasets.The comparison uses each CR result and its corresponding cloud-free reference to assess representational consistency and feature-level alignment.
  • E.6 Additional Visual Results: Additional visual results include detailed forward and reverse OAR-Flow processes and comparisons on CUHKCR-EXT-GZ, CUHKCR-EXT-CS, Potsdam-CR-thick, and Vaihingen-CR-thick.The process visualization appears in Fig. 9, while the dataset comparisons appear in Figs. 12–15.

E.7 Additional Ablation Studies

Additional ablations show that GCPA remains effective across multiple visual foundation model backbones, while OAR-Flow benefits from moderate observation anchoring. The selected default, ρ = 3, balances observation preservation with recovery of heavily obscured regions.

  • Effect of VFM backbones: All tested VFM variants outperform the baseline, with DINOv3 achieving the best overall performance and DINOv2 and CLIP also providing clear gains.The ablation replaces DINOv3 with DINOv2, CLIP, and MAE while keeping OAR-Flow unchanged, supporting the generality of semantic prior alignment.
  • Effect of the anchoring strength ρ: PSNR increases from 32.264 to 33.018 as anchoring strength ρ rises from 0 to 3.This indicates that moderate anchoring preserves reliable observation cues and stabilizes the reverse trajectory.
  • Effect of the anchoring strength ρ: Performance slightly drops when ρ is increased beyond 3, suggesting that overly strong anchoring can restrict recovery in heavily obscured regions.The study therefore adopts ρ = 3 by default.
Loading 2607.02471v1…