PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoCR: Learning a Generalist Cloud Removal Prior from Heterogeneous Observations

Jeonghyeok Do, Munchurl Kim

arXiv:2609.32510v1cs.CV

TL;DR

Cloud-removal models are often trained for individual datasets and input configurations, limiting reuse across observation settings. GeoCR learns a shared prior in one model from heterogeneous observations and datasets. It reports the best FID and DISTS on three benchmark settings, and supports inference without dataset-specific fine-tuning.

  • Problem

    Cloud-removal methods are typically specialized to individual datasets and input configurations, while applications vary in spectral bands, temporal coverage, and sensing modalities.

    Source: §1 · p. 2

  • Method

    GeoCR jointly pretrains a shared cloud-removal prior across ten datasets, using a common latent interface and a single generator for RGB and non-RGB latents conditioned on cloudy observations and optional SAR.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    GeoCR achieves the lowest FID and DISTS on full-band SEN12MS-CR and Sen2 MTC New and RGB-only CUHK-CR2 among compared methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support the feasibility of a reusable cloud-removal model across the evaluated heterogeneous settings, with conditioning ablations showing benefits from additional temporal and SAR evidence without retraining.

    Source: §5 · p. 9, §4.4 · p. 9

    Non-RGB stems are selected by spectral and radiometric configuration, with ten-band inputs distinguishing TOA reflectance from BOA imagery.

    Source: §3 · p. 4

Abstract

from arXiv · show

Cloud removal methods are typically specialized to individual datasets and input configurations, limiting reuse across sensors, spectral bands, and observation settings. We introduce GeoCR, a generalist model that unifies RGB-only-based CR and multispectral-based CR from single- or multi-temporal cloudy observations, with optional SAR guidance, within a single network. To accommodate different spectral and sensing domains, compact input and output stems extend a pretrained RGB autoencoder while keeping its encoder and decoder trunks frozen. This shared latent interface enables a single flow transformer to jointly model clean RGB and non-RGB latents, conditioned on separate cloudy-observation streams and optional SAR tokens. Through joint pretraining on the training splits of ten datasets comprising 883,331 cloud-free target images, GeoCR learns a shared cloud removal prior across these heterogeneous configurations. The same pretrained checkpoint supports direct inference without dataset-specific fine-tuning and efficient adaptation through low-rank adaptation (LoRA). We evaluate GeoCR against general image restoration and cloud removal methods on test splits of the contributing datasets under full-band and RGB-only settings. GeoCR achieves the best FID and DISTS on full-band SEN12MS-CR and Sen2_MTC_New and RGB-only CUHK-CR2, outperforming existing models and demonstrating the effectiveness of a reusable generative model across diverse settings.

1 INTRODUCTION

GeoCR addresses the challenge of reusing cloud-removal models across datasets and observation configurations by learning a shared prior for RGB-only and multispectral reconstruction, with optional SAR. It evaluates one pretrained checkpoint across five benchmarks and reports the best FID and DISTS on three.

1 INTRODUCTION

  • GeoCR (w/o FT) achieves the lowest FID and DISTS on full-band SEN12MS-CR and Sen2 MTC New and RGB-only CUHK-CR2.Figure 2 compares methods across five cloud-removal benchmarks; GeoCR uses the same checkpoint across all five.Source: Figure 2 · p. 2, §1 · p. 3
  • GeoCR unifies RGB-only and multispectral cloud removal across single- and multi-temporal observations, with optional SAR, using a corpus of 883,331 cloud-free targets from ten datasets.The corpus spans Sentinel-2, Landsat-8, and high-resolution optical imagery.Source: §1 · pp. 2–3
  • Compact configuration-specific stems connect non-RGB bands and SAR to a pretrained RGB autoencoder while its encoder and decoder trunks remain frozen.A flow transformer jointly models clean RGB and available non-RGB latents using separate cloudy-observation streams and optional SAR; the checkpoint also supports LoRA adaptation.Source: §1 · pp. 2–3
  • The evaluation compares GeoCR with general image-restoration and cloud-removal methods on held-out test splits of contributing datasets under full-band and RGB-only settings.The evaluation covers five primary benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior work approaches cloud removal through general image restoration, temporal and multisensor methods, and datasets with varied sensing and reconstruction settings. GeoCR is positioned as a shared prior and conditioning framework for learning across that diversity.

2 RELATED WORK

  • General image-restoration approaches include paired adversarial mappings, Brownian-bridge translation, and diffusion-assisted restoration, providing varied baselines for cloud removal.The passage describes pix2pix, pix2pixHD, BBDM, and HI-Diff.Source: §2 · p. 3
  • Cloud-removal work uses temporal redundancy, complementary sensors, and generative priors, while GeoCR focuses on a reusable prior spanning datasets and observation configurations.Related efforts include temporal attention, conditional diffusion, latent diffusion refinement, and mean-reverting dynamics.Source: §2 · p. 3
  • Cloud-removal datasets differ in spectral coverage, radiometric processing, resolution, temporal sampling, and SAR availability; GeoCR uses a common representation and conditioning framework for joint pretraining across them.The surveyed datasets include paired thin- and thick-cloud imagery, RGB–NIR, multispectral-plus-SAR, and multi-observation resources.Source: §2 · p. 3, Figure 3 · p. 4

3 GEOCR

GeoCR uses a staged shared-latent design to train one conditional flow generator across heterogeneous optical and optional SAR configurations. Its frozen encoder–decoder trunks and shared checkpoint support direct inference or LoRA adaptation.

3.1 STAGE 1: STEM ADAPTATION TO A SHARED LATENT INTERFACE

  • Compact stems adapt a pretrained RGB autoencoder’s shared encoder and decoder trunks to different measurement domains while those trunks remain frozen.The additional non-RGB and SAR stems are trained through reconstruction of their own observations, without requiring cloudy-to-clear mapping.Source: §3.1 · pp. 4–5

3.2 STAGE 2: JOINT CLOUD REMOVAL PRETRAINING

  • RGB and non-RGB components are encoded on matching latent grids and concatenated channel-wise, with zero latents standing in for unavailable non-RGB components.This preserves a common input layout for RGB-only and multispectral samples; optional SAR is encoded as a separate condition.Source: §3.2 · p. 5
  • A shared flow generator jointly models clean RGB and non-RGB latents, using separate cloudy-observation streams and optional SAR conditioning across input configurations.Multi-stream blocks share parameters across optical observations, while all streams interact through joint attention.Source: §3.2 · p. 5
  • Conditional flow matching trains the generator to predict target velocity from sampled noisy targets, conditioned on available observations.Only the generator is optimized in this stage, across the heterogeneous corpus and its available observation configurations.Source: §3.2 · p. 5

3.3 INFERENCE AND ADAPTATION

  • The Stage 2 checkpoint supports direct inference without dataset-specific optimization or target-dataset adaptation through LoRA.Autoencoder trunks and stems remain frozen in both modes; only LoRA adapters are optimized for adaptation.Source: §3.3 · p. 6

4 EXPERIMENTS

GeoCR is evaluated across pooled cloud-removal data and held-out benchmarks spanning multiple sensors and input settings. It leads on several distributional and perceptual comparisons, while coverage and conditioning analyses test its robustness across inputs.

4.1 PRETRAINING AND EVALUATION DATA

  • The shared pretraining corpus combines ten datasets and 883,331 cloud-free targets spanning Sentinel-2, Landsat-8, and high-resolution imagery with varied bands, temporal inputs, and optional SAR.The five primary benchmarks are SEN12MS-CR, Sen2 MTC New, CUHK-CR2, WHUS2-CRv, and T-CLOUD; full-band and RGB-only settings use identical test samples within each setting.Source: §4.1 · pp. 6–7

4.3 COMPARISON WITH THE STATE OF THE ART

  • GeoCR achieves the best FID and DISTS on full-band Sen2 MTC New and SEN12MS-CR and RGB-only CUHK-CR2; on full-band SEN12MS-CR, GeoCR (w/o FT) reduces FID by over 60% versus the strongest baseline.On RGB-only SEN12MS-CR, pix2pixHD retains lower DISTS and LPIPS.Source: Table 3 · p. 7, Table 4 · p. 7, §4.3 · p. 8

4.2 IMPLEMENTATION DETAILS AND EVALUATION PROTOCOL

  • Stage 1 trains 51.1K stem parameters around a frozen autoencoder for 400K updates; Stage 2 trains the 3.853B-parameter transformer for 500K updates, and inference uses four Euler steps.The evaluation also reports training and inference settings, including condition dropout and LoRA adaptation.Source: §4.2 · p. 7
  • The evaluation compares GeoCR with ten restoration and cloud-removal methods; FID and DISTS are primary metrics, supplemented by feature, perceptual, and reconstruction measures.Baselines are retrained on each dataset’s training split, while GeoCR (w/o FT) uses one pooled checkpoint and LoRA adapts it on the corresponding training split.Source: §4.2 · p. 7, §4.1 · p. 7

4.4 ABLATION STUDIES AND ANALYSIS

  • GeoCR variants have lower DISTS and LPIPS than compared baselines across SEN12MS-CR cloud-coverage groups, with an advantage that persists under heavy cloud cover.In nearly clear scenes, GeoCR improves on the unrestored input while all compared baselines worsen DISTS; the groups are defined from cloudy inputs.Source: Figure 5 · p. 8, §4.4 · p. 9
  • Using the same GeoCR (w/o FT) checkpoint, additional cloudy frames and SAR guidance improve reference fidelity, with less residual cloud and clearer terrain ridges, respectively.Tables 6 and 7 vary the observations on fixed test subsets.Source: Table 6 · p. 9, Table 7 · p. 9, Figure 6 · p. 9

5 CONCLUSION

GeoCR learns a shared cloud-removal prior from ten datasets and supports varied bands, temporal observations, and optional SAR. The appendix documents its interface and reports input-domain reconstruction diagnostics.

5 CONCLUSION

  • Joint pretraining on ten datasets with 883,331 cloud-free targets yields a shared cloud-removal prior that supports diverse bands, temporal observations, and optional SAR without dataset-specific fine-tuning.The same model also supports lightweight LoRA adaptation, and conditioning ablations find benefits from additional temporal and SAR evidence without retraining.Source: §5 · p. 9

APPENDIX

  • The appendix covers the shared latent interface, implementation details, additional results, and evaluation protocols, with its organization summarized in Table 8.Source: APPENDIX · p. 10, Table 8 · p. 10

A STEM ADAPTATION

  • Compact stems adapt the frozen RGB encoder–decoder trunks for ten-band optical and SAR reconstruction, while the original RGB route remains unchanged and the initialized NIR route requires no stem training.Source: A STEM ADAPTATION · p. 10
  • The initialized NIR route achieves reconstruction diagnostics of 42.06 dB on Sen2 MTC New, 40.41 dB on CUHK-CR, and 49.51 dB on Sen2 MTC Old, measuring input-domain reconstruction rather than cloud removal.Source: A STEM ADAPTATION · p. 10

B IMPLEMENTATION DETAILS

GeoCR uses shared frozen autoencoder trunks and lightweight stems to represent multiple sensing domains, then conditions a flow transformer on separate cloudy-observation and optional SAR streams.

Autoencoder

  • Each input and output stem is a single convolution; added stems for ten-band TOA, ten-band BOA, and VV/VH SAR are trained with an ℓ1 reconstruction loss around frozen encoder–decoder trunks.The added stems contain 53.5K parameters, of which 51.1K are trained; the original RGB projections and initialized single-band NIR stems are retained.Source: B IMPLEMENTATION DETAILS · p. 10
  • The encoder produces 32-channel posterior-mean latents at 1/8 resolution, packed into 128 channels at 1/16 resolution and normalized with fixed BatchNorm statistics before decoding reverses those operations.Source: B IMPLEMENTATION DETAILS · p. 10

Generator

  • The transformer keeps the noisy target, each of K cloudy observations, and optional SAR in separate token streams; cloudy frames interact through attention rather than temporal channel concatenation.Double-stream blocks jointly attend to target and conditioning tokens, later single-stream blocks process the combined sequence, and only target tokens enter the velocity head.Source: B IMPLEMENTATION DETAILS · p. 10

Reconstruction by configuration

  • The initialized NIR route reports 42.06 dB on Sen2 MTC New, 40.41 dB on CUHK-CR, and 49.51 dB on Sen2 MTC Old; these are pooled reconstruction diagnostics, not cloud-removal results.Table 9 computes PSNR from pooled reconstruction error on 64 training images per source, without clipping.Source: Table 9 · p. 11

C ADDITIONAL QUANTITATIVE RESULTS

Additional benchmarks show that GeoCR’s relative performance varies by dataset and metric, and that LoRA improvements are selective rather than consistent.

C ADDITIONAL QUANTITATIVE RESULTS

D ADDITIONAL QUALITATIVE RESULTS

The appendix presents qualitative comparisons across native and RGB-only settings, showing both GeoCR modes alongside baselines and clear references.

D ADDITIONAL QUALITATIVE RESULTS

  • GeoCR's examples recover scene layout and local appearance, but specialized baselines sometimes match the reference better.LoRA makes relatively small changes in the shown examples and does not improve every image.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 12
  • The two SEN12MS-CR rows in Figure 9 differ in both available observations and rendering, so their contrast does not isolate SAR's contribution.Figure 9 also includes representative outputs selected by dataset-level LPIPS, with per-image LPIPS values.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 12
  • Figures 11–19 provide four examples for each of six native dataset settings and three RGB-only settings, with all visualizations shown as RGB views.Each compares GeoCR without fine-tuning and with LoRA against available baselines and paired clear references.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 12

E FAILURE CASES AND LIMITATIONS

The examples expose limits when observations are incomplete or inconsistent with the target, and the evaluation does not establish transfer to unseen corpora or physical accuracy of obscured content.

E FAILURE CASES AND LIMITATIONS

  • When all optical observations are heavily obscured, generated details may look plausible yet differ from the reference; SAR adds structural evidence but cannot determine a unique optical appearance.Fine structures under complete cloud remain difficult to recover reliably, even when scene layout is convincing.Source: E FAILURE CASES AND LIMITATIONS · p. 12
  • Surface changes between acquisitions can make observations inconsistent with the target, and the model may preserve the input state rather than reconstruct an unobserved state.Photometric offsets also remain visible on T-CLOUD.Source: E FAILURE CASES AND LIMITATIONS · p. 12
  • PSNR and SSIM measure agreement with the supplied reference, but neither establishes the physical accuracy of obscured content.Figure 10 covers severe occlusion, temporal change, fine urban structure, and photometric mismatch; cases are selected using occlusion, change, texture, or error criteria.Source: Figure 10 · p. 13, E FAILURE CASES AND LIMITATIONS · p. 14
  • GeoCR's generalist behavior is evaluated on contributing sources; transfer to an entirely unseen corpus is unassessed, as are unseen sensors and geographically independent transfer.Further evaluation is also needed for applications requiring calibrated spectral fidelity.Source: E FAILURE CASES AND LIMITATIONS · p. 14

F DATASETS AND PREPROCESSING

The study combines ten training sources and documents dataset-specific input handling, acquisition sampling, and split audits for its dataset mixture and evaluations.

F DATASETS AND PREPROCESSING

  • Ten cloud-removal datasets contribute 883,331 cloud-free target images; six sources also provide evaluation benchmarks, while four contribute only pretraining data.Three RGB-only settings are derived from SEN12MS-CR, Sen2 MTC New, and WHUS2-CRv, and evaluation uses held-out samples from contributing sources.Source: F DATASETS AND PREPROCESSING · p. 14
  • Sentinel-2 digital numbers are converted to reflectance and normalized to [-1, 1], while eight-bit imagery and SAR decibel measurements receive separate normalization procedures.For separately rendered RGB-only datasets, reflectance is first converted to 8-bit values; SAR is clipped to [-30, 0] dB before mapping to [-1, 1].Source: F DATASETS AND PREPROCESSING · p. 14
  • WHUS2-CRv bands are aligned to a 10 m grid, and AllClear cloudy observations and SAR acquisitions are sampled within ±40 days of the target.AllClear targets are limited to at most 10% cloud, 10% shadow, and 1% no-data coverage; the target acquisition date is excluded from cloudy-observation sampling.Source: F DATASETS AND PREPROCESSING · p. 14
  • Split audits correct scene assignments and remove duplicate or overlapping samples; RICE1/2 are used only for pretraining because their supplied splits contain duplicate images.Audits also exclude overlapping Sen2 MTC Old samples and train/validation or within-test CUHK-CR1 duplicates.Source: F DATASETS AND PREPROCESSING · p. 14

G EVALUATION PROTOCOLS

Evaluation uses shared test samples and complementary distributional, perceptual, semantic, and reference-fidelity metrics, with qualitative comparisons shown as RGB views.

G EVALUATION PROTOCOLS

  • FID, DISTS, KID, DINO similarity, and LPIPS are computed on RGB views, including for native multispectral settings; PSNR and SSIM use the specified evaluation bands.Methods are compared on common samples and spatial extents for each metric, with FID and DISTS designated primary measures.Source: G EVALUATION PROTOCOLS · p. 15
  • FID and KID compare image-set distributions rather than correspondence to an individual cloudy input, using 2,048-dimensional Inception pool-3 features resized to 299 × 299.FID compares Gaussian approximations, while KID estimates squared maximum mean discrepancy with a polynomial kernel.Source: G EVALUATION PROTOCOLS · p. 15
  • DISTS measures structural and textural statistics, LPIPS compares learned deep-feature distances, and DINO similarity compares spatially aligned patch-token features against paired clear references.DISTS uses VGG-16, LPIPS uses AlexNet v0.1, and DINO similarity uses frozen DINOv3-SAT ViT-L/16 features.Source: G EVALUATION PROTOCOLS · p. 15
  • PSNR and SSIM quantify spatial and radiometric agreement with the reference and remain sensitive to misregistration and acquisition differences.SSIM uses different windows for native Sentinel-2 and display/RGB settings; PSNR is averaged over per-image scores.Source: G EVALUATION PROTOCOLS · p. 15
  • Figures 11–16 provide qualitative comparisons on six datasets, evaluating GeoCR without fine-tuning (w/o FT) and with LoRA adaptation; Figures 14 and 16 show the same four scenes across groups of competing methods.Figures 11–13 show native-setting RGB views with paired clear references, while Figure 15 shows RGB views with paired clear references in the RGB+NIR setting.Source: Figure 11 · p. 16, Figure 12 · p. 17, Figure 13 · p. 18
  • Figures 17–19 show RGB-only cases with one cloudy RGB input and no SAR guidance, comparing GeoCR without fine-tuning and with LoRA.The figures cover SEN12MS-CR, Sen2 MTC New, and WHUS2-CRv.Source: Figure 17 · p. 22, Figure 18 · p. 23, Figure 19 · p. 24

Related papers

Loading 2609.32510v1…