PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoSET: Generalist Foundation Model for SAR-to-EO Image Translation

Jeonghyeok Do, Munchurl Kim

arXiv:2609.37496v1cs.CV

TL;DR

Existing SAR-to-EO translation methods typically rely on separate models trained on limited target datasets, leaving a reusable cross-source prior as an open need. GeoSET pretrains a SAR-conditioned generator on a curated heterogeneous corpus with a speckle-robust SAR encoder, then adapts it across benchmarks. It achieves the best reported FID on all six benchmarks and DISTS on five, with LoRA updating 0.60% of generator parameters.

  • Problem

    Existing SET methods typically train separate dataset-specific models using only the SAR–EO pairs available in each target benchmark, leaving open whether heterogeneous sources can support a reusable prior.

    Source: §1 · p. 2

  • Method

    GeoSET pretrains a speckle-robust SAR encoder and SAR-conditioned flow generator on a filtered heterogeneous corpus, then adapts the shared parent to downstream datasets.

    Source: §1 · pp. 2–3

  • Results

    Across six benchmarks, GeoSET achieves the best reported FID on all six and the best DISTS on five, while LoRA remains competitive with full fine-tuning using 0.60% of generator parameters.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The results demonstrate the effectiveness of a reusable SET prior across heterogeneous SAR–EO domains, with LoRA enabling adaptation in approximately one hour per dataset.

    Source: §5 · p. 9, §4.4 · p. 9

Abstract

from arXiv · show

Paired synthetic aperture radar (SAR) and electro-optical (EO) imagery is increasingly available across sensors, resolutions, and geographic regions. Yet existing SAR-to-EO image translation (SET) methods are typically trained on a single, limited-scale dataset, producing models specialized to particular sensing conditions. We introduce GeoSET, the first generalist model for SET, built around a single pretrained parent that is adapted to downstream datasets under a common protocol. We curate over 3 million high-quality SAR--EO pairs from a collection of more than 10 million SAR observations, spanning diverse sensors, spatial resolutions, and ground sampling distances. To bridge the modality gap between SAR observations and a pretrained image generator, we develop a speckle-robust SAR encoder and pretrain the conditional generator on this heterogeneous corpus. The resulting parent supports efficient adaptation across downstream datasets through low-rank adaptation (LoRA), updating only 0.60% of the generator parameters and requiring approximately one hour per dataset. Across six downstream benchmarks, GeoSET achieves state-of-the-art results in FID and DISTS with full fine-tuning or LoRA, demonstrating effective transfer across heterogeneous SAR-EO domains.

1 INTRODUCTION

Existing SET methods typically train separate models on individual datasets, leaving open whether heterogeneous SAR–EO sources can support a reusable prior. GeoSET combines a curated multi-source corpus with speckle-robust SAR encoding and reports strong results across six downstream benchmarks.

1 INTRODUCTION

  • Existing SET methods typically train separate dataset-specific models, motivating the question of whether heterogeneous SAR–EO sources can support a reusable prior.SAR and EO capture different physical signals, so SET outputs are plausible SAR-conditioned visualizations rather than exact reconstructions of unobserved EO measurements.Source: §1 · p. 2
  • GeoSET pretrains once on over three million high-quality SAR–EO pairs selected from more than ten million SAR observations through source-aware filtering.A fixed downstream adaptation protocol lets each model inherit a shared cross-source prior rather than learning solely from its target dataset.Source: §1 · p. 2
  • Across six benchmarks, GeoSET achieves the best reported FID with full fine-tuning and LoRA, and the best DISTS on five benchmarks.LoRA remains competitive with full fine-tuning while updating only 0.6% of generator parameters.Source: §1 · p. 3
  • GeoSET is introduced as a generalist SET model under a pretrain-once, adapt-many framework, pretrained on a heterogeneous corpus curated through source-aware filtering.The paper describes this as the first generalist SET model under that framework, to the authors’ knowledge.Source: §1 · p. 3
  • GeoSET pairs speckle-robust SAR representation learning with multi-source pretraining of a SAR-conditioned flow model.The SAR encoder learns from speckle-perturbed inputs while the pretrained decoder remains fixed; the conditional generator uses a spatial SAR stream in place of text conditioning.Source: §1 · pp. 2–3

2 RELATED WORK

Conditional image generation has progressed from GAN-based translation to diffusion and flow-based approaches, while SET methods remain largely dataset-specific. GeoSET addresses this landscape with heterogeneous paired-data curation and a reusable SAR-conditioned generator.

2 RELATED WORK

  • Conditional translation methods developed from GANs and cycle consistency toward diffusion, spatial conditioning, latent diffusion, and flow matching.The cited approaches include coarse-to-fine generation and spatially adaptive normalization for synthesis and controllability.Source: §2 · p. 3
  • Recent SET methods use diffusion and bridge formulations, but typically train a separate model per target dataset without a shared prior learned from heterogeneous sources.Latent approaches often use natural-image-pretrained autoencoders for both SAR conditions and EO targets, which may not capture coherent radar statistics adequately without SAR-specific adaptation.Source: §2 · p. 3
  • GeoSET’s framework uses two pretraining stages—SAR encoder learning and SAR-conditioned generation—then adapts the parent to downstream benchmarks with LoRA or full fine-tuning.The SAR stream replaces the generator’s text condition and is initialized from its image stream.Source: Figure 3 · p. 4
  • Paired SAR–EO datasets span diverse regions, sensors, resolutions, and GSDs, but differ in registration, validity, crop density, and size, making direct pooling nontrivial.GeoSET consolidates sources using pair filtering and crop-equivalent source sampling.Source: §2 · pp. 3–4

3 GEOSET

GeoSET follows a pretrain-once, adapt-many framework: it trains a speckle-robust SAR encoder, pretrains a SAR-conditioned generator on a filtered multi-source corpus, then adapts the same parent to downstream benchmarks.

3.1 STAGE 1: SPECKLE-ROBUST SAR ENCODER

  • Stage 1 trains a SAR encoder to reconstruct the original observation from a speckle-perturbed input, keeping the pretrained decoder fixed to preserve compatibility with its latent interface.The objective uses only SAR images and requires neither paired EO images nor cross-modal alignment.Source: §3.1 · pp. 4–5
  • Domain-aware augmentation routes speckle perturbations according to each SAR source’s representation domain, with L ∈ {4, 8, 16} and variance 1/L for the sampled field.The augmentation is used for SAR conditioning during generalist pretraining and downstream adaptation, but disabled at inference.Source: §3.1 · p. 5

3.2 STAGE 2: GENERALIST SAR-TO-EO PRETRAINING

  • Stage 2 replaces the generator’s text-conditioning stream with a spatial SAR stream and trains it on the filtered multi-source corpus using conditional flow matching.The SAR stream is initialized from the pretrained image stream; the SAR and EO encoders remain frozen during generator pretraining.Source: §3.2 · p. 5
  • The corpus table reports drop rates relative to original SAR sample counts and crop-equivalent counts that determine the source sampling mixture.Crop-equivalent counts account for native image size and redundancy.Source: Table 1 · p. 6

3.3 STAGE 3: DOWNSTREAM ADAPTATION

  • Stage 3 adapts the same Stage 2 checkpoint to each downstream dataset through full fine-tuning or LoRA, while keeping the SAR and EO encoders and decoder frozen.LoRA freezes pretrained generator weights and adds low-rank updates to selected attention and MLP projections.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate GeoSET across heterogeneous training sources and six downstream benchmarks, comparing its full fine-tuning and LoRA variants with existing methods. The reported comparisons favor GeoSET on primary metrics, while ablations examine encoder reconstruction, adaptation efficiency, and component contributions.

4.1 DATASETS

  • Filtering 3,281,393 eligible SAR–EO pairs leaves 3,204,744 pairs for pretraining, alongside six downstream benchmarks spanning satellite and airborne platforms at 0.5–1.25 m GSD.The corpus combines multiple sources and filters out content, cloud, misregistration, compression, missing-tile, and no-data issues; two benchmarks are private.Source: §4.1 · pp. 6–7

4.3 COMPARISON WITH THE STATE OF THE ART

  • Across four public benchmarks, GeoSET achieves the best FID on all four and the best DISTS on three, while C-DiffSET retains the best DISTS on QXS-SAROPT.The comparison also reports gains in KID and DINO similarity, while competing methods often achieve stronger LPIPS, PSNR, or SSIM.Source: Table 2 · p. 7, Table 3 · p. 7, §4.3 · p. 8

4.2 IMPLEMENTATION DETAILS AND EVALUATION PROTOCOL

  • Baselines are retrained on the same training splits and evaluated on identical test pairs, with FID and DISTS as primary metrics and qualitative comparisons using identical SAR inputs.The qualitative protocol examines scene layout and local structures against paired EO references.Source: §4.2 · pp. 7–8, §4.3 · p. 8, Figure 5 · p. 8

4.4 ABLATION STUDIES AND ANALYSIS

  • Stage 1 encoder training increases average reconstruction PSNR by 3.54 dB on original SAR inputs and 5.23 dB on perturbed inputs.The larger gain under perturbation supports speckle robustness; the pretrained EO codec is preserved while representation adaptation focuses on SAR.Source: §4.4 · p. 9, Table 4 · p. 9
  • LoRA updates 0.6% of generator parameters and takes approximately one hour per dataset, reducing storage and peak training memory while remaining competitive with full fine-tuning.On SpaceNet6, LoRA achieves stronger FID, DISTS, and LPIPS under the evaluated settings.Source: §4.4 · p. 9, Table 4 · p. 9
  • Adding corpus pretraining reduces FID from 71.2 to 50.5 on SAR2EO and from 117.4 to 107.9 on SpaceNet6; SAR encoding and speckle augmentation add further gains across all metrics.The complete GeoSET (LoRA) configuration yields the best results in the cumulative comparisons.Source: §4.4 · p. 9, Table 5 · p. 9

5 CONCLUSION

GeoSET is presented as a reusable SAR-to-EO translation model pretrained on curated paired imagery and adapted to downstream datasets. Across six benchmarks it reports strong FID and DISTS results, with LoRA offering parameter-efficient adaptation.

5 CONCLUSION

  • GeoSET combines speckle-robust SAR encoding with generative pretraining on over three million curated SAR–EO pairs to learn a reusable SET prior.The paper presents it as the first generalist model for SET under a pretrain-once, adapt-many framework.Source: §5 · p. 9
  • Across six downstream benchmarks, GeoSET achieves state-of-the-art FID and DISTS results, while LoRA adapts in approximately one hour per dataset by updating 0.60% of generator parameters.The conclusion describes the shared prior's potential to advance SET beyond dataset-specific models.Source: §5 · p. 9

APPENDIX

The appendix collects additional comparisons, diagnostics, limitations, dataset and evaluation details, implementation descriptions, and metric definitions.

APPENDIX

  • Appendix sections cover additional quantitative and qualitative results, SAR conditioning diagnostics and limitations, pretraining sources, evaluation protocols, model components, and metrics.The overview organizes this material into sections A through D.Source: APPENDIX · p. 10, Table 6 · p. 10

A ADDITIONAL RESULTS AND DISCUSSIONS

Appendix analyses extend quantitative and qualitative comparisons and test whether generation uses SAR conditions and whether encoders preserve structure under speckle perturbation. The paper also limits its generalist claim: unseen domains do not consistently support strong zero-shot performance, and adaptation still requires target-domain paired data.

A ADDITIONAL RESULTS AND DISCUSSIONS

  • Additional examples on four benchmarks use identical SAR inputs and paired EO references to inspect scene layout, local structures, and texture consistency.The examples cover QXS-SAROPT, SAR2Opt, SAR2EO, and SpaceNet6.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 10
  • On pretraining sources, true SAR conditioning improves PSNR and LPIPS over shuffled and null conditions across all three sources.The diagnostic uses the Stage 2 parent after 500,000 updates and compares matched examples, integration steps, and random seed.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 10, Table 7 · p. 10
  • For speckle-perturbed SAR inputs (L = 4), reconstruction suppresses added perturbation and improves PSNR over the original SAR observation in every example.The reconstructions retain scene structures across diverse pretraining sources.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 10
  • GeoSET does not consistently achieve strong zero-shot performance across unseen SAR–EO domains; sensor, geometry, resolution, and radiometric-processing differences remain transfer challenges.The paper characterizes its generalist capability as a reusable prior for downstream adaptation rather than universal zero-shot translation.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 10
  • On CAP-BSG and KOMPSAT, GeoSET achieves the best FID, DISTS, KID, and DINO similarity on both datasets, while competing methods retain advantages in LPIPS, PSNR, and SSIM.Full fine-tuning provides the strongest FID and DISTS; LoRA remains competitive and matches its DINO similarity at reported precision.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 10, Table 8 · p. 11
  • LoRA adaptation still requires paired data from the target domain.This marks a boundary on the adaptation setting despite its practical efficiency.Source: A ADDITIONAL RESULTS AND DISCUSSIONS · p. 11

B DATASETS AND EVALUATION PROTOCOLS

The evaluation protocol documents the pretraining sources and downstream datasets, and evaluates competing methods on matched test pairs and crops.

B DATASETS AND EVALUATION PROTOCOLS

C IMPLEMENTATION DETAILS

The implementation combines representation-aware speckle augmentation with SAR-specific input paths and a pretrained multimodal generator; the section also describes its data streams and evaluation splits.

C IMPLEMENTATION DETAILS

  • Domain-aware speckle augmentation routes each SAR representation through an appropriate perturbation branch.TerraMesh uses the decibel branch, SARLO-80 the amplitude branch, and GUSO, SAR-1M, and both 3MOS streams the display branch.Source: C IMPLEMENTATION DETAILS · p. 11
  • For display imagery, augmentation uses an attenuated multiplicative perturbation with κ = 0.5, while unknown representations remain unchanged.Smaller L produces stronger perturbations and lower PSNR, while large-scale scene layout remains recognizable.Source: C IMPLEMENTATION DETAILS · p. 11
  • Channel-specific stems accommodate single-channel SAR and dual-polarization VV/VH inputs around shared encoder and decoder trunks.VV/VH inputs use a dedicated two-channel path that preserves separate polarization channels.Source: C IMPLEMENTATION DETAILS · p. 11
  • The pretraining corpus comprises five dataset families forming six streams, with 3MOS divided into mid- and high-resolution streams.Table 9 reports the streams’ spatial scales and tile sizes; display denotes rendered SAR imagery.Source: Table 9 · p. 12
  • The downstream protocol specifies dataset sizes and splits; SAR2EO uses 4,000 samples from the E3Diff test split and SAR2Opt uses a deterministic center crop.Source: Table 10 · p. 12
  • GeoSET uses FLUX.2 4B Base, with five double-stream and twenty single-stream blocks; both retain SAR and EO tokens for combined self-attention.Only EO tokens pass to the velocity head; the pretrained VAE encodes 32-channel latents at 1/8 resolution, packed to 128 channels at 1/16 resolution.Source: C IMPLEMENTATION DETAILS · p. 12

D EVALUATION METRICS

The evaluation uses seven complementary metrics, with FID and DISTS as primary measures, and includes qualitative comparisons and reconstruction visualizations.

D EVALUATION METRICS

  • FID and DISTS are the primary measures of distributional realism and paired perceptual fidelity, respectively.All methods use the same test pairs and spatial extents; lower or higher values are better as indicated for each metric.Source: D EVALUATION METRICS · p. 12
  • KID, DINO similarity, LPIPS, SSIM, and PSNR provide complementary distributional, satellite-feature, perceptual, structural, and radiometric assessments.SSIM is sensitive to residual misregistration, while PSNR measures pixelwise fidelity to the paired EO reference.Source: D EVALUATION METRICS · pp. 12–13
  • Figures 6–9 show qualitative comparisons on QXS-SAROPT, SAR2Opt, SAR2EO, and SpaceNet6 using identical SAR inputs and paired EO references.Source: Figure 6 · p. 14, Figure 7 · p. 15, Figure 8 · p. 16
  • Figure 10 compares SAR and EO reconstructions across pretraining sources, including reconstructions from speckle-perturbed SAR inputs at L = 4.Values indicate PSNR relative to the corresponding original observations.Source: Figure 10 · p. 18
  • Figure 11 compares original SAR inputs with perturbations at L = 16, 8, and 4; smaller L yields stronger perturbations.Values indicate PSNR relative to the original SAR observations.Source: Figure 11 · p. 18
Loading 2609.37496v1…