PaperlayerView exact revision on arXiv

Source-linked AI summary

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Jeonghyeok Do, Munchurl Kim

arXiv:2610.09317v1cs.CV

TL;DR

SAR and EO provide complementary observations, but learning modality-specific detail alongside transferable semantics raises the question of where semantic supervision should be applied. SAREO-FM assigns semantic alignment to learnable queries and reconstruction to modality tokens within a shared encoder. Its representations transfer across unimodal and joint settings, with joint observations benefiting tasks where the sensors provide complementary information.

  • Problem

    SAR and EO differ in sensing mechanisms and visual statistics, creating a representation-learning challenge: preserving modality-specific details while learning transferable semantics without coupling semantic guidance directly to pixel reconstruction.

    Source: §1 · p. 2

  • Method

    SAREO-FM uses a shared encoder for SAR tokens, EO tokens, and learnable semantic queries, assigning VFM alignment to queries and masked reconstruction to modality tokens.

    Source: §1 · p. 2

  • Results

    SAREO-FM transfers effectively across SAR-only, EO-only, and multimodal downstream tasks, with joint observations benefiting tasks where sensors provide complementary task-relevant information.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The model supports strong transfer from either modality alone while exploiting complementary SAR–EO evidence when it is relevant to a downstream task.

    Source: §1 · p. 1, §5 · p. 9

    The model assumes paired SAR and EO observations are sufficiently registered for patch-wise interaction and teacher alignment; misregistration and other local mismatches can weaken correspondence.

    Source: G LIMITATIONS · p. 15

Abstract

from arXiv · show

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR--EO observations on tasks that benefit from complementary sensing.

1 INTRODUCTION

SAR and EO offer complementary observations, but their different sensing mechanisms make it challenging to preserve modality-specific details while learning transferable semantics. SAREO-FM separates semantic and reconstruction supervision across token streams while retaining their interaction in a shared encoder.

1 INTRODUCTION

  • SAR and EO provide complementary views: SAR supports day-and-night, weather-resilient observation, while EO captures rich appearance and fine-grained semantic cues.A practical SAR–EO foundation model should learn from either modality alone and exploit complementary evidence when both are available.Source: §1 · p. 1
  • Prior methods align image tokens with EO-derived VFM features, coupling semantic guidance with the same tokens’ pixel-reconstruction objective.The paper frames where semantic supervision should be applied as a question of avoiding direct constraints on modality-specific reconstruction.Source: §1 · p. 2
  • SAREO-FM assigns EO-VFM alignment to learnable semantic queries and masked pixel reconstruction to modality tokens, while all streams interact in a shared encoder.The queries are excluded from the reconstruction decoders.Source: §1 · p. 2
  • Mixed modality masking exposes the encoder to paired, SAR-only, and EO-only inputs; evaluations report strong unimodal transfer and task-dependent gains from joint observations.The model is pretrained on raw SAR–EO observations from the million-scale SAR-1M corpus.Source: §1 · p. 2

2 RELATED WORK

Related work uses masked modeling, multimodal geospatial pretraining, and semantic feature alignment to learn from heterogeneous signals. SAREO-FM distinguishes itself by assigning pixel reconstruction and frozen-VFM supervision to separate token streams within a shared latent space.

2.1 MASKED MULTIMODAL MODELING

  • SAREO-FM follows masked multimodal modeling but assigns pixel-reconstruction and frozen-VFM supervision to separate token streams.MultiMAE and 4M-family approaches demonstrate masked modeling across heterogeneous signals.Source: §2.1 · p. 2

2.2 GEOSPATIAL FOUNDATION MODELS (GFMS)

  • Geospatial foundation models address sensor heterogeneity through alignment, representation partitioning, or architectural adaptation; SAREO-FM separates explicit supervision destinations in a shared latent space.The related work includes CROMA, SatMAE, MMEarth, Copernicus-FM, and DeCUR.Source: §2.2 · p. 3

2.3 SEMANTIC FEATURE ALIGNMENT

  • Unlike SARMAE and CoDe-MAE, SAREO-FM directs frozen-VFM features only to persistent semantic-query outputs, while modality-token outputs support modality-specific reconstruction.SARMAE uses paired EO imagery for teacher-side supervision without encoding EO as a student input modality.Source: §2.3 · p. 3

3 SAREO-FM

SAREO-FM uses a shared encoder to process SAR tokens, EO tokens, and semantic queries, assigning masked reconstruction and VFM alignment to separate streams. Its masking modes support paired and single-modality inputs, and the encoder transfers to SAR, EO, or joint observations.

3.1 OVERVIEW

  • The shared encoder processes SAR tokens, EO tokens, and semantic queries; queries aggregate cross-modal context and align with frozen-VFM patch features, while modality tokens feed separate reconstruction decoders.After pretraining, the VFM teacher and reconstruction decoders are discarded.Source: §3.1 · pp. 3–4

3.2 UNIFIED THREE-STREAM SAREO ENCODER

  • SAR and EO image tokens are patchified and jointly encoded with spatially organized learnable queries, while unavailable or withheld modality token sequences are omitted.The queries are never masked and interact with visible image tokens through joint encoding.Source: §3.2 · p. 4

3.3 DECOUPLED SEMANTIC SUPERVISION

  • VFM alignment supervises the semantic-query stream, leaving modality tokens unconstrained by direct EO-VFM targets while all streams continue exchanging information in the encoder.Contextualized queries are projected to the teacher feature space and aligned patch-wise using cosine distance.Source: §3.3 · pp. 4–5

3.4 MODALITY-SPECIFIC MASKED RECONSTRUCTION

  • Each modality has a reconstruction decoder that restores visible tokens to their spatial locations, inserts mask tokens, and evaluates pixel loss only at masked locations.Semantic-query latents are discarded before decoding, and the decoder predicts all image patches.Source: §3.4 · p. 5

3.5 MODALITY MASKING STRATEGY

  • Four masking modes combine independently or jointly masked SAR–EO inputs with SAR-only or EO-only inputs, supporting cross-modal interaction and single-modality use.Independent masking permits aligned visible cues; shared masking hides corresponding locations in both modalities.Source: §3.5 · p. 5

3.6 PRETRAINING OBJECTIVE

  • The objective combines modality-specific pixel reconstruction and semantic alignment, with unavailable or excluded supervision coefficients set to zero.Paired inputs with one modality masked use that modality’s reconstruction loss plus alignment; unpaired samples contribute only their present modality’s reconstruction loss.Source: §3.6 · p. 6

3.7 DOWNSTREAM TRANSFER

  • At transfer, the encoder accepts unmasked SAR, EO, or paired inputs without modality-specific backbones or separate inference pipelines, and task features can come from modality tokens, queries, or both.Modality tokens retain sensing-specific spatial detail, while query outputs provide contextually aggregated semantic features.Source: §3.7 · p. 6

4 EXPERIMENTS

Experiments assess in-corpus and out-of-corpus SAR transfer, EO-only transfer, and paired SAR–EO tasks under few-shot and higher-supervision settings. Results show label-efficient SAR gains, EO transfer beyond the frozen teacher, task-dependent benefits from paired inputs, and cumulative gains from the ablated components.

4.1 DATASETS AND DOWNSTREAM TASKS

  • Evaluation spans 18 SAR-1M source datasets, SAR classification, EO recognition, paired-image recognition, and qualitative BRIGHT dense transfer.SAR-ACD is the primary out-of-corpus SAR benchmark; MSTAR and FUSAR-Ship overlap with the pretraining corpus.Source: §4.1 · p. 6

4.2 IMPLEMENTATION DETAILS

  • Pretraining uses a ViT-B/16 encoder, 256 semantic queries, frozen DINOv3-7B guidance, and mixed modality masking; SARMAE∗ provides the matched controlled baseline.The model is trained for 300,000 steps, and the controlled comparison uses identical downstream protocols and SAR-1M SAR imagery.Source: §4.2 · pp. 6–7

4.3 SAR REPRESENTATION TRANSFER

  • 78.40% accuracy on 40-shot SAR-ACD, up from 68.50% with semantic queries (+9.90 points), while SARMAE∗ (Liu et al. (2025)) leads by 1.67 points under 30% supervision.In frozen transfer, SAREO-FM becomes stronger than SARMAE∗ from 5-shot on SAR-ACD and from 2-shot on FUSAR-Ship.Source: Table 1 · p. 7, §4.3 · pp. 7–8, §4.2 · p. 7

4.4 EO REPRESENTATION TRANSFER

  • EO-input SAREO-FM consistently surpasses frozen DINOv3 from 1 to 40 shots, while SAR-input EuroSAT transfer beats SARMAE∗ at every shot count.The EO-input gains are largest in the most label-scarce regimes and narrow as labels increase.Source: Table 2 · p. 8, §4.4 · p. 8

4.5 JOINT SAR–EO TRANSFER

  • 66.70 micro-AP on 5-shot BigEarthNet-MM for joint SAR–EO input; on BRIGHT, fusion recovers destroyed-building regions missed by single-modality models.The BRIGHT examples are qualitative evidence, not a full segmentation benchmark.Source: Table 3 · p. 8, §4.5 · pp. 8–9, Figure 5 · p. 9

4.6 ABLATION STUDY

  • 9.90 points on average over plain MAE in 10-shot SAR linear probing, with semantic guidance adding 5.39, decoupling 3.13, and raw paired EO 1.38 points cumulatively.The full model achieves the best result on both ablation benchmarks.Source: Table 4 · p. 9, §4.6 · p. 9

5 CONCLUSION

SAREO-FM separates EO-VFM supervision and masked reconstruction across semantic queries and modality tokens while preserving their interaction in a shared encoder. Its representations transfer across SAR-only, EO-only, and multimodal tasks, with joint observations benefiting tasks where sensor information is complementary.

5 CONCLUSION

  • SAREO-FM assigns EO-VFM supervision to semantic queries and masked reconstruction to modality tokens, separating explicit supervision while preserving interaction in a shared encoder.The conclusion reports transfer across SAR-only, EO-only, and multimodal downstream tasks, with joint observations benefiting tasks where the sensors offer complementary task-relevant information.Source: §5 · p. 9
  • Future work will extend the framework to additional sensing modalities and dense prediction tasks.Source: §5 · p. 9

APPENDIX

  • The appendix anchors additional analyses to main-paper results and documents implementation, evaluation, corpus, readout, deployment-cost, reproducibility, and empirical-scope details.Source: APPENDIX · p. 10

A.1 EXACT FROZEN FEW-SHOT SAR RESULTS

Frozen few-shot SAR results show SAREO-FM's strongest readout above SARMAE∗ at every evaluated FUSAR-Ship shot budget, while SAR-ACD results favor SARMAE∗ in the sparsest settings. The appendix also specifies the image-token and semantic-query sequence composition.

A.1 EXACT FROZEN FEW-SHOT SAR RESULTS

  • 1.60, 3.59, 5.71, 3.47, 2.06, and 1.46 points: SAREO-FM's strongest readout exceeds SARMAE∗ on in-corpus FUSAR-Ship at 1–40 shots.The largest separation is at 5 shots; Table 6 reports these numerical comparisons without inferring statistical significance.Source: §A.1 · p. 10
  • On out-of-corpus SAR-ACD, SARMAE∗ is stronger at 1 and 2 shots; the semantic-query readout crosses over at 5 shots and leads through 40 shots.Across twelve dataset–shot combinations, semantic-query features are the stronger SAREO-FM readout in ten.Source: §A.1 · p. 10

B.1 ENCODER SEQUENCE COMPOSITION

  • At 256×256 resolution, each modality has 256 image tokens; a 75% mask leaves 64 visible tokens per retained modality, while all 256 semantic queries remain active.The passage states that Table 7 reports the resulting encoder sequence lengths but does not supply those lengths here.Source: §B.1 · p. 10

B.2 MASKING MODES AND ACTIVE LOSSES

The appendix clarifies how masking, modality dropping, reconstruction, and semantic alignment operate in training, and documents corpus composition and benchmark interpretation. It also qualifies which transfer claims are supported by the corpus overlap.

B.2 MASKING MODES AND ACTIVE LOSSES

  • Retained image streams are masked by 75%, while a dropped stream is omitted from the student encoder and receives no pixel loss; all semantic queries remain active.When EO is dropped from a paired input, clean EO remains available to the frozen teacher for query targets. Configured sampling probabilities are 0.35/0.35/0.15/0.15; recorded frequencies closely match them.Source: §B.2 · pp. 10–11, Table 8 · p. 12

B.3 ONE PRETRAINING ITERATION

  • Each iteration samples paired SAR–EO or unpaired SAR, applies masking, encodes image tokens plus queries, reconstructs retained streams, and aligns query outputs to frozen-teacher features.Source: §B.3 · pp. 11–12

C.1 PRETRAINING CORPUS COMPOSITION

  • SAR-1M contains 1,312,902 SAR observations from 18 public datasets; 1,042,156 have geographically corresponding EO, and 270,746 are unpaired SAR.The corpus spans 57 target and scene categories, multiple SAR sensors and polarization modes, and ground resolutions from 0.1 to 60 m.Source: §C.1 · p. 12

C.2 DOWNSTREAM BENCHMARK SUMMARY

  • “In corpus” denotes SAR-1M source-image membership, not label leakage; the paired EuroSAT protocol adds geospatially matched SAR to standard EO-only imagery.Source: §C.2 · p. 12, Table 9 · p. 12

C.3 CORPUS MEMBERSHIP AND GEOGRAPHIC OVERLAP

  • Because BRIGHT contributes SAR-1M imagery, Figure 5 is an in-corpus qualitative dense probe, not evidence of unseen-event or unseen-scene generalization.The probe illustrates a dense prediction head using complementary pre-event EO and post-event SAR evidence.Source: §C.3 · pp. 12–13
  • Out-of-corpus claims require checking source, geographic, parent-scene, and event identifiers in addition to file hashes.Source: §C.3 · p. 13

C.4 PAIRED-DATASET SPLITS

The paired-dataset protocols constrain geographic overlap and specify how support samples, labels, and evaluation partitions are handled.

C.4 PAIRED-DATASET SPLITS

  • So2Sat LCZ42 uses a city-disjoint split; few-shot draws share class-balanced support identifiers across compared modes and readouts.Source: §C.4 · p. 13
  • BigEarthNet-MM uses 519,284 paired observations, 19 mapped classes, and micro-AP; support sampling stays within training, validation selects models, and test remains fixed.The passage gives train/validation/test counts of 269,695/123,723/125,866.Source: §C.4 · p. 13
  • The EuroSAT “SAR input” uses a geospatially paired SAR companion, with channel ordering, matching key, and paired-location split recorded in the manifest.Source: §C.4 · p. 13

D.1 ARCHITECTURE SPECIFICATION

SAREO-FM uses a shared encoder with modality-specific image streams and spatially organized semantic queries, while separating pretraining-only components from downstream readouts. The section specifies its reconstruction and semantic targets, feature pooling, and inference profiling scope.

D.1 ARCHITECTURE SPECIFICATION

  • The shared encoder uses 12 Transformer blocks, 768 dimensions, 12 attention heads, and separate SAR and EO patch projections; 256 spatial queries align with image patch locations.SAR and EO share a 16 × 16 positional grid, while learnable stream embeddings distinguish each stream.Source: §D.1 · p. 13

D.2 COMPONENT LIFECYCLE

  • The DINOv3 teacher, semantic projector, and reconstruction decoders are removed after pretraining, so the teacher adds no downstream inference parameters or computation.Source: §D.2 · p. 14

D.3 PIXEL AND SEMANTIC TARGETS

  • Masked pixel losses apply only to hidden patches, while all 256 query locations receive cosine alignment to corresponding teacher patches, including hidden locations.The design encourages queries to infer spatial semantics from visible evidence and context rather than copy teacher features from identically visible patches.Source: §D.3 · p. 14

E.1 FEATURE READOUTS

  • Downstream readouts pool each available modality separately before concatenation; the modality readout is 768-D for unimodal input and 1,536-D for paired input, while the semantic readout is 768-D.The combined readout is 1,536-D for unimodal input and 2,304-D for paired input.Source: §E.1 · p. 14, Table 12 · p. 14
  • The combined readout is treated as complementary because its linear head has more parameters than either modality-only or semantic-only heads.For K classes, the respective head sizes are K(768|A|+1), K(768+1), and K(768(|A|+1)+1).Source: §E.1 · p. 15

F INFERENCE COST

  • Unimodal input uses 512 encoder tokens and joint input 768; latency, throughput, memory, and FLOPs are measured under a fixed profiling protocol rather than inferred from token counts.The protocol uses 256 × 256 inputs, inference-only execution, fixed warm-up and measurement iterations, synchronization, and identical precision across modes.Source: F INFERENCE COST · p. 15

G LIMITATIONS

The evaluation and alignment setup impose scope boundaries: quantitative tests focus on image-level recognition, while dense-head examples are qualitative and paired observations assume sufficient registration.

G LIMITATIONS

  • Quantitative evaluation covers image-level classification and multi-label recognition; BRIGHT dense-head examples are qualitative, in-corpus diagnostics, not a complete segmentation benchmark.Source: G LIMITATIONS · p. 15
  • Paired SAR and EO observations are assumed sufficiently registered for patch-wise interaction and teacher alignment; misregistration and acquisition differences can weaken local correspondence.The passage also names clouds, SAR layover, and sensor-dependent resolution as potential sources of weaker correspondence.Source: G LIMITATIONS · p. 15
  • Because semantic targets come from an EO-pretrained VFM, semantics poorly represented by that teacher may transfer less effectively, despite SAR reconstruction preserving sensing-specific information.Source: G LIMITATIONS · p. 15
Loading 2610.09317v1…