PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal learning for SAR target recognition using optical vision foundation models

Lucas Hirsch, James R. Hopgood, Javid Khan, Yoann Altmann, Mike E. Davies

arXiv:2609.07753v1cs.CVcs.LGeess.IV

TL;DR

SAR classification is hindered by speckle, limited labeled data, and the gap from optical imagery, while strict EO/SAR pairing is a strong assumption. The paper trains a SAR classifier to align with class-level prototypes from a frozen optical DINOv3 encoder, without using pair identities. On UNICORNv2, prototype alignment outperforms the evaluated baselines, suggesting optical class structure can transfer to SAR.

  • Problem

    SAR ATR is difficult because speckle, geometry-dependent returns, and limited labeled data complicate classification, while one-to-one EO/SAR pairing is a strong training assumption.

    Source: §1 · pp. 1–2

  • Method

    A frozen EO DINOv3 encoder builds class-level prototypes, and a SAR model is trained to classify images while aligning its embeddings to the corresponding prototype.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    EO prototype alignment achieves the highest mean top-1 accuracy and macro-F1 among the evaluated methods, including frozen DINOv3, SAR-only finetuning, and unpaired MMD alignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest that optical foundation models can provide class-level information transferable to SAR ATR without pair identities or optical imagery at inference.

    Source: §6 · p. 10

    The method still requires class labels in both modalities, and experiments use a dataset containing coincident EO/SAR chips despite not using pair identities.

    Source: §6 · p. 10

Abstract

from arXiv · show

Synthetic Aperture Radar (SAR) is an important modality in a wide range of imaging applications due to its versatile, long range and near all weather operating capabilities. However, Automatic Target Recognition (ATR) remains a challenging problem due to limited labelled data, the strong speckle in SAR images and the significant domain gap between SAR and more abundant optical imagery. In contrast, electro-optical (EO) imagery benefits from massive datasets, clearer visual structure and powerful foundation models. In this work, we investigate how vision foundation models trained on optical data can provide class level supervision for SAR classification. We propose a cross-modal EO to SAR prototype alignment framework in which a frozen EO encoder, based on a DINOv3 vision foundation model, is used to construct class level optical prototypes without requiring strict EO/SAR pairs. A SAR model is then trained to classify SAR images while aligning its embeddings to the corresponding EO class prototype. At inference time, the SAR model operates independently, without access to optical imagery. We evaluate our approach on the UNICORNv2 dataset, an EO and SAR dataset of civilian vehicles with heavily speckled images and severe class imbalance. EO prototype alignment improves SAR classification accuracy over frozen DINOv3, SAR only finetuning and unpaired distribution alignment baselines, and t-SNE visualizations provide qualitative evidence of clearer separation among classes in the trained SAR embedding space. These results suggest that optical vision foundation models, despite being trained on visible spectrum imagery, provide transferable information for SAR image classification, offering a practical method for using large scale pretrained vision foundation models across challenging sensing modalities.

1. INTRODUCTION

SAR target recognition is difficult because speckle, viewing-angle-dependent returns, and limited labeled data complicate classification, while optical imagery supports powerful foundation models. This section introduces class-level EO prototype alignment as a way to transfer optical class structure to SAR without requiring matched EO/SAR pairs.

1. INTRODUCTION

  • SAR ATR is challenged by speckle, geometry- and viewing-angle-dependent returns, and smaller, narrower labeled datasets than those available for optical systems.EO imagery offers more interpretable visual structure and large datasets that have supported powerful self-supervised vision foundation models.Source: §1 · p. 1
  • The motivating question is whether optical foundation-model features can transfer to SAR classification even when optical imagery is absent at inference.The introduction suggests that target shape and geometry provide shared structure across the modalities despite their different sensing mechanisms.Source: §1 · pp. 1–2
  • One-to-one EO/SAR sample correspondence is a strong assumption; labeled EO examples from the same classes may instead come from another time or archive.The proposed approach therefore uses class-level EO references rather than sample-level pairs.Source: §1 · p. 2
  • The framework averages frozen EO DINOv3 features by class and trains a SAR classifier to align its embeddings with the corresponding prototype.At inference, the EO branch is discarded and only the SAR classifier is retained.Source: §1 · p. 2
  • EO prototype alignment achieves the highest mean top-1 accuracy and macro-F1 among the evaluated methods, including frozen DINOv3, SAR-only finetuning, and unpaired MMD alignment.A synthetic-prototype control suggests the gain is not explained exclusively by latent-space regularization, but also reflects class structure transferred from EO features.Source: §1 · p. 2

2. RELATED WORK

SAR target recognition faces ambiguity from radar returns and limited minority-class representation. Prior cross-modal work uses paired EO/SAR data, while this paper uses class-level supervision without pair correspondences and complements direct optical-backbone finetuning with frozen EO reference features.

2.1 SAR target recognition

  • SAR returns vary with viewing angle, scattering, and processing, while visually distinct classes can have overlapping signatures, especially in small chips and imbalanced datasets.These challenges motivate transfer learning and information from other sensing modalities.Source: §2.1 · p. 2

2.2 Cross-modal SAR and EO learning

  • Prior aerial recognition work often uses paired EO/SAR representations; this method instead aggregates EO data into class prototypes, requiring shared labels but not matched observations.The cited MAVIC-C challenge uses coincident EO/SAR training data and SAR-only evaluation.Source: §2.2 · pp. 2–3

2.3 Foundation models and finetuning

  • Rather than using an optical foundation model only as a SAR finetuning backbone, the approach also uses a frozen EO encoder for class-level reference features.LoRA freezes the pretrained SAR backbone and adds trainable low-rank updates.Source: §2.3 · p. 3

3. METHOD

The method transfers class-level optical references into SAR training without requiring matched EO/SAR image pairs. At inference, prediction uses only the SAR branch.

3.1 Problem setting and overview

  • Labelled SAR and EO examples from the same classes train a SAR-only predictor, without requiring individual EO/SAR matches.The EO data is discarded at inference; the prototype approach assumes the EO examples represent relevant target appearances.Source: §3.1 · pp. 3–4

3.2 EO prototype construction

  • Frozen optical DINOv3 features are averaged within each EO class and renormalized to form unit-norm reference prototypes.Averaging over EO examples removes pair identities.Source: §3.2 · p. 4

3.3 SAR encoder and classification head

  • The SAR branch uses a separate DINOv3 encoder adapted with LoRA and a linear classifier trained with cross-entropy.The base SAR model weights remain frozen while only LoRA parameters are updated.Source: §3.3 · p. 4

3.4 Prototype alignment loss

  • The training objective combines cross-entropy and prototype alignment, which pulls each SAR embedding toward its class’s EO prototype.The coefficient λ controls the contribution of the two loss terms.Source: §3.4 · p. 5

3.5 Baselines

  • Baselines compare frozen DINOv3, SAR-only LoRA finetuning, and unpaired global MMD alignment against class-level EO prototype supervision.MMD aligns global feature distributions rather than individual samples or class prototypes.Source: §3.5 · pp. 5–6

4. EXPERIMENTAL SETUP

The experiments use UNICORNv2 vehicle imagery, with heavily imbalanced training classes and a balanced test set. Evaluation reports accuracy and macro-F1 across two class groupings, with results averaged over repeated runs.

4.1 Dataset

  • UNICORNv2 contains ten vehicle classes and approximately 450,000 images per modality; its training split has 1000:1 class imbalance while the test split is balanced.The EO and wide-area SAR data were collected from an aircraft over Dayton, Ohio.Source: Table 1 · p. 6, §4.1 · p. 6
  • Alongside the original ten-class evaluation, accuracy is also computed over seven classes by merging sedans, SUVs, pickup trucks, and vans.Models remain trained on ten classes; merging is applied only for the seven-class accuracy calculation.Source: §4.1 · p. 6

4.2 Implementation and evaluation details

  • The frozen EO encoder precomputes class prototypes, while the SAR DINOv3 encoder is adapted with LoRA and trained with a linear classifier.Both image modalities are resized to 128 × 128 and normalized using ImageNet statistics.Source: §4.2 · p. 6
  • Evaluation reports top-1 accuracy and macro-F1, with accuracy averaged over five runs and uncertainty reported as one standard deviation.Macro-F1 gives equal weight to each class, complementing overall accuracy under class imbalance.Source: §4.2 · p. 7

5. RESULTS

EO prototype alignment achieved the best reported performance in the ten-class evaluation and improved the seven-class evaluation over frozen DINOv3. Visualizations showed better-separated clusters, while class-level overlap and uneven recall gains remained.

5.1 Classification performance

  • 33.3% accuracy and 0.281 macro-F1: EO prototype alignment achieved the highest mean performance in the original ten-class evaluation.The result is averaged across five runs.Source: §5.1 · p. 7, Table 2 · p. 7
  • 52.2% accuracy versus 41.1%, and 0.384 macro-F1 versus 0.305: EO prototype alignment improved both metrics over frozen DINOv3 in the seven-class evaluation.The evaluation merges sedans, SUVs, pickup trucks and vans into the light vehicle category.Source: §5.1 · p. 7, Table 2 · p. 7
  • One possible explanation for prototype alignment outperforming MMD is its direct class-specific target: each SAR embedding is pulled toward a fixed EO reference, whereas MMD aligns distributions.Source: §5.1 · p. 7

5.2 Feature space visualization

  • The t-SNE projection shows more separated clusters after prototype alignment, but sedans, SUVs, vans and pickup trucks still overlap.Buses, box trucks and pickup trucks with trailers form clearer clusters; the overlap among light vehicles motivates the seven-class evaluation.Source: §5.2 · p. 8, Figure 3 · p. 8
  • Recall gains were uneven: box trucks, SUVs, buses, pickup trucks and vans improved, while sedan recall decreased and several truck and motorcycle classes remained difficult.The difficult classes include flatbed trucks, motorcycles and semi trucks with trailers.Source: §5.2 · p. 8, Table 3 · p. 9
  • The t-SNE plots use training embeddings while classification is evaluated on test data, so cluster gaps are qualitative rather than evidence of linear separability in the original feature space.The projection prioritizes local neighborhoods and can exaggerate gaps between clusters.Source: §5.2 · pp. 8–9

5.3 Is EO alignment just regularization?

  • EO prototypes preserve optical class geometry through pairwise angles, unlike synthetic prototypes that impose uniform separation without EO information.The reported angles include 6.5° between pickup truck and sedan, 6.8° between SUV and sedan, and 56.9° between flatbed truck and motorcycle.Source: Figure 4 · p. 9, §5.3 · p. 10

6. DISCUSSION AND FUTURE WORK

The discussion reports that EO prototype alignment improves SAR classification without using pair identities during training, while identifying limits in supervision, data scope, and current evaluation. It proposes further tests of cross-modal supervision and transfer to unseen classes.

Discussion

  • 52.2% accuracy with prototype alignment versus 41.1% with frozen DINOv3 in the seven-class evaluation.The discussion also reports qualitative evidence of better separation for some vehicle classes, with remaining overlap among sedans, SUVs, pickup trucks, and vans.Source: §6 · p. 10
  • Synthetic-prototype alignment performs close to simple finetuning and worse than EO-prototype alignment, suggesting gains involve information encoded in EO prototypes.Source: §6 · p. 10
  • The method uses class labels in both modalities, evaluation remains on coincident EO/SAR data, scores remain low, and experiments cover civilian vehicles only.The authors call for testing with EO samples disjoint from SAR training instances, locations, or acquisition events, and for validation beyond civilian vehicles.Source: §6 · p. 10

Future Work

  • A systematic comparison of paired and unpaired approaches could clarify which shared information and supervision forms are most useful for transfer.Reported MAVIC-C results are broadly comparable, but protocol differences prevent direct comparison and make the benefit of pairing difficult to isolate.Source: §6 · p. 10
  • Zero-shot transfer is a proposed extension in which EO examples represent a new class without SAR examples, requiring the model to infer its mapping into SAR feature space.Source: §6 · p. 10

7. CONCLUSIONS

The conclusion presents EO class prototypes as training references for a SAR classifier, with the final model using SAR imagery alone at inference. On UNICORNv2, prototype alignment outperforms the reported baselines, and synthetic-prototype controls suggest the gains relate to EO-encoded structure.

Conclusions

  • A frozen DINOv3 encoder averages EO features into class prototypes, while a LoRA-adapted SAR encoder learns classification and alignment to the corresponding prototype.The EO branch is not required at inference; the retained classifier operates on SAR imagery alone.Source: §7 · p. 11
  • On UNICORNv2, EO prototype alignment improves over frozen DINOv3, SAR-only LoRA finetuning, and unpaired MMD alignment.Source: §7 · p. 11
  • Synthetic-prototype controls suggest the gains are related to EO-prototype structure rather than latent-space regularization alone.The t-SNE visualization provides qualitative evidence of improved embedding separation, while overlap among several light-vehicle classes remains.Source: §7 · p. 11

APPENDIX A. SIMULATION DETAILS

The appendix records the training setup and method-specific hyperparameters used for the reported experiments.

Appendix A. Simulation Details

  • Method settings are summarized in Table 5, and the full configurations, implementation details, and code for Table 2 results are available on GitHub.Source: Appendix A · p. 11, Table 5 · p. 11
  • Experiments used pretrained DINOv3 ViT-S/16+ features of dimension 384, with EO and SAR images resized to 128×128 and normalized using ImageNet statistics.Training used batch size 256, AdamW, cosine learning-rate scheduling, inverse-frequency weighted sampling with replacement, and no additional augmentations.Source: Appendix A · p. 11
  • The frozen baseline trained only a linear head; other methods also trained LoRA adapters on each block’s query, key, and value projections.LoRA used rank r = 28, scaling α = 56, and dropout 0.05.Source: Appendix A · p. 11
  • EO prototype alignment used λ = 0.60, while MMD alignment used λ = 0.45 with RBF kernels in place of the prototype loss.Both λ values were selected by sweeping 0.1 to 0.8 in increments of 0.05; models trained for at most 25 epochs and retained the checkpoint with highest validation macro-F1.Source: Appendix A · p. 11
Loading 2609.07753v1…