PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment

Angelos Zavras, Dimitrios Michail, Begüm Demir, Ioannis Papoutsis

arXiv:2402.09816v2cs.CV

TL;DR

Remote sensing presents CLIP with distribution shift and complementary non-RGB modalities, while suitable paired imagery and text are scarce. The paper patches CLIP and distills its visual and textual knowledge to align a remote-sensing encoder, reporting gains across classification and retrieval benchmarks.

  • Problem

    CLIP’s zero-shot performance falls short in remote sensing, where imagery differs from natural images, useful modalities extend beyond RGB, and suitable image-text pairs are scarce.

    Source: §1 · p. 2

  • Method

    The method patches CLIP using remote-sensing RGB composites, then distills knowledge from CLIP’s visual and text encoders to align a remote-sensing modality encoder.

    Source: §1 · pp. 2–3, §4.2 · p. 9

  • Results

    Both robust fine-tuning and cross-modal alignment yield performance gains across several remote-sensing classification benchmarks, and the enhanced model improves zero-shot classification and cross-modal retrieval.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approach uses CLIP’s large-scale pretraining to build a shared embedding space for remote-sensing modalities without text descriptions, task-specific parameters, or training from scratch.

    Source: §1 · p. 3, §5 · p. 14

    The authors report that EuroSAT performance is lacking, attributing this in part to aligning the self-supervised encoder only on BigEarthNet with a multi-label label-loss objective.

    Source: §5.1 · p. 15

Abstract

from arXiv · show

Deep Learning (DL) is undergoing a paradigm shift with the emergence of foundation models. In this work, we focus on Contrastive Language-Image Pre-training (CLIP), a Vision-Language foundation model that achieves high accuracy across various image classification tasks and often rivals fully supervised baselines, despite not being explicitly trained for those tasks. Nevertheless, there are still domains where zero-shot CLIP performance is far from optimal, such as Remote Sensing (RS) and medical imagery. These domains do not only exhibit fundamentally different distributions compared to natural images, but also commonly rely on complementary modalities, beyond RGB, to derive meaningful insights. To this end, we propose a methodology to align distinct RS image modalities with the visual and textual modalities of CLIP. Our two-stage procedure addresses the aforementioned distribution shift, extends the zero-shot capabilities of CLIP and enriches CLIP's shared embedding space with domain-specific knowledge. Initially, we robustly fine-tune CLIP according to the PAINT (Ilharco et al., 2022) patching protocol, in order to deal with the distribution shift. Building upon this foundation, we facilitate the cross-modal alignment of a RS modality encoder by distilling knowledge from the CLIP visual and textual encoders. We empirically show that both patching and cross-modal alignment translate to significant performance gains, across several RS imagery classification and cross-modal retrieval benchmark datasets. Notably, these enhancements are achieved without the reliance on textual descriptions, without introducing any task-specific parameters, without training from scratch and without catastrophic forgetting. We make our code implementation and weights for all experiments publicly available at https://github.com/Orion-AI-Lab/MindTheModalityGap.

1. Introduction

The introduction frames remote sensing as a domain where CLIP faces distribution shift, RGB-only information constraints, and scarce suitable image-text pairs. It presents a two-stage approach to adapt CLIP and align other remote-sensing modalities, reporting gains across classification and retrieval benchmarks.

1. Introduction

  • CLIP’s zero-shot performance is limited in remote sensing, where distribution shift, reliance on RGB, and scarce suitable image-text pairs constrain its use.Existing paired datasets are often commercial, very-high-resolution, and RGB-only, unlike free and open-access multispectral satellite data.Source: §1 · p. 2
  • The proposed two-stage method patches CLIP on remote-sensing RGB composites, then distills knowledge from CLIP’s visual and text encoders to align a remote-sensing encoder.The patching stage aims to address distribution shift without reducing natural-image zero-shot performance; the alignment stage adds domain-specific knowledge to CLIP’s shared embedding space.Source: §1 · pp. 2–3
  • The method aligns remote-sensing modalities without textual descriptions, task-specific parameters, training from scratch, or catastrophic forgetting.The authors state that this adaptation does not compromise CLIP’s performance in other domains or alter its shared embedding-space characteristics.Source: §1 · p. 3
  • The evaluation benchmarks patched and aligned models on remote-sensing classification and cross-modal retrieval, compares related CLIP adaptations, and probes representations with linear classifiers.The authors describe the evaluation as establishing new performance baselines for these tasks.Source: §1 · p. 3
  • The enhanced CLIP model improves zero-shot remote-sensing image classification and cross-modal retrieval by capturing remote-sensing semantics and using complementary information beyond RGB.The authors report that both robust fine-tuning and cross-modal alignment yield gains across several remote-sensing classification benchmarks.Source: §1 · p. 3

2. Related Work

Related work traces CLIP’s broad transfer potential and the challenge of adapting it to remote sensing with limited paired data. It distinguishes this work by aligning complementary remote-sensing modalities with CLIP’s image and text encoders.

2.1. Contrastive Language-Image Pre-training

  • CLIP’s large-scale pretraining and performance across downstream tasks motivate benchmarking newer pretrained models for remote sensing.The discussion describes advances in pretraining data filtering, architecture, and computational efficiency as reasons to assess these models for remote-sensing applications.Source: §2.1 · p. 3

2.2. Domain Specialized CLIP Models

  • Remote-sensing CLIP adaptation is constrained by scarce in-domain image-text pairs, and prior approaches use limited or synthetic data, data-efficient adaptation, and self-supervision.The authors position their approach as exploiting labeled remote-sensing imagery to address domain shift without task-specific parameters or catastrophic forgetting.Source: §2.2 · p. 4

2.3. Multi-modal CLIP-inspired models in Remote Sensing

  • Prior remote-sensing work has aligned multiple sensor modalities or geographic and audio information, while this work aligns multispectral Sentinel-2 imagery with CLIP’s image and text encoders.The authors present this alignment as a blueprint for other remote-sensing modalities and cross-modal retrieval and text-based zero-shot tasks.Source: §2.3 · p. 5

3. Method

The method first patches CLIP for satellite imagery, then aligns a pretrained remote-sensing modality encoder with CLIP’s visual and textual embeddings. RGB composites and labels anchor a shared space for associating remote-sensing imagery, RGB, and text.

3. Method

  • The method uses RGB composites and labels as anchors to align remote-sensing modalities in a shared embedding space for multimodal downstream tasks.The mapping brings each remote-sensing image, its RGB composite, and its label into that space.Source: §3 · p. 5

3.1. First Stage: Patching CLIP by interpolating weights

  • PAINT fine-tunes CLIP’s image encoder on a representative remote-sensing classification task, then interpolates its weights with the original weights to retain performance on supported tasks.The mixing coefficient is selected using held-out validation sets for both supported and patching tasks.Source: §3.1 · p. 6

3.2. Second Stage: Aligning RS imagery modalities with natural images and text

  • A pretrained satellite modality encoder is aligned to the patched CLIP image encoder using a joint objective that combines embedding-level mean squared error and label-supervised cross-entropy.The MSE encourages the student to mimic teacher embeddings; the cross-entropy uses CLIP’s frozen text encoder output as a classification anchor.Source: §3.2 · pp. 6–7

4. Experiments

The experiments evaluate CLIP patching and satellite-modality alignment on remote-sensing classification and retrieval benchmarks, including their effects on performance and supported tasks. Ablations examine dataset choice, alignment-loss variants, use of full Sentinel-2 multispectral input versus RGB, and transfer to Sentinel-1.

4.1. Datasets

  • The benchmark suite comprises three remote-sensing classification datasets with five nomenclatures, reflecting differences in satellite data and label schemes.The datasets are BigEarthNet, EuroSAT, and SEN12MS.Source: §4.1 · pp. 7–8

4.2. Experimental Setup

  • For alignment, a trainable Sentinel-2 satellite encoder learns from a frozen CLIP teacher using paired multispectral data and RGB composites, with CLIP’s text tower retained as the frozen classifier.The alignment stage uses a batch size of 256 for 5 epochs and sets λ = 0.05 in the loss.Source: §4.2 · p. 9

4.3. Results

  • MS→RGB retrieval outperforms RGB→MS across all tested models and datasets, while larger models generally perform better and BigEarthNet yields the highest scores.The retrieval evaluation covers BigEarthNet, EuroSAT, and SEN12MS using Recall@K metrics.Source: Table 2 · p. 11
  • Patched CLIP achieves the highest scores on 5 of 6 satellite classification datasets while remaining competitive on ImageNet.The comparison covers the original CLIP and RS-specialized CLIP variants using their published prompts.Source: §4.3 · p. 11, Table 3 · p. 12
  • Cross-modal alignment improves zero-shot classification on all evaluated multispectral datasets except EuroSAT, and the aligned encoder excels on nearly every linear-probing dataset.The linear-probing models were trained for 20 epochs.Source: §4.3 · p. 11, Figure 6 · p. 11, Table 4 · p. 12

4.4. Ablation Study

  • Patching with BigEarthNet variants outperforms EuroSAT and SEN12MS patching on supported and patching tasks; EuroSAT patching improves only EuroSAT, while SEN12MS patching mainly benefits itself.The authors associate BigEarthNet’s stronger results with its larger size and the relationship between class variety and generalization, while noting class imbalance and inconclusive evidence about multi-label versus multi-class effects.Source: §4.4 · p. 12, Figure 7 · p. 12
  • PAINT patching preserves ImageNet accuracy near the original CLIP baseline while improving remote-sensing performance, unlike fine-tuning, which reduces ImageNet accuracy across all model variants.Patching also generalizes across datasets better than fine-tuning, which the authors describe as prone to forgetting and overfitting to the adaptation task.Source: §4.4 · p. 13, Table 5 · p. 13
  • Using patched CLIP as the alignment teacher improves satellite-encoder classification and retrieval over using original CLIP weights, and the proposed loss performs best on both tasks.The comparison evaluates cross-entropy, mean-squared-error, and the proposed combined loss.Source: §4.4 · p. 13
  • Using CE tends to align the satellite encoder toward CLIP’s text encoder, whereas MSE tends to align it toward CLIP’s image encoder, disrupting residual-modality alignment.The passage describes this tendency as a consequence of the alignment loss choice.Source: §4.4 · p. 13
  • Aligning Sentinel-2 with all 13 bands significantly outperforms RGB-only alignment, whose loss plateaus after two epochs while the all-bands loss continues to decrease.The ablation reuses the same pretrained weights, retaining only RGB-channel weights for the RGB-only encoder and using all 13 input channels for the full-band encoder.Source: §4.4 · p. 13
  • For classification, CE outperforms MSE with and without patching, and patching generally improves results, particularly with combined MSE and CE loss.Table 6 reports the BigEarthNet-19 classification ablation.Source: Table 6 · p. 14
  • For cross-modal retrieval, patching generally improves performance, and MSE shows particular effectiveness across the reported retrieval settings.Table 7 reports R@10 for RGB→MS and MS→RGB retrieval on BigEarthNet, EuroSAT, and SEN12MS.Source: Table 7 · p. 14
  • Aligning Sentinel-1 after patching with Sentinel-1 data consistently outperforms the other tested approaches, while patching with Sentinel-2 data also beats aligning without patching.The experiments use false-color RGB composites and prompts adapted to describe them; EuroSAT is excluded because it has no Sentinel-1 imagery.Source: §4.4 · p. 14

5. Discussion and Conclusion

The method targets a joint embedding space for distinct remote-sensing modalities, using CLIP without text descriptions, task-specific parameters, training from scratch, or catastrophic forgetting. The discussion identifies limits in generalization and benchmarking, alongside possible refinements to patching and alignment.

5. Discussion and Conclusion

  • The method targets a joint embedding space for distinct RS modalities and provides preliminary baselines for cross-modal retrieval and text-based zero-shot tasks.The authors describe it as computationally efficient and as avoiding text descriptions, task-specific parameters, training from scratch, and catastrophic forgetting.Source: §5 · p. 14

5.1. A blueprint towards satellite vision-language models

  • Generalizability and comparison with established standards and benchmarks remain major challenges for the proposed RS VLM.The authors frame their methodology as a blueprint for resource-efficient development.Source: §5.1 · p. 14
  • A single proxy dataset may not cover RS variation in classification objectives, spatial coverage, atmospheric conditions, perspectives, and resolution.The authors suggest multi-dataset patching as a possible remedy, with expected improvements to patched models and subsequent cross-modal alignment.Source: §5.1 · pp. 14–15
  • EuroSAT performance is limited by aligning the satellite encoder using BigEarthNet alone and its multi-label loss objective; encoder capacity may also constrain transfer from larger CLIP models.The authors suggest adapting alignment to multiple datasets and potentially using a larger pre-trained satellite encoder.Source: §5.1 · p. 15
  • The authors point to EO benchmark suites as a way to support reliable assessment against established standards and streamline future EO applications.Such suites could provide ranked candidate CLIP models.Source: §5.1 · p. 15

5.2. Bridging modalities with distinct intrinsic features

  • The alignment loss combines MSE and CE to address the information gap between heterogeneous modalities, guiding the RS encoder toward teacher embeddings and shared-space alignment.The MSE term constrains the student to produce embeddings similar to the teacher's.Source: §5.2 · p. 15

Appendix A. Patched models zero-shot performance

The appendix reports zero-shot results for patched CLIP ViT models, using BigEarthNet-19 as the proxy task and ImageNet as a supported task.

Appendix A. Patched models zero-shot performance

  • The appendix reports zero-shot performance for four patched CLIP ViT architectures across satellite scene classification datasets and ImageNet.ImageNet serves as the representative supported task, while BigEarthNet-19 is the patching proxy task.Source: Appendix A · p. 16
  • The results compare different α weight interpolation coefficients across the four CLIP ViT models.The appendix directs readers to Table A.10.Source: Appendix A · p. 16
  • The section reports patching results for all four CLIP ViT-based models on BigEarthNet-19 and ImageNet across different α coefficients, and zero-shot performance of patched models across diverse satellite scene-classification nomenclatures and datasets.Source: Appendix A · p. 16

Appendix B. Per-class performance

The appendix presents per-class performance for the best CLIP-L-14 model before and after BigEarthNet-19 patching, alongside the aligned ViT-S-16. Its figures cover several satellite datasets and report improvements in zero-shot classification.

Appendix B. Per-class performance

  • The appendix plots per-class results for the best CLIP-L-14 model before and after BigEarthNet-19 patching, alongside the aligned ViT-S-16.The results span BigEarthNet-5, BigEarthNet-19, BigEarthNet-43, SEN12MS, and EuroSAT.Source: Appendix B · p. 16
  • The figures report significant zero-shot mAP improvements through patching and alignment for satellite scene classification on SEN12MS and BigEarthNet.BigEarthNet figures include comparisons involving the 5-, 19-, and 43-class nomenclatures.Source: Figure B.8 · p. 16, Figure B.10 · p. 17, Figure B.11 · p. 18
  • The EuroSAT figure reports significant zero-shot Accuracy improvements through patching and alignment.EuroSAT uses a 10-class nomenclature.Source: Figure B.9 · p. 17

Related papers

Loading 2402.09816v2…