PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data

Xiangyang Miao, Kelu Yao, Yekai Huang, Xiaogang Xu, Junxiao Xue, Minjun Shen, Chenghui Lv, Shanji Liu, Yaying Chen, Chao Li

arXiv:2609.03391v1cs.CVcs.AI

TL;DR

Existing remote sensing contrastive learning methods are largely tied to RGB-oriented CLIP interfaces, limiting unified modeling of heterogeneous sensors. OmniRSCLIP addresses this with SSBD, spectral-context-aware feature selection, and the OmniRS5M corpus, while experiments report preserved RGB performance and extension to heterogeneous modalities.

  • Problem

    Existing remote sensing contrastive learning frameworks remain centered on RGB-oriented CLIP architectures, making heterogeneous sensor inputs difficult to exploit in a unified manner.

  • Method

    OmniRSCLIP adapts arbitrary-channel inputs through SSBD, combines spectral-context-aware feature selection with CLIP-based alignment, and trains on the multi-modal OmniRS5M corpus.

  • Results

    Experiments on retrieval, zero-shot classification, semantic localization, and ablations show that OmniRSCLIP extends CLIP beyond RGB imagery while preserving strong semantic transferability.

  • Takeaways & Limitations

    A single CLIP-based model can align RGB, SAR, MSI, and HSI observations within a shared image-text semantic space.

Abstract

from arXiv · show

Contrastive language-image learning (CLIP) has become a key paradigm for remote sensing vision-language understanding. However, existing remote sensing contrastive learning methods are mostly built on RGB-oriented CLIP architectures, making it difficult to exploit heterogeneous sensors such as SAR, multi-spectral imaging (MSI), and hyperspectral imaging (HSI). To address this limitation, we propose OmniRSCLIP, an end-to-end contrastive learning framework that supports multi-source sensor inputs for remote sensing vision-language modeling. The key idea is to extend CLIP beyond its fixed RGB input interface without breaking the pretrained visual knowledge. To this end, OmniRSCLIP introduces Spectral-Spatial Basis Decomposition (SSBD), which formulates arbitrary-channel adaptation as a basis recomposition problem: pretrained CLIP patch embeddings provide transferable spatial bases, while wavelength-conditioned coefficients span sensor-specific embedding kernels within a constrained visual prior space. This design avoids forcing heterogeneous sensors into a fixed-channel input space, while aligning them in a unified image-text semantic space. We further introduce a spectral-context-aware mask-based contrastive learning scheme to suppress modality-specific redundant features and enhance fine-grained image-text alignment. Finally, to support multi-modal training, we construct OmniRS5M, the first large-scale remote sensing image-text corpus covering RGB, SAR, MSI, and HSI. Experiments on retrieval, zero-shot classification, and semantic localization show that OmniRSCLIP preserves strong RGB-domain performance while effectively extending CLIP to heterogeneous remote sensing modalities.

Introduction

Remote sensing contrastive learning remains constrained by RGB-oriented CLIP interfaces, motivating OmniRSCLIP and OmniRS5M for unified heterogeneous-sensor modeling. OmniRSCLIP uses SSBD to adapt arbitrary-channel inputs while retaining CLIP’s visual prior, and OmniRS5M supplies multi-modal image-text supervision.

  • Motivation: Existing remote sensing contrastive frameworks are mainly RGB-oriented, making unified use of optical, SAR, MSI, and HSI inputs difficult.These modalities differ in channel numbers, spectral responses, and imaging mechanisms.
  • Motivation: Projecting MSI and HSI into three-channel RGB space can discard sensor-specific spectral information, including subtle material absorption differences.HSI records hundreds of contiguous narrow bands, whereas PCA-based three-channel projections compress that information.
  • Method: OmniRSCLIP introduces SSBD, which recomposes wavelength-conditioned spectral-spatial kernels from pretrained CLIP spatial bases for arbitrary-channel sensor inputs.The constrained basis-recomposition design preserves transferable local visual primitives while adapting spectral responses.
  • Dataset: OmniRS5M provides multi-granularity supervision for multi-modal training by combining modality-aware caption distillation with global-local text synthesis.The corpus contains 4.67 million images and 73.90 million text candidates after filtering and manual quality inspection.
  • Results: The paper reports that OmniRSCLIP supports unified heterogeneous-sensor modeling and shows advantages across retrieval benchmarks and alternative arbitrary-channel adaptation strategies.The contribution summary covers short-text, long-text, and multimodal retrieval evaluations.

Related Work

Related work applies CLIP-style contrastive learning to remote sensing but remains dominated by RGB assumptions. Existing channel-adaptive approaches address variable-channel inputs with separate encoders, embedding heads, or constrained spectral mechanisms, leaving broader transferability challenges.

  • Remote Sensing Vision-Language Learning: Remote sensing vision-language methods are primarily designed for RGB imagery despite sensor diversity in channel numbers, spectral responses, and imaging mechanisms.Some MSI and HSI extensions require separate visual encoders or modality-specific embedding heads.
  • Channel-Adaptive Embeddings: Earth observation data range from single-channel SAR to hyperspectral imagery with hundreds of narrow bands, requiring channel-adaptive visual embeddings.Recent approaches explore wavelength-aware dictionaries and arbitrary-modality processing.
  • Channel-Adaptive Embeddings: HyperFree uses wavelength-aware dictionaries but has limited cross-modality transferability because of fixed spectral range constraints.

Method

OmniRSCLIP extends CLIP to heterogeneous remote-sensing sensors through SSBD, which combines pretrained spatial bases with wavelength-conditioned coefficients to produce compatible arbitrary-channel embeddings. It further uses spectral-context-aware feature masking for image-text alignment.

  • Architecture Overview: OmniRSCLIP replaces CLIP’s fixed RGB patch embedding with SSBD, a modality-adaptive interface for arbitrary-channel remote-sensing inputs.The framework retains CLIP’s dual-tower shared contrastive space while adapting RGB, SAR, MSI, and HSI inputs.
  • Spectral-Spatial Basis Decomposition: SSBD treats channel adaptation as basis recomposition, using learnable spatial bases and wavelength-conditioned coefficients instead of directly predicting full embedding kernels.This reduces the dynamic prediction target from CM full embedding parameters to CN wavelength-conditioned coefficients.
  • Spatial Basis Generation: The spatial bases inherit CLIP’s pretrained visual prior by perturbing a prototype extracted from the original patch embedding.The prototype preserves dominant spatial patterns, while small perturbations provide basis diversity for end-to-end specialization.
  • Wavelength-Conditioned Coefficient Generation: Wavelength tokens are processed jointly by a Transformer so each band’s coefficients depend on its position within the complete spectral profile.The resulting contextual representations produce wavelength-conditioned basis coordinates, including C-band wavelength information for SAR.
  • Dynamic Patch Embedding: The synthesized kernels are normalized, convolved with each input channel, and summed into feature maps that can be passed directly to the CLIP visual backbone.This avoids requiring a separate embedding layer for every sensor modality.
  • Spectral-Context-Aware Alignment: A spectral-context-aware Mask Network combines text features with SSBD-derived spectral context to select visual features for finer image-text alignment.The mask-based scheme extends text-conditioned feature selection to account for sensor characteristics.

Dataset Construction

OmniRS5M is a large-scale multimodal remote sensing image-text dataset built from multi-source data and modality-aware caption synthesis. It covers RGB, SAR, MSI, and HSI with short- and long-caption supervision.

  • OmniRS5M integrates multi-source data collection, modality-aware semantic distillation, XML organization, and dynamic text-supervised caption synthesis.
  • The corpus contains 4.67M images and 73.90M text candidates, including 27.82M short captions and 46.08M long captions.
  • OmniRS5M covers 4.32M RGB, 236.13K SAR, 71.87K MSI, and 47.71K HSI images.
  • The dataset jointly integrates RGB, SAR, MSI, and HSI modalities in a large-scale remote sensing image-text corpus.

Implementation Details

OmniRSCLIP uses CLIP-style ViT-B/16 and ViT-L/14 backbones with SSBD replacing fixed-channel patch embeddings, and evaluates retrieval using standardized comparison tables.

  • OmniRSCLIP uses ViT-B/16 and ViT-L/14 backbones, replacing each original fixed-channel patch embedding layer with SSBD.The number of spatial bases is set to 64.
  • Training uses two stages of 5 epochs each, beginning with RGB image-text pairs to mitigate training data imbalance.
  • Table 1 reports short-text/long-text Recall@1 for multi-source retrieval, with Mean averaging all Recall@1 values.
  • Table 2 compares cross-modal retrieval performance on the RSITMD and RSICD RGB benchmarks.

Cross-Modal Retrieval

OmniRSCLIP evaluates retrieval across RGB, SAR, MSI, and HSI while preserving strong RGB performance. Under a unified arbitrary-channel interface, it achieves the best mean multi-source retrieval performance among the compared methods.

  • The retrieval evaluation extends beyond RGB to SAR, MSI, and HSI modalities.
  • With ViT-L/14, OmniRSCLIP achieves the best image-to-text results on RSITMD and the best performance across all reported metrics on RSICD.
  • OmniRSCLIP preserves strong RGB-domain alignment while extending CLIP to arbitrary-modality remote sensing data.
  • OmniRSCLIP achieves the best mean performance when LongCLIP and HiMoCLIP are adapted with SSBD under the same multimodal setting.

Zero-shot Classification

Zero-shot classification tests category-level transfer without task-specific supervision. OmniRSCLIP performs strongly across four datasets while retaining recognition ability for heterogeneous sensor inputs.

  • Zero-shot classification evaluates category-level transferability without task-specific supervision.
  • ViT-B/16 achieves the best results among compared methods on all four zero-shot classification datasets.
  • ViT-L/14 remains consistently competitive across the four datasets.
  • The results indicate that OmniRSCLIP preserves category-level recognition while extending CLIP to heterogeneous sensor inputs.

Semantic Localization

OmniRSCLIP produces more concentrated responses on text-related regions and fewer accidental activations in qualitative semantic localization.

  • OmniRSCLIP produces more concentrated responses on text-related regions and fewer accidental activations than existing remote sensing VLMs.The method decomposes images into multi-scale local views, compares them with the query, and projects similarities into a heatmap.

Ablation Study

Ablations show that SSBD, Mask Network, CLIP-prior inheritance, and an adequately sized basis set jointly improve heterogeneous-sensor adaptation. Performance stabilizes around 48 to 64 bases, motivating 64 as the default.

  • Core Components: 34.89/43.87 to 38.16/48.94 average recall: SSBD improves MSI and HSI over fixed three-channel adaptation.Without SSBD, MSI and HSI use PCA while SAR channels are replicated; SSBD better exploits spectral information.
  • Core Components: The full configuration achieves the best overall results because SSBD and Mask Network provide complementary spectral embedding and alignment-relevant feature selection.SSBD supplies modality-adaptive spectral embedding, while Mask Network uses semantic and spectral context.
  • CLIP Prior: CLIP-initialized SSBD gains across all four modalities and outperforms DOFA, showing the benefit of retaining transferable visual knowledge.Prior inheritance improves both PCA- and SSBD-based adaptation relative to random initialization.
  • Number of Spatial Bases: 48 to 64 bases: performance curves for MSI and HSI become stable, while increasing to 128 yields limited additional gains.One basis causes a clear performance drop because it applies the same spatial transformation to every spectral band; 64 bases are adopted by default.

Conclusion

OmniRSCLIP aligns RGB, SAR, MSI, and HSI observations in a shared semantic space through a modality-adaptive embedding interface and spectral-text-aware feature selection. Experiments indicate that it extends CLIP beyond RGB imagery while preserving strong semantic transferability.

  • OmniRSCLIP enables a single CLIP-based model to align RGB, SAR, MSI, and HSI observations within a shared semantic space.The framework combines SSBD with spectral-text-aware feature selection.
  • Experiments on retrieval, zero-shot classification, semantic localization, and ablations show extension beyond RGB imagery with strong semantic transferability.
Loading 2609.03391v1…