PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data

Xiangyang Miao, Kelu Yao, Yekai Huang, Xiaogang Xu, Junxiao Xue, Minjun Shen, Chenghui Lv, Shanji Liu, Yaying Chen, Chao Li

arXiv:2609.03391v1cs.CVcs.AI

TL;DR

Existing remote sensing contrastive learning methods are largely tied to RGB-oriented CLIP interfaces, limiting unified modeling of heterogeneous sensors. OmniRSCLIP addresses this with SSBD, spectral-context-aware feature selection, and the OmniRS5M corpus, while experiments report preserved RGB performance and extension to heterogeneous modalities.

Abstract

from arXiv · show

Contrastive language-image learning (CLIP) has become a key paradigm for remote sensing vision-language understanding. However, existing remote sensing contrastive learning methods are mostly built on RGB-oriented CLIP architectures, making it difficult to exploit heterogeneous sensors such as SAR, multi-spectral imaging (MSI), and hyperspectral imaging (HSI). To address this limitation, we propose OmniRSCLIP, an end-to-end contrastive learning framework that supports multi-source sensor inputs for remote sensing vision-language modeling. The key idea is to extend CLIP beyond its fixed RGB input interface without breaking the pretrained visual knowledge. To this end, OmniRSCLIP introduces Spectral-Spatial Basis Decomposition (SSBD), which formulates arbitrary-channel adaptation as a basis recomposition problem: pretrained CLIP patch embeddings provide transferable spatial bases, while wavelength-conditioned coefficients span sensor-specific embedding kernels within a constrained visual prior space. This design avoids forcing heterogeneous sensors into a fixed-channel input space, while aligning them in a unified image-text semantic space. We further introduce a spectral-context-aware mask-based contrastive learning scheme to suppress modality-specific redundant features and enhance fine-grained image-text alignment. Finally, to support multi-modal training, we construct OmniRS5M, the first large-scale remote sensing image-text corpus covering RGB, SAR, MSI, and HSI. Experiments on retrieval, zero-shot classification, and semantic localization show that OmniRSCLIP preserves strong RGB-domain performance while effectively extending CLIP to heterogeneous remote sensing modalities.

Introduction

Remote sensing contrastive learning remains constrained by RGB-oriented CLIP interfaces, motivating OmniRSCLIP and OmniRS5M for unified heterogeneous-sensor modeling. OmniRSCLIP uses SSBD to adapt arbitrary-channel inputs while retaining CLIP’s visual prior, and OmniRS5M supplies multi-modal image-text supervision.

Motivation

  • Existing remote sensing contrastive frameworks are mainly RGB-oriented, making unified use of optical, SAR, MSI, and HSI inputs difficult.These modalities differ in channel numbers, spectral responses, and imaging mechanisms.Source: Introduction · p. 1
  • Projecting MSI and HSI into three-channel RGB space can discard sensor-specific spectral information, including subtle material absorption differences.HSI records hundreds of contiguous narrow bands, whereas PCA-based three-channel projections compress that information.Source: Introduction · pp. 1–2

Method

  • OmniRSCLIP introduces SSBD, which recomposes wavelength-conditioned spectral-spatial kernels from pretrained CLIP spatial bases for arbitrary-channel sensor inputs.The constrained basis-recomposition design preserves transferable local visual primitives while adapting spectral responses.Source: Introduction · p. 2

Dataset

  • OmniRS5M provides multi-granularity supervision for multi-modal training by combining modality-aware caption distillation with global-local text synthesis.The corpus contains 4.67 million images and 73.90 million text candidates after filtering and manual quality inspection.Source: Introduction · p. 2

Results

  • The paper reports that OmniRSCLIP supports unified heterogeneous-sensor modeling and shows advantages across retrieval benchmarks and alternative arbitrary-channel adaptation strategies.The contribution summary covers short-text, long-text, and multimodal retrieval evaluations.Source: Introduction · p. 2

Related Work

Related work applies CLIP-style contrastive learning to remote sensing but remains dominated by RGB assumptions. Existing channel-adaptive approaches address variable-channel inputs with separate encoders, embedding heads, or constrained spectral mechanisms, leaving broader transferability challenges.

Remote Sensing Vision-Language Learning

  • Remote sensing vision-language methods are primarily designed for RGB imagery despite sensor diversity in channel numbers, spectral responses, and imaging mechanisms.Some MSI and HSI extensions require separate visual encoders or modality-specific embedding heads.Source: Contrastive Language-Image Pre-training · pp. 2–3

Channel-Adaptive Embeddings

  • Earth observation data range from single-channel SAR to hyperspectral imagery with hundreds of narrow bands, requiring channel-adaptive visual embeddings.Recent approaches explore wavelength-aware dictionaries and arbitrary-modality processing.Source: Channel-Adaptive Embedding · p. 3
  • HyperFree uses wavelength-aware dictionaries but has limited cross-modality transferability because of fixed spectral range constraints.Source: Channel-Adaptive Embedding · p. 3

Method

OmniRSCLIP extends CLIP to heterogeneous remote-sensing sensors through SSBD, which combines pretrained spatial bases with wavelength-conditioned coefficients to produce compatible arbitrary-channel embeddings. It further uses spectral-context-aware feature masking for image-text alignment.

Architecture Overview

  • OmniRSCLIP replaces CLIP’s fixed RGB patch embedding with SSBD, a modality-adaptive interface for arbitrary-channel remote-sensing inputs.The framework retains CLIP’s dual-tower shared contrastive space while adapting RGB, SAR, MSI, and HSI inputs.Source: Architecture Overview · p. 3, Method · p. 3

Spectral-Spatial Basis Decomposition

  • SSBD treats channel adaptation as basis recomposition, using learnable spatial bases and wavelength-conditioned coefficients instead of directly predicting full embedding kernels.This reduces the dynamic prediction target from CM full embedding parameters to CN wavelength-conditioned coefficients.Source: Motivation · pp. 3–4

Spatial Basis Generation

  • The spatial bases inherit CLIP’s pretrained visual prior by perturbing a prototype extracted from the original patch embedding.The prototype preserves dominant spatial patterns, while small perturbations provide basis diversity for end-to-end specialization.Source: Spatial Basis Generation · p. 4

Wavelength-Conditioned Coefficient Generation

  • Wavelength tokens are processed jointly by a Transformer so each band’s coefficients depend on its position within the complete spectral profile.The resulting contextual representations produce wavelength-conditioned basis coordinates, including C-band wavelength information for SAR.Source: Wavelength-Conditioned Coefficient… · pp. 4–5

Dynamic Patch Embedding

Spectral-Context-Aware Alignment

  • A spectral-context-aware Mask Network combines text features with SSBD-derived spectral context to select visual features for finer image-text alignment.The mask-based scheme extends text-conditioned feature selection to account for sensor characteristics.Source: Figure 2 · p. 4, Spectral-Context-Aware Mask-Based… · p. 5

Dataset Construction

OmniRS5M is a large-scale multimodal remote sensing image-text dataset built from multi-source data and modality-aware caption synthesis. It covers RGB, SAR, MSI, and HSI with short- and long-caption supervision.

  • OmniRS5M integrates multi-source data collection, modality-aware semantic distillation, XML organization, and dynamic text-supervised caption synthesis.Source: Dataset Construction · p. 5
  • The corpus contains 4.67M images and 73.90M text candidates, including 27.82M short captions and 46.08M long captions.Source: Dataset Construction · p. 5
  • OmniRS5M covers 4.32M RGB, 236.13K SAR, 71.87K MSI, and 47.71K HSI images.Source: Dataset Construction · p. 5
  • The dataset jointly integrates RGB, SAR, MSI, and HSI modalities in a large-scale remote sensing image-text corpus.Source: Dataset Construction · p. 5

Implementation Details

OmniRSCLIP uses CLIP-style ViT-B/16 and ViT-L/14 backbones with SSBD replacing fixed-channel patch embeddings, and evaluates retrieval using standardized comparison tables.

  • OmniRSCLIP uses ViT-B/16 and ViT-L/14 backbones, replacing each original fixed-channel patch embedding layer with SSBD.The number of spatial bases is set to 64.Source: Implementation Details · p. 5
  • Training uses two stages of 5 epochs each, beginning with RGB image-text pairs to mitigate training data imbalance.Source: Implementation Details · p. 5
  • Table 1 reports short-text/long-text Recall@1 for multi-source retrieval, with Mean averaging all Recall@1 values.Source: Table 1 · p. 6
  • Table 2 compares cross-modal retrieval performance on the RSITMD and RSICD RGB benchmarks.Source: Table 2 · p. 6

Cross-Modal Retrieval

OmniRSCLIP evaluates retrieval across RGB, SAR, MSI, and HSI while preserving strong RGB performance. Under a unified arbitrary-channel interface, it achieves the best mean multi-source retrieval performance among the compared methods.

  • The retrieval evaluation extends beyond RGB to SAR, MSI, and HSI modalities.Source: Cross-Modal Retrieval · p. 6
  • With ViT-L/14, OmniRSCLIP achieves the best image-to-text results on RSITMD and the best performance across all reported metrics on RSICD.Source: Cross-Modal Retrieval · p. 6
  • OmniRSCLIP preserves strong RGB-domain alignment while extending CLIP to arbitrary-modality remote sensing data.Source: Cross-Modal Retrieval · p. 6
  • OmniRSCLIP achieves the best mean performance when LongCLIP and HiMoCLIP are adapted with SSBD under the same multimodal setting.Source: Cross-Modal Retrieval · p. 6

Zero-shot Classification

Zero-shot classification tests category-level transfer without task-specific supervision. OmniRSCLIP performs strongly across four datasets while retaining recognition ability for heterogeneous sensor inputs.

Semantic Localization

OmniRSCLIP produces more concentrated responses on text-related regions and fewer accidental activations in qualitative semantic localization.

  • OmniRSCLIP produces more concentrated responses on text-related regions and fewer accidental activations than existing remote sensing VLMs.The method decomposes images into multi-scale local views, compares them with the query, and projects similarities into a heatmap.Source: Semantic Localization · p. 7

Ablation Study

Ablations show that SSBD, Mask Network, CLIP-prior inheritance, and an adequately sized basis set jointly improve heterogeneous-sensor adaptation. Performance stabilizes around 48 to 64 bases, motivating 64 as the default.

Core Components

  • 34.89/43.87 to 38.16/48.94 average recall: SSBD improves MSI and HSI over fixed three-channel adaptation.Without SSBD, MSI and HSI use PCA while SAR channels are replicated; SSBD better exploits spectral information.Source: Effect of Core Components · p. 7
  • The full configuration achieves the best overall results because SSBD and Mask Network provide complementary spectral embedding and alignment-relevant feature selection.SSBD supplies modality-adaptive spectral embedding, while Mask Network uses semantic and spectral context.Source: Effect of Core Components · p. 7

CLIP Prior

  • CLIP-initialized SSBD gains across all four modalities and outperforms DOFA, showing the benefit of retaining transferable visual knowledge.Prior inheritance improves both PCA- and SSBD-based adaptation relative to random initialization.Source: Effect of CLIP Prior · p. 7

Number of Spatial Bases

  • 48 to 64 bases: performance curves for MSI and HSI become stable, while increasing to 128 yields limited additional gains.One basis causes a clear performance drop because it applies the same spatial transformation to every spectral band; 64 bases are adopted by default.Source: Effect of the Number of Spatial Bases · p. 7

Conclusion

OmniRSCLIP aligns RGB, SAR, MSI, and HSI observations in a shared semantic space through a modality-adaptive embedding interface and spectral-text-aware feature selection. Experiments indicate that it extends CLIP beyond RGB imagery while preserving strong semantic transferability.

  • OmniRSCLIP enables a single CLIP-based model to align RGB, SAR, MSI, and HSI observations within a shared semantic space.The framework combines SSBD with spectral-text-aware feature selection.Source: Conclusion · p. 7
  • Experiments on retrieval, zero-shot classification, semantic localization, and ablations show extension beyond RGB imagery with strong semantic transferability.Source: Conclusion · p. 7

Related papers

Loading 2609.03391v1…