PaperlayerView exact revision on arXiv

Source-linked AI summary

SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery

Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, Huimei He, Jian Wang, Jingdong Chen, Ming Yang, Yongjun Zhang, Yansheng Li

arXiv:2312.10115v2cs.CV

TL;DR

Prior Remote Sensing Foundation Models often focus on single modalities without temporal and geo-context modeling, limiting their breadth for Earth Observation tasks. SkySense addresses this gap with a billion-scale multimodal spatiotemporal model trained on temporal optical and SAR data using multi-granularity and geo-context learning. Across 16 datasets and 7 tasks, it outperforms 18 recent RSFMs in all test scenarios.

  • Problem

    Prior RSFMs primarily focus on single modalities and omit temporal and geo-context modeling, despite Earth Observation relying on diverse multimodal temporal data and tasks.

  • Method

    SkySense uses a factorized multimodal spatiotemporal encoder with Multi-Granularity Contrastive Learning and Geo-Context Prototype Learning, trained on 21.5 million temporal RSI sequences.

  • Results

    SkySense outperforms 18 advanced RSFMs across 16 datasets and 7 tasks in all test scenarios, including single- and multimodal and static-to-temporal settings.

  • Takeaways & Limitations

    Its modular design supports flexible use across modalities, spatial granularities, and task types, while the released pretrained weights are intended to facilitate future research and Earth Observation applications.

Abstract

from arXiv · show

Prior studies on Remote Sensing Foundation Model (RSFM) reveal immense potential towards a generic model for Earth Observation. Nevertheless, these works primarily focus on a single modality without temporal and geo-context modeling, hampering their capabilities for diverse tasks. In this study, we present SkySense, a generic billion-scale model, pre-trained on a curated multi-modal Remote Sensing Imagery (RSI) dataset with 21.5 million temporal sequences. SkySense incorporates a factorized multi-modal spatiotemporal encoder taking temporal sequences of optical and Synthetic Aperture Radar (SAR) data as input. This encoder is pre-trained by our proposed Multi-Granularity Contrastive Learning to learn representations across different modal and spatial granularities. To further enhance the RSI representations by the geo-context clue, we introduce Geo-Context Prototype Learning to learn region-aware prototypes upon RSI's multi-modal spatiotemporal features. To our best knowledge, SkySense is the largest Multi-Modal RSFM to date, whose modules can be flexibly combined or used individually to accommodate various tasks. It demonstrates remarkable generalization capabilities on a thorough evaluation encompassing 16 datasets over 7 tasks, from single- to multi-modal, static to temporal, and classification to localization. SkySense surpasses 18 recent RSFMs in all test scenarios. Specifically, it outperforms the latest models such as GFM, SatLas and Scale-MAE by a large margin, i.e., 2.76%, 3.67% and 3.61% on average respectively. We will release the pre-trained weights to facilitate future research and Earth Observation applications.

1. Introduction

Remote Sensing Foundation Models aim to support diverse Earth Observation interpretation, but prior models are constrained by single-modal pre-training and limited geo-context modeling. SkySense addresses these needs with a modular billion-scale multimodal spatiotemporal model and broad evaluation.

  • Earth Observation involves diverse tasks such as crop monitoring and natural disaster management, motivating generic Remote Sensing Foundation Models.
  • An ideal RSFM should model multimodal temporal optical and SAR imagery, support multiple spatial granularities, and use space-time geo-context.Optical imagery provides spectral and texture information but is weather-sensitive, whereas SAR operates in all weather conditions.
  • Prior RSFMs have shown success on specific datasets but remain limited by single-modal pre-training and neglect of geo-context.
  • SkySense is a 2.06-billion-parameter MM-RSFM trained on 21.5 million temporal sequences from high-resolution optical, multispectral, and SAR imagery.Its factorized encoder separates spatial feature extraction from multimodal temporal fusion, enabling modular use for static single-modal through temporal multimodal tasks.
  • SkySense outperforms 18 advanced RSFMs across 16 datasets and all evaluated scenarios spanning diverse modalities, tasks, and spatial scales.The evaluation covers classification to localization and static to temporal settings.
  • SkySense combines Multi-Granularity Contrastive Learning with Geo-Context Prototype Learning to represent varied modal and spatial granularities and integrate regional geo-knowledge.

2. Related Work

Related RSFM research extends vision-foundation-model techniques to remote sensing data with space-time coordinates, diverse scales, and multimodal inputs. SkySense is positioned as a comprehensive multimodal foundation model addressing gaps in prior single-modal approaches.

  • Remote sensing foundation models adapt vision foundation-model techniques to space-time remote sensing data with diverse spatial scales.Contrastive learning and related pretraining strategies are used to learn representations for remote sensing applications.
  • Prior work explores geo-location prediction, temporal multispectral imagery, scale-aware encoding, and combined contrastive-learning and masked-image-modeling strategies.
  • CROMA and DeCUR investigate multimodal pretraining for static imagery, while SkySense targets a more comprehensive multimodal remote sensing foundation model.

3. SkySense

SkySense combines aligned multi-modal temporal remote-sensing inputs with factorized spatiotemporal encoding, multi-granularity contrastive learning, cross-modal alignment, and optional geo-context prototypes. Its modular pre-training framework supports independently or jointly deployed components for varied downstream tasks.

  • Pre-training Dataset: 21.5 million training samples combine static high-resolution optical imagery, temporal multispectral imagery, temporal SAR imagery, and geo-context metadata.The inputs are geo-spatially aligned; Sentinel-2 and Sentinel-1 sequences are sampled temporally for training.
  • Model Architecture: The factorized encoder extracts spatial features independently for each modality before fusing them into a multi-modal spatiotemporal representation.Spatial feature maps are aligned in their spatial dimensions and concatenated across modality-specific sequence lengths.
  • Model Architecture: Date-specific positional encoding and temporal self-attention incorporate acquisition dates and fuse multi-modal temporal data.The date encoding represents different days of the year, supporting tasks affected by seasonal variation.
  • Pre-training: The teacher-student framework uses augmented views, EMA-updated teacher parameters, cross-modal alignment, and modular downstream reuse with optional prototype enhancement.The teacher branch supplies downstream parameters, while pre-trained modules may be used alone or combined and frozen or fine-tuned.
  • Pre-training: Multi-Granularity Contrastive Learning trains representations across pixel-, object-, and image-level spatial features for individual and fused modalities.Object features are obtained by unsupervised clustering of pixel-level features, while image-level features use average pooling.
  • Pre-training: Geo-Context Prototype Learning extracts region-aware prototypes from fused representations and optionally concatenates attended prototypes with the fused feature.Prototype subsets are selected by geographic location, and their weighted combination supplies semantic representations such as water bodies or cropland.

4. Experiments

SkySense is evaluated across diverse single- and multi-modal EO tasks, with modules usable individually or jointly. It shows strong performance across the reported experiments, including benefits from temporal and multi-modal inputs.

  • SkySense is evaluated on 16 datasets spanning different modalities, tasks, and spatial scales.
  • SkySense’s pre-trained modules can be combined or used individually, and frozen or fine-tuned for different tasks.
  • Single-Modal Tasks: SkySense consistently improves semantic segmentation results across the evaluated datasets.The model is reported to generalize better across datasets than CACo, which is mainly proficient on the Dyna.-S2 validation set.
  • Multi-Modal Tasks: 11.1% OA increase is observed when using temporal S2-Ts instead of static multispectral data for crop mapping.
  • Multi-Modal Tasks: 13% higher performance is achieved with Sentinel-1 when cloud ratios exceed 50%.The comparison uses S2-Ts with and without Sentinel-1, highlighting the value of SAR under cloud coverage.
  • Multi-Modal Tasks: More than 2.5% mAP improvement over the previous SOTA is reported for both evaluated BEN-MM modality settings.Adding Sentinel-2 imagery substantially improves performance over Sentinel-1 alone.
  • Multi-Modal Tasks: Multi-modal downstream tasks show notable gains, supporting the effectiveness of SkySense’s multi-modal pre-training.

5. Discussions & Ablation Studies

The ablation studies examine multi-modal pre-training, modal alignment, and geo-context prototype learning. Results show progressive improvements and finer-grained geographic representations from the proposed components.

  • Geo-Context Prototype Learning: Geo-Context Prototypes segment geographic areas more finely than the ESRI LandCover Map in the visualization.Unsupervised prototypes identify cropland within a town that the LandCover Map overlooks.
  • Ablation Studies: Multi-modal pre-training improves Dyna.-MM performance to 47.0% mIoU.
  • Ablation Studies: Modal alignment adds a 0.7% mIoU improvement by encouraging cross-modal feature interactions.
  • Ablation Studies: Geo-Context Prototype Learning further increases performance to 48.2% mIoU.

6. Conclusion & Future Work

SkySense is presented as a large-scale multi-modal RSFM with flexible modules and strong performance across varied EO interpretation tasks. Future work will extend it to language-modality applications.

  • SkySense’s modules can be flexibly used across scenarios while the model consistently outperforms other models on varied tasks.
  • Future work will incorporate the language modality to extend SkySense’s applications to more EO tasks.

Supplementary Material

The supplementary materials contain 12 sections covering additional evaluations, convergence analyses, and comparisons supporting SkySense’s pre-trained features.

  • The supplementary materials are divided into 12 sections.
  • Frozen-backbone evaluations cover scene classification, object detection, and semantic segmentation, including data from satellite sensors differing from the pre-training data.
  • Convergence-rate comparisons examine SkySense against Scale-MAE, Satlas, CMID, and GFM across three downstream tasks.

B. Experimental results of the frozen backbone tuning and various satellite sensors

Frozen-backbone and cross-sensor evaluations test whether SkySense’s learned representations transfer across downstream tasks and satellite sources. The experiments also compare convergence behavior with recent RSFMs.

  • Frozen backbone tuning: SkySense is evaluated with frozen backbone features on scene classification, object detection, and semantic segmentation.Task-specific heads are tuned while backbone parameters remain fixed.
  • Various satellite sensors: SkySense is tested on three additional Gaofen and Landsat datasets to assess generalization beyond its pre-training sensors.The model greatly outperforms other RSFMs in these experiments.
  • Convergence analysis: Convergence rate is treated as evidence of representation quality because robust pre-trained features facilitate faster downstream convergence.This evaluation connects pre-training quality with downstream optimization behavior.
  • Convergence analysis: SkySense is compared with Scale-MAE, Satlas, CMID, and GFM using convergence curves for scene classification, object detection, and semantic segmentation.The curves are reported in Figure C6 across AID, DIOR, and iSAID tasks.

D. Experimental results of SkySense with fewer parameters

SkySense retains strong downstream performance when its spatial backbone is reduced from Swin-H to Swin-L. The results attribute its effectiveness to architecture and pre-training, not parameter scaling alone.

  • Parameter reduction: 69.8% fewer parameters result from replacing Swin-H (654M) with Swin-L (197M), with only a slight downstream performance decrease.The comparison covers scene classification, object detection, and semantic segmentation.
  • Comparison with RSFMs: 2.62% and 1.71% OA gains on RESISC-45 at TR=10% show SkySense with Swin-L outperforming SatMAE and Scale-MAE, respectively.SkySense with Swin-L uses 197M parameters, versus 307M for each comparison model.
  • Interpretation: SkySense’s smaller-backbone results indicate that innovative architecture design and advanced pre-training strategies contribute beyond parameter size.The authors explicitly reject performance being explained simply by scaling model parameters.

E. Comparison with random initialization and Vision Foundation Model

SkySense is compared with randomly initialized models and DINOv2 across remote-sensing tasks, while supplementary analyses examine architectural and pre-training components. The reported results consistently favor SkySense, with a small MAE-related exception.

  • Random initialization: SkySense substantially outperforms randomly initialized models across five datasets spanning scene classification, detection, segmentation, and change detection.The comparison uses identical networks fine-tuned from scratch or SkySense pre-trained weights.
  • Vision Foundation Model: SkySense outperforms DINOv2 on all five evaluated remote-sensing datasets.The authors relate this to remote-sensing-specific data, pre-training methods, and model structures.
  • Architecture validation: The F-ST encoder is compared with UNet3D and TSViT using Sentinel-2 data from PASTIS-R, with all models trained from scratch.This isolates the space-time processing architecture rather than pre-training effects.
  • Cross-modal alignment: Cross-Modal Alignment is visualized through Transformer attention maps to assess interactions among the extra, HSROI, TMsI, and TSARI tokens.The comparison contrasts attention with and without alignment.
  • Additional pre-training analysis: -0.7% mIoU is observed on the DynamicEarthNet test set when MAE is added to pre-training.The authors state that pixel-level modeling is already addressed by Multi-Granularity Contrastive Learning.

H. Qualitative results of downstream tasks

Qualitative visualizations compare SkySense with recent RSFMs across segmentation, detection, change detection, and classification. The examples emphasize more complete boundaries, fewer mistaken detections, and attention focused on relevant scene structures.

  • Object detection: SkySense produces more accurate plane detections and bounding boxes than SatLas and Scale-MAE in complex FAIR1M scenes.The FAIR1M test labels are unavailable, so the visualization contains no ground truth.
  • Change detection: SkySense provides change boundaries that better match ground truth than SatLas and Scale-MAE on LEVIR-CD examples.The comparison highlights rougher boundaries from the other methods and more detailed SkySense boundaries.
  • Scene classification: SkySense’s Grad-CAM visualizations focus on primary structures such as teaching buildings and viaducts, unlike competing methods’ scattered or overly local attention.The examples use the AID scene-classification dataset.
  • Semantic segmentation: Multimodal training improves segmentation of difficult agricultural areas and continuous rivers compared with single-modal inputs.The authors also report minor errors, including soil mistaken for water and incomplete wetland segmentation.
  • Overall comparison: The qualitative results collectively report stronger segmentation, detection, and classification performance for SkySense across downstream tasks.These visualizations are presented as evidence of learning capacity and generalization.

I. Pre-training dataset

SkySense is pre-trained on a globally sourced, multi-modal dataset of temporal optical and SAR imagery designed to provide complementary spatial, spectral, and temporal information. The dataset contains 21.5 million sequences spanning diverse geographic regions and is used with substantial computational resources.

  • Modalities and sources: The dataset combines HSROIs, TMsI, and TSARI from WorldView-3/4, Sentinel-2, and Sentinel-1 sources.HSROIs provide high-resolution optical RGB imagery, TMsI provides ten multispectral bands, and TSARI provides VV and VH SAR polarization data.
  • Dataset composition: 21.5 million training sequences each combine one static HSROI, a 20-frame TMsI sequence, and a 10-frame TSARI sequence.The dataset occupies around 300 Terabytes.
  • Geographic coverage: The pre-training data covers 8.78 million square kilometers across 40 countries and areas on six continents.Its geographical distribution is illustrated in Figure I13.
  • Modalities and sources: The three modalities complement one another through differences in temporal information, spatial resolution, and imaging mechanisms.The dataset includes high-resolution optical, medium-resolution multispectral, and SAR imagery.
  • Pre-training cost: SkySense training uses 875k steps on 80 A100-80GB GPUs, requiring 24600 A100 GPU hours and 4488.69 GFLOPs.Training uses a batch size of 240 and AdamW optimization.

K. Dataset and implementation details of downstream tasks

The downstream evaluation covers diverse remote-sensing tasks, modalities, spatial scales, and temporal settings using task-specific datasets and heads. SkySense's pre-trained modules can be used independently or combined to match downstream input and task requirements.

  • Downstream tasks: The evaluation spans semantic segmentation, oriented object detection, change detection, scene classification, and multi-modal mapping tasks.The experiments use datasets including DynamicEarthNet, DIOR-R, FAIR1M, LEVIR-CD, AID, and BigEarthNet.
  • Oriented object detection: FAIR1M evaluates fine-grained oriented object detection using more than 40,000 high-resolution images and over 1 million annotated instances.The benchmark contains five categories and 37 fine-grained subcategories, with mAP reported on the official leaderboard.
  • Change detection: LEVIR-CD, OSCD, and DynamicEarthNet-Sentinel2 are used to assess bi-temporal, multispectral, and semantic change detection settings.The LEVIR-CD evaluation uses 637 pairs of visible images with 0.5-meter GSD.
  • Multi-modal semantic segmentation: DynamicEarthNet-MM and PASTIS-MM evaluate time-insensitive land-cover mapping and time-sensitive crop mapping with aligned optical and SAR data.PASTIS-MM adds high-resolution RGB imagery to Sentinel-2 multispectral and Sentinel-1 SAR data.
  • Module deployment: SkySense provides spatial encoders, a Multi-Modal Temporal Fusion Transformer, and an Attentional Geo-Context Integration module for flexible downstream use.These modules can be used alone or combined according to downstream modality and task requirements.
Loading 2312.10115v2…