PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models for Remote Sensing and Earth Observation: A Survey

Aoran Xiao, Weihao Xuan, Junjue Wang, Jiaxing Huang, Dacheng Tao, Shijian Lu, Naoto Yokoya

arXiv:2410.16602v3cs.CV

TL;DR

Remote sensing models face heterogeneous data and task-specific training demands, while general foundation models often struggle with RS domain differences. This survey reviews and benchmarks RS foundation models across approaches and modalities. It reports emerging capabilities alongside limited dataset scale and diversity and continuing performance gaps for some model families.

  • Problem

    Task-specific RS models require substantial data collection, annotation, and computation, while general foundation models face challenges from RS sensor, viewpoint, resolution, spectral, and temporal differences.

    Source: §I · p. 1, B. Generative Models and Data… · p. 13

  • Method

    The survey systematically reviews RSFMs across model types, benchmarks them across modalities and tasks, and discusses challenges and future directions.

    Source: §I · p. 2

  • Results

    The survey reports new VFM capabilities in cross-modal generation, accelerated semantic-mask annotation, and segmentation with minimal fine-tuning, while LLMs and generative FMs in RS still trail their general-domain counterparts.

    Source: C. Summary and Discussion · p. 9, F. Summary and Discussion · p. 14

  • Takeaways & Limitations

    The survey concludes that tailored RSFMs integrating advanced AI with domain-specific geoscience knowledge remain important for broader Earth-observation applications.

    Source: F. Summary and Discussion · p. 14, §VIII · p. 17

    VFM development remains constrained by the limited scale and diversity of current RS self-supervised datasets, for both individual and multimodal applications.

    Source: C. Summary and Discussion · p. 9

Abstract

from arXiv · show

Remote Sensing (RS) is a crucial technology for observing, monitoring, and interpreting our planet, with broad applications across geoscience, economics, humanitarian fields, etc. While artificial intelligence (AI), particularly deep learning, has achieved significant advances in RS, unique challenges persist in developing more intelligent RS systems, including the complexity of Earth's environments, diverse sensor modalities, distinctive feature patterns, varying spatial and spectral resolutions, and temporal dynamics. Meanwhile, recent breakthroughs in large Foundation Models (FMs) have expanded AI's potential across many domains due to their exceptional generalizability and zero-shot transfer capabilities. However, their success has largely been confined to natural data like images and video, with degraded performance and even failures for RS data of various non-optical modalities. This has inspired growing interest in developing Remote Sensing Foundation Models (RSFMs) to address the complex demands of Earth Observation (EO) tasks, spanning the surface, atmosphere, and oceans. This survey systematically reviews the emerging field of RSFMs. It begins with an outline of their motivation and background, followed by an introduction of their foundational concepts. It then categorizes and reviews existing RSFM studies including their datasets and technical contributions across Visual Foundation Models (VFMs), Visual-Language Models (VLMs), Large Language Models (LLMs), and beyond. In addition, we benchmark these models against publicly available datasets, discuss existing challenges, and propose future research directions in this rapidly evolving field. A project associated with this survey has been built at https://github.com/xiaoaoran/awesome-RSFMs .

I. INTRODUCTION

Remote sensing foundation models address the mismatch between general-purpose models and heterogeneous Earth-observation data, and this survey reviews their methods, benchmarks, challenges, and research directions. It also frames RSFM learning as pretraining followed by downstream use, with several adaptation choices.

I. INTRODUCTION

  • Task-specific RS models demand costly data collection, annotation, and computation, while their limited cross-task transfer constrains broader adoption amid heterogeneous RS data.RS data are large-scale, complex, dynamic, and heterogeneous.Source: §I · p. 1
  • General foundation models generalize across tasks, but differences in sensors, viewpoints, resolutions, spectral bands, and temporal patterns challenge their direct use in RS.The survey describes RSFMs as a response to these domain differences.Source: §I · p. 1
  • The survey reviews RSFM advances across VFMs, VLMs, LLMs, and generative models, benchmarks them across modalities and tasks, and identifies challenges and research directions.Its taxonomy groups methods as VFMs, VLMs, and other RSFMs, including LLMs and generative FMs.Source: §I · p. 2

A. RS Learning Paradigms

  • RSFM learning has two stages: pretraining for transferable representations, then utilization on downstream tasks.Unsupervised pretraining can use unlabeled RS data, which is advantageous where annotation is impractical; it may not directly suit a specific task.Source: A. RS Learning Paradigms · pp. 2–3
  • Downstream use ranges from fully fine-tuning all parameters to PEFT with a frozen backbone or zero-shot prediction, which often underperforms for general-domain FMs in RS.PEFT adds lightweight learnable parameters, whereas full fine-tuning can overwrite the FM’s original representations; the survey notes no RSFM yet matches general-domain FMs’ robust zero-shot capabilities.Source: A. RS Learning Paradigms · p. 3

B. Common RS Sensor Modalities

RSFMs encounter inputs that vary substantially in spectral content, geometry, dimensionality, and physical measurement. These properties create distinct compatibility and interpretation challenges for general foundation models.

B. Common RS Sensor Modalities

  • Optical RGB captures visible red, green, and blue light, while multispectral imagery adds near-infrared and shortwave-infrared bands beyond RGB.General FMs can be applied directly to RGB, though performance may be suboptimal; MSI also faces input incompatibilities and domain gaps.Source: B. Common RS Sensor Modalities · p. 3
  • Hyperspectral imagery records tens of narrow, contiguous bands for detailed material identification, but its dimensionality raises computational complexity and overfitting risks.Its spectral characteristics and domain gaps also make it difficult to use with general FMs.Source: B. Common RS Sensor Modalities · p. 3
  • SAR uses active microwave signals for all-weather, day-and-night imaging, but slant-range geometry and speckle noise challenge general FMs.PolSAR measures radar-wave polarization; InSAR compares multiple images through phase differences to detect small surface deformations.Source: B. Common RS Sensor Modalities · p. 3
  • LiDAR point clouds provide accurate 3D terrain and object representations, while DSMs encode surface elevation including buildings, vegetation, and infrastructure.LiDAR point clouds are irregular and sparse; DSMs have a grid-like structure and elevation-specific data, both posing challenges for general FMs.Source: B. Common RS Sensor Modalities · pp. 3–4
  • Thermal infrared imagery measures emitted heat and supports nighttime thermal mapping, but lower spatial resolution, atmospheric conditions, and surface emissivity complicate interpretation.The passage identifies environmental monitoring, urban heat analysis, vegetation assessment, and wildfire detection as applications.Source: B. Common RS Sensor Modalities · p. 4

C. Relevant Surveys

The survey positions its review against earlier work and introduces the foundations needed to understand RSFMs, from core architectures to common interpretation tasks. It emphasizes the diversity of both model approaches and the geospatial inputs they must handle.

C. Relevant Surveys

  • Earlier surveys cover natural-domain FMs or narrower RS topics; the survey addresses RSFM types and modalities with a broader, up-to-date review.The cited comparison notes that prior geoscience reviews lacked in-depth dataset and technical analysis and were predominantly qualitative.Source: C. Relevant Surveys · p. 4

III. FOUNDATIONS OF RSFMS

  • FM learning relies on transfer learning through pretraining and on scale supported by computing hardware, architectures, and large training datasets.The section introduces RSFM pretraining methods and datasets before reviewing architectures and downstream tasks.Source: §III · p. 4

A. Deep Network Architectures

  • CNNs learn spatial hierarchies through local receptive fields, whereas transformers use self-attention for global dependencies and can integrate visual and nonvisual modalities.The survey describes ResNet skip connections as a way to address vanishing or exploding gradients.Source: A. Deep Network Architectures · pp. 4–5

B. Typical RS Interpretation Tasks •

  • Scene classification assigns whole scenes to land-use and land-cover classes, while semantic segmentation labels each pixel and must handle spectral ambiguity and multisensor variation.Segmentation also faces intricate spatial patterns and differences in resolution across datasets.Source: B. Typical RS Interpretation Tasks • · p. 5
  • Object detection locates targets with horizontal or rotated boxes, while change detection compares images over time and depends on handling alignment, seasonal, lighting, and sensor variation.The passages identify scale variation and occlusion as detection challenges, and precise multi-temporal alignment as a change-detection challenge.Source: B. Typical RS Interpretation Tasks • · p. 5
  • RS visual-language tasks include answering questions about imagery, generating image captions, and grounding text queries in image regions or objects.Their challenges include linking language to relevant visual evidence across modalities and aligning text with spatial and spectral features.Source: B. Typical RS Interpretation Tasks • · p. 5

IV. VISUAL FOUNDATION MODELS FOR RS

Remote sensing visual foundation models (VFMs) use supervised and self-supervised pre-training to handle diverse sensor modalities and visual tasks. This section reviews their datasets and methods, including contrastive learning, masked image modeling, and hybrid objectives.

1) Pre-training Datasets:

  • RS pre-training datasets are growing in size and modality coverage, but remain smaller and less diverse than general foundation-model datasets; annotation costs also motivate unlabeled SSL data.The datasets span varied sources and ground sample distances.Source: 1) Pre-training Datasets · p. 6

2) Supervised Pre-training:

  • ImageNet pre-training can be suboptimal for RS because natural and remote-sensing imagery differ in modality, perspective, color, texture, and layout.MillionAID-based pre-training helps mitigate these discrepancies, though some can persist by task and the examined pre-training was limited to RGB classification; multi-task pre-training across seven label types improved downstream performance across image resolutions.Source: 2) Supervised Pre-training · p. 6

3) Self-Supervised Pre-training:

  • SSL learns transferable representations from unlabeled data before downstream fine-tuning, an attractive approach where RS data are abundant but costly to label.The review distinguishes its focus on FM pre-training from earlier RS SSL work.Source: 3) Self-Supervised Pre-training · p. 6
  • Contrastive pre-training brings augmented views of the same sample together and separates different samples, producing robust, transferable representations.RS work includes optical RGB approaches using geolocation, temporal structure, aligned unchanged regions, and combinations of unlabeled RS with labeled natural images; other studies extend contrastive learning beyond optical RGB.Source: 3) Self-Supervised Pre-training · p. 7
  • Masked image modeling (MIM) reconstructs randomly masked image patches in pixel space, with reconstruction error evaluated only on masked patches.RS adaptations address temporal and spatial resolution, object size and orientation, and spectral dimensions; examples include temporal embeddings, multi-scale reconstruction, angle-aware masking, and spectral masking.Source: 3) Self-Supervised Pre-training · p. 7
  • Hybrid SSL methods combine contrastive and generative objectives to learn complementary representations for downstream RS tasks.Examples include contrastive masked-image distillation, cross-modal contrastive learning with masked reconstruction, and cross-scale consistency.Source: 3) Self-Supervised Pre-training · p. 8

1) SAM Overview:

SAM combines image and prompt encoders with a mask decoder for promptable segmentation, and RS research adapts it to domain-specific tasks and sensor modalities. The review notes limits in dataset diversity and multimodal integration, alongside capabilities such as cross-modal generation and reduced-data segmentation.

1) SAM Overview:

  • SAM generates segmentation masks from geometric prompts, using a heavyweight image encoder, a lightweight prompt encoder, and a lightweight mask decoder.Its reported training set, SA-1B, contains over 11 million RGB images and 1.1 billion mask annotations.Source: 1) SAM Overview · p. 8, Figure 8 · p. 8

2) SAM-based studies for RS:

  • RS adaptations extend SAM toward semantic segmentation, change detection, and instance segmentation, including approaches that generate datasets, refine pseudo-labels, or use learnable prompts.These adaptations address SAM’s limited semantic understanding and its primary design for RGB, while RS imagery includes complex land-cover patterns and non-RGB modalities.Source: 2) SAM-based studies for RS · p. 8
  • SAM has also been adapted for semantic segmentation across RGB, SAR, PolSAR, DSM, and MSI by freezing encoders, applying LoRA, and using modality-specific decoders.Another approach combines SAM with GroundDINO for text-based segmentation of RGB imagery.Source: 2) SAM-based studies for RS · p. 8
  • CAT-SAM offers few-shot adaptation using a prompt bridge to jointly fine-tune SAM’s image encoder and mask decoder, with reported segmentation performance across multiple RS modalities using very limited tuning samples.The approach targets tuning imbalance between encoder and decoder.Source: 2) SAM-based studies for RS · pp. 8–9

C. Summary and Discussion

  • RS VFM research emphasizes supervised and self-supervised pre-training, especially SSL, while limited scale and diversity in RS datasets constrain widely adopted pre-trained models.The constraint applies to datasets for individual modalities as well as multimodal applications.Source: C. Summary and Discussion · p. 9
  • Only a few RSFMs accept multiple sensor modalities simultaneously, and reported gains over single-modality fine-tuning remain relatively modest.The review identifies differences in spatial resolution, spectral characteristics, and sensing principles as challenges to effective integration.Source: C. Summary and Discussion · p. 9
  • VFM examples include cross-modal generation, SAM-based mask annotation pipelines, and high segmentation performance with minimal fine-tuning data.The review says RSFM studies have not yet fundamentally reshaped a specific traditional RS application, although it describes future potential.Source: C. Summary and Discussion · p. 9

V. VISION-LANGUAGE MODELS FOR RS

RS vision-language models (VLMs) connect visual inputs with textual representations for semantic interpretation and interaction. The section reviews their architectures and training objectives, RS datasets, and CLIP adaptations, while noting that dataset development remains limited in scale and modality coverage.

V. VISION-LANGUAGE MODELS FOR RS

  • VLMs combine visual and language encoders whose embeddings are fused or aligned, supporting tasks such as classification, segmentation, visual grounding, and captioning.Contrastive learning aligns image-text pairs, while generative objectives include masked reconstruction and autoregressive next-token prediction.Source: §V · p. 9, A. Preliminaries of VLMs in Natural… · pp. 9–10

B. Vision-language datasets for RS

  • RS vision-language datasets cover question answering, image-text pre-training, captioning, and visual grounding, using resources such as EarthVQA, RemoteCLIP, and GeoVG.Some datasets repurpose existing annotations or connect satellite imagery with ground-level imagery or geographic data.Source: B. Vision-language datasets for RS · pp. 10–11
  • RS vision-language datasets are progressing but lag general domains and remain concentrated on RGB, with limited exploration of SAR, MSI, and HSI.Source: B. Vision-language datasets for RS · p. 11

C. VLMs with Contrastive Objectives

  • CLIP-style VLM pre-training uses a symmetric image-text contrastive loss to align matching pairs above mismatched pairs, enabling representations usable without task-specific fine-tuning.The review notes that general-purpose internet image-text training may not capture RS-specific characteristics, motivating RS-tailored adaptations.Source: C. VLMs with Contrastive Objectives · p. 11

1) Visual-Text Models:

RS contrastive VLM research adapts CLIP through RS-specific datasets and benchmarks, and also aligns imagery with geographic location information. These approaches cover visual-text and visual-location learning.

1) Visual-Text Models:

  • RS visual-text models extend CLIP through continual pre-training, fine-tuning, or image alignment strategies, with evaluations spanning classification, retrieval, counting, localization, and segmentation.GRAFT aligns satellite and ground-level imagery without textual annotations and reports considerable improvements, especially in zero-shot settings.Source: 1) Visual-Text Models · p. 11

2) Visual-Location Models:

  • Visual-location models align RS imagery with geographic context, using approaches such as geo-tagged image-location contrastive learning, image-to-GPS retrieval, globally sampled imagery, or metadata captions.Examples include CSP, GeoCLIP, SatCLIP, and SatMIP, which use different location representations and sampling strategies.Source: 2) Visual-Location Models · pp. 11–12

D. VLMs with Generative Objectives

The survey reviews generative VLMs for RS, their developing task and modality coverage, and other RS foundation-model directions including LLMs. It also describes GeoLLM’s use of map context for geospatial prediction.

D. VLMs with Generative Objectives

  • RS generative VLMs largely build on LLaVA, which combines pretrained Vicuna and CLIP encoders through a linear projector and generates responses autoregressively.LLaVA also used 150k synthetic visual-instruction samples for instruction fine-tuning.Source: D. VLMs with Generative Objectives · p. 12
  • GeoChat, LHRS-Bot, and SkysenseGPT extend instruction-following VLMs toward multi-task, progressively tuned, and fine-grained RS scene understanding.GeoChat aggregates RS datasets and uses LoRA fine-tuning; LHRS-Bot tunes vision and language models with volunteered geographic information and RS images.Source: D. VLMs with Generative Objectives · p. 12

E. Summary and Discussion

  • RS VLM research spans contrastive image-text or image-location alignment and generative image-conditioned text, while most work remains focused on optical RGB and transfer from general-domain FMs.The survey notes broader datasets, temporal and multisensor assistants, and performance limitations on ultrahigh-resolution imagery.Source: E. Summary and Discussion · pp. 12–13, D. VLMs with Generative Objectives · p. 12

VI. OTHER FOUNDATIONS FOR RSFMS

  • The survey also covers RS LLMs, generative foundation models, and weather forecasting models beyond VFMs and VLMs.It describes LLM geospatial prediction as underexplored because LLMs lack visual perception needed for many RS applications.Source: §VI · p. 13, A. Large Language Models for RS · p. 13

A. Large Language Models for RS

  • GeoLLM enriches prompts with OpenStreetMap data, improving LLM performance on population density, asset wealth, mean income, and women’s education indicators.The cited study found that geographic coordinates alone did not yield accurate predictions for these indicators.Source: A. Large Language Models for RS · p. 13

B. Generative Models and Data Synthesis for RS

The survey presents RS generative models as a response to scarce, heterogeneous data and describes diffusion, multimodal reconstruction, and simulation approaches. It also discusses efficiency constraints and weather-forecasting models.

B. Generative Models and Data Synthesis for RS

  • RS data’s varying spectral bands, irregular temporal sampling, and complex spatiotemporal structure motivate specialized generative models for synthesis and tasks such as super-resolution, cloud removal, and temporal inpainting.Synthetic data may expand dataset quantity and diversity and provide automatic annotations.Source: B. Generative Models and Data… · p. 13
  • DiffusionSat conditions satellite-image generation on text and metadata, supporting synthesis, super-resolution, inpainting, and temporal generation; another diffusion model addresses probabilistic change detection.DiffusionSat’s metadata includes coordinates, timestamps, and GSD.Source: B. Generative Models and Data… · p. 13
  • TerraMesh [149] combines optical imagery, SAR, DEM, and land-cover data with masked reconstruction to generate multimodal outputs and cross-modal training samples.The survey notes that such samples can be difficult and expensive to acquire at large scales.Source: B. Generative Models and Data… · p. 13
  • SynRS3D uses Blender, GPT-4-generated text, and Stable Diffusion textures to create high-resolution optical data with land-cover types, height information, and annotated building changes.The survey describes physics-based simulation as a scalable, flexible alternative for making RS datasets.Source: B. Generative Models and Data… · p. 13

C. Computational Efficiency and Scalability

  • High-resolution, multimodal, and temporally rich inputs, alongside real-time and edge requirements, make RSFMs difficult to scale efficiently.The survey says model compression is critical but relatively underexplored, particularly for reducing size, inference latency, and memory footprint while preserving task performance.Source: C. Computational Efficiency and… · p. 14

D. Weather Forecasting Foundation Models

  • Pangu-Weather uses a three-dimensional Earth-specific transformer and hierarchical temporal aggregation, and consistently outperforms leading traditional numerical weather prediction systems.It was trained on 39 years of global data.Source: D. Weather Forecasting Foundation Models · p. 14

E. Responsible RSFMs

The survey reports benchmark progress alongside limits in comparability and VLM transfer performance. It also highlights responsible-AI concerns and concludes that RS-specific LLMs and generative models still trail general-domain counterparts.

E. Responsible RSFMs

  • RSFM training and deployment raise environmental and ethical concerns as well as potential societal benefits, with security, privacy, bias, and fairness among responsible-AI dimensions identified in cited work.The survey refers readers to Ghamisi et al. for a detailed discussion of responsible AI for EO.Source: E. Responsible RSFMs · p. 14

F. Summary and Discussion

  • The survey concludes that RS LLMs and generative FMs still trail their general-domain counterparts, while recent adaptation progress is promising and weather forecasting underscores the value of domain-specific RSFMs.Source: F. Summary and Discussion · p. 14

VII. BENCHMARK PERFORMANCE

  • The reviewed studies differ in modality, setup, and task, and comprehensive cross-task and cross-modality evaluation frameworks are not yet established.The survey therefore focuses on key dimensions rather than claiming a unified fair comparison.Source: §VII · p. 14

A. Performance of VFM Pre-training

  • Across downstream RS tasks, increasingly advanced pretraining methods benefit diverse tasks, though application-specific targets complicate fair comparison and motivate standardized benchmarks.The survey compares seven tasks and notes that metrics are sourced from the respective papers.Source: A. Performance of VFM Pre-training · pp. 14–15, Table V · p. 15

B. Performance of VLM Transfer Learning

  • RS-specific VLMs improve on CLIP’s zero-shot performance on RS benchmarks, but tend to be less robust across benchmarks than CLIP.The comparison concerns RS image classification and retrieval with Sentinel-2 data.Source: B. Performance of VLM Transfer Learning · p. 15, Table VI · p. 15
  • GEOBench-VLM [164] shows a substantial gap between open- and closed-source VLMs, while both groups perform suboptimally across geospatial recognition tasks.The benchmark spans 31 fine-grained tasks across eight categories; the survey attributes limitations chiefly to limited geospatial data in large multimodal training datasets.Source: B. Performance of VLM Transfer Learning · p. 15, Figure 11 · p. 16

VIII. FUTURE DIRECTIONS

The survey identifies data, capability, and deployment gaps that shape future RSFM research, then concludes by presenting its review and benchmarks as a resource for guiding that work.

VIII. FUTURE DIRECTIONS

  • Current RS datasets lack the scale and diversity for comparable emergent capabilities, while construction needs broader modalities, richer resolution and context, and stronger semantic enrichment.Large-scale multimodal annotation remains a bottleneck; data management also needs integration, quality assurance, governance, and privacy controls.Source: §VIII · p. 16
  • Future systems need multimodal and spatiotemporal processing, efficient transfer through adaptation and continuous learning, and lower computational demands for deployment.Large, high-resolution, multimodal workloads strain memory and processing, especially for onboard or edge use; time-sensitive EO also calls for real-time or near-real-time inference.Source: §VIII · p. 16
  • Broader EO applications, including weather forecasting, environmental monitoring, and urban planning, require varied inputs and outputs plus expertise in remote sensing and geoscience.Inputs may include imagery, GPS, temperature, and NDVI maps; outputs range from atmospheric predictions to land-use assessments.Source: §VIII · pp. 16–17
  • Most RSFMs emphasize perception, while complex EO problems call for spatial, temporal, and causal reasoning over noisy, high-dimensional, sparsely labeled data.The survey points to domain knowledge, causal and temporal models, and cross-modal, cross-scale fusion as research needs.Source: §VIII · p. 17
  • As RSFMs enter critical EO applications, modality-aware explainability is needed to trace predictions to geospatial and physical evidence.The passage highlights heterogeneous non-RGB inputs and sensitive uses such as policy, disaster assessment, and climate monitoring.Source: §VIII · p. 17
  • Seasonal, geographic, and acquisition shifts challenge RSFM generalization, and reliance on task-specific fine-tuning limits it.Suggested directions include domain-agnostic pretraining, robust cross-modal alignment, and augmentation that simulates real-world variability.Source: §VIII · p. 17
  • Mixture-of-Experts systems could specialize across tasks or modalities and activate only needed experts, but efficient routing and coordination remain challenges.The passage also describes potential scaling across geospatial applications without extensive retraining.Source: §VIII · p. 17

IX. CONCLUSION

  • The survey reviews RSFM background, datasets, technical approaches, benchmarks, and future directions as a resource for guiding research and understanding remaining EO challenges.It aims to support future studies and advance remote sensing interpretation.Source: §IX · p. 17

APPENDIX FIGURE CREDITS

The appendix credits example images for several remote-sensing modalities and lists selected acronyms.

APPENDIX FIGURE CREDITS

  • Figure 4 credits example images for Optical RGB, HSI, LiDAR, TIR, and DSM.The caption identifies the figure as examples of different RS modalities.Source: APPENDIX FIGURE CREDITS · p. 17

ACRONYMS

Related papers

Loading 2410.16602v3…