PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality

Danfeng Hong, Chenyu Li, Xuyang Li, Gustau Camps-Valls, Jocelyn Chanussot

arXiv:2603.00988v1cs.CVcs.SE

TL;DR

RS data are becoming too diverse and complex for traditional approaches, while researchers still face gaps in understanding, categorizing, and applying foundation models. This survey organizes RS foundation models around the shift from unimodality to multimodality and adds tutorial and resource guidance. It reports that multimodal pretrained models consistently outperform unimodal models across representative classification, segmentation, and change-detection tasks.

  • Problem

    The increasing volume and diversity of RS data, together with gaps in researchers’ knowledge, taxonomy, and technical guidance, creates a need for more capable and accessible modeling approaches.

  • Method

    The paper surveys RS foundation models from unimodal and multimodal perspectives and provides tutorial guidance covering selection, configuration, loading, fine-tuning, and deployment.

  • Results

    Multimodal pretrained models consistently achieve superior results across classification, segmentation, and change detection on the PANGAEA-Bench benchmark.

  • Takeaways & Limitations

    The survey offers an accessible pathway for understanding and applying RS foundation models while helping readers account for computational constraints and domain-specific challenges.

Abstract

from arXiv · show

Remote sensing (RS) techniques are increasingly crucial for deepening our understanding of the planet. As the volume and diversity of RS data continue to grow exponentially, there is an urgent need for advanced data modeling and understanding capabilities to manage and interpret these vast datasets effectively. Foundation models present significant new growth opportunities and immense potential to revolutionize the RS field. In this paper, we conduct a comprehensive technical survey on foundation models in RS, offering a brand-new perspective by exploring their evolution from unimodality to multimodality. We hope this work serves as a valuable entry point for researchers interested in both foundation models and RS and helps them launch new projects or explore new research topics in this rapidly evolving area. This survey addresses the following three key questions: What are foundation models in RS? Why are foundation models needed in RS? How can we effectively guide junior researchers in gaining a comprehensive and practical understanding of foundation models in RS applications? More specifically, we begin by outlining the background and motivation, emphasizing the importance of foundation models in RS. We then review existing foundation models in RS, systematically categorizing them into unimodal and multimodal approaches. Additionally, we provide a tutorial-like section to guide researchers, especially beginners, on how to train foundation models in RS and apply them to real-world tasks. The survey aims to equip researchers in RS with a deeper and more efficient understanding of foundation models, enabling them to get started easily and effectively apply these models across various RS applications.

I. INTRODUCTION

Remote sensing data are growing in volume, diversity, and complexity, exposing limitations in traditional models. This survey reviews foundation models in RS, identifies adoption barriers, and positions its broader unimodal-to-multimodal perspective against earlier reviews.

  • RS data increasingly combine imagery, spectral data, temporal sequences, LiDAR, and other modalities that traditional models struggle to represent.
  • Foundation models can be pretrained on diverse datasets and fine-tuned to transfer representations across geospatial tasks, including classification, monitoring, disaster management, and agriculture.
  • B. Challenges and Difficulties: Many RS researchers lack familiarity with foundation-model principles and practical adaptation strategies, creating a barrier to applying these models.
  • B. Challenges and Difficulties: A comprehensive, category-clear, tutorial-oriented taxonomy and standardized evaluation framework remain lacking, complicating model selection and comparison.
  • B. Challenges and Difficulties: Researchers also face uncertainty about fine-tuning, performance, scalability, robustness, and integration protocols for real-world RS applications.
  • C. Comparison with Existing Reviews: Earlier reviews focus mainly on specific subdomains, methods, or applications, whereas this survey examines the transition from unimodality to multimodality and adds practical guidance.

D. Goals and Contributions

The paper aims to make RS foundation models understandable and practically usable. It surveys their evolution toward multimodality, explains core learning paradigms, and provides tutorial guidance for applying pretrained models to downstream tasks.

  • The survey creates an accessible pathway for RS researchers and practitioners to understand and apply foundation models.
  • It provides a detailed RS-focused survey explaining foundation-model principles, operation, and differences from traditional RS processing.
  • The review frames the evolution from unimodal models to multimodal models integrating optical, LiDAR, and SAR data.
  • A tutorial covers applying pretrained models through fine-tuning and integration into existing RS workflows.
  • Foundation-model workflows use self-supervised pretraining on unlabeled RS data followed by transfer to downstream tasks, with weights frozen or fine-tuned.

E. Paper Organization

The paper introduces foundation models in remote sensing, explains their generalized representations and transferability, and organizes a survey of datasets, models, and practical applications.

  • Paper scope: The survey reviews how foundation models function in remote sensing and traces their evolution from unimodal to multimodal approaches.It also presents a tutorial-style guide for applying pretrained models to remote-sensing workflows.
  • Paper scope: Foundation models learn generalized representations by pretraining at large scale on vast, diverse datasets, then adapting to downstream tasks.Their architectures capture complex patterns across heterogeneous data and require limited additional data for specific applications.
  • Remote-sensing focus: Remote-sensing foundation models address diverse modalities, including optical RGB, SAR, spectral, LiDAR, and spatiotemporal data.Their pretraining datasets emphasize sensing modality, spatial resolution, scale, and downstream suitability.
  • Learning and applications: Self-supervised or unsupervised pretraining on unlabeled remote-sensing data reduces dependence on extensive labeled datasets and supports efficient transfer learning.Fine-tuning adapts pretrained representations to downstream applications with relatively small amounts of labeled data.
  • Learning and applications: Foundation models can handle varying resolutions, sensor types, and temporal frequencies while scaling to continuous satellite imagery and global climate data.These properties support scientific research and practical applications requiring large-scale, timely analysis.

B. Learning Paradigm in RS

Remote sensing is shifting from task-specific supervised learning toward self-supervised pretraining followed by fine-tuning, using abundant unlabeled data to improve transfer across applications.

  • Traditional paradigm: Traditional remote-sensing pipelines either extract features before task-specific prediction or directly map inputs to outputs with end-to-end models.These approaches are typically organized around individual tasks such as classification, detection, or environmental monitoring.
  • Foundation-model paradigm: Foundation models introduce a “self-supervised pretraining + fine-tuning” paradigm that learns generalized representations from massive unlabeled remote-sensing data.Pretraining identifies inherent patterns, structures, and relationships in the data.
  • Foundation-model paradigm: Fine-tuning adapts pretrained representations to downstream objectives using relatively small labeled datasets.The process aligns representations with tasks such as land-cover classification, change detection, and object recognition.
  • Benefits: This paradigm reduces dependence on extensive labels while maintaining high performance across varied remote-sensing applications.It addresses annotation scarcity caused by the difficulty of pixel-level labeling over large and complex geographic areas.
  • Benefits: The roadmap shows increasing attention to multimodal foundation models, particularly after 2024, as diverse remote-sensing data sources become integrated.The shift includes optical imagery, SAR, spectral data, and other modalities.

A. Overview: Statistical Analysis

The survey analyzes 105 remote-sensing foundation-model papers and finds rapid growth, a shift toward multimodality, and broad geographic and venue participation.

  • Statistical analysis: The authors collected 105 papers and analyzed modality distribution, publication venues, and country-wise contributions.Figure 2 summarizes these three statistical perspectives.
  • Statistical analysis: Remote-sensing foundation-model development shifted clearly toward multimodal approaches, especially in 2024, with optical, SAR, and spectral data increasingly integrated.The trend is presented as reflecting demand for more comprehensive and robust remote-sensing solutions.
  • Publication venues: ArXiv hosts the largest number of papers, while IEEE TGRS ranks second and CVPR third among the reported publication venues.The venue distribution reflects both early dissemination and increasing overlap between remote sensing, geoscience, AI, and computer vision.
  • Geographic distribution: China leads country-wise contributions, followed by the USA and Australia.The distribution indicates participation from established AI hubs and emerging remote-sensing AI leaders.
  • Unimodal models: Unimodal foundation models cover RGB, radar, point-cloud, spectral, and meteorological data from individual sources or sensor types.Examples target modality-specific features and challenges.
  • Unimodal models: Scale-MAE emphasizes scale-invariant representation learning, whereas SatMAE++ emphasizes resolution-aware feature extraction.The former supports cross-scale generalization, while the latter is advantageous for high-resolution applications.

2) Spectral Data:

Remote-sensing foundation models increasingly combine spectral, temporal, geospatial, optical, SAR, and other heterogeneous information through specialized pretraining and fusion strategies.

  • 2) Spectral Data:: Spectral foundation models learn spatial-spectral coupling and spectral sequentiality from multispectral or hyperspectral observations.SpectralGPT uses 3D masking and spectral consistency constraints to learn structured spectral representations.
  • 2) Spectral Data:: Multi-temporal and multi-band models improve spatial and spectral information reconstruction by modeling channel, temporal, and spatial relationships.Examples include grouping masks, contrastive learning, channel masking, and adaptive fine-tuning.
  • Multimodal integration: Multimodal foundation models integrate complementary data types, including time series, geospatial information, text, language models, and audio.They extend beyond single modalities and can combine homogeneous or heterogeneous inputs.
  • Multimodal integration: Cross-modal attention, graph-based reasoning, and co-learning dynamically capture complementarities across modalities for complex downstream tasks.These are presented as advanced interactive fusion strategies.
  • Heterogeneous data: Optical-SAR and broader multimodal models use contrastive learning, masked modeling, modality-specific modules, and decoupled representations.These designs support combinations involving optical, SAR, spectral, DSM, weather, elevation, and depth data.
  • Heterogeneous data: Holistic models such as OmniSat integrate high-resolution RGB, spectral time series, and radar time series through heterogeneous modules and contrastive masking.The cited models target broad multimodal remote-sensing applications.

D. A Concise Summary of Foundation Models in RS

The survey categorizes RS foundation models into unimodal and multimodal approaches and compares their architectures, applications, code availability, and benchmark performance. Multimodal models consistently outperform unimodal models across representative RS tasks.

  • The survey organizes unimodal and multimodal RS foundation models by publications, pretrained architectures, downstream applications, and code availability.
  • Multimodal pretrained models consistently achieve superior results across classification, segmentation, and change detection tasks.
  • Table IV compares unimodal and multimodal models on the PANGAEA benchmark, which covers 11 datasets spanning semantic segmentation, change detection, and regression.
  • Multimodal models occupy top-performing positions across nearly all tasks and deliver substantial gains over unimodal models on datasets such as CTM-SS.

IV. HOW TO USE PRETRAINED FOUNDATION MODELS IN RS DOWNSTREAM TASKS: A TUTORIAL

The tutorial presents a step-by-step workflow for applying pretrained RS foundation models to downstream tasks, from model selection and environment setup through fine-tuning and deployment.

  • The workflow covers model selection, environment setup, model loading, fine-tuning, and deployment for applying pretrained RS foundation models.The workflow is illustrated in Fig. 4.

A. Choose Right RS Foundation Models

Selecting and preparing a pretrained RS foundation model requires matching its data, architecture, features, environment, and input dimensions to the downstream task. These choices support reliable model loading and later fine-tuning.

  • Model selection should account for pretraining data, network architecture, model features, and downstream-task requirements.
  • Pretraining data should be evaluated by sensor type, satellite platform, image dimensions, channel count, and data format.These factors indicate alignment between the model’s pretraining data and downstream inputs.
  • Architecture selection should consider the backbone type, parameter count, computational complexity, lightweight deployment, and real-time performance requirements.
  • Model-specific features, including multimodal integration and temporal information, should be matched to specialized task needs.
  • The runtime environment must provide compatible libraries, frameworks, and pretrained weights to support model loading and fine-tuning.RS workflows commonly use Python, deep learning frameworks, and geospatial data-processing tools.
  • Pretrained models may require adaptation when downstream image dimensions differ from fixed input sizes such as 224 × 224 × 3 versus 128 × 128 × 3.Suggested approaches include image resizing or interpolation and positional-encoding interpolation for transformer models.

2) Handling Mismatches w.r.t Channel Dimension:

RS channel mismatches arise because data may range from single-channel SAR imagery to images with dozens or hundreds of spectral channels. The tutorial presents several adaptation strategies before fine-tuning and deployment.

  • 2) Handling Mismatches w.r.t Channel Dimension:: RS inputs can have channel counts that differ substantially from pretrained-model requirements, including single-channel SAR and dozens or hundreds of spectral channels.
  • 2) Handling Mismatches w.r.t Channel Dimension:: Channel mismatches can be addressed through channel duplication, PCA dimensionality reduction, added convolutional layers, or direct input-layer modification.
  • 2) Handling Mismatches w.r.t Channel Dimension:: Fine-tuning may freeze selected pretrained parameters and train task-specific modules or update the entire model.Resource-aware fine-tuning reduces computational overhead, whereas comprehensive fine-tuning targets high precision when resources are available.
  • 2) Handling Mismatches w.r.t Channel Dimension:: After fine-tuning, deployment combines pretrained generalization with task-specific optimization for predictions on new data.
  • 2) Handling Mismatches w.r.t Channel Dimension:: The tutorial extends the workflow to vision-language foundation models because real-world RS applications may require joint reasoning over imagery and textual or metadata information.

1) Step 1: Clarify Task Type and Data Pairing: •

Multimodal RS applications require matching task types with suitable image–text data, model architectures, token alignment, and adaptation strategies. The workflow emphasizes resource-aware design and task-specific supervision.

  • Task taxonomy: Downstream tasks include image-to-text, text-to-image retrieval, visual question answering, instruction following, and multimodal classification or detection.
  • Data pairing: Because RS corpora rarely contain rich captions, practitioners can combine metadata, language-model refinement, teacher-model pseudo-captioning, and human filtering.
  • Data pairing: Text granularity should match the task, since fine-grained supervision can improve VQA and grounding while increasing annotation cost.
  • Model selection: Candidate architectures include encoder–decoder models, unified transformers, and RS-specific vision-language foundation models.
  • Multimodal alignment: Connectors must accommodate RS vision tokens, while spectral descriptors can expose band semantics to the language component.
  • Fine-tuning: Lightweight adaptation freezes the vision encoder and language model while tuning connectors, LoRA or adapters, and spectral projections; instruction tuning supports VQA and dialogue.
  • Fine-tuning: Multitask training can combine contrastive learning, image-conditioned language modeling, and multilingual targets to support generalization.

5) Step 5: Deployment and Prompting: •

Effective RS foundation-model deployment requires resource-aware evaluation, careful adaptation, and prompting that reflects sensor and geographic context. The review also identifies unresolved questions about generality, transferability, scalability, retention, robustness, and uncertainty.

  • 5) Step 5: Deployment and Prompting: •: Structured prompts that specify RS context, sensor, resolution, and bands are recommended to reduce hallucination.
  • 5) Step 5: Deployment and Prompting: •: Deployment efficiency can improve through language-head distillation, key-value caching, and precomputed image embeddings for retrieval.
  • 5) Step 5: Deployment and Prompting: •: Evaluation should combine standard NLP metrics with RS-aware accuracy, geospatial grounding, and expert assessment for safety-critical tasks.
  • F. Computational Resources for Foundation Models: Resource comparisons standardize forward-pass assumptions while reporting FLOPs, parameters, and empirical inference-memory ranges that vary with implementation conditions.
  • V. CONCLUSION: The field has progressed from unimodal representations toward multimodal models combining optical, SAR, spectral, LiDAR, and auxiliary information.
  • V. CONCLUSION: The tutorial organizes model selection, environment configuration, loading, fine-tuning, and deployment while accounting for RS characteristics and computational constraints.
  • VI. FUTURE DIRECTIONS AND CHALLENGES: Open questions include scaling laws, criteria for general RS foundation models, and why random or cross-domain pretrained weights can sometimes match RS-specific models.
  • VI. FUTURE DIRECTIONS AND CHALLENGES: Further challenges concern catastrophic forgetting, data-management bottlenecks, multimodal collaboration, sensor and domain robustness, and uncertainty-aware deployment.

VII. MAIN ABBREVIATION

The paper compiles abbreviations spanning neural architectures, remote-sensing foundation models, adaptation methods, datasets, and geospatial concepts.

  • The abbreviation list covers convolutional networks, masked autoencoders, self-supervised learning, geospatial foundation models, and related RS terminology.
Loading 2603.00988v1…