PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Unified Copernicus Foundation Model for Earth Vision

Yi Wang, Zhitong Xiong, Chenying Liu, Adam J. Stewart, Thomas Dujardin, Nikolaos Ioannis Bountos, Angelos Zavras, Franziska Gerken, Ioannis Papoutsis, Laura Leal-Taixé, Xiao Xiang Zhu

arXiv:2503.11849v3cs.CV

TL;DR

Existing EO foundation models are limited by fixed sensor modalities, surface-focused data, and underused metadata. The paper introduces a unified dataset, adaptable foundation model, and hierarchical benchmark, reporting improved downstream performance across surface and atmospheric tasks while defining a scope limited to Sentinel data from about one year.

  • Problem

    Existing EO foundation models largely use fixed spectral sensors, focus on Earth’s surface, and overlook metadata beyond imagery.

  • Method

    The paper combines Copernicus-Pretrain, Copernicus-FM with dynamic hypernetworks and metadata encoding, and Copernicus-Bench with 15 hierarchical downstream tasks.

  • Results

    The unified approach improves downstream performance on Sentinel-3 and Sentinel-5P tasks and outperforms supervised training on 11/15 tasks.

  • Takeaways & Limitations

    The work supports unified multimodal EO pretraining and creates opportunities to connect EO representations with weather and climate research.

  • Takeaways & Limitations

    The study is restricted to the Sentinel series and a temporal range of about 1 year.

Abstract

from arXiv · show

Advances in Earth observation (EO) foundation models have unlocked the potential of big satellite data to learn generic representations from space, benefiting a wide range of downstream applications crucial to our planet. However, most existing efforts remain limited to fixed spectral sensors, focus solely on the Earth's surface, and overlook valuable metadata beyond imagery. In this work, we take a step towards next-generation EO foundation models with three key components: 1) Copernicus-Pretrain, a massive-scale pretraining dataset that integrates 18.7M aligned images from all major Copernicus Sentinel missions, spanning from the Earth's surface to its atmosphere; 2) Copernicus-FM, a unified foundation model capable of processing any spectral or non-spectral sensor modality using extended dynamic hypernetworks and flexible metadata encoding; and 3) Copernicus-Bench, a systematic evaluation benchmark with 15 hierarchical downstream tasks ranging from preprocessing to specialized applications for each Sentinel mission. Our dataset, model, and benchmark greatly improve the scalability, versatility, and multimodal adaptability of EO foundation models, while also creating new opportunities to connect EO, weather, and climate research. Codes, datasets and models are available at https://github.com/zhu-xlab/Copernicus-FM.

1. Introduction

Existing EO foundation models remain constrained by limited sensor diversity, rigid modality-specific architectures, and incomplete evaluation. The paper addresses these gaps with a unified dataset, adaptable model, and hierarchical benchmark spanning surface and atmospheric observations.

  • Research gaps: Current pretraining datasets mainly use high- or medium-resolution surface sensors, excluding temporally rich Sentinel-3 and Sentinel-5P observations.These missions provide near-daily global coverage of land, oceanic, and atmospheric variables.
  • Research gaps: Most EO foundation models use rigid architectures that cannot dynamically accommodate new spectral bands or non-spectral variables.Recent spectral-flexibility methods still lack mechanisms for non-spectral EO inputs.
  • Contributions: Copernicus-Pretrain integrates 18.7 million aligned observations from all major operational Sentinel missions, including atmospheric variables and coarse-scale observations.The dataset is designed to support holistic modeling of Earth system interactions with wider and more frequent coverage.
  • Contributions: Copernicus-FM processes spectral and non-spectral sensors with dynamic hypernetworks and incorporates metadata for practical applications.The model is presented as a unified foundation model for heterogeneous EO modalities.
  • Contributions: Copernicus-Bench evaluates foundation models through 15 hierarchical tasks spanning preprocessing, base applications, and specialized applications across Sentinel missions.The benchmark supports systematic assessment at different levels of practical application.

2. Related work

Related work has progressed from large single-sensor datasets and specialized foundation models toward multimodal pretraining, but existing benchmarks and datasets remain limited in sensor and task diversity. Copernicus-Pretrain is positioned to support joint surface–atmosphere modeling at scale.

  • EO pretraining datasets: Early EO pretraining datasets established large-scale learning, while later efforts expanded coverage, temporal diversity, and dataset size for Sentinel-1/2 and Landsat imagery.Examples include fMoW, Million-AID, SEN12MS, SeCo, SSL4EO, SatlasPretrain, and Major TOM.
  • Research direction: Recent dataset efforts aim at joint surface–atmosphere modeling at scale.This direction motivates broader multimodal EO pretraining beyond surface observations.
  • EO foundation models: Most EO foundation models focus on optical data and use contrastive, masked-image-modeling, or hybrid pretraining strategies.The related work categorizes representative single-sensor approaches by their pretraining objectives.
  • EO benchmarks: Existing EO benchmarks generally cover limited sensor and task diversity, with many concentrating on optical imagery or specialized application areas.SustainBench, GEO-Bench, PhilEO Bench, and FoMo-Bench illustrate different but bounded evaluation scopes.

3. Copernicus-Pretrain

Copernicus-Pretrain organizes aligned Sentinel observations and elevation data in ERA5-consistent geographic grids. Its curation pipeline combines global grid construction, modality-specific sampling and filtering, and quality checks to produce a large multimodal dataset.

  • Dataset design: Copernicus-Pretrain aligns Sentinel-1 SAR, Sentinel-2 reflectance, Sentinel-3 radiance, Sentinel-5P atmospheric variables, and Copernicus DEM GLO30 in dense regional grids.The dataset is intended to extend both the scale and modality range of EO foundation-model research and connect EO with weather and climate studies.
  • Grid construction: The dataset follows ERA5 coordinates by dividing the globe into about 1M 0.25° × 0.25° cells, then retaining about 393K land grids after coastal buffering.Each cell covers approximately 28 km × 28 km.
  • Modality-specific collection: Sentinel-3 and Sentinel-5P are sampled across whole cells with modality-specific patch sizes, cloud or NaN filtering, and variable sequence lengths.Sentinel-3 uses eight randomly selected yearly images after cloud filtering, while Sentinel-5P includes four atmospheric variables.
  • Modality-specific collection: Sentinel-1 and Sentinel-2 use Gaussian local-patch sampling around populated cities, with additional patches added to cells lacking coverage.This strategy addresses excessive data volume and redundancy while filling spatial gaps.
  • Dataset scale: Quality checks yield about 310K grids with at least one sensor and about 220K grids containing all modalities.The final dataset contains approximately 18M images across eight Sentinel modalities and elevation data.

4. Copernicus-FM

Copernicus-FM uses sensor-aware hypernetworks and metadata encoding to process heterogeneous spectral and non-spectral modalities within one architecture. It is pretrained with masked reconstruction and auxiliary continual distillation across aligned modality inputs.

  • Unified architecture: Copernicus-FM processes spectral and non-spectral inputs with varied spatial resolutions while flexibly integrating geolocation, area, and time metadata.Dynamic hypernetworks patchify modalities, and Fourier-encoded metadata is added to patch tokens when available.
  • Dynamic patch embedding: Spectral hypernetworks generate patch-embedding weights from wavelength and bandwidth encodings, then dynamically reshape kernels for each modality’s patch size.This accommodates substantial resolution differences without requiring one fixed patch size across modalities.
  • Dynamic patch embedding: Variable hypernetworks extend dynamic patch embedding to non-spectral modalities by generating weights from encoded variable names.A frozen language model encodes variable names offline into vectors used by the hypernetwork.
  • Metadata integration: Metadata encodings represent geolocation, spatial coverage, and acquisition time, with learnable tokens used when metadata is unavailable.The encodings are processed, expanded across patch dimensions, and added to positional encodings.
  • Training objectives: Pretraining combines MAE-style masked image reconstruction for each modality with auxiliary continual distillation from specialized teachers such as DINOv2.Distillation uses a small loss weight to refine the latent space and improve out-of-the-box representation quality.
  • Ablation studies: Adding spectral bandwidth, variable hypernetworks, metadata encoding, and continual distillation produces consistent ablation improvements, with metadata encoding especially beneficial for non-optical modalities.The ablations use a ViT-Small on a 10K subset of Copernicus-Pretrain.

5. Copernicus-Bench

Copernicus-Bench evaluates EO foundation models across 15 hierarchical datasets spanning preprocessing, base applications, and specialized Sentinel-mission tasks. The benchmark uses standardized frozen-encoder and supervised-training protocols and reports broad downstream gains.

  • Datasets: Copernicus-Bench contains 15 datasets organized into preprocessing, base applications, and specialized applications across Sentinel modalities.The levels include 2 cloud detection, 8 land-use/land-cover, and 5 specialized tasks.
  • Datasets: Six newly curated datasets address gaps in ML-ready Sentinel-3 and Sentinel-5P tasks, while nine derive from existing permissively licensed datasets.The benchmark’s dataset characteristics and task abbreviations are summarized in Table 3.
  • Evaluation protocol: Evaluation combines linear probing for classification with frozen-encoder UPerNet decoders for segmentation and regression, plus feature-map differences for change detection.Supervised baselines are trained from scratch, and reported metrics use three runs with validation-selected learning rates.
  • Benchmark results: Copernicus-FM largely improves downstream performance on Sentinel-3 and Sentinel-5P tasks and outperforms supervised training on 11/15 tasks.The results support cross-modal pretraining benefits for both surface and atmospheric applications.

6. Bridging EO & climate via grid embeddings

Copernicus-Pretrain’s grid alignment connects EO representations with ERA5 climate parameters. Combining grid embeddings with coordinates yields the best overall climate-task performance, motivating global embedding resources and future weather applications.

  • EO–climate alignment: Copernicus-Pretrain grid embeddings are directly aligned with ERA5 climate parameters through shared geographic grid cells.The alignment enables climate tasks using EO-derived representations alongside geographic information.
  • Climate prediction: Combining grid embeddings with coordinates yields the best overall performance, demonstrating complementary spatial context and EO-derived semantic features.Grid embeddings also outperform location encodings from a specialized EO-trained location encoder.
  • Future applications: The authors extend a global embedding dataset at 0.25° and identify medium-range weather forecasting as a future application.They propose using EO-derived representations for static or dynamic variables in weather models.

7. Conclusion

The work advances EO foundation modeling through a unified dataset, adaptable model, and hierarchical benchmark spanning Sentinel observations of Earth’s surface and atmosphere. It also demonstrates climate-oriented potential while remaining limited to Sentinel data collected over about one year.

  • Copernicus-Pretrain, Copernicus-FM, and Copernicus-Bench advance EO foundation modeling across data, model, and evaluation.The dataset covers major Sentinel missions, the model adapts to spectral and non-spectral modalities, and the benchmark organizes hierarchical downstream tasks.
  • EO grid embeddings show potential for connecting EO foundation models with weather and climate studies.The work demonstrates this potential in a simple climate prediction task.
  • The study is restricted to the Sentinel series and a temporal range of about 1 year.The authors identify efficient satellite expansion and native multimodal fusion and time-series processing as future directions.

Supplementary Material

The supplementary material provides additional implementation details, analyses, visualizations, and ablation studies for the paper’s datasets and foundation model.

  • The supplementary material reports additional implementation details, analyses, visualizations, and ablation studies.

A.1. Comparison to existing EO pretraining datasets

The supplementary material compares Copernicus-Pretrain with existing EO pretraining datasets and details its aligned-grid organization, coverage statistics, local patches, and time-series lengths.

  • Comparison to existing EO pretraining datasets: Copernicus-Pretrain is compared in detail with several existing EO pretraining datasets.
  • Dataset statistics: The supplementary statistics include histograms of local patch numbers for S1/2 in both union and joint datasets.
  • Dataset statistics: Time-series length histograms are provided for S1, S2, S3, and S5P across union and joint subsets where specified.S1 and S2 use separate union and joint figures; S3 and S5P likewise have union and joint visualizations.
  • Dataset statistics: The joint subset’s global distribution is presented in Figure 7.

B.1. Subtractive ablation

The supplementary material examines Copernicus-FM through subtractive ablations, metadata and variable encodings, training details, and benchmark documentation. These analyses describe how the model represents heterogeneous sensors and metadata and how missing metadata is handled.

  • Subtractive ablation: The subtractive ablation isolates the benefits of individual Copernicus-FM components regardless of their order.The study complements the incremental ablation from the main paper.
  • Encoding modalities: Fourier encoding of wavelengths and bandwidths supplies the spectral hypernetwork with inputs for generating patch-embedding weights.The encoding covers spectral channels from the dataset’s sensors.
  • Encoding modalities: Language-model encodings of variable names generate patch-embedding weights for non-spectral modalities and preserve semantic relationships between variables.The t-SNE visualization reports clustering of S5P variables and separation from out-of-domain concepts.
  • Encoding modalities: Language encoding is more flexible than random hashing and more flexible than spectroscopy, while spectroscopy retains semantics.The comparison distinguishes flexibility from semantic structure across the encoding options.
  • Metadata ablations: Metadata is Fourier-encoded and added to positional encodings, with geolocation providing the largest reported improvement, followed by area and time.The metadata ablations also evaluate dropping ratios and alternative geographic-coordinate formats.
  • Implementation and benchmark: Copernicus-FM is pretrained on 220K aligned grids using masked image modeling and continual distillation, with training details and benchmark curation documented.The supplementary material specifies preprocessing, model configuration, loss design, optimization, and comparisons with existing EO benchmarks.

C.2. Benchmark curation

Copernicus-Bench organizes 15 datasets into three task levels spanning all primary Sentinel missions, combining adapted existing datasets with newly curated S3 and S5P resources. The benchmark includes classification, segmentation, regression, multimodal, temporal, and climate-oriented evaluation settings with documented metadata and experimental procedures.

  • Benchmark scope: 15 datasets are organized into three task levels covering all primary Copernicus Sentinel missions.Nine datasets are adapted from existing permissively licensed sources, while six are newly curated to address S3/5P gaps.
  • Dataset sources: Nine datasets are adapted from existing datasets, and six are newly curated for machine-learning-ready Sentinel-3 and Sentinel-5P coverage.
  • Sourced datasets: Cloud-S2, EuroSAT, BigEarthNet, DFC2020, Flood-S1, and LCZ-S2 provide adapted classification, segmentation, and change-detection datasets across Sentinel-1 and Sentinel-2.Paired Sentinel-1/2 datasets can support multimodal evaluation, while Flood-S1 uses pre- and post-event SAR imagery.
  • New datasets: New datasets extend evaluation to Sentinel-3 and Sentinel-5P through cloud segmentation, land-cover classification and segmentation, biomass regression, and air-quality regression.Cloud-S3 supports both multi-class and binary segmentation; the S3 and S5P datasets use time-series, annual, or seasonal imagery modes.
  • Documentation and evaluation: Figures visualize dataset examples, geographical distribution, metadata availability, and task-specific benchmark settings.The benchmark documentation notes that some datasets lack geolocation or time metadata, and reports experiments using supervised baselines and frozen-encoder transfer learning.

D.2. Copernicus embedding dataset

Copernicus-Embed-025deg converts multimodal Copernicus-FM representations into a global 0.25° embedding map. The dataset is intended to connect Earth-observation imagery with weather and climate data at high spatial coverage and compression.

  • Dataset construction: Copernicus-Embed-025deg averages Copernicus-FM embeddings across modalities for 0.25° grid cells, producing a global map with shape 721x1440x768.Empty ocean grids are filled with zero, and the map is visualized using its top three principal components as RGB channels.
  • Representation: The embedding map provides a semantic representation that integrates multiple satellite-observation sources at an extremely high compression ratio.
  • Applications: Copernicus-Embed-025deg facilitates alignment between Earth-observation imagery and weather or climate data, including use as static variables alongside ERA5.The paper presents this alignment as an opportunity for developing weather and climate foundation models.
Loading 2503.11849v3…