PaperlayerView exact revision on arXiv

Source-linked AI summary

UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation

Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu

arXiv:2606.23503v1cs.CV

TL;DR

Earth Observation models often rely on rigid patch projectors that constrain heterogeneous sensor inputs, resolutions, and modalities. UniverSat replaces them with a Universal Patch Encoder and achieves transfer across diverse tasks while a single shared backbone matches or surpasses modality-specific foundation models.

  • Problem

    Rigid patch projectors constrain Earth Observation models to specific modalities, resolutions, and preprocessing choices despite heterogeneous sensor inputs.

  • Method

    UniverSat uses a Universal Patch Encoder with axial cross-attention to map arbitrary-spatial, spectral, and temporal patches into shared embeddings for self-supervised multimodal training.

  • Results

    Across 13 sensors and 7 heterogeneous datasets, UniverSat transfers effectively across downstream Earth Observation tasks, matching or surpassing modality-specific foundation models.

  • Takeaways & Limitations

    A single shared backbone can provide modality- and resolution-agnostic representations across diverse Earth Observation data and tasks.

  • Takeaways & Limitations

    UniverSat trades specialization for generality and may be less accurate or efficient than modality-specific models in standard settings, while unseen non-optical sensors require additional modality encoding.

Abstract

from arXiv · show

Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space with a shared set of weights. This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and segmentation on standard EO benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.

1 Introduction

UniverSat addresses the limitations of rigid ViT patch projectors in heterogeneous Earth Observation by using a Universal Patch Encoder for arbitrary spatial, spectral, and temporal inputs. Its unified architecture supports multimodal self-supervised training across diverse sensors while preserving flexible output resolution and competitive performance.

  • Motivation: EO foundation models typically fix band counts, temporal sampling, and patch size, limiting their use across sensors with varying resolutions, densities, and extents.This rigidity is adequate for narrow modality sets but becomes limiting for heterogeneous EO workflows.
  • Architecture: UniverSat replaces fixed patch projectors with a Universal Patch Encoder that embeds patches of arbitrary spatial, spectral, and temporal dimensions into a shared latent space.The encoder uses linear-complexity axial cross-attention within a transformer operating over spatialized tokens.
  • Architecture: A single set of weights processes modality combinations and arbitrary resolutions without input resampling or channel filtering.The model is modality-agnostic across heterogeneous EO inputs.
  • Architecture: The architecture decouples output feature-map resolution from input patch size, preserves fine spatial details through a skip connection, and remains compatible with standard self-supervised objectives.These properties provide resolution flexibility, granularity, and SSL compatibility while retaining the core ViT structure.
  • Training and evaluation: 7 datasets spanning 13 sensors across four modalities are jointly used for self-supervised training, covering 0.1–300 m GSD, 1–150 timestamps, and 1–396 spectral channels.The modalities are optical, hyperspectral, radar, and elevation, and the model is designed to incorporate unseen sensor configurations.
  • Contributions: UniverSat provides a unified EO architecture, multimodal self-supervised framework, competitive performance across diverse datasets and tasks, and demonstrated generalization to unseen sensors and modality combinations.The stated evaluation spans high-resolution RGB, radar time series, and hyperspectral benchmarks.

2 Related work

Related work spans self-supervised, multimodal, and flexible EO foundation models. Despite progress across objectives, modalities, and input variability, many approaches remain limited in generality or still resample inputs.

  • Self-supervised Learning for EO: EO self-supervision progressed from contrastive learning to generative masking and predictive objectives, while incorporating spectral, temporal, and spatial priors.Several methods integrate EO priors into the loss or encoder.
  • Self-supervised Learning for EO: Many EO self-supervised pipelines remain monomodal and monotemporal, while some use semantic products as inputs and thereby move toward semi-supervision.These limitations reduce generality and blur the distinction between self-supervision and semi-supervision.
  • Multimodal SSL for EO: Multimodal EO pretraining uses geo-registration for masked cross-modal reconstruction and cross-sensor contrast, with teacher–student hybrids improving stability at scale.Most studies focus on Sentinel-1/2 time-series and single-date inputs, while some also cover elevation and hyperspectral data.
  • Flexible Models: Flexible EO backbones address temporal, spectral, and spatial variability through variable-length series, band strategies, wavelength encodings, and scale-aware mechanisms.Nevertheless, many pipelines still resample inputs.

3 Method

UNIVERSAT processes heterogeneous Earth observations with a shared Universal Patch Encoder that produces resolution-agnostic embeddings, then fuses modalities into dense feature maps at user-specified output resolutions. It is trained self-supervised using cross-modal contrast and latent multimodal masked modeling.

  • Universal Patch Encoder: The Universal Patch Encoder maps patches from arbitrary sensor modalities and spatial, spectral, or temporal resolutions into a shared fixed-width embedding space.It replaces the fixed patch projector of a standard ViT and progressively collapses patch dimensions using axial cross-attention.
  • Universal Patch Encoder: UPE sequentially collapses pixel, channel, time, and sub-patch dimensions, producing a global patch embedding and sub-patch embeddings for high-resolution skip connections.The outputs are f ∈ R8D and f_sub ∈ R^{S×4D}.
  • Universal Patch Encoder: Axial cross-attention aggregates atomic tokens one axis at a time, avoiding impractical MLP projection and prohibitively expensive full self-attention.Restricting attention to a single axis gives linear complexity in the number of atomic tokens.
  • UNIVERSAT Backbone: UNIVERSAT fuses co-registered modality embeddings with axial cross-attention, processes the resulting patch tokens with Transformer blocks, and preserves fine details through sub-token cross-attention.Its output resolution is specified at inference time by resampling patch embeddings to a target grid and attending to sub-patch embeddings.
  • Self-Supervised Training: Self-supervised training combines patch-level cross-modal contrast with latent multimodal masked modeling on spatially aligned, heterogeneous datasets.Aggressive masking drops modalities, timestamps, channels, and patches to improve robustness across scales and sensor configurations.

4 Experiments

UNIVERSAT is trained on a heterogeneous seven-dataset corpus spanning 13 sensors and broad spatial, temporal, and spectral conditions, then evaluated across 16 EO datasets. It transfers robustly to unseen sensor configurations and specialized hyperspectral tasks, while ablations show trade-offs among generality, resolution control, fusion, and efficiency.

  • Probing Experiments: UNIVERSAT achieves state-of-the-art results on BrickKiln and Sen1Flood11 under strict kNN and linear probing, despite greater generality than sensor- or timestamp-specific competitors.The evaluation covers 16 datasets from GeoBench, PangaeaBench, and SpectralEarth.
  • Probing Experiments: Using a linear probe with 3700–5000ˆ fewer supervised parameters, UNIVERSAT remains competitive on PangaeaBench and reaches state-of-the-art performance on PASTIS-R and AI4Farms.Competing approaches use heavyweight decoder heads such as UperNet, whereas UNIVERSAT exploits dense embeddings.
  • Probing Experiments: UNIVERSAT maintains strong performance on unseen mono-temporal Sentinel-1/2, reduced-band Sentinel-2, and synthetic HLS configurations, demonstrating robustness to new sensors.These configurations were not present during training.
  • Hyperspectral Data: On EnMAP-based SpectralEarth tasks, UNIVERSAT consistently outperforms DOFA without EnMAP training, surpasses SpatSIGMA, and approaches the specialized SpectralEarth-L model.The model remains competitive in hyperspectral analysis despite using probing rather than task-specific training.
  • Ablation Study: Replacing the UPE with modality-specific MLPs causes a marked performance drop, increases parameters by 58%, removes unseen-sensor support, and halves training time; other ablations show resolution control, skip connections, late fusion, and contrastive loss matter.Removing the skip connection helps PASTIS but harms other benchmarks, while disabling resolution control particularly hurts generalization to unseen datasets and removing contrastive loss harms multimodal segmentation.

5 Conclusion

UNIVERSAT is a multimodal Earth Observation transformer that replaces fixed patch projectors with a Universal Patch Encoder. Self-supervised training across 13 sensors and 7 heterogeneous datasets produces modality- and resolution-agnostic representations that transfer to downstream EO tasks.

  • Architecture: UNIVERSAT replaces fixed patch projectors with a Universal Patch Encoder for multimodal Earth Observation inputs of any resolutions.This enables a single model to process multimodal inputs across varying resolutions.
  • Training: 13 sensors and 7 heterogeneous datasets were used for self-supervised training.The training setup spans multimodal Earth Observation data from diverse sensors and datasets.
  • Transfer: UNIVERSAT learns modality- and resolution-agnostic representations that transfer effectively to a wide range of downstream Earth Observation tasks.The shared representation is intended to support transfer across downstream tasks despite differences in modality and resolution.

Supplementary Material

The supplementary material covers compute cost, licensing, reproducibility, methodological insights into the Universal Patch Encoder, training datasets, and extended comparisons with competing methods.

  • Supplementary Material: The appendix details compute cost and licensing, reproducibility, Universal Patch Encoder methodology, training datasets, and comparisons with competing methods.These topics are organized into Sections A.1–A.4.

A.1 Compute Cost and License

UniverSat’s code and pretrained models are released under the MIT License, while experiments rely on substantial H100 GPU resources, including approximately 240 GPU-hours for self-supervised training.

  • License: Code and pretrained models will be released under the MIT License, permitting unrestricted use, modification, and distribution.All datasets are publicly available and used according to their respective licenses.
  • Compute Resources: Training UNIVERSAT in the self-supervised setting requires approximately 240 GPU-hours on H100 GPUs.Linear probing requires 1–10 GPU-hours per dataset, totaling 60 GPU-hours; the full project required approximately 30K GPU-hours.

A.2 Reproducibility Details

This section summarizes the implementation and evaluation details required to reproduce the experiments.

  • The section presents implementation and evaluation details needed to reproduce the experiments.

A.2.1 Model Configurations

Table A.1 presents the model configuration details needed to reproduce UNIVERSAT-Tiny and UNIVERSAT-B.

  • Table A.1 provides configuration details for reproducing UNIVERSAT-Tiny and UNIVERSAT-B.

A.2.2 Self-Supervised Pretraining

This section specifies the self-supervised pretraining setup and the modality- and dataset-level configurations used to prepare inputs and targets. The details cover optimization hyperparameters, modality preprocessing, and spatial settings across datasets.

  • Pretraining optimization: Self-supervised pretraining is defined through explicit hyperparameters and optimization details.These settings are summarized in Table A.2.
  • Modality preprocessing: Modality preprocessing specifies native resolution, input bands or descriptors, subpatch factors, and the datasets using each modality.These modality-level settings are summarized in Table A.3.
  • Dataset spatial configuration: Dataset spatial configuration specifies image size, input and target output scales, and the modalities used for each dataset.These dataset-level settings are summarized in Table A.4.

A.3 Additional Methodological Details … A.4 Data and Models

UniverSat encodes heterogeneous EO inputs with learnable metadata-aware modules, axial cross-attention, aggressive multimodal masking, and LM3 self-supervision. Training uses seven heterogeneous datasets and supports unseen configurations with flexible outputs through shared weights.

  • A.3.1 Learnable Fourier Features (LFF).: Learnable Fourier Features lift scalar values such as radiance, time, and wavelength into D-dimensional embeddings using learnable frequencies and phases.When applied to tensors, the embeddings operate element-wise.
  • A.3.2 Metadata Encoding.: Metadata encoding represents optical wavelengths and timestamps with dedicated LFFs, non-optical channels with learned lookups, and spatial positions with GSD-scaled RoPE.Additive metadata enters keys and values, while RoPE transforms projected queries and keys, making ACA modality- and geometry-aware.
  • A.3.3 Axial Cross-Attention (ACA).: Axial cross-attention collapses one tensor axis at a time because direct MLP projection is impractical and full self-attention is too expensive.Each query attends only along the collapsed axis, giving cost linear in the number of atomic tokens per patch.
  • A.3.4 Scale Augmentation and Masking.: Scale augmentation samples input patch sizes and target resolutions from dataset-specific presets, while masking independently drops modalities, timestamps, channels, and patches.Modality dropping occurs with probability 30%, time-series timestamps with probability 50%, and approximately 90% of input atoms are removed overall.
  • A.3.5 Linear Multimodal Masked Modeling (LM3): LM3 predicts masked-patch representations in a frozen random-projection latent space, using modality-specific targets, four cloud-minimized timestamps, and a self-attention predictor.Optical patch-size variation uses kernel interpolation, whereas SAR uses separate projections per patch size; the projections remain frozen after initialization.
  • A.3.5 Linear Multimodal Masked Modeling (LM3): LM3 uses a contrastive loss between predicted embeddings and corresponding random-projection targets, avoiding collapse without momentum teachers while coupling modalities and time.The objective combines masked latent-space prediction with contrastive training.
  • A.4 Data and Models: Table A.5 compares multimodal EO foundation models by training modalities, handling of unseen spatial, temporal, and spectral configurations, and feature-map granularity.UniverSat supports the broadest modality mix, handles unseen configurations, and offers flexible output resolution with a single set of weights.
  • A.4 Data and Models: Table A.6 summarizes seven heterogeneous pretraining datasets by semantic labels, spatial extent, sensor modalities, spatial resolution, temporal resolution, spectral resolution, and atomic-token volume.The datasets provide the heterogeneous training data used for UniverSat pretraining.
Loading 2606.23503v1…