PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation

Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, Xiao Xiang Zhu

arXiv:2403.15356v3cs.CV

TL;DR

EO foundation models are often specialized for particular sensors, limiting their flexibility across heterogeneous modalities and spectral bands. DOFA uses a wavelength-conditioned dynamic hypernetwork and continual pretraining across EO modalities; the paper reports strong results across tasks, including generalization to unseen sensors with reduced computational costs.

  • Problem

    Existing EO foundation models are often specialized for individual modalities or fixed spectral-band configurations, while EO tasks require data from diverse sensors and spectral characteristics.

    Source: §1 · p. 1

  • Method

    DOFA uses a wavelength-conditioned dynamic hypernetwork in a unified Transformer and continually pretrains across EO modalities; DOFA+ adds a pretrained vision prior and distillation.

    Source: §1 · p. 2

  • Results

    DOFA and DOFA+ achieve state-of-the-art performance across a wide range of EO tasks and modalities and generalize to unseen sensors and spectral configurations.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The models offer flexibility for open-world EO settings with reduced computational costs, without retraining for diverse sensor and spectral configurations.

    Source: §1 · p. 2, §5 · p. 15

Abstract

from arXiv · show

Earth observation (EO) in open-world settings presents a unique challenge: different applications rely on diverse sensor modalities, each with varying ground sampling distances, spectral ranges, and numbers of spectral bands. However, existing EO foundation models are typically tailored to specific sensor types, making them inflexible when generalizing across the heterogeneous landscape of EO data. To address this, we propose the Dynamic One-For-All (DOFA) model, a unified, multimodal foundation framework designed for diverse vision tasks in EO. Inspired by neural plasticity, DOFA utilizes a wavelength-conditioned dynamic hypernetwork to process inputs from five distinct satellite sensors flexibly. By continually pretraining on five EO modalities, DOFA achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities. Enhanced with hybrid continual pretraining, DOFA+ requires significantly fewer computational resources while outperforming counterparts trained with extensive GPU budgets. Experiments on diverse datasets highlight DOFA's potential as a foundation for general-purpose vision models in the sensor-diverse EO domain. The code and pre-trained weights are publicly available at https://github.com/zhu-xlab/DOFA.

1 Introduction

DOFA addresses the rigidity of EO foundation models by using wavelength-conditioned dynamic weights to accommodate diverse and unseen sensor bands. Its continual-pretraining variants are reported to perform strongly across many EO tasks while reducing computational costs.

1 Introduction

  • Existing EO foundation models are often tied to fixed spectral-band configurations or individual modalities, with reported challenges for intersensor relationships, unseen sensors, and computing costs.These limitations motivate models that can adapt to flexible input modalities.Source: §1 · pp. 1–2
  • DOFA uses wavelength-conditioned dynamic weights in a unified Transformer to process varying and unseen spectral bands across EO modalities.A shared vision backbone provides feature learning across heterogeneous inputs.Source: §1 · p. 2
  • DOFA uses continual pretraining with masked image modeling and knowledge distillation to reduce computational cost, while DOFA+ adds a vision-model prior and hierarchical distillation.DOFA+ retains the underlying architecture and uses local reconstruction to learn EO-specific patterns while preserving semantic priors.Source: §1 · p. 2
  • DOFA and DOFA+ support parameter-efficient, label-efficient adaptation across classification, segmentation, detection, and environmental change detection tasks.The paper presents the models as a unified backbone for downstream EO tasks.Source: §1 · pp. 1–2
  • Across 20+ EO tasks, DOFA demonstrates robust adaptability, efficiency, and scalability, with state-of-the-art performance and generalization to unseen sensors and spectral configurations.The contribution summary also reports reduced computational costs and flexibility in open-world settings.Source: §1 · pp. 2–3

2 Related Work

Prior EO foundation-model research spans single-modality learning, spatial-resolution adaptation, and multimodal fusion, with newer efforts aiming to cover broader sensor diversity. DOFA and DOFA+ distinguish themselves by dynamically encoding spectral bands into weights, addressing cases where downstream band counts change without retraining.

2 Related Work

  • Early EO foundation models often learned from single modalities, while other work addressed ground-sample-distance adaptation and representations for differing spatial resolutions.Examples include temporal representation learning and GSD-aware positional or patch strategies.Source: §2 · p. 3
  • Multimodal approaches include early multispectral-SAR fusion, spectral grouping, modality-specific embeddings, and cross-attention or shared-common representation learning.These approaches seek unified representations across EO data types.Source: §2 · p. 3
  • Recent models target broader EO diversity through modality prompts, scale-adaptive encoders, sensor-agnostic attention, or combined global and local objectives.SkySense V2, AnySat, Panopticon, and Galileo illustrate these directions.Source: §2 · p. 3
  • Changing the number of spectral bands downstream remains difficult for existing models without retraining; DOFA uses a dynamic weight generator to encode bands into weights.Its architecture is described as inspired by neuroplasticity and adaptive network-weight changes for novel modalities.Source: Figure 2 · p. 3, §2 · p. 4

3 Methodology

DOFA’s methodology combines a shared Transformer framework with wavelength-conditioned dynamic weights to handle diverse EO channels. Its pretraining pairs image reconstruction with teacher-feature distillation, and DOFA+ adapts this recipe using DINOv2 priors.

3.1 Preliminary

  • In the preliminary ViT setup, flattened image patches are projected into D-dimensional embeddings, then combined with a class token and positional embeddings.The resulting sequence is supplied to subsequent ViT layers.Source: §3.1 · p. 4

3.2 Architecture overview

  • DOFA uses a dynamic weight generator to predict patch-embedding weights from each image channel’s spectral wavelength, then processes the resulting representations with shared Transformer blocks.For new spectral ranges or sensors, the generator interpolates in weight space; a dynamic decoder reconstructs the target modality’s bands.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 Wavelength-conditioned dynamic patch embedding

  • Wavelength-conditioned dynamic patch embedding generates weights and biases compositionally from channel wavelengths, making patch embedding independent of input band count.The generated parameters are used in the patch-embedding convolution.Source: §3.3 · pp. 5–6

3.4 Multimodal continual pretraining

  • Multimodal continual pretraining combines masked-image reconstruction from full-channel inputs with cosine-similarity distillation between teacher and student features.A proxy input derived from each modality is fed to a frozen ImageNet-pretrained teacher; the dynamic generator is initialized to mimic the teacher’s patch-embedding weights.Source: §3.4 · p. 7

3.5 DOFA+: Hybrid Continual Pre-training

  • DOFA+ starts from DINOv2 weights, uses local reconstruction to learn EO-specific patterns, and distills intermediate representations across multiple teacher layers.The design combines local detail learning with global semantic alignment in a single-branch setup without auxiliary network modules.Source: §3.5 · p. 7
  • DOFA+ combines masked-image reconstruction with hierarchical DINOv2 feature distillation and reaches state-of-the-art performance after 410k EO image tiles.The recipe is presented as a compact pretraining regime that reduces sample, cost, and carbon-footprint demands.Source: §3.5 · p. 7

4 Experiments

The experiments evaluate DOFA and DOFA+ across diverse EO classification and segmentation datasets. Both models perform strongly, with reported advantages over other foundation models across multiple classification datasets and all segmentation datasets.

4 Experiments

  • Across 22 datasets, the experiments assess classification, segmentation, detection, and change detection under linear-probing, decoder-finetuning, and full-finetuning settings.The protocols limit training epochs and freeze pretrained backbones wherever possible.Source: §4 · p. 8

4.1.1 GEO-Bench Classification Experiments

  • The evaluation spans six classification datasets from varied domains and satellite platforms, using dataset-specific learning rates and 50-epoch linear probing.Classification uses 224×224 crops except SatMAE (96×96) and CROMA (120×120), following their official setups.Source: §4.1.1 · p. 8

4.1.2 GEO-Bench Segmentation Experiments

  • Segmentation covers six tasks across RGB, multispectral, and hyperspectral data, using frozen encoders with UPerNet decoders.The models are trained for 20 epochs; Table 2 reports mean intersection over union (mIoU).Source: §4.1.2 · p. 8, Table 2 · p. 9

4.1.1 GEO-Bench Classification Experiments

  • DOFA and DOFA+ show superior performance across multiple GEO-Bench classification datasets, including strong m-forestnet results despite no Landsat-8 pretraining.Table 1 reports linear-probing results on six classification tasks; m-bigearthnet uses micro-averaged multilabel average precision rather than top-1 overall accuracy.Source: §4.1.1 · p. 8, Table 1 · p. 9

4.1.2 GEO-Bench Segmentation Experiments

  • DOFA and DOFA+ outperform other foundation models across all six segmentation datasets, with DOFA+ improving significantly over DOFA.The reported improvement is attributed to multi-layer distillation, which the authors say helps learn fine-grained spatial patterns.Source: §4.1.2 · pp. 8–9, Table 2 · p. 9

4.2 PANGEA Benchmark Experiments

On PANGEA, DOFA+ achieves the highest average mIoU and strong task rankings with fewer pretraining images and computational resources than TerraMindv1-L. On RESISC45, DOFA and DOFA+ achieve high classification accuracy under frozen-backbone and fine-tuned evaluation.

4.2 PANGEA Benchmark Experiments

  • PANGEA evaluation freezes the pretrained backbones and fine-tunes decoders for 100 epochs, using only datasets with direct download links in the original benchmark.For DOFA+, inputs smaller than 224 pixels are upscaled to 224; larger inputs retain their original size.Source: §4.2 · p. 9
  • DOFA ViT-Base performs competitively across eight PANGEA tasks, outperforming RemoteCLIP and SatlasNet and reaching CROMA-comparable average mIoU with fewer computational costs.The reported consistency includes burned-area mapping, flood mapping, and crop monitoring.Source: §4.2 · p. 9
  • 59.81 average mIoU gives DOFA+ the highest PANGEA score, and it ranks best or second-best on 6 of 8 tasks despite fewer pretraining images and lower compute than TerraMindv1-L.DOFA+ has the best or second-best scores on BurnSr (86.53), CTM-SS (57.47), and SN7 (63.06); TerraMindv1-L leads MADOS, where DOFA+ is second-best.Source: §4.2 · p. 10

4.2.1 Image Classification on RESISC45

  • DOFA ViT-Base achieves 91.3% frozen and 97.3% fine-tuned accuracy on RESISC45, surpassing existing methods in both settings.The benchmark contains 31,500 remote-sensing images across 45 scene categories.Source: §4.2.1 · p. 10
  • On RESISC45, DOFA+ ViT-Large achieves the best overall results, with 95.3% frozen and 98.1% finetuned accuracy, significantly outperforming all prior methods.Source: §4.2.1 · p. 10

4.3 Object Detection Experiments

On DIOR detection, DOFA+ achieves the reported state-of-the-art mAP50 while using a more computationally efficient backbone than SkySense V2. DOFA and DOFA+ use one shared parameter set across tasks and adapt to different image resolutions and sensor inputs.

4.3 Object Detection Experiments

  • The DIOR evaluation uses Faster R-CNN backbones and mAP50, the mean Average Precision at an Intersection-over-Union threshold of 0.5.The benchmark contains 23,463 images, 192,472 annotated instances, and 20 categories; all models are fully fine-tuned.Source: §4.3 · p. 10
  • 79.73 mAP50 gives DOFA+ the reported new SOTA, exceeding SkySense V2 (79.50) and other listed baselines.DOFA scores 76.21 mAP50; compared with SkySense V2, DOFA+ uses ViT-L rather than Swin-Huge, with significantly smaller pretraining data and GPU hours.Source: §4.3 · p. 10, Table 5 · p. 11
  • DOFA and DOFA+ share one parameter set across tasks while adapting to DIOR’s high-resolution images and lower-resolution Sentinel-1 and Sentinel-2 imagery.The passage identifies flexibility across sensors, ground sampling distances, and spectral-band combinations.Source: §4.3 · p. 10
  • The authors conclude that DOFA+ achieves superior high-resolution detection without task-specific architectural modifications.They characterize the approach as flexible and scalable across datasets with widely differing properties.Source: §4.3 · p. 10

4.4 Ablation Studies

The ablations test spectral-band combinations and pretraining duration. Full-band inputs achieve the best reported segmentation metrics, while the epoch study reports diminishing gains beyond 80 epochs.

4.4 Ablation Studies

  • The ablations examine spectral diversity and pretraining duration as contributors to DOFA’s effectiveness and as guidance for different modalities and training regimes.The spectral-band experiment evaluates DOFA+ (ViT-Base) on 13-band Sentinel-2 imagery for Sen1Floods11; the duration experiment probes DOFA representations on RESISC45.Source: §4.4 · p. 11

4.4.1 Number of Spectral Bands

  • Adding Red Edge bands raises Water IoU from 49.77 to 57.51 for RGB + CA + RE1, while adding Coastal Aerosol alone does not improve on RGB.Performance continues to rise as further Red Edge and NIR bands are introduced; Fig. 6 shows improvement with added bands, and Fig. 7 shows finer segmentation maps.Source: §4.4.1 · p. 12, Figure 6 · p. 12, Figure 7 · p. 13
  • With the full spectral-band combination, DOFA+ achieves an F1 score of 92.51, mIoU of 86.58, and Water IoU of 76.83.The full-band configuration is reported as best overall; the spectral-input study evaluates a frozen backbone without requiring architecture changes or retraining.Source: §4.4.1 · p. 12, Table 6 · p. 12

4.4.2 Training Epochs

  • Accuracy steadily rises as pretraining increases from 20 to 80 epochs, while gains beyond 80 epochs are marginal, suggesting diminishing returns from excessively long training for DOFA.Source: §4.4.2 · pp. 12–13

4.5 Pre-training efficiency comparison

DOFA and DOFA+ are reported to achieve competitive or superior performance with fewer resources than recent EO foundation models. DOFA+ uses a compact pretraining setup, while DOFA’s training uses a progressively expanded multimodal dataset.

4.5 Pre-training efficiency comparison

  • DOFA pretraining progresses from 50K images for 100 epochs, to 410K images (100K each from Sentinel-1, Sentinel-2, NAIP, and Gaofen; 10K EnMAP) for 20 epochs, then one epoch on the full approximately 11.5M-image, five-modality dataset.Source: §4.5 · pp. 13, 15
  • DOFA+ reaches 98.1% accuracy on RESISC-45 after 150 epochs on 410K EO images, using 8 NVIDIA L40 GPUs for three days.The passage contrasts this setup with models using millions of images and up to 800 epochs, and reports 9,216 GPU hours for TerraMindv1-B.Source: §4.5 · p. 15
  • DOFA+ uses a single ViT-L backbone at lower computational cost while achieving competitive or superior performance to task-specific counterparts.The authors describe DOFA and DOFA+ as requiring significantly lower resources while retaining competitive or superior performance.Source: §4.5 · pp. 13, 15

4.6 Visualizations

The visualizations examine dynamically generated sensor weights and compare learned feature representations across models and datasets. DOFA’s representations are reported to cluster better than those of compared models on the shown datasets.

4.6 Visualizations

  • DOFA dynamically generates weights for different input sensors, as illustrated by selected kernel weights for inputs with more than four channels.Fig. 8a visualizes generated weights for different modalities.Source: §4.6 · p. 15
  • The visualizations cover two distinct representations: generated sensor kernels and feature embeddings from foundation models across datasets.The kernel weights are shown in Fig. 8a and the cross-model feature plots in Fig. 8b.Source: §4.6 · p. 15
  • On m-pv4ger, m-so2sat, and m-eurosat, both DOFA versions show better-clustered pretrained features than the compared models.Fig. 8b uses t-SNE plots of extracted features, with colors representing semantic categories.Source: §4.6 · p. 15

5 Conclusion

DOFA and DOFA+ achieve state-of-the-art results across multiple benchmarks and generalize to unseen sensors and spectral configurations without retraining. Their wavelength-conditioned architecture supports flexible EO inputs, while DOFA+ adds a lightweight distillation pipeline for constrained compute.

5 Conclusion

  • DOFA and DOFA+ achieve state-of-the-art results across multiple benchmarks.Experiments cover multiple benchmarks.Source: §5 · p. 15
  • Both models generalize to unseen sensors and spectral configurations without retraining.Source: §5 · p. 15
  • A wavelength-conditioned dynamic hypernetwork lets one model process multimodal satellite inputs across sensors, spectral bands, and spatial resolutions.Source: §5 · p. 15
  • Continual and hybrid pretraining across five EO modalities yields spectral-spatial representations that transfer to downstream tasks.Source: §5 · p. 15
  • DOFA+ adds a lightweight distillation pipeline and performs strongly under constrained computational budgets.Source: §5 · p. 15

6 Data availability statements

The paper provides download information for a five-modality EO dataset and for evaluation datasets and benchmarks.

6 Data availability statements

  • The constructed multimodal dataset contains five modalities with distinct spectral and spatial characteristics.Source: §6 · p. 15
  • Download information is provided for Sentinel-1, Sentinel-2, Gaofen, NAIP, and EnMAP data.Source: §6 · pp. 15–16
  • Evaluation resources include RESISC45, GEO-Bench datasets for 12 downstream tasks, and the PANGEA benchmark.GEO-Bench tasks comprise six classification and six segmentation tasks.Source: §6 · p. 16

7 Code availability

Training and inference scripts and trained models are publicly available; the paper also acknowledges its funding sources.

7 Code availability

  • Training and inference scripts are publicly available in the DOFA GitHub repository.Source: §7 · p. 16
  • Trained models are publicly available through the Earthflow DOFA Hugging Face repository.Source: §7 · p. 16
  • The work acknowledges support from German ministries, Munich Center for Machine Learning, the European Commission, and the European Research Council.Source: §7 · p. 16
Loading 2403.15356v3…