PaperlayerView exact revision on arXiv

Source-linked AI summary

EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation

Kelsey Doerksen, Hannah Kerner

arXiv:2605.29330v1cs.CV

TL;DR

Earth observation benchmarks have largely measured generalization to samples similar to training data, while comprehensive evidence on robustness to natural deployment shifts has been lacking. EarthShift benchmarks models across five remote-sensing shift types and 11 tasks. The evaluation finds 20% average out-of-distribution performance degradation and no robustness advantage for geospatial foundation models over generic vision or fully supervised models.

  • Problem

    Most satellite machine-learning benchmarks test generalization to samples closely related to training data, and no comprehensive study has quantified robustness to natural shifts because representative benchmark datasets are limited.

    Source: §1 · p. 1

  • Method

    EarthShift benchmarks models across 11 tasks and five remote-sensing shift types, comparing in-distribution and out-of-distribution performance across model families and transfer protocols.

    Source: §1 · p. 2

  • Results

    20% average performance degradation was observed out-of-distribution, and geospatial foundation models showed no robustness advantage over generic vision models or fully supervised baselines.

    Source: §7 · pp. 9–10

  • Takeaways & Limitations

    EarthShift provides a flexible testbed for reproducible robustness evaluation as new tasks and models are added.

    Source: §6.3 · p. 9, §7 · p. 10

    EarthShift does not exhaustively analyze shifts across its task scenarios, limiting broad claims about robustness to these shift types without a more extensive experiment suite.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Current Earth observation benchmarks focus on measuring performance on diverse tasks and applications, typically measuring generalization in-distribution. But when models are deployed, they must generalize to myriad out-of-distribution scenarios, such as new time periods, geographies, scales, and sensors. We introduce EarthShift: the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing. EarthShift enables users to measure distributional robustness by comparing performance in- and out-of-distribution using datasets from paired datasets from different sources, temporal windows, geographic locations, and sensors. Our experiments on 8 geospatial foundation models (GFMs) and 11 tasks covering 5 shift types show that GFMs consistently perform 15-20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy. We show that GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models. This highlights a need for future research to strive for improvements in distributional robustness, not just performance, which can be benchmarked using EarthShift. We release our code and datasets to provide a testbed to guide future work to create foundation models that are robust and reliable in real-world applications. Code and data for EarthShift are available at: https://earthshift.github.io

1 Introduction

Earth observation models face common natural distribution shifts in deployment, but existing benchmarks largely test near-training data and no comprehensive study had quantified robustness across realistic shifts. EarthShift evaluates five shift types and finds that robustness varies by shift, with no consistent advantage from geospatial foundation models or fine-tuning.

1 Introduction

  • Existing satellite benchmarks largely test samples close to training data, despite frequent deployment shifts and the need for robust monitoring in a changing climate.Source: §1 · pp. 1–2
  • EarthShift evaluates 11 tasks across spatial-resolution, temporal, geographic, sensor, and source shifts, comparing geospatial, vision, and supervised models across classification, segmentation, and two transfer protocols.Source: §1 · p. 2
  • Models are robust to temporal shifts but not geographic, scale, sensor, or source shifts.Source: §1 · p. 2
  • Geospatial foundation models have similar or sometimes worse robustness than vision foundation models, and neither family outperforms traditional supervised models.Source: §1 · p. 2
  • Fine-tuning encoders does not improve robustness and sometimes worsens it.Source: §1 · p. 2
  • EarthShift is intended as a flexible framework for evaluating robustness gaps and encouraging models that generalize across out-of-distribution conditions.Source: §1 · p. 2

2 Related Work

Prior robustness work often studies synthetic shifts or isolated shift types, while natural-shift benchmarks relevant to deployed satellite machine learning remain limited. Earlier remote-sensing studies reveal robustness challenges but do not provide a holistic evaluation across settings and models.

2 Related Work

  • Robustness research often uses synthetic corruptions or augmentations, while adaptation methods assume target-domain data and EO test-time adaptation remains unexplored.Source: §2 · p. 2
  • Natural-shift benchmarks remain limited for deployed satellite machine learning, although prior examples report up to 30% accuracy loss at a new disaster site and failures across countries.Source: §2 · p. 3
  • Remote-sensing studies have examined individual shifts or tasks, finding limited cross-satellite gains for specialized models and poor geographic or disaster generalization in other settings.Source: §2 · p. 3

3 EarthShift benchmark

EarthShift measures robustness by comparing in-distribution and out-of-distribution performance across five natural shift categories. Its benchmark pairs datasets to represent changes in scale, time, geography, sensors, and data sources.

3.1 Measuring distributional robustness

  • Effective robustness measures OOD performance relative to a baseline that accounts for ID capability, with the baseline fit separately for each shift type.The baseline function β is estimated by linear regression of OOD versus ID scores across models and datasets within a shift type.Source: §3.1 · p. 3

3.2 Distribution Shifts

  • EarthShift evaluates five shift categories using paired ID and OOD datasets, with 11 test cases across the benchmark.Models are fine-tuned on ID data and tested on held-out samples from both the ID and shifted OOD datasets.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 Scale shift

  • Scale shift tests changes in image resolution while retaining shared classes and geographic coverage, including RESISC45–UCMerced in both ID/OOD directions.The paired datasets share 19 retained classes.Source: §3.3 · p. 4

3.4 Temporal Shift

  • Temporal shift compares images from the same locations across seasons or years, using FTW season-to-season data and Germany data from 2018 versus 2019.The evaluation includes the global FTW dataset, a South Africa subset, and a Germany year-to-year subset.Source: §3.4 · pp. 4–5

3.5 Geographic Shift

  • Geographic shift evaluates transfer between FTW country subsets, including Germany-to-Cambodia and Germany-to-Denmark pairings.The pairings differ in location, field characteristics, seasons, or agroecological conditions.Source: §3.5 · p. 5

3.6 Sensor Shift

  • Sensor shift tests radiometry changes from Sentinel-2 to Sentinel-1 and partial-band overlap, including RGB-to-RGE1 inputs.The benchmark includes BigEarthNetv2 multi-label classification, Sen1Floods11 segmentation, and m-EuroSat partial-band overlap.Source: §3.6 · p. 5

3.7 Data source shift

  • Data source shift compares datasets with the same task, sensor channels, classes, resolution, and geographic coverage but different generation methodologies.The DeepGlobe–DFC2022 segmentation pairing uses four overlapping classes.Source: §3.7 · pp. 5–6

4 Methods

The experiments compare remote-sensing-pretrained geospatial models with generic vision and randomly initialized models under two fine-tuning settings. Training uses task-specific heads and evaluates performance with task-appropriate metrics.

4.1 Model architectures

  • The comparison includes eight remote-sensing-pretrained GFMs, ImageNet-pretrained ResNet50, ViT-b-16 and CLIP, plus randomly initialized ResNet50 and ViT-b-16 models.The randomly initialized models provide fully supervised comparisons.Source: §4.1 · p. 6

4.2 Training setup

  • Models are trained either by updating all pretrained parameters and the task head or by freezing the backbone and training only a new task-specific head.A learnable 1x1 convolution adapts input channels, while simple convolutional segmentation and linear classification heads limit adaptation hyperparameters.Source: §4.2 · p. 6
  • Learning rates are selected on representative tasks and reused within each shift type; frozen-backbone and full fine-tuning run for 40 and 50 epochs, respectively.Both settings use batch size 32, AdamW, a 20-epoch warmup followed by cosine decay, and task-appropriate cross-entropy losses.Source: §4.2 · pp. 6, 8
  • The ImageNet ResNet50 is the most robust model overall in both full fine-tuning and frozen-backbone settings, though no single model dominates every shift.The randomly initialized ResNet50 ranks second in the frozen-backbone setting.Source: Table 2 · p. 8
  • Evaluation uses accuracy for classification, micro-F1 for multilabel classification, and mIoU for segmentation, with channel-wise normalization.Figure 2 plots each model-task combination for frozen-backbone and full fine-tuning; the dashed y=x line marks perfect robustness.Source: Figure 2 · p. 7, §4.2 · p. 8

5 Results

Models show little effective robustness beyond that predicted by in-distribution performance, with robustness varying substantially across shift types. Temporal shifts are comparatively easy, while sensor and scale shifts cause marked degradation.

  • GFMs show ∼0% average effective robustness across tasks, with no advantage over generic vision models or random baselines.Baseline slopes vary by shift type, from 1.02 and 0.88 for temporal shifts to 0.39 and 0.59 for sensor shifts, for frozen and full settings respectively.Source: §5 · p. 8
  • All models show strong robustness to temporal shifts, with little to no performance drop and a baseline slope near 1.The authors note this may reflect shift difficulty rather than a model-family advantage.Source: §5 · p. 8
  • All models suffer severe degradation under sensor shifts and high heterogeneity, with GFMs performing worst on average.Source: §5 · p. 8
  • Models trained on FTW-Germany generalize significantly worse to Cambodia than to Denmark, where cropping patterns are more similar.The Germany–Cambodia comparison spans different continents and dominant crops, while Germany and Denmark share a continent and similar crop types.Source: §5 · pp. 8–9
  • Under scale shift, the average model loses 43% of its in-distribution performance, with OOD losses ranging from ∼32–63%.Even models with in-distribution performance of 0.95 show significant OOD degradation.Source: §5 · p. 9

6 Discussion

The evaluated models show substantial out-of-distribution losses, and Earth-observation pretraining does not provide a robustness advantage over generic vision or fully supervised models. The authors qualify these findings by noting that their model selection and shift coverage are not exhaustive.

6 Discussion

  • The 13 evaluated architectures, including 8 GFMs, represent currently dominant architectural and pretraining paradigms, but are not an exhaustive model survey.Source: §6 · p. 9

6.1 Evaluated models lack distributional robustness for Earth observation

  • Models lose 20–25% of performance on average out-of-distribution, with losses varying from ∼2% for temporal shifts to up to ∼60% for sensor shifts.These gaps are comparable to the 14–30% gaps reported for ImageNet to ImageNet-V2 [17]; EO pretraining confers no robustness advantage over ImageNet-pretrained or fully supervised models.Source: §6.1 · p. 9

6.2 Pre-training on EO data does not help robustness

  • The authors speculate that EO pretraining data may encode sensor and geographic biases, whereas ImageNet’s photographic diversity may support robustness.Source: §6.2 · p. 9

6.3 Limitations and Future Work

  • EarthShift does not exhaustively analyze shifts across task scenarios, limiting broad claims about robustness to these shift types.The authors say additional datasets would strengthen the claims, while expecting the consistent trends across models and tasks to remain.Source: §6.3 · p. 9

7 Conclusion

EarthShift benchmarks robustness across realistic distribution shifts, and the evaluation finds substantial OOD degradation with no robustness advantage for GFMs over generic vision or fully supervised baselines. The authors release code and datasets for reproducible evaluation.

  • Across 13 models, including 8 GFMs, and 11 tasks spanning 5 shift types, current models show 20% average OOD performance degradation.Source: §7 · p. 9
  • GFMs show no robustness advantage over generic vision models or fully supervised baselines among the 13 models tested.Source: §7 · p. 10
  • The authors release code, datasets, and onboarding instructions for contributing to EarthShift to support reproducible robustness evaluation.Source: §7 · p. 10

A.2 Dataset License Details

Table 3 lists the dataset licenses used in EarthShift.

A.3 Geospatial Foundation Model Details

Table 4 inventories the compared models and lists their backbone parameter counts, learning techniques, and pre-training datasets.

A.3 Geospatial Foundation Model Details

  • Table 4 names the models included in the comparison.Source: Table 4 · p. 14
  • The comparison reports each model’s number of backbone parameters.Source: Table 4 · p. 14
  • The table also lists each model’s learning technique and pre-training dataset.Source: Table 4 · p. 14

A.4 Experimental Details

The experiments report learning-rate sweeps and repeated runs across classification and segmentation tasks. The extended results cover effective robustness and absolute ID/OOD performance across shift settings, including comparisons by model type, fine-tuning approach, and size.

A.4 Experimental Details

  • The experiments used 8,190 learning-rate sweep runs, then retrained task-model pairs with five seeds for reporting.The sweep covered 13 models, two fine-tuning settings, seven shifts, and 18 learning rates; classification and segmentation used 520 and 910 runs, respectively.Source: §A.4 · p. 15

A.5 Extended Results - Effective Robustness

A.6 Extended Results - Absolute Performance Scores

Loading 2605.29330v1…