REOBench: Benchmarking Robustness of Earth Observation Foundation Models

1 Introduction

• EOFMs are pretrained models intended for adaptation to downstream Earth-observation tasks, and include both unimodal and vision-language variants.Source: §1 · pp. 1–2
• Weather conditions and sensor discrepancies can introduce noise and variability in remote-sensing images, motivating systematic robustness comparisons.Such benchmarking can guide development of EOFMs that adapt to these conditions and produce stable results.Source: §1 · p. 2
• REOBench evaluates EOFMs across six remote-sensing tasks and twelve appearance-based and geometric perturbation types at varying severity levels.The benchmark focuses on high-resolution optical imagery and covers models based on masked image modeling, contrastive learning, and large language models.Source: §1 · p. 2
• Existing EOFMs suffer noticeable performance degradation under common image corruptions, with particularly sharp drops for masked image modeling models.The results also show that vulnerability differs across tasks, architectures, and perturbation types.Source: §1 · p. 2
• Performance drops range from less than 1% to over 20%, varying across tasks, model architectures, and perturbation types.Source: §1 · p. 2
• Vision-language foundation models are more robust than vision-centric models, particularly in image-level scene classification.Source: §1 · p. 2

2 REOBench Dataset

• The benchmark draws on AID, ISPRS Potsdam, DIOR, and three VRSBench subsets for six image-understanding tasks.The tasks include scene classification, semantic segmentation, object detection, image captioning, visual question answering, and visual grounding.Source: §2 · p. 2

2.1 Corruptions in Remote Sensing Images

• Twelve synthetic corruptions span environmental, sensor-induced, and geometric degradations, each applied at five severity levels.Examples include haze, sensor noise, rotation, scaling, and translation.Source: §2.1 · pp. 2–3

2.2 Definition of Corruption Robustness

• Relative task performance drop (RTP) measures performance degradation on corrupted inputs relative to clean-data accuracy; smaller RTP indicates greater robustness.Source: §2.2 · p. 4

3.1 Implementation Details

• MIM- and CL-based models use pretrained vision backbones with task-specific heads, while LLM-based models are evaluated zero-shot.Scene classification uses a linear layer; segmentation and detection use UperNet and Oriented R-CNN, respectively.Source: §3.1 · p. 4

3 Benchmark Robustness on EOFMs

• Across scene classification, segmentation, detection, captioning, VQA, and grounding, evaluated models show performance degradation under image corruptions.The reported severity and most robust model differ by task and metric.Source: §3.2 · p. 5, §3.3 · p. 5, §3.4 · p. 6

3.2 Scene Classification

• GeoRSCLIP achieves the best corrupted-image scene classification performance, while VHM has the smallest performance drop.CL-based models generally outperform MIM- and LLM-based models on clean and noisy scene classification images.Source: Table 1 · p. 5, §3.2 · p. 5

3.3 Semantic Segmentation

• ScaleMAE achieves 60.02% average mIoU under corruption, while GeoRSCLIP has the smallest drop, 10.01% of mIoU.LLM-based methods are omitted because open-source models for this task were unavailable.Source: Table 2 · p. 5, §3.3 · p. 5

3.4 Object Detection

• RVSA reaches 70.96% mAP on clean images but has the most severe corruption-related decline; SatMAE++ is most robust under noisy conditions.Both MIM- and CL-based methods drop more than 8% in mAP under image corruptions.Source: Table 3 · p. 6, §3.4 · p. 6

3.5 Image Captioning

• RS-LLaVA has the smallest captioning CLAIR-score drop at 2.03%, while Falcon performs best overall but drops 6.28%.Geometric distortions are excluded for captioning, VQA, and grounding; fine-tuned GeoChat improves absolute captioning performance with a similar corruption drop.Source: Table 4 · p. 6, §3.5 · p. 6

3.6 Visual Question Anaswering

• VHM achieves the best VQA accuracy on clean and noisy images, while fine-tuned GeoChat improves on zero-shot models with a slightly smaller drop.LHRS-Bot, RS-LLaVA, and Falcon show less corruption sensitivity despite lower overall accuracy.Source: Table 5 · p. 7, §3.6 · p. 7

3.7 Visual Grounding

• GeoGround leads visual grounding on clean and perturbed images, with 75.93% accuracy and the smallest drop of 4.48%.Fine-tuned GeoChat improves substantially over its zero-shot counterpart and reduces its drop under noisy conditions.Source: Table 6 · p. 7, §3.7 · p. 7

4 Discussion

• MIM-, CL-, and LLM-based models vary in robustness across tasks, architectures, backbone sizes, and corruption types; compound corruptions and multispectral inputs also expose degradation.The discussion analyzes these dimensions across the benchmark and preliminary multispectral experiments.Source: §4 · p. 7, §4.5 · p. 9, §4.6 · p. 9

4.2 Robustness Across Different Tasks

• LLM-based models typically drop below 5% on captioning and VQA, while MIM-based models are especially sensitive in classification.CL-based models are more stable across classification, segmentation, and detection; the text attributes this possibly to contrastive learning of invariant representations.Source: §4.2 · p. 8

4.1 Vision-Centric vs. Vision-Language Foundation Models

• MIM-based models drop more than 25% on average in scene classification, while vision-language models stay below 10% on most tasks.The robustness gap is less pronounced for segmentation and detection; complementary language grounding is offered as a possible explanation.Source: §4.1 · p. 7, Figure 2 · p. 8

4.3 Robustness Across Different Backbone Sizes

• ViT-L is more robust than ViT-B for scene classification, but has larger corruption-related drops on segmentation and detection.The reported backbone-size pattern differs between high-level recognition and pixel- or region-level tasks.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 Robustness Across Perturbation Types

• Motion blur causes the largest reported degradation, with MIM losing around 60%; translation has the least impact.LLM-based models maintain drops below 10% in most corruption types.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 Robustness Across Compound Perturbations

• RemoteCLIP detection performance falls from 56.72% under Brightness to 40.58% under three combined perturbations.Brightness plus Compression produces a decline exceeding the sum of the individual perturbation effects.Source: Table 7 · p. 9, §4.5 · p. 9

4.6 Robustness in Multispectral Remote Sensing

• Preliminary experiments on fMoW-Sentinel2 and BigEarthNet find substantial corruption-related performance drops for SatMAE and EarthDial.These experiments extend evaluation beyond optical imagery to multispectral data.Source: Table 8 · p. 9, §4.6 · p. 9

Robustness Research in Remote Sensing

• Prior remote-sensing robustness work studies adversarial attacks, classification, or object detection, while REOBench evaluates foundation-model robustness.The cited studies include Kazmi et al., Mei et al. [21], Lian et al. [63] [22], and.Source: §5 · p. 9

Foundation Models in Remote Sensing

• MIM-based foundation models use an encoder to mask image tokens and a decoder to reconstruct masked pixels in a self-supervised manner; SatMAE [5] is a notable remote-sensing approach.Source: §5 · p. 9
• CL methods align image and text embeddings, while LLM-based methods use multimodal language models; diffusion methods learn text-image distributions through noising and denoising.Examples include RemoteCLIP [9], GeoRSCLIP [10], and RSGPT; diffusion studies span satellite [75] [76], aerial, hyperspectral [78], and multi-resolution imagery [79].Source: §5 · p. 10

6 Conclusion and Future Work

• REOBench evaluates EOFMs across six core tasks and twelve perturbation types, finding corruption-related degradation and variation across model types, tasks, and backbone sizes.The authors hope it will serve as a standard benchmark for developing more robust and reliable Earth observation models.Source: §6 · p. 10
• The evaluation is limited to high-resolution optical imagery, excluding multispectral, hyperspectral, and SAR data.The authors note that benchmark datasets may not fully represent global variation in geography, resolution, or sensor types.Source: §6 · p. 10
• Change detection, region captioning, and object counting are not included among the benchmark tasks.The authors also state that the dataset and task coverage is not exhaustive.Source: §6 · p. 10

A.1 Overview

• The dataset overview pairs AID, Potsdam, DIOR, and VRSBench with scene classification, segmentation, detection, captioning, VQA, and visual grounding.Source: §A.1 · p. 16

A.2 Data Organization

• The data organization separates training assets from corrupted test images and provides task-specific VRSBench evaluation annotations.For DIOR, test annotations cover clean, rotated, scaled, and translated images; clean annotations are reused for non-spatial corruptions.Source: §A.2 · pp. 16–17

A.3 Intended Uses

• REOBench is intended for academic and research use, including evaluating and understanding robustness across tasks, noise types, and model architectures.Source: §A.3 · p. 17

A.4 Use Cases

• The benchmark is intended to support robustness research with standardized evaluation under realistic corruptions.Its use cases include fine-grained diagnosis of degradation across model types, tasks, and corruption types.Source: §A.4 · p. 17
• The benchmark aims to guide development of more resilient remote-sensing models for applications including disaster response and environmental monitoring.Source: §7 · p. 10, §A.4 · p. 17

A.5 Limitations

• REOBench covers optical imagery but excludes multispectral, hyperspectral, and SAR modalities.Source: §A.5 · p. 17
• Its six core tasks omit change detection, region captioning, and object counting.Source: §A.5 · p. 17
• The use of AID, DIOR, Potsdam, and VRSBench may not capture the geographic, temporal, and sensor diversity of global remote-sensing data.Source: §A.5 · p. 17

A.6 Ethical Considerations

• The dataset uses publicly available data and includes no personal, private, or sensitive information.The authors encourage ethical use in applications that may affect policy or resource allocation.Source: §A.6 · p. 17

A.7 Documentation and Maintenance

• The authors plan versioned updates for bug fixes, added corruptions, and task expansions, with community feedback invited.Source: §A.7 · p. 18

A.8 Accountability Framework

• An open accountability process invites reports of dataset-quality, annotation, or robustness-evaluation issues for review and possible incorporation into future updates.Source: §A.8 · p. 18

• REOBench can be downloaded from Hugging Face, and its metadata is documented using the Croissant framework.Source: B URL to Data and Metadata · p. 18

C Author Statement and Data License

• The dataset is released under CC-BY-4.0, which permits use, distribution, and reproduction with attribution; the authors accept responsibility for rights or ethical concerns.Source: C Author Statement and Data License · p. 18

D Hosting and Accessibility

• The dataset is hosted and archived on Hugging Face, with updates planned bi-annually or after significant source-data changes, and annotations organized in JSON.Source: D Hosting and Accessibility · p. 18

E.1 Source datasets

• AID, Potsdam, DIOR, and VRSBench supply data for scene classification, semantic segmentation, object detection, and three vision-language tasks.Source: §E.1 · p. 18

E.2 Image Pertubations

• Corruptions use severity levels S1–S5, from mildest to most severe, with type-specific parameters such as noise amplitude, blur kernel size, and geometric offsets.The parameter configurations are organized by corruption type and severity.Source: §E.2 · p. 19, Table 10 · p. 19

E.3 Perturbation Fidelity Analysis

• FID scores between corrupted and clean datasets remain consistently low across severity levels, supporting their visual similarity to clean imagery.The analysis covers DIOR, VRSBench, Potsdam, and AID, using clean datasets as reference distributions across five severity levels.Source: §E.3 · p. 19, Table 11 · p. 20
• Cloud FID rises with severity on SEN12MS-CR, while haze FID on RRSHID is lowest around severity 5, indicating the closest match to real haze there.Haze severity is evaluated through level 9; scores remain below 130 across all severity levels for both weather perturbations.Source: §E.3 · pp. 19–20, Table 12 · p. 20

F.1 More Implementation Details

• Training configurations specify task-specific datasets, data splits, model heads, and optimization settings for scene classification, semantic segmentation, and object detection.Scene classification uses AID; segmentation uses Potsdam patches and UPerHead with an auxiliary FCNHead; detection uses DIOR-R with an Oriented R-CNN head.Source: §F.1 · p. 20, Table 13 · p. 21

F.2 Evaluation Prompts

• Caption evaluation uses GPT-4o-mini to score whether candidate captions describe the same image as reference captions on a 0–100 scale.The prompt requests a JSON score.Source: §F.2 · p. 21
• VQA evaluation uses GPT-4o-mini for semantic answer matching, treating synonyms as matches and returning 1 for a match or 0 otherwise.The prompt compares the predicted answer with the ground truth answer for each question.Source: §F.2 · p. 21

F.3 Qualitative Results

• Figures 5 and 6 compare corrupted-image segmentation and detection outputs with clean-image results for Gaussian noise, salt-and-pepper noise, cloud, and brightness-and-contrast corruption.Each figure uses severity 5 and places corrupted-image results above clean-image ground truth and results.Source: Figure 5 · p. 22, Figure 6 · p. 23
• Under data gaps, SkySenseGPT answers horizontal and GeoChat answers right for road orientation, while VHM matches the ground-truth north-south.The example reports the ground truth as North-South.Source: §F.3 · pp. 24–25
• In the storage-tank caption example, the ground truth describes three tanks, while VHM reports eight and Falcon adds unsupported scene details.RS-LLaVA gives a broader description of a factory with white and silver tanks.Source: §F.3 · p. 24
• Figures 7 and 8 mark correct captioning and VQA information in green and incorrect information in red.These figures provide selected examples rather than aggregate scores.Source: Figure 7 · p. 24, Figure 8 · p. 25
• For a cloud-corrupted image, GeoChat and SkySenseGPT answer teal instead of the ground-truth yellow, while VHM gives the correct color.For the small-vehicle count question, GeoChat answers 11 versus the ground truth of 2; SkySenseGPT answers 2 and VHM 3.Source: §F.3 · p. 25
• For whether a vehicle is at the top-most position, GeoChat and SkySenseGPT answer yes, matching ground truth, while VHM answers no.The example also poses a road-orientation question but supplies no answers for it.Source: §F.3 · p. 25
• Figure 9 contrasts visual-grounding examples on clean images with results under severity-5 clouds and Gaussian blur.The examples ask about objects and their attributes or locations, including fields, a chimney, courts, and ships.Source: Figure 9 · p. 26