Source-linked AI summary
GEO-Bench: Toward Foundation Models for Earth Monitoring
Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, Xiao Xiang Zhu
TL;DR
Earth-monitoring foundation models lack a diverse, comparable benchmark spanning relevant downstream tasks and modalities. GEO-Bench addresses this gap with curated classification and segmentation tasks, standardized aggregate evaluation, and baseline comparisons; results highlight strong general-purpose models and substantial data-efficiency differences.
Problem
Existing Earth-observation pre-training studies use different, often narrow task sets and inadequately report evaluation uncertainty, preventing direct comparison.
Method
GEO-Bench provides six image-classification and six semantic-segmentation tasks, with normalized aggregate scoring, confidence intervals, and reproducible evaluation code.
Results
ConvNeXt and SwinV2 achieve the strongest aggregated performance, while ConvNeXt-B-timm reaches performance comparable to ResNet18-Rnd using 2% of its training data.
Takeaways & Limitations
The benchmark offers a systematic basis for measuring Earth-monitoring models and is intended to stimulate foundation-model development across downstream tasks.
Takeaways & Limitations
The benchmark does not extensively evaluate temporal fine-tuning or fusion with text and weather, and its geographic coverage could include more countries and biomes.
Abstract
from arXiv · showhide
Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks. Such models, recently coined foundation models, have been transformational to the field of natural language processing. Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited. To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results to enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models. We believe that this benchmark will be a driver of progress across a variety of Earth monitoring tasks.
1 Introduction
Earth monitoring applications increasingly use machine learning, but general-purpose vision pre-training transfers poorly to remote sensing. GEO-Bench addresses the lack of diverse, comparable downstream evaluations for Earth-monitoring foundation models.
- Earth monitoring machine learning supports climate mitigation, adaptation, and science across applications including methane detection, forest carbon quantification, weather prediction, and crop monitoring.
- CLIP under-performs on Earth monitoring tasks because it is trained mainly on ground-level RGB images captured at a single time.
- Earth observation data differs from typical image data through overhead views, satellite revisit times, cloud occlusions, and multisensor measurements.
- Existing Earth-observation pre-training studies use different downstream task sets, limiting direct comparison, while often lacking task diversity and uncertainty reporting.
- GEO-Bench provides six image-classification and six semantic-segmentation tasks spanning countries and sensor modalities, with smaller transformed datasets and single-GPU reproducibility.
- The benchmark aims to stimulate foundation-model development, systematically measure model quality, identify effective pre-trained models, and support open evaluation.
- The paper evaluates 20 state-of-the-art models to establish reference points and characterize existing pre-trained models.
2 Remote sensing data for self-supervision
Earth-observation data offer multimodal and temporal structure for self-supervised learning. Potential sources include multispectral satellite imagery, SAR, elevation, and georeferenced semantic data.
- The benchmark does not provide pre-training data because foundation-model training typically depends on model-specific data-source choices.
- Multispectral satellite sources such as Sentinel-2 and Landsat 8 provide multiple spectral bands with periodic revisits.
- Foundation models can encode multimodal data through self-supervised training before fine-tuning across climate-related remote-sensing tasks.
- Satellite imagery forms a four-dimensional longitude–latitude–wavelength–time array that supports self-supervision through adjacent-tile prediction or seasonal contrast.
- SAR and terrain-elevation data can be geolocated with other sources and provide modalities complementary to optical spectral bands.
- Georeferenced text and non-image layers such as Wikipedia and OpenStreetMap can be predicted or contrasted with satellite imagery to learn transferable semantic representations.
3 GEO-Bench
GEO-Bench adapts geospatial datasets into an accessible, diverse benchmark for evaluating models across classification and segmentation. Its design prioritizes consistent use, sensor and task diversity, and fair assessment.
- GEO-Bench contains 6 classification and 6 segmentation tasks, with dataset characteristics, examples, and spatial coverage documented across the benchmark materials.
- The benchmark modifies and gathers geospatial datasets to support accessibility, usability, and effective cross-task model assessment.
- A consistent data schema and harmonized datasets reduce engineering effort while preserving sensor-type and resolution diversity.
- The benchmark covers multispectral, SAR, hyperspectral, elevation, and cloud-probability modalities at spatial resolutions from 0.1 to 30 m/pixel.
- Detection and counting datasets are transformed into semantic segmentation, yielding six classification and six segmentation task sets.
- Original dataset splits are preserved when available; otherwise, validation and test sets are generated without spatial overlap.
- Only datasets with permissive licenses are included after adaptation.
- Datasets larger than 20000 samples are randomly subsampled to represent data-scarce downstream tasks and reduce evaluation cost.
4 Using The Benchmark
GEO-Bench standardizes fine-tuning and reporting so model comparisons are reproducible across tasks. The procedure combines bounded tuning, repeated seeds, normalized aggregated scores, and bootstrap uncertainty estimates.
- Fine-tuning is preferred over fixed representations because it can mitigate dependence on the pre-training task and often improves downstream generalization.
- Hyperparameter tuning is recommended within a maximum budget of 16 trials per task, with early stopping based on validation metrics.
- Recommended augmentations are limited to 90° rotations and vertical or horizontal flips, while augmentation choices remain open for further study.
- The open-source toolbox provides dataset loading, result visualization, and PyTorch-Lightning-based training tools.
- 4.1 Reporting Results: Results should be reported per task and in aggregate with reliable confidence intervals, using code supplied to generate figures from raw results.
- 4.1 Reporting Results: At least 10 seeds are recommended because 3–5 seeds do not provide reliable confidence intervals.
- 4.1 Reporting Results: IQM trims the highest and lowest 25% of values before averaging, reducing outlier influence relative to the mean and variance relative to the median.
- 4.1 Reporting Results: Task results are normalized between strong-baseline minimum and maximum values, so a future score above 1 indicates benchmark progress.
5 Related Works
Related benchmarks support remote-sensing evaluation through diverse datasets, standardized interfaces, and task-specific procedures, but GEO-Bench emphasizes broader coverage and aggregated uncertainty reporting.
- SustainBench covers 15 public datasets spanning seven sustainable development goals and includes a public leaderboard for tracking task performance.
- TorchGeo provides PyTorch data loaders for 52 remote-sensing datasets across classification, regression, change detection, segmentation, and detection tasks.
- EarthNets analyzes metadata from 400 remote-sensing datasets and recommends two classification, two segmentation, and two detection datasets for benchmarking.
- Unlike EarthNets’ six recommended datasets, GEO-Bench provides 12 datasets with aggregated results and statistical uncertainty reporting.
- AiTLAS benchmarks 22 classification datasets with standardized splits, fine-tuning, cross-dataset label overlap, and experiments across 10 model families.
6 Experiments
The experiments compare baseline architectures, training-set sizes, multispectral inputs, segmentation models, and computational resource use across GEO-Bench. Results show strong differences among models, data-efficiency trade-offs, limited or architecture-dependent multispectral gains, and practical resource costs.
- 6.2.2 Comparing Baselines on RGB only: ConvNeXt and SwinV2 are the strongest RGB-only baselines, with ConvNeXt and SwinV2 outperforming other models across aggregated and most dataset results.Scratch ResNet18 performs substantially worse than pretrained ResNet18, while remote-sensing-pretrained ResNet variants show no improvement over timm weights in aggregate.
- 6.2.3 Accuracy vs training set size: ConvNeXt-B-timm reaches aggregated performance comparable to ResNet18-Rnd using only 2% of the training set, a 50x data-efficiency difference.ConvNeXt often surpasses SwinV2 as training data decreases, whereas transformer architectures tend to perform better in the high-data regime.
- 6.2.3 Accuracy vs training set size: Reducing training data increases dataset discriminativity, particularly for m-eurosat, although m-eurosat remains weakly discriminative when most models achieve very high accuracy.GeoLifeClef 2022 was discarded because all models performed equally badly, limiting its utility for distinguishing baselines.
- 6.2.4 Leveraging Multispectral Information: Sentinel-2 pretraining with DINO or MoCo modestly improves ResNet50 performance on average, whereas multispectral input systematically decreases ViT-S performance.Randomly initializing missing first-layer channels for an RGB-only model does not produce systematic improvement and extends fine-tuning time.
- Segmentation benchmark: ResNet101-DeepLabV3 performs best in aggregate on segmentation, but still underperforms on some datasets.The segmentation experiments cover six combinations of ResNet18, ResNet50, and ResNet101 with U-Net and DeepLabV3.
- 6.4 Resource Usage: SwinV2 and ConvNeXt-B use up to 4x the memory of ResNet50, while their forward passes are only twice as slow.The benchmark reports parameters, memory usage, forward-pass time, and downstream fine-tuning convergence time for evaluated algorithms.
7 Conclusion
The paper introduces a reproducible benchmark for evaluating pre-trained models on remote-sensing downstream tasks. It aims to stimulate foundation-model development while reducing experimental resource demands, but does not cover all desired Earth-monitoring capabilities.
- GEO-Bench adapts remote-sensing datasets into a conventional pipeline and provides code for fine-tuning and evaluating individual tasks.
- Training on all seven subsets takes on average about 1.88 times longer than training on the full training set.
- The benchmark does not extensively evaluate temporal fine-tuning or fusion with text and weather data.
- Its spatial coverage includes most continents but could be expanded to more countries and biomes.
Checklist
The checklist records affirmative responses for accurately presenting contributions, documenting limitations and societal impacts, and reporting reproducible experimental details. It also records appropriate handling of assets, privacy, and non-applicable human-subject and theoretical-result requirements.
- The checklist identifies no applicable theoretical proofs, crowdsourcing procedures, participant risks, or compensation requirements.
- The checklist confirms that the paper describes its contributions and limitations and discusses potential negative societal impacts.
- The paper reports code, data, training details, and error bars needed to reproduce the experimental results.
- The authors report compute resources and cite, license, and release the assets used in the work.
A.1 Benchmark Coverage
The appendix examines benchmark coverage, baseline behavior, training-size effects, and convergence. The benchmark spans nearly all continents, while smaller training subsets support additional analyses but increase experiment count and computation.
- Benchmark coverage: The benchmark covers all continents except Antarctica, although China, Russia, North Africa, and South America remain substantially uncovered.
- Segmentation benchmark: U-Net performs more strongly overall, while ResNet101 is generally slightly worse and less stable than ResNet50.
- Segmentation benchmark: The segmentation benchmark reports normalized and raw IoU across baseline seeds, with violin plots summarizing bootstrap IQM or seed distributions.
- Training-size analysis: Training-size experiments evaluate normalized IoU as the training set grows from 1% to 100%, using confidence intervals from bootstrap IQM samples.
- Training-size analysis: Using exponentially decreasing training subsets makes the sevenfold experiment increase more computationally modest, while all subsets take about 1.88 times longer than the full set.
- Convergence analysis: Convergence time is overall proportional to training-set size, despite a mild behavior change near 1%.
A.5 Discriminativity of Datasets
The appendix defines dataset discriminativity as how well a dataset distinguishes algorithm performance and studies how training-set size affects that property. Smaller datasets usually improve discriminativity slightly, but estimates remain noisy and dataset-dependent.
- Metric definition: Discriminativity measures how well a dataset reveals whether one algorithm outperforms another, using repeated random-seed comparisons.
- Metric definition: The dataset-level measure averages pairwise discriminativity across model pairs and uses stratified bootstrap repetitions to estimate uncertainty.
- Results: Reducing dataset size almost systematically improves discriminativity, but only by a small amount.
- Results: BigEarthNet is the most discriminative dataset, whereas full-size pv4ger has poor discriminativity that can improve when reduced.
- Results: Discriminativity estimates are noisy and sensitive to random seeds.
- Resource usage: The appendix also reports model resource usage, including parameter count, memory, forward time, and convergence time.
B Remote Sensing Data Schema
The schema represents heterogeneous Earth-monitoring data through typed bands, sensor metadata, dataset specifications, and labels. It is designed to support flexible model construction and preprocessing across datasets.
- Data schema: The schema addresses Earth-monitoring heterogeneity by supporting modality, temporal, positional, and band encodings.Band encoding communicates data sources, while temporal and positional encodings support transformer-based processing.
- Data schema: Band combines spatial data with BandInfo metadata such as band name, resolution, and spectral range.A sensor-class hierarchy further supports introspection and flexible band encodings for transformer architectures.
- Data schema: Sample packages one or more Bands with a label, which may itself be a Band.
- Dataset specification: TaskSpecifications describes a dataset’s name, label type, band metadata, and shapes without loading samples.This enables procedural generation of a suitable machine-learning model at training start.
- Preprocessing: Band statistics include minimum, maximum, mean, variance, and percentiles for adapting inputs to pretrained-model expectations.
C.3 Emissions of large pre-trained models
The paper weighs the substantial emissions from training large models against their reuse and potential climate benefits. It recommends reducing emissions through cleaner infrastructure, efficient computation, and transparent reporting.
- Emissions: 284 tCO2e can result from training a large transformer on largely fossil-fuel-powered computers.The paper contrasts this with 2.8 tCO2e for a roundtrip San Francisco–London passenger flight.
- Trade-offs: The authors judge that climate-mitigation benefits from improved Earth monitoring likely outweigh foundation-model emissions.
- Mitigation measures: Selecting renewable-powered, carbon-neutral data centers with good PUE can reduce emissions by 50×.
- Mitigation measures: The paper recommends minimizing computationally intensive runs, choosing efficient hardware, monitoring emissions, reporting them, and offsetting cumulative emissions.
- Mitigation measures: Efficient code and network sparsification can reduce emissions by up to 10-fold.