Source-linked AI summary
No One Knows the State of the Art in Geospatial Foundation Models
Isaac Corley, Nils Lehmann, Caleb Robinson, Gabriel Tseng, Anthony Fuller, Hamed Alemohammad, Evan Shelhamer, Jennifer Marcus, Hannah Kerner
TL;DR
Geospatial foundation model research lacks standardized evaluations, protocols, weights, and pretraining controls needed to determine which models fit which tasks. The paper audits 152 papers and proposes six community expectations; it finds substantial cross-paper disagreement and limited comparability. It concludes that shared evaluation and reporting standards are needed for a common understanding of GFM progress.
Problem
GFM literature lacks standardized evaluations, protocols, released weights, and pretraining controls needed to compare models for specific tasks.
Method
The paper audits a transparent corpus of 152 GFM papers and proposes six recommendations for authors, reviewers, venues, and benchmark maintainers.
Results
46 of 301 multi-paper tuples show at least 10-point metric spreads under nominally fixed models, benchmarks, and protocols.
Takeaways & Limitations
The paper calls for shared core evaluations, a versioned evaluation harness, reproducibility annotations, variance reporting, and controls separating data, architecture, and algorithm effects.
Takeaways & Limitations
Reproducible evaluation can still misrepresent deployment utility because datasets and patch-level scores may not reflect operational distributions or map-level accuracy.
Abstract
from arXiv · showhide
Geospatial foundation models (GFMs) have been proposed as generalizable backbones for disaster response, land-cover mapping, food-security monitoring, and other high-stakes Earth-observation tasks. Yet the published work about these models does not give reviewers or users enough information to tell which model fits a given task. We argue that nobody knows what the current state of the art is in geospatial foundation models. The methods may be useful, but the GFM literature does not standardize evaluations, training and testing protocols, released weights, or pretraining controls well enough for anyone to compare or rank them. In a 152-paper audit, we find 46 cross-paper disagreements of at least 10 points for the same model, benchmark, and protocol; 94/126 papers with extractable pretraining data use a configuration no other paper uses; and 39% of GFM papers release no model weights. This lack of community standards can be solved. We propose six concrete expectations: named-license weight release, shared core evaluations, copied-versus-rerun baseline annotations, variance reporting, one shared evaluation harness, and data-vs-architecture-vs-algorithm controls. These gaps are a coordination failure, not a fault of any individual lab; the authors of this paper, like many others in the GFM community, have contributed to them. Rather than just critiquing the community, we aim to provide concrete steps toward a shared understanding of how to innovate GFMs.
1 Introduction
Geospatial foundation models promise reusable backbones across Earth-observation tasks, but inconsistent evaluations, reporting, and pretraining controls prevent reliable comparisons or identification of the state of the art. This paper audits the literature and proposes recommendations to improve comparability without claiming the methods themselves are useless.
- Motivation: GFMs promise one pretrained Earth-observation backbone transferable across sensors, geographies, label regimes, and downstream tasks, making evaluation harder than ordinary model comparison.The cited downstream examples include crop, flood, building, forest, and land-cover-change mapping.
- Contributions: The authors release a 152-paper systematic review, identify three troubling trends, and propose six recommendations labeled R1 through R6 for improving GFM research.The recommendations target authors, reviewers, venues, and benchmark maintainers.
- Problem: 46 disagreements of at least 10 points occur across papers for the same model and benchmark, including Scale-MAE accuracy reports of 33.0 and 89.6 on NWPU-RESISC45.Both reports use the same released checkpoint and nominal protocol, so readers cannot determine which result to trust.
- Scope: The paper critiques how GFMs are reported and compared, not whether pretrained satellite-imagery backbones work, and does not require public or identical pretraining data.Private and diverse data sources remain compatible with the foundation-model concept when evidence and comparisons are adequate.
2 Publication corpus
The authors build a transparent, reproducible 152-paper corpus of geospatial foundation-model literature using survey seeding, citation expansion, and structured extraction with human validation. They include both explicitly labeled foundation models and earlier self-supervised remote-sensing models, while excluding incomplete-year, paywalled, metadata-poor, and poorly reusable candidates.
- Corpus construction: 152 papers form an audited corpus seeded from prior GFM surveys and expanded through OpenAlex and Semantic Scholar citation graphs.The expansion adds papers from 2024–2025 not covered by the surveys.
- Corpus construction: 140 papers use downloaded LaTeX sources, while the remaining 12 are converted from PDFs into structured markdown for extraction.The pipeline extracts metadata on models, architectures, pretraining, downstream tasks, and code and weight releases.
- Validation and reproducibility: The extraction, validation, analyses, code, and intermediate outputs are documented or released to support reproducibility.A second LLM pass flags disagreements, followed by manual human review for validation.
- Corpus composition: 46% of the 152 papers explicitly call their proposed model a foundation model in the title, abstract, or contributions.The corpus also includes earlier self-supervised remote-sensing models included by prior GFM surveys.
- Scope and exclusions: The authors exclude 2026 papers, paywalled or metadata-poor venues, and broader-search candidates lacking weights, code, or pretraining data.The latter exclusions may make the reported headline numbers conservative because such papers would likely move them further in the same direction.
3 Troubling Trends in GFM Comparisons · 3.1 Model weights are not published · 3.2 The field does not have a shared set of core benchmarks
GFM comparisons are hindered by unreleased model weights and the absence of shared core benchmarks. These repeated reporting choices prevent reliable reuse, reruns, and literature-wide rankings.
- 3 Troubling Trends in GFM Comparisons: The paper identifies repeated reporting choices that make GFM claims harder to understand and pairs each troubling trend with a recommendation.Section 4 translates these trends into actions for authors, reviewers, and the community.
- 3.1 Model weights are not published: 39% of papers release no model weights, preventing downstream reuse, model comparison, and baseline reruns.Another 19% provide code without a model artifact, requiring attempted retraining from the authors’ codebase.
- R1. Release weights under a named license (§2, §3.5): Pretrained satellite-imagery models intended for reuse should release weights under a named license by camera-ready publication or name the release constraint.This recommendation establishes a concrete condition for reproducible downstream use.
- 3.2 The field does not have a shared set of core benchmarks: 152 papers report evaluations on 401 distinct benchmarks, yielding 1,046 evaluation experiments and an average of 2.6 evaluations per benchmark.Benchmark aliases were merged, while auxiliary label-source evaluations were excluded.
- 3.2 The field does not have a shared set of core benchmarks: The mean fraction of downstream benchmarks overlapping the top-10 is 0.27 (95% CI [0.23, 0.32]), while 50/143 papers (35%) have zero overlap.The year-by-year Gini coefficient is stable after 2023, indicating no improving convergence on shared benchmarks.
- 3.2 The field does not have a shared set of core benchmarks: No GFM in the corpus can credibly claim a literature-wide ranking because published results lack enough shared benchmarks and fixed protocols for fair comparison.The paper characterizes this as a benchmark-lottery problem requiring community coordination.
- R2. Report on a shared core set of evaluation datasets (§3.2): Authors comparing pretrained satellite-imagery models should report shared core evaluations with a clear protocol and add tests addressing other claimed aspects of novelty.A shared core set is intended to make comparisons more interpretable while preserving room for additional novelty-specific evaluations.
3.3 Reported metric values diverge by tens of points across papers, at fixed protocol · 3.4 Aggregated benchmarks provide dataset bundles, not evaluation harnesses
Across shared geospatial benchmarks, papers report metric values that diverge by tens of points under nominally fixed protocols, while benchmark bundles lack the common harness needed to make evaluations comparable. The paper therefore calls for explicit provenance and variability annotations, a shared evaluation tool, and caution that reproducibility alone may not ensure operational validity.
- 3.3 Reported metric values diverge by tens of points across papers, at fixed protocol: 46 of 301 multi-paper tuples differ by at least 10 points, including Scale-MAE accuracy 33.0 versus 89.6 on NWPU-RESISC45 linear probing.The comparison used 10,817 normalized results across 152 papers and excluded generic and classical-ML baselines.
- 3.3 Reported metric values diverge by tens of points across papers, at fixed protocol: Such gaps can reflect unannotated changes in splits, channels, classes, normalization, adaptation recipes, baseline sweeps, or vision-language settings.For linear probing, optimizer, head learning rate, and evaluation crop may also be unspecified; prompt templates, verbalizers, API snapshots, and temperature add hidden axes.
- 3.3 Reported metric values diverge by tens of points across papers, at fixed protocol: Every result-table entry should identify whether it is ⟳RERUN with disclosed configuration or COPIED with its source paper and protocol cited.This annotation is intended to distinguish copied numbers from rerun baselines that may use different procedures.
- 3.3 Reported metric values diverge by tens of points across papers, at fixed protocol: Headline comparisons should report repeated-run mean±std when affordable, or state clearly when a result comes from one run.The recommendation is especially important for benchmarks whose results vary substantially across random seeds.
- 3.4 Aggregated benchmarks provide dataset bundles, not evaluation harnesses: LLM evaluation harnesses standardize versioned task definitions, reference protocols, CI submission checks, and common task configurations, unlike geospatial benchmark bundles.The cited example is lm-evaluation-harness, which powers the Open LLM Leaderboard.
- 3.4 Aggregated benchmarks provide dataset bundles, not evaluation harnesses: Geospatial repositories provide curated splits, dataloaders, task lists, and example code, but no canonical tool with versioned protocols, CI-gated submissions, and a community-wide execution standard.Examples include GEO-Bench, PANGAEA, FoMo-Bench, and PhilEO Bench.
- 3.4 Aggregated benchmarks provide dataset bundles, not evaluation harnesses: Shared datasets remain incomparable when optimizer, head learning rate, evaluation crop, or macro-versus-weighted-per-class IoU averaging differs.The paper argues that curating additional datasets cannot close this protocol gap and connects it to the disagreements reported in §3.3.
- 3.4 Aggregated benchmarks provide dataset bundles, not evaluation harnesses: Community benchmark maintainers and venues should build one shared evaluation tool with fixed task definitions and automatic submission checks.Even this is insufficient if datasets omit operational task, geography, sensor, and label distributions, or if spatial autocorrelation inflates accuracy estimates.
3.5 Architecture and pretraining-data improvements are confounded
Architecture, pretraining-data, and algorithm changes are often confounded, preventing readers from attributing gains to a specific modification. The paper therefore calls for controlled comparisons and more complete accounting of pretraining data to improve comparability.
- Attribution problem: Changing model architecture and pretraining data together prevents readers from identifying which change caused an observed gain.Private, proprietary, and diverse pretraining data remain acceptable when the dataset itself is the claimed contribution or architecture-only attribution is not requested.
- Pretraining-data diversity: 87 distinct named primary pretraining datasets span 126 papers that identify a specific dataset, while 26 papers describe pretraining data only generically.These counts follow merging dataset aliases and excluding unnamed or misnamed datasets.
- Pretraining-data diversity: 9 papers use MillionAID, compared with 8 for SSL4EO-S12, 6 for fMoW, and 5 for fMoW-RGB.Named sensor labels can conceal dozens of overlapping derived pretraining datasets with different scene compositions.
- Pretraining-data comparability: Naming the same dataset does not ensure comparable pretraining because one model may use BigEarthNet alone while another uses it within a custom mixture.The paper therefore computes each paper’s full pretraining set as the deduplicated set of all named datasets used for pretraining.
- Controlled comparisons: New GFMs should isolate claimed changes across pretraining data, architecture, and algorithm through controlled comparisons.Without these controls, stronger changes can conceal weaker results and obscure which modification produced an improvement.
4 Recommendations
The paper turns six documented evaluation and reproducibility gaps into actionable expectations for GFM authors: release reusable weights, use shared benchmarks and tooling, clarify baseline provenance, report uncertainty, and control data changes. It also urges venues to treat these expectations as default review requirements.
- R1: Release reusable weights: 39% of the corpus releases no weights, so reusable models should ship weights under a named permissive-by-default license or explain release constraints.Restrictions may arise from sensor licensing, data residency, export control, or partner requirements.
- R2: Shared core evaluations: Authors should evaluate models on a minimum shared core of benchmarks with explicit protocols, adding task-specific tests for claimed capabilities.The listed task families include classification, segmentation, change detection, regression, time series, multimodal/SAR, and map-level evaluation.
- R3: Baseline provenance: Every baseline should be labeled COPIED or ⟳RERUN, with source or new configuration details disclosed and large rerun discrepancies flagged.Rerunning the proposed method while copying all baselines without protocol notes is not a fair comparison.
- R4: Uncertainty reporting: Authors should report mean±std over repeated runs when affordable, and otherwise disclose single-run status and randomness sources before claiming improvements.For expensive runs, gains should not be presented as clear unless likely larger than normal run-to-run differences.
- R5: Shared evaluation harness: The community needs one third-party, CI-gated evaluation harness with versioned protocols, a common submission format, and automated checks.GEO-Bench, PANGAEA, TerraTorch, and related bundles provide starting points, but no single shared tool yet runs the same protocol for everyone.
- R6: Control pretraining changes: Papers combining new pretraining data with new architectures or objectives should include a comparison fixing the data source, preferably within the same sensor family.Suggested public controls include SSL4EO-S12, MillionAID, fMoW, and MajorTOM-Core at similar token or image budgets.
5 Alternative views
The authors argue that GFM evaluation problems will not self-correct through time or input heterogeneity alone. They frame the findings as describing the public record while allowing proprietary-data constraints when releasing artifacts.
- Alternative views: 56.6-pt within-model divergence, 35% zero top-10 overlap, and 75% unique pretraining configurations indicate the measured failures will not fix themselves with time.The authors agree that a consolidating leaderboard is the right self-correction route, but reject waiting for concentration and divergence to resolve organically.
- Alternative views: Input heterogeneity does not explain disagreement within an axis, as shown by the Scale-MAE/RESISC45 disagreement on a single benchmark.The authors support remote-sensing-native standards without mandating one input format.
- Alternative views: The trends concern the public record visible to reviewers, leaderboard maintainers, and downstream users, though stricter “top papers only” filters could produce different results.The corpus already excludes metadata-poor venues and obvious non-reuse papers, but no obvious stricter filter exists.
- Alternative views: Proprietary or private-data constraints can justify closed weights or data, but absent such constraints, authors should release all artifacts and state the constraints blocking release.AlphaEarth is cited as an example of a closed-weight or closed-data model that can still contribute to the research community.
6 Conclusion
The conclusion argues that geospatial foundation models cannot yet be ranked reliably because published studies lack shared comparison controls. It calls for coordinated standards and envisions maintained, protocol-consistent leaderboards for users and researchers.
- Conclusion: A 152-paper audit supports the claim that current GFM state-of-the-art comparisons are unreliable because studies lack shared benchmarks, consistent scores, and controlled pretraining setups.These gaps make model gains difficult to attribute across papers.
- Limitations: The analysis is limited by reliance on LLM-based extraction, public APIs, and public data, although humans manually verified every extracted field against source papers.The authors note that human error remains possible.
- What the future looks like if we are successful: If successful, a maintained leaderboard would let users select task, sensor, region, and date range while trusting that top entries share the same evaluation protocol.Researchers could avoid comparing against every backbone from the previous seven years solely because baseline knowledge is fragmented.
B Extraction Pipeline
The study extracts structured records from LaTeX and PDF-only papers using LLM-assisted processing, then reviews, normalizes, and manually verifies the results.
- Extraction Pipeline: Records come directly from LaTeX sources using Claude Opus 4.7 and GPT 5.5 Codex.PDF-only papers are converted to structured markdown with Docling before applying the same pipeline.
- Extraction Pipeline: A second LLM review pass flags extraction disagreements for manual inspection.
- Extraction Pipeline: Names are normalized to deduplicate repeated models, benchmarks, and pretraining datasets before manual verification of extracted fields.
C Weight & Code Release Audit
The audit defines weight release conservatively from explicit paper claims or publicly hosted artifacts, excluding dataset pointers. It finds that 39% of papers release no model weights, while 19% release code without a model artifact.
- Weight-release criteria: Weight-release status is determined from explicit PDF or LaTeX claims, with public hosting checked when claims are unclear.Pointers to pretraining datasets or HuggingFace dataset entries do not count as weight releases.
- Audit results: 59/152 = 39% of papers release no model weights.The corpus includes 93 papers releasing weights, 26 explicitly declining, and 33 remaining uncertain.
- Audit results: 29/152 = 19% ship a public code repository without releasing a model artifact.This is a separate subset from the overall weight-release audit.
D Reported-number divergence: harvest details
The analysis groups reported results only when model, benchmark, metric, evaluation regime, and training-fraction bucket match. It further restricts comparisons to full-training results, excludes classical-ML baselines, and omits detection benchmarks with inconsistent mAP definitions.
- Reported-number divergence: harvest details: Table 1 groups numbers by the strictest extractable combination of model, benchmark, metric, evaluation regime, and training-fraction bucket.Evaluation regimes include linear, kNN, fine-tune, zero-shot, and few-shot.
- Reported-number divergence: harvest details: The released code keeps only full-train results, drops classical-ML baselines, and excludes detection benchmarks where mAP definitions differ.Excluded definitions include mAP, mAP@50, and oriented-object-detection mAP.