Source-linked AI summary
OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation
Henry Herzog, Favyen Bastani, Yawen Zhang, Gabriel Tseng, Joseph Redmon, Hadrien Sablon, Ryan Park, Jacob Morrison, Alexandra Buraczynski, Karen Farley, Joshua Hansen, Andrew Howe, Patrick Alan Johnson, Mark Otterlee, Ted Schmitt, Hunter Pitelka, Stephen Daspit, Rachel Ratner, Christopher Wilhelm, Sebastian Wood, Mike Jacobi, Hannah Kerner, Evan Shelhamer, Ali Farhadi, Ranjay Krishna, Patrick Beukema
TL;DR
Earth observation foundation models are difficult to train, deploy, and apply to real-world tasks, particularly for non-profit and humanitarian organizations. OlmoEarth addresses these challenges with a stable multimodal spatio-temporal architecture and training regime, achieving the best performance on 15 of 24 embedding evaluations and 19 of 29 fine-tuning evaluations while supporting an open end-to-end platform.
Problem
Earth observation foundation models are complex and expensive to train and deploy, while training instability and representation collapse can limit their practical use.
Method
OlmoEarth combines a multimodal spatio-temporal encoder-decoder with modality-aware masking and Latent MIM Lite, which uses frozen random projections for stable latent-space training.
Results
15 of 24 embedding evaluations and 19 of 29 full fine-tuning evaluations identify OlmoEarth as the best-performing model.
Takeaways & Limitations
OlmoEarth is deployed through an open end-to-end platform used by partner organizations for environmental, conservation, and food-security applications.
Takeaways & Limitations
OlmoEarth Large does not always outperform OlmoEarth Base and is significantly worse on embedding-based pixel time-series tasks.
Abstract
from arXiv · showhide
Earth observation data presents a unique challenge: it is spatial like images, sequential like video or text, and highly multimodal. We present OlmoEarth: a multimodal, spatio-temporal foundation model that employs a novel self-supervised learning formulation, masking strategy, and loss all designed for the Earth observation domain. OlmoEarth achieves state-of-the-art performance compared to 12 other foundation models across a variety of research benchmarks and real-world tasks from external partners. When evaluating embeddings OlmoEarth achieves the best performance on 15 out of 24 tasks, and with full fine-tuning it is the best on 19 of 29 tasks. We deploy OlmoEarth as the backbone of an end-to-end platform for data collection, labeling, training, and inference of Earth observation models. The OlmoEarth Platform puts frontier foundation models and powerful data management tools into the hands of non-profits and NGOs working to solve the world's biggest problems. OlmoEarth source code, training data, and pre-trained weights are available at $\href{https://github.com/allenai/olmoearth_pretrain}{\text{https://github.com/allenai/olmoearth_pretrain}}$.
1 Introduction
OlmoEarth addresses the gap between promising Earth observation foundation-model research and limited real-world adoption by introducing a stable training regime, comprehensive evaluation, and an open platform for organizations using these tools.
- 1 Introduction: Foundation models are promising in research settings, but their size, training complexity, deployment cost, and instability limit real-world adoption, especially among non-profits.The paper identifies training instability, representation collapse, and underperformance relative to stated potential when reproducing existing work.
- 1 Introduction: OlmoEarth introduces Latent MIM Lite, replacing Latent MIM’s target encoder with a frozen random linear projection to stabilize training and unify supervised and self-supervised learning.The same token-space loss is applied to observational data and labeled maps without adding modality-specific predictor heads.
- 1 Introduction: OlmoEarth’s modality-aware masking and modality-restricted contrastive objective make multimodal reconstruction harder without relying on extreme masking ratios or easy negatives.The strategy combines random token masking with full modality reconstruction and contrasts tokens only within their respective bandsets.
- 1 Introduction: The evaluation compares OlmoEarth with 12 other foundation models across 18 research benchmarks and 19 partner datasets using transfer learning and full fine-tuning.The study is designed to address the lack of a standard evaluation suite for remote sensing models.
- 1 Introduction: OlmoEarth provides an end-to-end platform that brings foundation models to non-profit, humanitarian, and environmental organizations.The platform supports data gathering, alignment, preprocessing, labeling, fine-tuning, and inference for Earth observation tasks.
2 OlmoEarth
OlmoEarth is a multimodal, spatio-temporal Vision Transformer that processes aligned satellite observations and maps with flexible tokenization, masking, latent prediction, and contrastive objectives. Its training design uses modality-aware masking and frozen random projections to support stable multimodal learning.
- 2 OlmoEarth: OlmoEarth is a Vision Transformer encoder-decoder that attends across space, time, and modalities in aligned satellite image timeseries and derived maps.It adds positional, temporal, and modality embeddings before full self-attention, while a decoder predicts masked-token latents.
- 2 OlmoEarth: The pretraining dataset contains 285,288 global samples covering 2.56km×2.56km regions and one-year time ranges, with up to 12 monthly timesteps.Samples are selected from tiles containing 120 OpenStreetMap feature categories and use uniformly resampled 10-meter data.
- 2 OlmoEarth: The modality-aware masking strategy assigns bandsets to not selected, encode only, decode only, or encode-and-decode categories, including maps only as decoding targets or ignored inputs.It reframes reconstruction as predicting missing bandsets from partial views of other bandsets and avoids the overly easy all-bandset setting.
- 2 OlmoEarth: Latent MIM Lite predicts masked inputs in latent space using frozen, randomly initialized modality-specific projections, which avoids representation collapse and unifies supervised and self-supervised losses.The same architecture and loss handle observations and maps, while empirical ablations report benefits from additional supervised data.
3 Experiments
OlmoEarth is evaluated against other foundation models on research benchmarks and partner tasks using frozen-encoder transfer and full fine-tuning. It performs strongly overall, while ablations and scaling results reveal both the benefits of its training modifications and limitations for larger models.
- 3.2 Research Benchmarks and Partner Tasks: The evaluation compares 12 foundation models using shared training recipes, hyperparameter sweeps, and frozen-encoder or end-to-end fine-tuning protocols.Models are evaluated only on tasks compatible with their input modalities, and large models are not fine-tuned on every task because of resource constraints.
- 3.2 Research Benchmarks and Partner Tasks: OlmoEarth is best on 15 of 24 tasks with kNN/linear probing and 19 of 29 tasks with full fine-tuning.The comparisons cover research benchmarks and real-world partner tasks.
- 3.5 Results: OlmoEarth Large does not consistently outperform Base and is significantly worse on embedding-based pixel time series tasks.The authors suggest this may reflect more extensive recipe exploration for Base and the challenges of scaling Earth observation models.
- 3.5 Results: Panopticon performs strongly on embedding tasks, Terramind on fine-tuning tasks, and DINOv3 lags on tasks requiring temporal understanding.Galileo also performs strongly on many benchmarks, especially agriculture-related tasks.
- 3.6 Ablations: Latent MIM Lite substantially improves over Latent MIM, whose poor performance is attributed to representation collapse, while further modifications improve all evaluated tasks.The development path starts from standard Latent MIM and tests masking, loss, and map-data modifications.
- 3.7 Environmental Impact: Training OlmoEarth uses an estimated 4,307 kWh, equivalent to powering an average U.S. household for five months.The reported carbon estimate is equivalent to an economy flight from Seattle to Portugal, and the environmental estimate is a lower bound.
4 Related Work
Remote sensing pretraining has shifted from contrastive learning and pixel reconstruction toward latent-space masked modeling. Existing approaches improve representation quality but introduce instability or accessibility and flexibility limitations.
- 4 Related Work: Latent-space masked modeling is effective for remote sensing but has documented training instabilities.Recent remote sensing methods increasingly follow latent reconstruction approaches from natural-image research.
- 4 Related Work: TerraMind avoids instability by using a frozen tokenizer built from a quantized autoencoder for image modalities.The tokenizer is used during multimodal masked modeling.
- 4 Related Work: Precomputed embeddings improve accessibility but still require expertise and may prevent real-time or sub-annual predictions.OlmoEarth embeddings match or outperform AEF embeddings on partner tasks, while full fine-tuning enables better results.
5 Discussion
OlmoEarth is released through an open, end-to-end platform that supports environmental organizations from data curation through model inference. The paper also acknowledges misuse risks and outlines expansion toward climate, weather, and non-geospatial data.
- The OlmoEarth Platform supports data curation, labeling, model fine-tuning, and inference for organizations working on environmental challenges.Partner organizations include Global Mangrove Watch, Global Ecosystem Atlas, and the International Food Policy Research Institute.
- Global Mangrove Watch improved mangrove classification from a 95.3% F1 random-forest baseline to 98.1% after fine-tuning OlmoEarth.
- OlmoEarth’s open license permits free use, modification, and sharing while restricting military, defense-related, and extractive-industry applications.
- Planned extensions include climate and weather forecasting across wider spatial and temporal resolutions, plus geolocated natural images for fine-grained recognition.
A Research Benchmarks
The research-benchmark suite combines GEO-Bench with additional classification, segmentation, and time-series datasets. Its authors note that several benchmarks have important limitations affecting realism, comparability, or metric stability.
- The modified m-brick-kiln benchmark may be too easy because many negative examples appear to contain only dark pixels.
- m-forestnet includes more modified-subset examples than the original dataset, but the source of the additional examples could not be determined.
- Several benchmarks have dataset-specific concerns, including large-polygon labels affecting m-cashewplant and weak cross-benchmark correlation for Sen1Floods11 accuracy.
- GEO-Bench tasks generally use single satellite images or image pairs even when labels change slowly over time, although multiple inputs often improve remote-sensing performance.
- MADOS uses custom-processed images and contains rare classes that make test-set mIoU highly variable across runs.
B Partner Tasks
The partner-task suite spans conservation, agriculture, wildfire risk, renewable energy, marine infrastructure, and ecosystem mapping. Tasks vary substantially in modality, temporal context, label type, and geographic focus.
- The suite includes regression of live fuel moisture content, whose predictions are used to understand wildfire risk.
- Mangrove classification supports mapping mangrove growth and loss across different years.
- Ecosystem type mapping in North Africa classifies 2,361 examples into 110 level-3 IUCN Global Ecosystem Typology categories.
- Forest-loss driver classification uses four pre-loss and four post-loss Sentinel-2 images to predict causes such as mining, logging, and agriculture.
- Partner tasks include marine infrastructure detection, vessel detection and characterization, solar-farm segmentation, wildlife land-cover classification, and crop-type classification.
- Solar-farm maps are used to understand the global rate of renewable-energy deployment over time.
C Additional Ablations
Additional ablations evaluate individual components of the final OlmoEarth model and training recipe. They show that strong performance remains possible without map data or labeled data, while the final model prioritizes broad consistency over winning every metric.
- Table 6 removes components individually from the OlmoEarth base model and includes a MAE baseline, with models trained for 300,000 steps.
- The “No Maps” model retains relatively high performance using only observational data, whereas the Sentinel-2-only model performs relatively poorly.
- Pure self-supervised training still produces good performance, although the model can benefit from labeled data.
- The final OlmoEarth model has the best average score and lowest average per-task rank, although it is not best on every metric.
D Comparison to AlphaEarth Foundations
OlmoEarth is compared with AlphaEarth as both a frozen feature extractor and a fine-tunable model, but AlphaEarth’s unreleased status limits evaluation. OlmoEarth outperforms AlphaEarth after full fine-tuning across all five comparable tasks.
- AlphaEarth is evaluated on only five tasks because exporting its embeddings is costly, including 26 EECU-seconds per 32 × 32 crop and an estimated $290 for 100K crops.
- With KNN classification, OlmoEarth outperforms AlphaEarth on Nandi and AWF, while AlphaEarth outperforms OlmoEarth on Ecosystem mapping.
- Full fine-tuning lets OlmoEarth outperform the best possible AlphaEarth results on all five evaluated tasks.AlphaEarth cannot be evaluated under fine-tuning because Google has not released the model.
- The comparison evaluates AlphaEarth embeddings against OlmoEarth embeddings using KNN, frozen-backbone decoders, and full fine-tuning strategies.
E Patch Size Analysis for m_cashew_plant
For m_cashew_plant, larger patch sizes improve performance for models supporting variable patch sizes, an unusual pattern attributed to the dataset’s coarse polygonal labels.
- Larger patch sizes improve performance for variable-patch-size models such as OlmoEarth and Galileo on m_cashew_plant.Table 8 reports linear-probing and fine-tuning results across patch sizes.
- The observed trend is unusual because smaller patches typically improve performance in comparable settings.
- The authors hypothesize that coarse polygon labels, rather than pixel-level labels, explain the preference for larger patches.Figure 5 illustrates the dataset’s coarse, polygonal labels.
F OlmoEarth Platform
OlmoEarth Platform combines foundation models with data-management tools to support environmental organizations through the workflow from data collection and labeling to model deployment and map publishing.
- The platform supports satellite data collection, labeling, model fine-tuning, inference, and final map publishing in one workflow.
- It targets organizations working on environmental challenges, including conservation, climate action, and food security.
- The platform removes the need for organizations to manage GPU infrastructure or possess deep learning expertise.