Source-linked AI summary
OlmoEarth v1.3: Embeddings
Gabriel Tseng, Favyen Bastani, Patrick Alan Johnson, Yawen Zhang, Piper Wolters, Ando Shah, Hadrien Sablon, Joseph Redmon, Heather DeHaven, Rachel Ratner, Sasha Levy, Jordan Steward, Katrina Midgley, Andrew Howe, Hunter Pitelka, Ted Schmitt
TL;DR
Embeddings can be reused across downstream tasks, but earlier OlmoEarth models were principally deployed through finetuning. OlmoEarth v1.3 makes embeddings a first-class deployment target and produces annualized 10 m releases. Its 128-dimensional embeddings are best on 10/11 tasks against two prominent products, while its 64-dimensional embeddings beat both on 7/11.
Problem
Method
Results
Takeaways & Limitations
The release provides annualized 10 m embeddings globally for 2023–2025 and for select countries for 2017–2022.
Source: §6 · p. 15Coverage for 2017–2022 is limited to select countries, and the reported coverage is a snapshot while computation continues.
Source: §4 · p. 10
Abstract
from arXiv · showhide
We make some OlmoEarth embeddings. To do this, we train a new model: OlmoEarth v1.3. OlmoEarth v1.3 is designed to condense multimodal, multitemporal remote sensing data into a single digestible 64 or 128-dimensional vector. OlmoEarth v1.3 produces state-of-the-art embeddings. In addition, OlmoEarth v1.3 is a strong encoder backbone -- competitive with OlmoEarth v1 and v1.2 -- while requiring 18% and 53% of the GPU hours to train, respectively. We use OlmoEarth v1.3 to compute global, annualized 10m embeddings, which we release under the OlmoEarth artifact license.
1 Introduction
OlmoEarth v1.3 makes embeddings a first-class deployment target, addressing the reuse and large-scale analysis advantages of embeddings. Its 128-dimensional embeddings lead on 10 of 11 tasks against two prominent 10m products, while its 64-dimensional embeddings beat both on 7 of 11.
1 Introduction
- Embeddings let practitioners pay inference costs once and reuse representations across downstream tasks; they also enable large-scale aggregation unavailable through finetuning [19].For large-scale mapping with OlmoEarth v1.2, inference accounts for 98% of cost and data export and preprocessing for 2% [27].Source: §1 · p. 1
- OlmoEarth v1.3 makes embedding a first-class deployment target, unlike earlier OlmoEarth models principally deployed through finetuning [12] [27].The model addresses embedding design considerations of aggregation, high-resolution efficiency, and compactness.Source: §1 · pp. 1–2
- OlmoEarth v1.3’s 128-dimensional embeddings are best on 10/11 tasks, while its 64-dimensional embeddings beat Alpha Earth Foundations and TESSERA v2 [6] on 7/11.The model also maintains OlmoEarth v1.2’s quality as a finetuning model while requiring about half the GPU hours to train.Source: §1 · p. 2
- The report evaluates OlmoEarth v1.3 embeddings and its backbone, then describes global multi-year embedding generation, serving, and environmental impact.Source: §1 · p. 2
2 OlmoEarth v1.3
OlmoEarth v1.3 adapts the v1.2 encoder-decoder for embedding deployment by adding attention pooling and token distillation. These components address aggregation, output-resolution efficiency, and compact embeddings.
2.1 Aggregation
- ViT tokens span one modality, timestep, and spatial patch; learned Attention Pooler queries aggregate them, while RoPE gives queries spatial identity without dependence on window position.The same learned location query is repeated across spatial patches, and direct supervision trains each location token to reconstruct the map pixels it covers.Source: §2.1 · p. 3
- PASTIS validation mIoU rises from 51.3 without direct supervision to 58.4 with supervision on location tokens.The supervision trains each token to reconstruct map pixels in its spatial footprint.Source: §2.1 · pp. 3–4
2.2 Efficiency at high resolutions
- 7.7× lower cost (308 vs. 2,378 GMACs per window) comes from running the ViT at 20m while the Attention Pooler outputs 10m embeddings.The Attention Pooler adds negligible cost while enabling the lower-cost forward pass.Source: Figure 4 · p. 4, §2.2 · p. 4
- Sampling one pixel per masked patch for pixel-resolution PixLatentMIM Lite targets yields a +0.5 mIoU improvement over standard LatentMIM Lite on PASTIS validation.Pooling patch sizes are sampled such that pp ≤ pe, allowing location tokens at different resolutions.Source: §2.2 · p. 4
2.3 Compaction
- Distillation projects 768-dimensional location tokens to 128 dimensions, with Matryoshka losses also training the first 64 dimensions as a smaller embedding.The chosen recipe preserves a 768-dimensional teacher backbone and provides nested embedding sizes.Source: §2.3 · p. 5
2.4 Putting it together
- Because the Attention Pooler aggregates tokens during both pretraining and deployment, OlmoEarth v1.3 removes instance contrastive loss without a performance regression.This reduces gradient updates to one forward pass per microbatch instead of two.Source: §2.4 · p. 6
- OlmoEarth v1.3 requires 53% of the GPU hours of v1.2 and 18% of v1’s GPU hours to train.It otherwise follows the same training recipe as OlmoEarth v1 and v1.2.Source: §2.4 · p. 6
3 Experiments & Results
OlmoEarth v1.3 produces strong 128- and 64-dimensional embeddings and remains a competitive backbone. Evaluations show that its embedding quality and compute trade-offs depend on patch size and task, while the Attention Pooler can affect both performance and cost.
3.1 Embedding evaluations
- Embedding evaluations use annualized, quantized inputs from all three modalities, with 12 timesteps, 16 × 16 windows, encoder patch size 2, and 10m output patches.The AEF protocol samples balanced training sets and reports mean accuracy across repeated draws; PLANTEUR also tests leave-one-island-out geographic generalization.Source: §3.1 · pp. 6–7
3.1.1 Comparison with AEF and Tessera v2
- 128-dimensional embeddings outperform both baselines on 7/8 AEF datasets; 64-dimensional embeddings outperform both on 5–7/8, with TESSERA v2 consistently ahead on Ethiopia.On crop-type segmentation, the 128-dimensional embeddings score 58.6 mIoU versus TESSERA v2’s 58.3 on PASTIS and are best on PLANTEUR.Source: §3.1.1 · p. 7
- Across linear classifiers, the 128-dimensional embeddings remain best on 7/8 AEF datasets; they are best on 6/8 under random forests and MLPs.Regularized linear models perform best for OlmoEarth and TESSERA, while AEF benefits from MLP and random-forest readouts.Source: §3.1.1 · pp. 7–8
3.1.2 Embedding quality as a function of ViT-patch size
- Patch size 2 reduces PASTIS inference cost 7.8×, from 2,396 GMACs to 308 GMACs, while mIoU decreases only from 60.2 to 59.8.This is the setting used for evaluations and deployment; Figure 6 plots PASTIS validation mIoU against inference cost.Source: Figure 6 · p. 7, §3.1.2 · p. 8
3.2 Non-embedding evaluations
- The Attention Pooler adds 2% compute on PASTIS, 16% on single-timestep, single-modality tasks, and 9% averaged across non-embedding evaluations.These evaluations use encoder and pooler patch size 4, unlike the embedding evaluations’ encoder patch size 2.Source: §3.2 · p. 8
3.2.1 kNN and Linear Probing Results
- OlmoEarth v1.3 Base without the Attention Pooler has the highest mean rank across 24 kNN/LP tasks: 4.1 versus 4.3 for v1.2 and 4.2 for v1.Adding the pooler lowers v1.3 performance by 2.1 points on average; the authors hypothesize it fares worse on tasks lacking tokens to aggregate or spatial context.Source: §3.2.1 · p. 9
3.2.2 Finetuning Results
- OlmoEarth v1.3 without Attention Pooling has a 4.34 mean finetuning rank, behind v1.2 Base at 4.02 and ahead of v1 Base at 4.74.Attention Pooling lowers v1.3’s mean rank to 5.88.Source: §3.2.2 · p. 9
- v1.3 ranks second on both research and partner finetuning tasks, with mean ranks of 4.50 and 4.23, respectively.v1.2 leads on research benchmarks, while v1 leads on partner tasks.Source: §3.2.2 · p. 9
4 Creating & Serving Global Scale Embeddings
OlmoEarth v1.3 embeddings are released as a precomputed, readable-from-object-storage dataset of annual 10 m vectors. A queued, parallel inference pipeline builds the coverage area, which is still incomplete for earlier years.
4 Creating & Serving Global Scale Embeddings
- The release contains 128-dimensional int8 vectors at 10 m resolution, one per pixel per reference year, readable directly from object storage.Embeddings are computed from one calendar year of imagery; the store reserves slots for 2017–2025, with uncomputed years taking no storage.Source: §4 · p. 10
- 2025, 2024, and 2023 embeddings are global; 2017–2022 cover select countries and remain in progress, so the stated coverage is a snapshot.A coverage layer records the extent each year will reach when complete.Source: §4 · p. 10
4.1 Inference at scale
- Inference is queued as independent spatial work units, and workers materialize imagery, run the model, and write results directly to the store.Work units can be resumed: completion markers are written after embeddings land, and enumeration skips units already marked.Source: §4.1 · pp. 10–11
- Each UTM zone is stored as a sparse Zarr array, with shards aligned to inference windows and chunks serving as the transfer unit.Chunking the embedding axis at 64 lets a 64-dimension Matryoshka prefix fetch one chunk rather than two.Source: Figure 8 · p. 11
- A 480-pixel crop does not fit in 40 GB on A100 workers, so they use 384-pixel crops, matching H100 output at mean cosine similarity 0.9999.H100 workers complete a full unit in a median of 30 minutes; A100 workers take a median of 84 minutes.Source: §4.1 · p. 11
4.2 Removing tiling artifacts using Searchlight
Tiled inference can create artifacts where crops meet because edge pixels have less, directionally biased spatial context. Searchlight uses neighborhood attention over a large domain to produce translation-invariant inference and remove crop seams.
4.2 Removing tiling artifacts using Searchlight
- Tiled inference artifacts arise because crop-edge pixels have less spatial context, which can create discrepancies between pixels processed by different crops.The artifacts are especially visible on homogeneous areas such as water.Source: §4.2 · p. 11, Figure 9 · p. 12
- Searchlight applies neighborhood attention to one large domain, with each pixel attending to its centered 16 × 16 window, making inference translation-invariant apart from boundary effects.Its compute cost grows linearly with input size.Source: §4.2 · p. 11
- PASTIS validation mIoU is 60.4 with Searchlight and 59.8 with tiled inference, remaining stable despite differences between training and inference attention and input size.Searchlight removes artifacts between crops.Source: §4.2 · p. 11, Figure 9 · p. 12
4.3 Seamless windows
A 48-pixel margin around inference windows reduces seams while leaving interiors unchanged. The embedding store is organized for scalable writes and area reads, with chunking that makes 64-dimension prefixes cheaper to retrieve.
4.3 Seamless windows
- 0.960 to 0.994: mean cosine similarity across window edges rises with a 48-pixel margin, while window interiors remain unchanged at cosine similarity ≥0.9999.A 112-pixel margin gives no further gain; 48 pixels adds 9.6% to pixels computed per window.Source: §4.3 · p. 12, Figure 10 · p. 12
4.4 Dataset layout
- Each UTM zone is stored in one array using its northern CRS, with northing continuing below zero south of the equator to keep coordinates monotonic across it.The conventional paired northern and southern CRS approach would require reads crossing the equator to join two arrays.Source: §4.4 · p. 12
- 2048 × 2048-pixel shards align with inference windows, so each window writes one object owned by one worker without coordination.The store uses Zstandard level 3; because chunks are not seekable, a single-pixel query moves the whole chunk, roughly 3 MB, so the layout favors area reads.Source: §4.4 · p. 13
- 3.04 MB instead of 6.05 MB per tile: chunking the embedding axis at 64 lets a Matryoshka 64-dimension prefix fetch one chunk rather than two.The chunk size is the narrowest width the model is trained to emit.Source: §4.4 · p. 13, Figure 11
4.5 Quantization
The embeddings are stored as int8 using signed-power quantization, retaining emitted magnitudes rather than L2-normalizing. Dequantized vectors closely match the float embeddings.
4.5 Quantization
- The embeddings are stored as int8 using a signed-power scheme with scale s = 127.5 and exponent γ = 1/2.The scheme encodes and recovers float values; −128 is reserved as nodata.Source: §4.5 · p. 13
- 0.99997: median cosine similarity between a float embedding and its dequantized form.The values are quantized as emitted rather than L2-normalized, preserving magnitude used by downstream classifiers.Source: §4.5 · p. 13
- Nearly all coordinates of the student head’s LayerNorm output fall inside the scheme’s [−1, 1] range, and clamping handles the rest.Source: §4.5 · p. 13
4.6 Provenance
The provenance records document which scenes contributed to each 30-day mosaic and where the imagery came from, and are released as GeoParquet tables by reference year.
4.6 Provenance
- Each window and modality record lists the scenes composited into each 30-day mosaic, in compositing order.Source: §4.6 · p. 13
- Records include each window’s materialized bounds and margin, and resolve scenes to their public imagery source and product name.Source: §4.6 · p. 13
- For 2025, the GeoParquet release contains about 450,000 windows, 21 million mosaic entries, and 730,000 scenes in 150 MB.The tables are released per reference year, with rows for windows and scenes in the mosaics.Source: §4.6 · p. 13
4.7 Distribution
The embeddings are distributed openly from object storage, with a browser explorer for visualizing and analyzing them; its false-color layer is a separate visualization rather than a reduced embedding dataset.
4.7 Distribution
- At 10 m global coverage, 128 int8 dimensions amount to roughly 150 TB per reference year and well over a petabyte for 2017–2025; Source Cooperative hosts the dataset openly.It can be read directly through an S3-compatible endpoint with any Zarr client, without an API key or tile server.Source: §4.7 · pp. 13–14
4.8 Interactive exploration
- The browser application maps the false-color layer and supports similarity search and clustering, with results exportable as reproducible Python scripts.Source: §4.8 · p. 14
4.9 False-color visualization
- The false-color layer projects a stratified-sample PCA basis to three uint8 RGB components, capturing only a fifth to two fifths of local variance.It is a visualization, not a reduced-dimension embedding dataset, and is stored separately after reprojection to Web Mercator.Source: §4.9 · p. 14
5 Environmental Impact
OlmoEarth v1.3 substantially reduces pretraining cost relative to earlier versions, although generating annual global embeddings costs far more than pretraining; patch size offers a compute–performance trade-off.
5.1 Pretraining
- OlmoEarth v1.3 Base required 18% / 53% of OlmoEarth v1 / v1.2 GPU hours and emitted 22% / 50% of their tCO2eq.The pretraining impact estimate is a lower bound because it excludes hardware manufacturing, transportation, and other factors.Source: §5.1 · p. 14, Table 5 · p. 14
5.2 Computing embeddings
- One year of global annualized embeddings costs approximately 14,000 H100 hours, about 26× the pretraining cost per year of annualized embeddings.This is an estimate because inference mixes H100s and A100s.Source: §5.2 · p. 14
- Patch size 4 uses roughly 5× less compute than patch size 2, while PASTIS validation mIoU drops from 59.8 to 58.6.Patch size 2 was selected for inference as a balance between performance and compute cost.Source: §5.2 · p. 14
6 Discussion
OlmoEarth v1.3 supports the project’s goal of making Earth observation insights easier to obtain through lower-cost pretraining and precomputed embeddings, which are released for specified years and regions under a restricted-use license.
6 Discussion
- The authors hope halving pretraining costs will broaden geospatial foundation-model research, while precomputed embeddings make high-quality mapping possible without running the model.Source: §6 · p. 15
- Annualized 10 m embeddings are released globally for 2023–2025 and for select countries for 2017–2022; users can choose 128 dimensions or the 64-dimensional prefix.For other time ranges, users can run the model through OlmoEarth Studio or generate embeddings locally.Source: §6 · p. 15
- OlmoEarth v1.3 and its embeddings are released under the OlmoEarth Artifact License, which restricts military, defense-related, and extractive-industry use.Source: §6 · p. 15
A Full kNN / LP results
Table 6 provides full kNN and linear-probe results across research benchmarks and real-world partner tasks. It compares OlmoEarth-Base variants, including v1.3 with and without Attention Pooling.
A Full kNN / LP results
- Table 3 compares OlmoEarth-Base models with the best non-OlmoEarth models, while Table 6 provides the full results.Source: A Full kNN / LP results · p. 18
- Table 6 reports kNN and linear-probe results on research benchmarks and real-world partner tasks.Not all models can run on all tasks because of incompatible input modalities.Source: Table 6 · p. 18
- The table compares v1.3 with Attention Pooling and without it.Source: Table 6 · p. 18
- The table highlights the winning ViT Base model among v1, v1.2, and v1.3.Source: Table 6 · p. 18