PaperlayerView exact revision on arXiv

Source-linked AI summary

Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications

Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Þorsteinn Elí Gíslason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Oliveira, Joao Lucas de Sousa Almeida, Rocco Sedona, Yanghui Kang, Srija Chakraborty, Sizhe Wang, Carlos Gomes, Ankur Kumar, Myscon Truong, Denys Godwin, Hyunho Lee, Chia-Yu Hsu, Rohit Lal, Ata Akbari Asanjan, Besart Mujeci, Disha Shidham, Trevor Keenan, Paulo Arevalo, Wenwen Li, Hamed Alemohammad, Pontus Olofsson, Christopher Hain, Robert Kennedy, Bianca Zadrozny, David Bell, Gabriele Cavallaro, Campbell Watson, Manil Maskey, Rahul Ramachandran, Juan Bernabe Moreno

arXiv:2412.02732v3cs.CV

TL;DR

Geospatial foundation models have limited support for multi-temporal data, diverse validation, and accessible fine-tuning workflows. Prithvi-EO-2.0 addresses these gaps with global decade-spanning HLS pretraining, spatiotemporal modeling, metadata embeddings, and SME-guided evaluation. It improves on its predecessor across GEO-Bench and performs strongly across diverse real-world Earth observation applications.

  • Problem

    Existing geospatial foundation models often do not model Earth observation’s multi-temporal structure, lack comprehensive cross-task validation, and provide limited fine-tuning guidance.

  • Method

    Prithvi-EO-2.0 is pretrained with masked autoencoding on a global HLS dataset and evaluated through GEO-Bench and SME-led downstream applications.

  • Results

    Prithvi-EO-2.0 improves up to 8% over its predecessor across standard benchmarks and shows state-of-the-art results across SME-led applications.

  • Takeaways & Limitations

    The model provides a versatile foundation for classical Earth observation and high-resolution applications spanning disaster response, land cover and crop mapping, and ecosystem dynamics monitoring.

Abstract

from arXiv · show

This paper presents Prithvi-EO-2.0, a new geospatial foundation model that offers significant improvements over its predecessor, Prithvi-EO-1.0. Trained on 4.2 million global time series samples from NASA's Harmonized Landsat and Sentinel-2 data archive at 30-m resolution, the new model incorporates temporal and location embeddings for enhanced performance across various geospatial tasks. Through extensive benchmarking with GEO-Bench, the model outperforms the previous Prithvi-EO model by 8% across a range of tasks. It also outperforms six other geospatial foundation models when benchmarked on remote sensing tasks from different domains and resolutions (i.e. from 0.1 m to 15 m). The results demonstrate the versatility of the model in both classical Earth observation and high-resolution applications. Early involvement of end-users and subject matter experts (SMEs) allowed constant feedback on model and dataset design, enabling customization across diverse SME-led applications in disaster response, land cover and crop mapping, and ecosystem dynamics monitoring. Prithvi-EO-2.0 is available as an open-source model on Hugging Face and IBM TerraTorch, with additional resources on GitHub. The project exemplifies the Trusted Open Science approach embraced by all involved organizations.

I. INTRODUCTION

Prithvi-EO-2.0 addresses gaps in geospatial foundation models by combining multi-temporal, global pretraining with extensive benchmarking and SME-guided applications. Its design emphasizes long-term Earth observation dynamics, broad evaluation, and practical fine-tuning support.

  • Motivation: Available geospatial foundation models often neglect multi-temporal data, provide limited diverse validation, or lack streamlined fine-tuning guidance.These limitations restrict users’ ability to assess model suitability and broaden adoption.
  • Contribution: Prithvi-EO-2.0 uses a global decade-spanning HLS dataset and spatiotemporal modeling to capture seasonal and long-term dynamics.The pretraining data consists of medium-resolution 30 m satellite imagery sampled to represent diverse land cover and ecosystems.
  • Validation and adoption: SMEs participated in dataset and model design, GEO-Bench benchmarking, and implementation and assessment of diverse downstream applications.The evaluation spans disaster response, land cover and crop mapping, and ecosystem dynamics monitoring.
  • Validation and adoption: TerraTorch integrates Prithvi-EO-2.0 and its data loaders to simplify customization of geospatial foundation models for downstream applications.The toolkit is powered by PyTorch Lightning and TorchGeo.
  • Related work: The model extends prior work by combining spatial representation with temporal modeling over more than 4M samples, exceeding the temporal coverage of earlier models.Prior approaches either use temporal frames as augmentation, operate on pixel time series, or rely on small and geographically restricted datasets.

B. Model architecture and pretraining

Prithvi-EO-2.0 adapts masked autoencoding for satellite image sequences by replacing 2D representations with 3D spatiotemporal embeddings and adding optional time and location metadata. The architecture is trained in 300M- and 600M-parameter variants, with metadata-aware and metadata-free versions.

  • Pretraining framework: Prithvi-EO-2.0 uses masked autoencoding with an asymmetric Vision Transformer encoder-decoder that reconstructs randomly masked image patches.The encoder processes visible tokens, while the decoder predicts masked tokens using mean squared error in pixel space.
  • Spatiotemporal embeddings: The model replaces 2D patch and positional embeddings with 3D versions to represent temporal, height, and width dimensions.A 3D convolution divides the input into non-overlapping cubes, with t = 1 because satellite acquisitions occur at irregular, relatively low frequency.
  • Spatiotemporal embeddings: 3D positional encodings combine separate 1D sine/cosine encodings for time, height, and width.The encoding uses position, dimension index, and embedding dimension size as defined in the accompanying formulation.
  • Metadata integration: Optional metadata encodes each sample’s latitude, longitude, year, and day-of-year, adding learned time and location biases to embedded tokens.Randomly dropping metadata during pretraining helps the model handle cases where temporal or geolocation information is unavailable.
  • Model variants: The model is available in 300M- and 600M-parameter variants, each trained with or without temporal and location information.The variants use ViT-L and ViT-H backbones, respectively, and were trained for 400 epochs.

C. Evaluation

The evaluation combines standardized benchmarking against published competitors with SME-led testing on real-world downstream applications. Classification and segmentation are assessed using task-appropriate metrics, including accuracy, mIoU, precision, recall, and F1-score.

  • Foundation-model usefulness is evaluated through standardized competitor benchmarking and comparisons with state-of-the-art methods on real-world downstream tasks.
  • GEO-Bench provides the standardized framework for benchmarking Prithvi-EO-2.0 against published Earth observation foundation models.
  • SMEs assess the models across disaster response, land cover and crop mapping, and ecosystem dynamics applications.
  • Classification tasks use accuracy, while segmentation tasks use mean Intersection over Union to quantify overlap between predicted and ground-truth regions.
  • Some tasks additionally report precision, recall, and F1-score.
  • All fine-tuning experiments require a single GPU.

1) Benchmarking:

The benchmarking protocol uses GEO-Bench to compare geospatial foundation models across classification and segmentation datasets, while also describing disaster-response task setups and evaluation metrics. Comparisons use controlled tuning and repeated experiments across diverse sensors, resolutions, and application datasets.

  • 1) Benchmarking:: GEO-Bench contains six classification and six semantic-segmentation datasets spanning varied spatial resolutions and domains.
  • 1) Benchmarking:: Ten hyperparameter trials and ten repeated experiments per task support fairer, reproducible model comparisons.
  • 1) Benchmarking:: Prithvi-EO-2.0 is compared with six recent optical Earth observation foundation models and the earlier US-trained Prithvi model.
  • 1) Benchmarking:: Aggregated benchmark scores average mIoU for segmentation, micro-averaged accuracy for classification, and micro-averaged F1 for m-big-earthnet.
  • 2) Disaster response:: Disaster-response evaluations cover flood mapping, wildfire scar mapping, burn-intensity mapping, and landslide detection.
  • 2) Disaster response:: The burn-intensity dataset contains 5,692 paired 224 × 224 images across pre-burn, during-burn, and postburn stages.
  • 2) Disaster response:: Burn-intensity fine-tuning uses a Prithvi encoder with an UperNet decoder, LoRA, and weighted cross-entropy, with U-Net baselines.

3) Land cover and crop mapping:

Land-cover and crop mapping evaluations test Prithvi-EO models on multi-temporal datasets with different spatial, spectral, and temporal characteristics. The experiments compare against conventional, temporal, and other foundation-model baselines under standardized fine-tuning procedures.

  • a) Multi-Temporal Crop Segmentation in the United States:: The US crop-segmentation task compares newer Prithvi-EO models with the earlier model and a U-Net baseline.
  • a) Multi-Temporal Crop Segmentation in the United States:: The US crop dataset uses thirteen segmentation classes and reports mIoU and aggregated accuracy.
  • b) Multi-Temporal Land Cover and Crop Classification in Europe:: Sen4Map contains 335,125 Sentinel-2 time series, and the evaluation feeds models sequences of 12 monthly composites at 10 m resolution.
  • b) Multi-Temporal Land Cover and Crop Classification in Europe:: Sen4Map inputs are cropped to 15 × 15 regions and upscaled to 224 × 224 pixels to preserve the model's patch-processing setup.
  • b) Multi-Temporal Land Cover and Crop Classification in Europe:: Sen4Map differs from Prithvi pretraining in spatial scale, image context, and spectral bands, potentially complicating fine-tuning.
  • c) Multi-Temporal Crop Segmentation with PASTIS:: PASTIS provides 2,433 irregularly sampled Sentinel-2 sequences from four French regions for segmentation of 18 crop types plus background.
  • c) Multi-Temporal Crop Segmentation with PASTIS:: PASTIS comparisons include PASTIS, U-TAE, Satlas, DOFA, and Presto, using identical fine-tuning procedures and mIoU evaluation.

4) Ecosystem dynamics:

The ecosystem-dynamics applications evaluate Prithvi-EO-2.0 for above-ground biomass and gross primary productivity estimation using multimodal, multi-temporal Earth observation data. The workflows combine pretrained Prithvi representations with task-specific inputs and fine-tuning configurations.

  • Above Ground Biomass Estimation: BioMassters provides yearly above-ground biomass targets for Finnish forests from 2016–2022 using 11,462 reference images paired with 12 months of Sentinel-1 and Sentinel-2 imagery.AGB measurements were derived from LiDAR data calibrated with in-situ measurements.
  • Above Ground Biomass Estimation: Five input configurations vary Sentinel-2 bands, Sentinel-1 inclusion, and four versus 12 temporal frames for biomass estimation.The configurations include six- or 11-band Sentinel-2 inputs, optional Sentinel-1 data, and four or 12 frames.
  • Above Ground Biomass Estimation: The highest-performing biomass configuration was selected for subset training on 50%, 20%, 10%, and 5% of the original training set.The four timestamps were selected from 12 Sentinel-2 observations using data-quality priorities.
  • Estimation of Gross Primary Productivity: GPP estimation integrates daily measurements from 37 globally distributed eddy covariance flux towers with HLS reflectance and MERRA-2 environmental variables.The GPP data were obtained from FLUXNET, AmeriFlux, and ICOS networks and processed through the ONEFLUX pipeline.
  • Estimation of Gross Primary Productivity: The GPP model freezes the Prithvi encoder for six-band HLS inputs, processes ten MERRA-2 features convolutionally, and combines both branches to predict GPP.The resulting representations are concatenated before a linear prediction layer.

IV. RESULTS AND DISCUSSION

The evaluation section presents benchmarking results first, followed by three categories of downstream tasks.

  • The evaluation reports benchmarking results before discussing three categories of downstream tasks.

A. Benchmarking

Across GEO-Bench, Prithvi-EO-2.0 achieves strong aggregate and task-level performance, with the larger models leading overall and multi-temporal variants performing especially well. The comparisons also indicate benefits from global pretraining, temporal and location embeddings, and transfer to high-resolution tasks.

  • Aggregate GEO-Bench Performance: Prithvi-EO-2.0-600M-TL and Prithvi-EO-2.0-600M achieve the best combined performance across all 12 GEO-Bench datasets.They are also the top-performing models for segmentation, while DOFA and the two 600M variants lead classification.
  • Task-Level Performance: Prithvi-EO-2.0-600M-TL and Prithvi-EO-2.0-600M outperform other models in four of six medium-resolution tasks.These tasks use Sentinel-2 imagery at 10 m resolution.
  • Task-Level Performance: Accuracies reach ≥97% on some datasets, including m-brik-klin and m-pv4ger, where multiple models perform equally well.
  • Evaluation Considerations: The m-cashew-plant aggregate score varies because the test set lacks class 0, and predicting that absent class lowers average IoU.
  • Model and Pretraining Effects: Prithvi-EO-2.0-100M improves the overall GEO-Bench score by 3% over the architecturally identical Prithvi-EO-1.0-100M.The newer model was pretrained on a larger global dataset.
  • Model and Pretraining Effects: Models with temporal and location embeddings achieve higher overall performance than their non-TL counterparts, while Prithvi-EO-2.0 also performs well on high-resolution tasks despite 30 m pretraining.The reported high-resolution applications include tree crown identification and cattle tracking using drone imagery.

B. Disaster response

In disaster-response applications, Prithvi-EO-2.0 generally improves over earlier models and U-Net baselines across flood, wildfire, burn-intensity, and landslide-related evaluations. Performance advantages are particularly visible for target-class IoU and under limited training data, although burn-severity discrimination remains difficult.

  • Flood Mapping: Prithvi-EO-2.0-600M-TL achieves a water-class IoU 3.5 points higher than Prithvi-EO-1.0-100M on Sen1Floods11.Average-metric gains are limited by the dataset’s predominant and easily identifiable land class.
  • Wildfire Scar Mapping: The wildfire-scar IoU gap between Prithvi-EO-1.0 and Prithvi-EO-2.0-600M-TL is 5.6 points.The newer model versions show the same improvement trend as in flood detection.
  • Burn Intensity Mapping: Prithvi-EO-2.0 models outperform U-Net variants on burn-intensity mIoU, with Prithvi-EO-2.0-600M reaching 75.9% IoU for class 0.All models struggle to distinguish among the different burn-severity classes.
  • Landslide Detection: Prithvi-EO-2.0-300M trained with Lovasz loss achieves the best overall L4S performance by mIoU and F1 score, while the 600M model has higher recall but lower precision.The 600M model reports 70.4% mIoU and 58.6% F1 score.
  • Limited-Data Evaluation: With only 50 training images, Prithvi-EO-2.0 models show smaller performance drops than U-Net and U-Net++, and Prithvi-EO-2.0-600M performs best overall.U-Net’s mIoU falls from 70.4% to 59.7%, while its F1 score falls from 59.7% to 35.0%.
  • Visual Comparison: Prithvi-EO-2.0-300M produces fewer false positives and broader true-positive coverage than U-Net, whereas the 600M model detects more true positives but generates more false positives on the small subset.

C. Land cover and crop mapping

Prithvi-EO models outperform baselines across land-cover and crop-mapping tasks, with Prithvi-EO-2.0-600M generally strongest. The models also handle longer temporal inputs than used during pretraining.

  • C. Land cover and crop mapping: 50.7% mIoU and 68.8% mAcc were achieved by Prithvi-EO-2.0-600M on United States crop segmentation.All Prithvi-EO models outperformed U-Net, and Prithvi-EO-2.0 improved over Prithvi-EO-1.0.
  • C. Land cover and crop mapping: Prithvi-EO-2.0-600M consistently achieved the highest weighted-averaged F1 scores for European land-cover classification across data-availability scenarios.All Prithvi-EO models outperformed the ViViT baseline and the newer versions surpassed Prithvi-EO-1.0.
  • C. Land cover and crop mapping: Prithvi models outperformed the baseline on crop-type classification, with Prithvi-EO-2.0-300M and 600M both surpassing Prithvi-EO-1.0-100M.The 600M model had marginally better results on two data ratios and slightly lower performance on three others than the 300M model.
  • C. Land cover and crop mapping: Prithvi-EO models pretrained with four frames remained robust on Sen4Map’s 12-frame inputs, indicating they can handle larger time-series inputs.The crop-segmentation examples illustrate performance with both the full training set and a 50-image subset.

3) Multi-Temporal Crop Segmentation with PASTIS:

Prithvi-EO-2.0-600M achieved the strongest PASTIS crop-segmentation performance, while comparisons in biomass estimation show both its strengths and optical-data limitations. The model benefits from longer time series and efficient fine-tuning, but higher biomass remains difficult to estimate.

  • 3) Multi-Temporal Crop Segmentation with PASTIS:: Prithvi-EO-2.0-600M achieved the highest mIoU on PASTIS with both 100% and 10% of the training data.All models experienced a significant performance drop when trained on approximately 200 images.
  • 3) Multi-Temporal Crop Segmentation with PASTIS:: Prithvi-EO-2.0 learned long sequences better than other GFMs on PASTIS, capturing temporal patterns through its pretraining data and architecture.DOFA and Satlas had lower scores, while Presto performed worst, which the authors attribute partly to its input image size.
  • 1) Above Ground Biomass Estimation:: The best BioMassters performance used 12 Sentinel-2 timestamps and LoRA, while fine-tuning on 5% of training data differed by 10.36% from the full-data result.The full training set contained 6,951 images, compared with 348 images for the 5% subset.
  • 1) Above Ground Biomass Estimation:: Both biomass models underestimated AGB above 400, while the baseline predicted high AGB values more accurately and with greater granularity than fine-tuned Prithvi.The baseline had a tighter distribution around the identity line and performed better in the 300–400 range.
  • 1) Above Ground Biomass Estimation:: Prithvi approached multimodal-model performance using multispectral imagery alone and benefited from longer time series, while direct SAR integration did not improve biomass estimation.The authors suggest fusing a separate SAR encoder with Prithvi’s outputs as an alternative multimodal strategy.

2) Estimation of Gross Primary Productivity (GPP) at Globally Distributed Sites:

Prithvi-EO-2.0 improves GPP estimation across globally distributed flux-tower sites by exploiting spatial and temporal information in HLS and MERRA-2 data. It outperforms conventional baselines and ResNet in the reported evaluations.

  • 2) Estimation of Gross Primary Productivity (GPP) at Globally Distributed Sites:: Up to 20% improvement in R2 over random forest and XGBoost without vegetation indices was achieved by Prithvi-EO-2.0 on GPP estimation.Prithvi-EO-2.0-600M-TL achieved the highest scores across all testing years without using vegetation indices as inputs.
  • 2) Estimation of Gross Primary Productivity (GPP) at Globally Distributed Sites:: The evaluation used leave-one-year-out cross-validation over 975 samples from 37 globally distributed flux towers.Table XV reports R2 values for the tested models and identifies the held-out test years.
  • 2) Estimation of Gross Primary Productivity (GPP) at Globally Distributed Sites:: Prithvi-EO’s spatial HLS information improves representation within flux-measurement footprints relative to spatially averaged baseline inputs.The baselines use 1.5 × 1.5 km2 spatial averages around tower locations.
  • 2) Estimation of Gross Primary Productivity (GPP) at Globally Distributed Sites:: Prithvi-EO-2.0-600M-TL achieved an average R2 of 0.81, compared with 0.75 for ResNet.The authors relate the advantage to generalizable spatial context from massive pretraining together with temporal and location embeddings.

APPENDIX A GEO-BENCH DETAILED RESULTS

The appendix provides detailed GEO-Bench results for classification and segmentation datasets, using repeated evaluations and reporting multiple summary statistics. The tables cover six classification and six segmentation datasets across varied domains and resolutions.

  • APPENDIX A GEO-BENCH DETAILED RESULTS: GEO-Bench results report mean, standard deviation, maximum, and minimum test-set metrics over ten repeated runs using each model’s best hyperparameters.These results correspond to the boxplots shown in Figure 7.
  • APPENDIX A GEO-BENCH DETAILED RESULTS: The detailed classification tables cover m-bigearthnet, m-brick-kiln, m-eurosat, m-forestnet, m-pv4ger, and m-so2sat.Tables A1 and A2 organize the classification results by dataset group.
  • APPENDIX A GEO-BENCH DETAILED RESULTS: The detailed segmentation tables cover m-NeonTree, m-SA-crop-type, m-cashew-plant, m-chesapeake, m-nz-cattle, and m-pv4ger-seg.Tables A3 and A4 organize the segmentation results by dataset group.
Loading 2412.02732v3…