Source-linked AI summary
Foundation Models for Generalist Geospatial Artificial Intelligence
Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurung, Sam Khallaghi, Hanxi, Li, Michael Cecil, Maryam Ahmadi, Fatemeh Kordi, Hamed Alemohammad, Manil Maskey, Raghu Ganti, Kommy Weldemariam, Rahul Ramachandran
TL;DR
Earth observation AI often depends on costly labeled data and task-specific models with limited spatial-temporal generalization. The paper introduces a scalable framework and uses it to pretrain and fine-tune the Prithvi geospatial foundation model across multiple tasks. Prithvi shows strong cross-task performance and data efficiency, while its pretraining remains limited by U.S.-focused sampling and weaker seasonal-change performance.
Problem
Earth observation models commonly require expensive labeled data and task-specific approaches, motivating foundation models that can generalize across geoscience tasks with fewer labels.
Method
The paper develops a scalable framework for self-supervised pretraining and downstream fine-tuning, using a masked-autoencoder transformer to create Prithvi from multispectral HLS imagery.
Results
Prithvi performs strongly across segmentation and generation tasks, with low sensitivity to the number of labeled images used for fine-tuning.
Takeaways & Limitations
Prithvi is accurate, fast to fine-tune, data-efficient, and generalizes across resolutions and global georegions using few labeled fine-tuning examples.
Takeaways & Limitations
Pretraining focused on the USA, one year of data, and a three-step time interval, and the model did not significantly improve seasonal-change handling.
Abstract
from arXiv · showhide
Significant progress in the development of highly adaptable and reusable Artificial Intelligence (AI) models is expected to have a significant impact on Earth science and remote sensing. Foundation models are pre-trained on large unlabeled datasets through self-supervision, and then fine-tuned for various downstream tasks with small labeled datasets. This paper introduces a first-of-a-kind framework for the efficient pre-training and fine-tuning of foundational models on extensive geospatial data. We have utilized this framework to create Prithvi, a transformer-based geospatial foundational model pre-trained on more than 1TB of multispectral satellite imagery from the Harmonized Landsat-Sentinel 2 (HLS) dataset. Our study demonstrates the efficacy of our framework in successfully fine-tuning Prithvi to a range of Earth observation tasks that have not been tackled by previous work on foundation models involving multi-temporal cloud gap imputation, flood mapping, wildfire scar segmentation, and multi-temporal crop segmentation. Our experiments show that the pre-trained model accelerates the fine-tuning process compared to leveraging randomly initialized weights. In addition, pre-trained Prithvi compares well against the state-of-the-art, e.g., outperforming a conditional GAN model in multi-temporal cloud imputation by up to 5pp (or 5.7%) in the structural similarity index. Finally, due to the limited availability of labeled data in the field of Earth observation, we gradually reduce the quantity of available labeled data for refining the model to evaluate data efficiency and demonstrate that data can be decreased significantly without affecting the model's accuracy. The pre-trained 100 million parameter model and corresponding fine-tuning workflows have been released publicly as open source contributions to the global Earth sciences community through Hugging Face.
1 Introduction
The paper addresses costly labeled-data requirements and weak spatial-temporal generalization in geoscience AI by proposing a scalable framework for geospatial foundation models. It uses the framework to train and fine-tune Prithvi across Earth observation applications while investigating pretraining design, efficiency, and data requirements.
- Geoscience and remote sensing models commonly require expensive labeled data and generalize poorly across space and time.
- The paper proposes a first-of-its-kind framework spanning geospatial data processing, self-supervised pretraining, downstream fine-tuning, and inference.
- Prithvi is trained from multispectral Harmonized Landsat Sentinel-2 measurements and fine-tuned for cloud imputation, flood mapping, fire-scar segmentation, and crop segmentation.
- The study asks how to design and evaluate geoscience foundation models, efficiently pretrain noisy redundant remote-sensing data, and generalize with fewer labels.
- The model architecture, pretraining weights, and inference service are shared openly through Hugging Face.
2 Background
The background frames foundation models as self-supervised generalist models adapted to downstream tasks, while highlighting the region-specific, label-intensive nature of existing Earth observation approaches. Prior remote-sensing foundation models also have limitations in data modality and task coverage.
- Foundation models pretrain on large unlabeled datasets through self-supervision and then fine-tune for different downstream tasks.
- Existing remote-sensing foundation models have focused on aerial benchmark imagery and visible red, green, and blue bands.
- Supervised models remain common for fire-scar identification, flood mapping, and crop identification, providing baselines for comparison with Prithvi.
- Earlier crop-segmentation studies report labeled-data-dependent performance across kappa, mean IoU, F1, and overall accuracy measures.
- Earth observation methods often require ground inspection and local knowledge, remain region-specific, and are difficult to scale without substantial cost.
3 Data for Pretraining
The pretraining data pipeline combines HLS satellite observations with representative stratified sampling and offline quality filtering. It addresses cloud contamination, missing values, and data-loading overhead through indexed subregions and storage workflows.
- 3.1 Harmonized Landsat Sentinel-2 Dataset: HLS provides harmonized Landsat and Sentinel-2 surface-reflectance observations at approximately two- to three-day intervals.
- 3.1 Harmonized Landsat Sentinel-2 Dataset: The HLS archive was selected for its 3.61 Petabyte scale, coverage reaching back to 2015, and broad user base.
- 3.2 Efficient Data Sampling: The sampling method groups low-resolution tiles by aggregate geospatial statistics and samples uniformly across groups to improve representativeness.
- 3.2 Efficient Data Sampling: Temperature and precipitation statistics divide the contiguous United States into 20 regions for stratified sampling.
- 3.3 Preprocessing Routines: Offline preprocessing computes cloud and missing-value percentages in subregions and stores indices for selected tile, timestamp, and coordinate locations.
- 3.4 Data Loading: Zarr files store selected subregions to address the overhead of opening many GeoTiff files during training.
4 Model Architecture and Pretraining
Prithvi uses a masked autoencoder with a Vision Transformer backbone, adapted to multi-temporal and multispectral satellite inputs. The pretraining task reconstructs masked tokens from visible patches using mean squared error.
- Prithvi pretraining uses an asymmetric masked autoencoder with a Vision Transformer backbone and focuses on the open-sourced 100M-parameter version.
- The encoder processes unmasked patches, while the decoder reconstructs masked tokens from encoded visible patches and mask tokens.
- The reconstruction objective is mean squared error between masked input tokens and predicted tokens in pixel space.
- Spatiotemporal design: The architecture is modified for temporal and multispectral imagery using three-dimensional positional and patch embeddings.
- Training configuration: Pretraining uses six HLS bands, 224 × 224 inputs, and patches sized 1 × 16 × 16 across time and spatial dimensions.
- Data loading: Zarr-based data loading is approximately two times faster than corresponding GeoTiff loading.
5 Downstream Tasks
Prithvi is applied to several Earth observation tasks using task-specific fine-tuning, including temporal imagery reconstruction and segmentation. The workflow reuses pretrained encoder weights and adds lightweight decoder components tailored to each task.
- Prithvi is evaluated on multi-temporal cloud gap imputation, flood mapping, wildfire scar segmentation, and multi-temporal crop segmentation.
- Multi-Temporal Cloud Gap Imputation: Cloud-gap imputation uses 18-channel inputs formed from six spectral bands across three temporal snapshots.Each image chip covers a 224 x 224 pixel region at 30-meter spatial resolution.
- Multi-Temporal Cloud Gap Imputation: The cloud-imputation dataset contains 7,852 image chips from the contiguous United States after cloud-coverage and missing-value screening.Scenes span 2022, with temporal gaps ranging from 1 to 200 days.
- Fine-tuning reuses or updates pretrained encoder weights while learning decoder weights for each downstream task.The customized pipeline supports spatiotemporal data and segmentation, regression, and classification tasks.
- The task-specific decoder uses a lightweight neck with four ConvTranspose2D layers followed by one two-dimensional convolutional layer.Segmentation is evaluated using mean IoU, mean F1-score, and mean accuracy.
6 Results
Prithvi was evaluated across pretraining and four downstream Earth-observation tasks, including cloud gap imputation, flood mapping, wildfire scar segmentation, and crop segmentation. Results show faster convergence, competitive or superior performance, and substantial labeled-data efficiency.
- 6.1 Pretraining Results: Prithvi’s masked-pixel reconstruction pretraining converged with training MSE 0.0283 and validation loss 0.0364.Reconstructions successfully captured RGB and infrared bands across time and masking ratios.
- 6.2.1 Multi-Temporal Cloud Gap Imputation: Prithvi achieved SSIM above 0.9 after only a few fine-tuning epochs for multi-temporal cloud gap imputation.It consistently outperformed CGAN in Mean Absolute Error and SSIM across all training subsets.
- 6.2.1 Multi-Temporal Cloud Gap Imputation: With only 400 fine-tuning samples, Prithvi exceeded CGAN trained on 6,231 samples by 1.2% in SSIM.The comparison used matched validation data and training subsets ranging from 400 to 6,231 image chips.
- 6.2.2 Flood Mapping: Pretraining reduced flood-mapping convergence time from 55 to 25 epochs for reaching the reference performance, exceeding a factor-of-two acceleration.The pretrained model also generalized from 30m HLS pretraining data to 10m Sentinel-2 fine-tuning data and global flood events.
- 6.2.2 Flood Mapping: Prithvi preserved flood-mapping performance with half the labeled images, from 252 to 126, while further reductions still reached average IoU above 80%.The 126-image setting represented approximately 11 labeled images per geographic region across eleven regions.
- 6.2.3 Wildfire Scar Mapping: Prithvi achieved wildfire-scar IoU 73.62, surpassing U-Net by 2.61pp and a ViT baseline by 4.58pp.Pretrained weights also produced the fastest convergence and remained effective with substantially reduced labeled-data quantities.
- 6.2.4 Multi-Temporal Crop Segmentation: For crop segmentation, Prithvi exceeded U-Net in mean accuracy, 64.06% versus 61.91%, and mean IoU, 0.426 versus 0.420.Prithvi performed better for several crop classes, including Corn, Winter Wheat, Alfalfa, Fallow/Idle Cropland, and Cotton.
7 Discussion
The paper presents a data-preparation and self-supervised pre-training approach for Prithvi, while showing strong downstream performance and data efficiency alongside defined scope limitations. The model generalizes broadly, but its U.S.-focused, one-year pre-training setup leaves room for improvement on seasonal and task-specific performance.
- Data preparation and pre-training: The pipeline collates and stratifies HLS data for uniform sampling, balancing data quality and distribution before MAE-style self-supervised pre-training.The mechanism masks 75% of the data and trains the model to reconstruct the masked information.
- Downstream performance: A compact 100 million-parameter Prithvi model performs strongly across segmentation and generation tasks and is relatively insensitive to the number of labeled fine-tuning images.The reported results compare Prithvi with state-of-the-art models across several downstream task categories.
- Relationship to other models: Prithvi complements Presto: Prithvi targets surface-dependent remote-sensing predictions, whereas Presto uses longer pixel time series for classification and regression.Presto has fewer than 1 million parameters, making it more than 100 times smaller than Prithvi-100M.
- Limitations and future directions: Pre-training assumes U.S. coverage, a time step of 3, and one year of data, and the model does not significantly improve performance on seasonal changes.The paper expects global HLS pre-training could further increase performance.
- Limitations and future directions: Although Prithvi generalizes strongly to global data during fine-tuning, it did not surpass state-of-the-art models on some downstream tasks.The authors propose architectural advances and a larger globally pretrained model as future directions.
8 Conclusion
The conclusion characterizes Prithvi as an accurate, fast-to-fine-tune, and data-efficient geospatial foundation model that generalizes across resolutions and regions with few labeled examples. The authors open-source the framework and associated model resources to accelerate further geoscience and remote-sensing research.
- Conclusion: Prithvi is accurate, fast in fine-tuning, and data-efficient after large-scale self-supervised pre-training on HLS data.The conclusion presents these properties as experimental findings.
- Conclusion: Prithvi generalizes across different resolutions and global geographic regions using few labeled data during fine-tuning.This conclusion concerns the model's demonstrated generalization scope.
- Open-source contribution: The authors open-source the code, architecture, pretrained weights, fine-tuning workflows, and downstream-application demos.These resources are intended to accelerate AI work in geoscience and remote sensing.