Foundation Models for Generalist Geospatial Artificial Intelligence

1 Introduction

• Geoscience and remote-sensing models often require expensive labeled data and generalize poorly across space and time, driving interest in self-supervised foundation models.These models pre-train on large unlabeled datasets and can then be fine-tuned for varied downstream tasks with smaller labeled datasets.Source: §1 · p. 2
• The paper proposes a scalable framework for training, fine-tuning, and inference, and uses it to train Prithvi on HLS multispectral satellite measurements for four downstream applications.The framework connects to geospatial data sources through discovery operators and covers data sampling, filtering, and preprocessing, pretraining, fine-tuning, and inference.Source: §1 · p. 2, Figure 1 · p. 3
• The introduction asks how foundation models should be designed and evaluated for geoscience.This is presented as the paper’s first research question.Source: §1 · p. 2
• The paper asks how to efficiently pre-train on large remote-sensing datasets while removing noise and avoiding redundant information.The question notes that data may repeat information across ecosystems and landscapes and may contain acquisition noise from cloud occlusion or sensor malfunction.Source: §1 · p. 2
• The paper asks whether foundation models can generalize across geoscience domains while requiring substantially less labeled data to reach benchmark performance.This is framed as the third research question.Source: §1 · p. 2
• The paper makes its model architecture, pretraining weights, and inference service openly available on Hugging Face to support AI for geoscience.The authors present open-source access as a way to share their insights and support the field.Source: §1 · p. 2

Background

• U-Net uses feature-extraction and upsampling legs that fuse adjacent-layer outputs, and has been applied to segmentation with visible and hyperspectral imagery.The cited examples include building and waterbody segmentation.Source: §2 · p. 3
• MAE pre-trains by masking image patches and reconstructing their pixel values, using a Vision Transformer encoder that processes only visible patches and MSE to measure reconstruction error.Its decoder uses encoded visible patches and mask tokens to reconstruct masked patches; the approach avoids specialized pre-training operations.Source: §2 · pp. 3–4

Related work

• Existing remote-sensing foundation models focus on RGB aerial imagery and object detection, whereas Presto is a smaller, complementary approach for longer satellite time series.The paper argues that these approaches do not fully address satellite imagery’s geospatial, multispectral, temporal, and spatial-patch properties; Presto has fewer than 1 million parameters, approximately 100 times fewer than Prithvi.Source: §2 · p. 4
• Prior supervised studies cover fire-scar identification, flood mapping, and crop segmentation, while task-specific and region-dependent methods and limited ground truth constrain scalability.The paper positions these studies as baselines for assessing whether Prithvi can adapt across the tasks with fewer labeled data points.Source: §2 · p. 5

3.1 Harmonized Landsat Sentinel-2 Dataset

• The HLS project combines Landsat 8/9 and Sentinel-2 data into surface-reflectance observations roughly every two to three days; training uses six shared spectral channels.The selected channels are Sentinel bands 2, 3, 4, 8A, 11, and 12.Source: §3.1 · pp. 5–6

3.2 Efficient Data Sampling

• To avoid overrepresenting common landscapes, the method groups tiles by low-resolution geospatial statistics and samples uniformly; its example divides the contiguous U.S. into 20 temperature-precipitation regions.The example uses 42 years of PRISM data to define the regions.Source: §3.2 · pp. 6–7, Figure 2 · p. 6

3.3 Preprocessing Routines

• Cloud masks identify valid, cloud-free areas; offline preprocessing partitions each HLS tile into training-sized windows, filters by cloud and missing-pixel thresholds, and records retained coordinates.The index file records the tile name, timestamp, and top-left pixel coordinates for each retained sub-image.Source: §3.3 · p. 7

3.3.1 Zarr preprocessing

• A batch of 128 samples with three timesteps and six bands requires opening 2304 GeoTIFFs, so the authors stored selected sub-regions in concurrently readable Zarr files for pretraining.A cloud-native, multiprocess method generates the Zarr training data.Source: §3.3.1 · pp. 7–8

4 Model Architecture and Pretraining

• Prithvi pretraining uses a masked autoencoder that reconstructs randomly masked patches with an asymmetric ViT encoder-decoder and masked-token pixel-space MSE loss.The paper focuses on the open-sourced 100M-parameter version; the encoder processes only unmasked patches, while the decoder receives latent and masked tokens.Source: §4 · p. 8

4.1 Spatiotemporal Considerations

• The model adapts ViT to spatiotemporal inputs with 3D sine-cosine positional encodings and 3D-convolutional embeddings of non-overlapping data cubes.The temporal tubelet size is 1 to accommodate the relatively low-frequency satellite observations.Source: §4 · p. 8, §4.1 · p. 9

4.2 Pretraining

• Using one-fourth the workers, Zarr loading reduces epoch time by over 40% on 8 GPUs and is comparable to GeoTiff loading on 64 GPUs.The authors report that filtered-sample storage reduces data moved to the training environment; Table 1 compares average epoch times for preprocessing and loading.Source: §4.2 · p. 9, Table 1 · p. 9
• Pretraining uses six HLSL30 bands, 224 × 224 inputs, 1 × 16 × 16 patches, and ViT-base or ViT-large backbones trained for 1000 epochs.Runs used AdamW, batch size 1024, a one-cycle cosine scheduler with maximum learning rate 5e-4, and up to 64 NVIDIA A100 GPUs; data were loaded from Zarr with PyTorch DataLoader, xarray, and dask.Source: §4.2 · p. 9

5 Downstream Tasks

• The downstream suite covers multi-temporal cloud-gap imputation, flood mapping, wildfire-scar segmentation, and multi-temporal crop segmentation.The paper introduces the task data and modeling used to apply the pretrained MAE model.Source: §5 · p. 10

5.1 Datasets

• Cloud-gap imputation uses 7,852 HLS chips across the contiguous United States, each with three 224 × 224 snapshots, 18 bands, and less than 5% cloud cover per scene.The scenes span time differences of 1–200 days and were checked for missing values.Source: §5.1 · p. 10
• Flood mapping uses Sen1Floods11: 4,831 512 × 512 chips covering 120,406 km² across 14 biomes, 357 ecoregions, six continents, and 11 flood events.The benchmark includes fully convolutional network results under different input and labeled-data setups, as well as radar-backscatter thresholding.Source: §5.1 · p. 10
• Wildfire-scar mapping uses 805 HLS scenes from 2018–2021, selected one to three months after fires and screened for cloud cover, tile coverage, and visible scars.Each 512 × 512 scene is centered on the scar and paired with a GeoTIFF mask derived from fire shapefiles.Source: §5.1 · p. 10
• Crop segmentation uses 3,854 CONUS chips with 18 bands from early-, mid-, and late-season 2022 scenes, aligned to USDA Crop Data Layer labels.Chips are 224 × 224 pixels at 30-meter resolution; quality checks discard clouds, shadows, cloud proximity, and missing values.Source: §5.1 · pp. 10–11

5.2 Downstream Modeling Considerations

• The pretrained masked-autoencoder encoder is reused or fine-tuned with task-specific decoder weights, using an enhanced mmsegmentation-based PyTorch module.The library is extended for spatiotemporal inputs and customized to support tasks beyond semantic segmentation, including regression and classification.Source: §5.2 · pp. 11–12
• Segmentation uses a lightweight decoder with four ConvTranspose2D layers and a final 2D convolution; cloud imputation fine-tunes the encoder without a decoder head.Segmentation is evaluated with mean IoU, mean F1-score, and mean accuracy; imputation uses RMSE, MAE, and SSIM. Training settings vary by task: flood and crop use weighted cross-entropy, while wildfire uses unweighted dice loss.Source: §5.2 · p. 12

6.1 Pretraining Results

• Pretraining reconstruction reached a minimum training MSE loss of 0.0283 and validation loss of 0.0364.The model reconstructed RGB and infrared bands across time and masking ratios; reconstruction declined as masking increased, but remained accurate around visible pixels even with 90% masked.Source: §6.1 · pp. 12–13, Figure 5 · p. 13

6.2.1 Multi-Temporal Cloud Gap Imputation

• 1.2% higher SSIM than CGAN fine-tuned on all 6,231 samples was achieved by Prithvi fine-tuned on just 400 samples.Prithvi also exceeded CGAN in MAE and SSIM across all tested training subsets and reached SSIM above 0.9 after a few epochs.Source: §6.2.1 · p. 14, Figure 7 · p. 15

6.2.2 Flood Mapping

• Pretraining cut the flood-mapping fine-tuning time to reach the ViT-base reference performance from 55 epochs with random weights to 25 epochs on average.This transfer was from 30m HLS pretraining to 10m Sentinel-2 fine-tuning on global flood events.Source: §6.2.2 · p. 16, Figure 9 · p. 16
• Prithvi reduced flood-mapping labels from 252 images to 126 with similar performance, and still averaged over 80% IoU after a nearly 90% reduction.The 126-image subset represented about 11 labeled images per geographic region across eleven regions.Source: §6.2.2 · p. 16, Figure 9 · p. 16
• 82.99 IoU on the water class was Prithvi’s top flood-mapping score after 500 epochs; performance of pretrained and nonpretrained Prithvi converged after extended fine-tuning.At 50 epochs, water-class IoU was 81.26. The comparison includes the Sen1Floods11 baseline and off-the-shelf vision transformers.Source: §6.2.2 · pp. 16–17, Table 2 · p. 17

6.2.3 Wildfire Scar Mapping

• Prithvi maintained comparable wildfire-scar mapping performance with 135 labeled images versus 540, while pretrained weights converged fastest.Leaving encoder weights unupdated lowered performance; 135 images are one quarter of the original 540-image set.Source: §6.2.3 · p. 17
• 73.62 IoU on the wildfire-scar class surpassed U-Net by 2.61pp and ViT by 4.58pp, with HLS pretraining improving class IoU by up to 1.36pp.The pretrained model also surpassed the randomly initialized Prithvi and the reported U-Net- and Transformer-based baselines.Source: §6.2.3 · p. 17, Table 3 · p. 18

6.2.4 Multi-Temporal Crop Segmentation

• 64.06% mean accuracy for Prithvi versus 61.91% for U-Net accompanied mean IoU scores of 0.426 and 0.420, respectively, with no significant IoU difference.Prithvi had higher accuracy in Corn, Winter Wheat, Alfalfa, Fallow/Idle Cropland, and Cotton, and higher IoU in Corn and Soybeans.Source: §6.2.4 · p. 19, Table 4 · p. 19

7 Discussion

• The framework stratifies HLS data for uniform sampling and uses MAE self-supervision with 75% of data masked for reconstruction.Its pipeline is designed to address data redundancy and imbalance while supporting sampling, data balance, and quality control.Source: §7 · p. 19
• The 100-million-parameter model performs strongly across segmentation and generation tasks, compares with state-of-the-art models, and has low sensitivity to fine-tuning label quantity.The discussion also notes that fine-tuning and inference require notably less data than other deep-learning algorithms.Source: §7 · p. 20
• On Sen1Floods11, [63] found Prithvi 2.5% IoU below dedicated U-Net on known-region water mapping but 8.6% IoU above it on unseen regions.The discussion presents this unseen-region result as evidence of Prithvi’s generalization in flood mapping, while noting room to improve performance on data from the same distribution.Source: §7 · p. 20
• Prithvi’s U.S.-focused, one-year, three-step pretraining did not significantly improve seasonal-change handling, while Presto [21] complements it with longer pixel time series and lightweight computation.Presto has fewer than 1 million parameters—over 100 times smaller than Prithvi-100M—and is particularly strong in classification and regression; Prithvi focuses on remote-sensing predictions tied to surfaces in image subregions.Source: §7 · p. 20
• Although Prithvi generalizes to global data during fine-tuning, the authors expect global HLS pretraining to improve performance and acknowledge that some tasks do not surpass state of the art.They open-sourced the framework and model to encourage further development; future directions include larger global models, 3D Swin backbones, additional bands, and high-resolution imagery below 10 m.Source: §7 · p. 20

• Foundation models should not be expected to outperform state-of-the-art models across all application categories, but they excel where labeled data is limited.Source: §8 · p. 21
• Prithvi is accurate, fast to fine-tune, and data-efficient, and it generalizes across resolutions and global regions with few labeled fine-tuning examples.These results follow large-scale self-supervised pretraining on HLS data.Source: §8 · p. 21
• The framework supports development of large-scale geospatial foundation models, and the authors open-sourced Prithvi's code, architecture, pretrained weights, fine-tuning workflows, and downstream-task demos.Source: §8 · p. 21