Source-linked AI summary
SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding
Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, Aniruddha Kembhavi
TL;DR
Remote sensing automation needs a broad, large-scale dataset because applications span many categories, feature sizes, and sensing conditions, while existing benchmarks are limited and fragmented. SatlasPretrain combines Sentinel-2 and NAIP imagery with 302M labels across 137 categories and seven label types, and evaluates baseline and unified models. Pre-training improves average downstream accuracy by 18% over ImageNet and 6% over the next best baseline, while results indicate substantial room to improve time-series and long-range-context processing.
Problem
Existing remote sensing datasets are small and fragmented, limiting generalizable methods across the field's diverse monitoring tasks.
Method
SatlasPretrain combines Sentinel-2 and NAIP images with 302M labels across 137 categories and seven label types, alongside evaluation of baselines and SATLASNET.
Results
18% over ImageNet and 6% over the next best baseline, pre-training on SatlasPretrain improves average accuracy across seven downstream tasks.
Takeaways & Limitations
SatlasPretrain can improve accuracy on niche remote sensing tasks that require costly expert annotation.
Takeaways & Limitations
The evaluation finds substantial room for improvement in processing remote sensing image time series.
Abstract
from arXiv · showhide
Remote sensing images are useful for a wide variety of planet monitoring applications, from tracking deforestation to tackling illegal fishing. The Earth is extremely diverse -- the amount of potential tasks in remote sensing images is massive, and the sizes of features range from several kilometers to just tens of centimeters. However, creating generalizable computer vision methods is a challenge in part due to the lack of a large-scale dataset that captures these diverse features for many tasks. In this paper, we present SatlasPretrain, a remote sensing dataset that is large in both breadth and scale, combining Sentinel-2 and NAIP images with 302M labels under 137 categories and seven label types. We evaluate eight baselines and a proposed method on SatlasPretrain, and find that there is substantial room for improvement in addressing research challenges specific to remote sensing, including processing image time series that consist of images from very different types of sensors, and taking advantage of long-range spatial context. Moreover, we find that pre-training on SatlasPretrain substantially improves performance on downstream tasks, increasing average accuracy by 18% over ImageNet and 6% over the next best baseline. The dataset, pre-trained model weights, and code are available at https://satlas-pretrain.allen.ai/.
1. Introduction
Remote sensing supports broad, frequent Earth monitoring, but automation is limited by accuracy barriers and scarce labels across a fragmented task landscape. SatlasPretrain addresses this gap with a large, diverse multi-task dataset and reports stronger downstream performance while exposing specialized modeling challenges.
- Remote sensing images support monitoring applications including mapping infrastructure, renewable energy deployment, deforestation, glacier loss, illegal fishing, and natural disasters.
- Accuracy barriers and scarce expert labels keep many remote sensing applications manual or semi-automated.Road extraction remains an accuracy challenge, while niche applications often have few labeled examples.
- Existing remote sensing datasets are small and fragmented across categories and tasks, limiting transfer learning and use of large architectures.The cited datasets contain fewer than 10K images each and typically target individual categories or label types.
- SatlasPretrain combines Sentinel-2 and NAIP imagery with 302M labels spanning 137 categories and seven label types.Its labels include points, polygons, polylines, segmentation, regression, object properties, and patch classification.
- 18% over ImageNet and 6% over the next best baseline, pre-training on SatlasPretrain improves average performance across seven downstream tasks.
- SATLASNET jointly models all seven label types and improves average performance by 7.1% over training separately on each label type.The unified model uses task-specific output heads to learn from every category.
2. Related Work
Earlier remote sensing datasets generally specialize in one label type, a limited category set, or single-image prediction. SatlasPretrain differs by combining broader label diversity with spatio-temporal imagery that supports multi-image learning.
- General-purpose datasets range from land-cover and scene classification to object detection and instance segmentation, but each targets limited category or task scopes.
- Existing datasets typically make predictions for a single label type and usually from a single image.
- These limitations reduce category coverage, label diversity, and opportunities to synthesize features across image time series.
- Domain-specific datasets extend beyond single-task settings, including vessel attributes in SAR imagery and crop segmentation from SAR-optical time series.
- Self-supervised remote sensing methods use temporal augmentations to encourage similar representations for images of the same location captured at different times.
3. SatlasPretrain
SATLASPRETRAIN is a very-large-scale remote sensing dataset combining diverse labels with spatio-temporal imagery across high- and low-resolution modes. Its scale, geographic coverage, label diversity, and temporal structure support research on remote sensing understanding.
- Spatio-temporal structure: Labels are linked to geographic coordinates and time ranges, enabling predictions that use multiple images over time and long-range spatial context.Slow-changing labels correspond to the most recent image, whereas dynamic labels reference a specific image and time.
- Scale and coverage: SATLASPRETRAIN spans all continents except Antarctica and contains 40x more image pixels and 150x more labels than the largest existing dataset.Its 137 categories are 2x more than those in the largest existing dataset.
- Dataset structure: SATLASPRETRAIN contains 856K tiles with image time series and labels spanning 137 categories.Each tile covers a disjoint spatial region of up to 25 km^2.
- Imagery: The dataset combines low-resolution Sentinel-2 imagery and high-resolution NAIP imagery, with separate train and test splits evaluated independently by image mode.Sentinel-2 provides 10 m/pixel multispectral imagery, while NAIP provides 1 m/pixel aerial imagery in the United States.
- Labels: SATLASPRETRAIN provides seven label types, including segmentation, regression, points, polygons, polylines, and object properties.Examples include land cover, bathymetry, wind turbines, buildings, roads, and rotor diameter.
- Label collection: The dataset derives labels from expert and AMT annotation plus five existing sources, while iterative precision-and-recall analyses guide annotation and source processing.The existing sources include OpenStreetMap, NOAA lidar scans, WorldCover, Microsoft Buildings, and C2S.
4. SatlasNet
SATLASNET is a unified model for learning from SATLASPRETRAIN’s seven label types. It processes spatially aligned image time series and uses task-specific output heads for the different prediction types.
- Unified modeling: SATLASNET learns from all seven label types in SATLASPRETRAIN through a unified model.Off-the-shelf models cannot jointly support all label types, including object properties and image classification.
- Architecture: SATLASNET processes spatially aligned image time series with a Swin-Base backbone and max temporal pooling.The backbone produces multi-scale feature maps for each image before temporal aggregation.
- Architecture: Seven output heads produce predictions for the seven label types.The architecture uses task-specific heads, including separate heads for outputs such as segmentation and polygons.
5. Evaluation
The evaluation compares SATLASNET and eight baselines across SATLASPRETRAIN’s diverse label types, then tests pre-training on seven downstream tasks. SATLASNET benefits from unified and multi-image training, while SATLASPRETRAIN pre-training improves performance with limited downstream data.
- Evaluation on SATLASPRETRAIN: SATLASNET is compared with eight baselines across classification, semantic segmentation, instance segmentation, regression, points, polygons, and polylines.The baselines cover subsets of the seven label types, whereas SATLASNET supports all seven.
- Evaluation on SATLASPRETRAIN: Joint training reduces average performance on several label types but provides large inference-efficiency gains by computing backbone features once per image.Fine-tuning the jointly trained parameters recovers performance and supports transfer between label types.
- Evaluation on SATLASPRETRAIN: 7.1% relative improvement in average performance across label types and image modes comes from fine-tuning jointly trained SATLASNET parameters rather than per-type training.This result supports transfer-learning opportunities between label types.
- Qualitative Results: Qualitative results show high accuracy for wind turbines and water towers, but missed oil wells and noisy road and railway segments remain.The authors suggest specialized models for polyline outputs could improve accuracy.
- Downstream Performance: 18% improvement over ImageNet pre-training and 6% over the next best baseline occur in average accuracy across seven downstream tasks trained with 50 examples.The downstream tasks include classification, semantic segmentation, and instance segmentation.
6. Use in AI-Generated Geospatial Data
The authors deployed SATLASPRETRAIN models in Satlas, an AI-generated global geospatial data platform. The platform targets decision-relevant products that are difficult to curate manually at global scale.
- Deployment: SATLASPRETRAIN models support Satlas, a platform generating global geospatial data from satellite imagery with AI.The platform provides data such as wind turbine and solar farm positions for emissions reduction, disaster relief, and urban planning.
7. Conclusion
SATLASPRETRAIN improves existing remote sensing datasets through greater scale and label diversity. Its pre-trained models improve downstream accuracy and have been used for multiple geospatial detection tasks.
- Conclusion: 18% higher average downstream accuracy than ImageNet and 6% higher than existing remote sensing datasets demonstrate SATLASPRETRAIN’s effectiveness for remote sensing methods.The conclusion links these gains to applications with few labeled examples.
- Conclusion: SATLASPRETRAIN models have been used to detect wind turbines, solar farms, offshore platforms, and tree cover in the Satlas platform.These applications are reported as deployments of models pre-trained on SATLASPRETRAIN.