Source-linked AI summary
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, Stefano Ermon
TL;DR
Satellite imagery offers abundant unlabeled data but has temporal and multi-spectral structure that existing natural-image pre-training methods do not adequately capture. SatMAE adapts masked autoencoders with temporal or spectral positional encodings and dimension-specific masking, achieving improvements over prior state-of-the-art methods across benchmark and downstream remote-sensing tasks. These results support tailored self-supervised pre-training for satellite imagery, although the approach addresses data with irregular temporal sampling through a specialized design.
Problem
Existing self-supervised methods are mainly designed for natural images, while satellite data contain temporal and multi-spectral structure that existing satellite methods cannot jointly represent.
Method
SatMAE adapts masked autoencoders with temporal or spectral positional encodings, independently masked patches across the temporal or spectral dimension, and grouped spectral-band embeddings.
Results
SatMAE outperforms previous state-of-the-art methods across benchmark and downstream remote-sensing tasks, including a 6.29% improvement in fMoW top 1 classification accuracy.
Takeaways & Limitations
Tailoring masked-autoencoder pre-training to satellite temporal and spectral structure yields useful representations for supervised and downstream remote-sensing evaluation.
Takeaways & Limitations
Naively concatenating temporal images along the channel dimension cannot represent temporal ordering, variable time gaps, fine-grained temporal information in deeper layers, or temporal-shift invariance.
Abstract
from arXiv · showhide
Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/
1 Introduction
Satellite imagery offers abundant unlabeled data but includes temporal, geographical, resolution, and multi-spectral structure that natural-image pre-training methods do not adequately address. SatMAE adapts masked-autoencoder pre-training to these dimensions and reports improvements across benchmark and downstream remote-sensing tasks.
- Self-supervised learning is attractive for remote sensing because unlabeled satellite data are abundant while human labels are scarce.
- Existing self-supervised approaches mainly target natural images, whereas satellite imagery contains meaningful temporal and geographical information, multiple spectral bands, and variable resolution.
- SatMAE extends masked autoencoders to temporal and multi-spectral satellite imagery with temporal or spectral positional encodings and independently masked patches.
- SatMAE introduces fMoW-Sentinel, a Sentinel-2 dataset cross-referenced with fMoW for benchmarking multi-spectral satellite-image models.
- SatMAE achieves significant improvements over previous state-of-the-art methods on benchmark datasets and downstream remote-sensing tasks.
2 Related Work
Prior remote-sensing work includes task-specific supervised architectures and self-supervised contrastive methods, while MAE-based extensions do not optimally handle satellite temporal and multi-spectral inputs. SatMAE is positioned as a framework tailored to those data characteristics.
- Supervised satellite-image time-series studies use architectures such as LSTMs, self-attention, and transformers for specialized tasks but are often not directly applicable across remote-sensing datasets.
- Contrastive satellite-imagery methods use spatially aligned or cross-sensor images as positive pairs, but their performance relies heavily on controlling positive-pair quality.
- MAE reconstructs masked input patches and largely avoids designing specific data augmentation, motivating extensions to additional modalities.
- VideoMAE, GMAE, and MultiMAE extend MAE to other modalities, but the cited approaches do not optimally handle temporal and multi-spectral satellite inputs.
- VideoMAE assumes equally spaced frames, unlike satellite sequences whose observations can be temporally irregular and discontinuous.
3 Background
The background describes masked autoencoders as asymmetric encoder-decoder systems that reconstruct masked image patches. Transformer positional encodings provide position awareness for patch tokens using spatial coordinates.
- Masked Autoencoder: MAE divides an image into non-overlapping patches, embeds them as tokens, and masks a fraction before encoding the visible tokens with a Vision Transformer.
- Masked Autoencoder: The MAE decoder restores masked positions with learnable mask tokens and reconstructs the image using all token positions.
- Masked Autoencoder: The reconstructed image is compared with the original using mean-squared error computed per pixel on masked regions.
- Positional encoding: Positional encoding makes transformer representations position-aware by encoding each patch’s spatial location.
- Positional encoding: The standard encoding uses sinusoidal functions of position and feature dimension, with Ω normally set to 10000.
4 Method
SatMAE adapts masked autoencoders to satellite imagery by preserving temporal and spectral structure through specialized patch embeddings, positional encodings, and masking strategies. The method addresses irregular temporal sampling and heterogeneous multi-spectral bands while retaining information needed for pre-training and fine-tuning.
- Temporal SatMAE: SatMAE represents temporal sequences as patch tokens for each image, reusing a shared spatial patch embedding across the time series.The temporal patch size is set to PT = 1 because satellite sequences are shorter than typical video sequences.
- Temporal Encoding: Temporal encodings combine learned year, month, and hour features with spatial positional encodings before transformer encoding.The selected timestamp components reflect long-term landscape change, seasonality, climate, daylight, and temperature.
- Masking Strategies: Independent masking allows different spatial regions to be masked in different images, while consistent masking keeps masked regions aligned across the time series.Both strategies mask a fraction pm of patches; independent masking can expose an unmasked version of a region in another image.
- Multi-spectral SatMAE: For multi-spectral imagery, SatMAE groups bands and applies a separate patch embedding to each group before concatenating the resulting tokens.Grouping is intended to preserve fine-grained information across bands with different wavelengths and spatial resolutions.
- Spectral Encoding: Spectral group encodings identify each user-defined band group and are concatenated with spatial positional encodings before the encoder.The combined grouping and group-encoding configuration is denoted SatMAE+Group.
- Masking Strategies: The multi-spectral masking experiments compare consistent masking and independent masking within the SatMAE+Group configuration.These variants are denoted SatMAE+Group+CM and SatMAE+Group+IM.
5 Experiments
SatMAE is evaluated across benchmark classification, multi-spectral imagery, ablations, and remote-sensing transfer tasks. The experiments show gains from temporal encoding, independent masking, spectral grouping, and SatMAE pre-training.
- fMoW RGB classification: SatMAE improves fMoW single-image top-1 classification accuracy by 6.29% over the previous state-of-the-art self-supervised method.Without SatMAE pre-training, ViT-Large reached 62.48% after 50 finetuning epochs, below ResNet-50 trained from scratch at 69.05%.
- Temporal fMoW classification: SatMAE improves temporal fMoW top-1 accuracy by 4.48% over the previous state-of-the-art and by 2.06% over its non-temporal result.It also outperforms UTAE by 18%; naive channel stacking performs worse than non-temporal SatMAE.
- Temporal ablations: Temporal SatMAE gains are mainly associated with temporal encoding and independent masking rather than consistent masking.Consistent cropping slightly decreases performance, while test-time augmentation is beneficial.
- Multi-spectral classification: SatMAE+Group+IM achieves the highest fMoW-Sentinel accuracy, outperforming supervised training from scratch by 6.27% and ImageNet initialization by 4.84%.The experiments group bands by wavelength and spatial resolution, and report Top-1 and Top-5 validation accuracy.
- Multi-spectral ablations: Band-group choice changes fMoW-Sentinel performance by about 0.6%, and independent masking performs best in the spectral ablation.The ablation compares ViT sizes, grouping strategies, masking strategies, and spectral group encoding.
6 Conclusion
SatMAE is a masked-autoencoder framework tailored to remote-sensing data, using joint positional, temporal/spectral encoding and masking to handle temporal and multi-spectral imagery. Across pre-training and downstream datasets, it outperforms previous state-of-the-art results by large margins.
- SatMAE adapts masked autoencoders to satellite imagery with joint positional, temporal, and spectral encodings.The framework is designed to handle temporal and multi-spectral inputs while learning representations useful for finetuning.
- Its masking strategy operates in a joint positional, temporal/spectral space for temporal and multi-spectral inputs.
- The authors report effectiveness across pre-training datasets and multiple downstream datasets, with large-margin improvements over previous state-of-the-art results.
- More efficient transformer architectures remain needed because longer token sequences can strain computational resources.The paper also identifies positional encodings, spectral-band grouping, and architectures for detection and segmentation as areas for future work.
Broader Impact
SatMAE addresses sparse labelled data for satellite-imagery tasks relevant to measuring social, economic, and environmental phenomena. The authors present pre-training as a way to amplify the usefulness of limited labels and support rapid imagery-feature extraction.
- Sparse labelled data can leave policy decisions without timely or accurate measurements of economic, social, and environmental phenomena.
- Satellite-imagery pre-training could reduce dependence on labelled data for downstream tasks.
- SatMAE could amplify the usefulness of sparse labels for key human outcomes.
- The approach could enable rapid and accurate extraction of imagery features for downstream tasks such as poverty prediction and infrastructure development.
A.1 Datasets
The paper uses fMoW RGB imagery and constructs temporal and Sentinel-2 multi-spectral datasets aligned to fMoW locations. The Sentinel dataset contains 13 bands and large train, validation, and test image collections.
- fMoW RGB: fMoW is a worldwide high-resolution satellite-image time-series dataset for classifying 62 architecture categories.It includes RGB imagery and metadata such as location, time, and sun angles.
- Temporal fMoW: The temporal fMoW version associates each image with two co-located images from different timestamps when possible, producing sequences of length three.Images without enough distinct co-located timestamps are duplicated, and the split prevents leakage across locations.
- fMoW Sentinel: fMoW Sentinel pairs fMoW locations with 90-day Sentinel-2 surface-reflectance composites to reduce cloud-coverage effects.The collection uses geo-coordinate bounding boxes to gather Sentinel-2 time series at fMoW datapoint locations.
- fMoW Sentinel: The Sentinel dataset contains all 13 Sentinel-2 frequency bands and 712,874 training, 84,939 validation, and 84,966 test images.Mean image dimensions are approximately 45 × 60 pixels.
- Dataset statistics: The dataset materials include figures showing category and location distributions, geographic distribution by country, and per-band pixel-count distributions.
- Dataset statistics: Table 10 reports mean and standard deviation pixel values for each channel across the fMoW Sentinel training dataset.Band B10 lacks bottom-of-atmosphere information and is no longer accessible on Google Earth Engine.
A.3 Training Details
Training spans non-temporal and temporal fMoW RGB, fMoW Sentinel, and several remote-sensing transfer datasets, using ViT backbones with dataset-specific temporal or spectral processing. Experiments also vary masking, augmentation, grouping, and pre-training duration.
- Datasets and evaluation: The experiments cover fMoW RGB, temporal fMoW RGB, fMoW Sentinel, NAIP, EuroSAT, BigEarthNet, and SpaceNet v1.
- fMoW RGB: Non-temporal fMoW RGB uses resized 224-pixel images, 16-pixel patches, dataset normalization, and ViT-Large pre-training.
- Temporal fMoW: Temporal fMoW samples are sequences of three co-located images, duplicating an image when two distinct timestamps are unavailable.The train/validation split keeps co-located images within the same split.
- Temporal SatMAE: Temporal pre-training triples the input patches, uses temporal encoding, and applies independent masking with a fixed masked-patch ratio pm = 0.75 per image.The setup trains for 100 epochs with batch size 1024 on eight NVIDIA V100 GPUs.
- Multi-spectral SatMAE: For Sentinel imagery, ViT-Large uses 96 × 96 inputs and 8-pixel patches, producing 144 patches when stacked or 432 patches for three spectral groups.Grouping bands by spatial resolution and wavelength similarity avoids the unstable 1,440-patch single-band configuration.
- Sentinel finetuning: Additional augmentation during Sentinel finetuning increases performance by around 2.18%.The configuration adds weight decay, drop path, reprob, mixup, and cutmix changes while reporting the best validation Top 1 accuracy.
- Pre-training duration: Training SatMAE for 200 rather than 50 epochs further improves final Top 1 accuracy after finetuning, and a smaller ViT-Base can outperform ViT-Large with longer pre-training.
A.4 Impact of masking ratio and patch size on fMoW-RGB-temporal
On fMoW-RGB-temporal data, SatMAE performs best with a 0.75 masking ratio, while larger patches reduce accuracy despite improving computational efficiency.
- A 7.19% performance drop occurs with a 0.6 masking ratio compared with the default pm = 0.75.Lower masking ratios expose more visible patches, making reconstruction easier but weakening the pretext task.
- The default pm = 0.75 is roughly optimal for temporal SatMAE.A masking ratio of 0.9 may make reconstruction difficult because too few patches remain visible, potentially requiring longer training.
- Larger patch sizes reduce performance by providing less granular spatial information to the model.Larger patches are more computationally efficient, creating an accuracy–resource trade-off.
A.5 Impact of masking ratio and patch size on fMoW-Sentinel
On fMoW-Sentinel multi-spectral data, lower masking ratios weaken representations, while higher masking may remain viable; larger patches reduce performance but can accelerate training.
- Lower masking ratios produce less useful representations because more visible patches make reconstruction too easy.Unlike the temporal setting, pm = 0.9 does not cause a large performance drop for multi-spectral data with independent masking.
- A masking ratio of pm = 0.9 may be usable for multi-spectral independent masking because fewer encoded tokens could quicken pre-training.The passage presents this as a suggestion based on the observed performance pattern.
- Larger patch sizes result in worse performance by providing less granular spatial information to deeper model layers.The accuracy loss should be weighed against the potential gain in training speed.
- Bands B9 and B10 show the smallest accuracy drop when masked, while B1 shows a larger drop in the ablation.The authors nevertheless drop B1 and retain RGB and other multi-spectral bands as highly relevant.
C.2 Spectral SatMAE
Spectral SatMAE reconstructs masked multi-spectral imagery more sharply when bands are grouped and masked independently, and it learns correlations across bands.
- SatMAE+Group+IM produces clearer reconstructions than SatMAE+Group+CM and SatMAE+Stack.Independent masking yields sharper outputs, whereas consistent masking and channel stacking produce fuzzier reconstructions.
- The visualizations cover RGB, NIR, and SWIR band groups across multiple SatMAE spectral settings.RGB, NIR, and SWIR correspond to distinct Sentinel-2 band groups described in the figure caption.
- Independent masking enables reconstruction of masked spectral content using information from other band groups.The model reconstructs a masked bright spot in the SWIR group from information in other groups.
- The authors hypothesize that longer pre-training could further improve reconstruction quality and learned representations.