Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery

• Remote-sensing imagery varies in ground sample distance, object scale, and spectral bands, making multi-scale information relevant to remote-sensing tasks.Source: §1 · p. 1
• Multi-scale information is scarcely exploited; SatMAE does not use it, while ScaleMAE uses GSD-based positional encoding limited to optical imagery.Source: §1 · p. 1
• SatMAE++ sets state-of-the-art performance on all six datasets and gains 3.6% absolute over baseline on land-cover classification.Source: §1 · p. 2
• SatMAE++ combines standard positional encodings with multi-scale reconstruction using convolution-based upsampling blocks.Source: §1 · p. 2

2.1. Representation Learning for Satellite Imagery

• Satellite-image pre-training includes contrastive learning, in-domain representation learning, continual learning, and masked image modeling; ScaleMAE uses GSD positional encoding but is limited to optical imagery.Source: §2.1 · p. 2

2.2. Multi-scale Information

• Multi-scale information is used in classification, detection, and segmentation, and prior approaches include hierarchical masking and multi-scale autoencoders.Source: §2.2 · p. 2
• SatMAE++ uses convolution-based upsampling blocks to reconstruct feature maps at multiple scales, while Sentinel-2 bands span three GSD resolutions from 10 m to 60 m.Source: §2.2 · p. 2, Table 1 · p. 2

• MAE divides an image into nonoverlapping patches, masks a fraction of tokens, encodes the visible tokens, and reconstructs the image with mean squared error.Source: §3 · p. 2
• Positional encodings represent patch-token locations, and the background gives sinusoidal encodings using position and feature-dimension terms.Source: §3 · p. 2
• The illustrated SatMAE++ framework reconstructs the lowest-resolution image with MAE, then upsamples features to reconstruct higher-resolution images with L1 losses.Source: Figure 1 · p. 3

4.1. Baseline Framework

• Baseline SatMAE groups multispectral channels by GSD and uses 96×96 inputs with patch size 8, but lacks multiscale modeling; ScaleMAE’s GSD embeddings are limited to same-GSD RGB.Source: §4.1 · p. 3

4.2. Overall Architecture

• SatMAE++ feeds the lowest-resolution image through SatMAE, then uses convolution-based upsample blocks to reconstruct higher-resolution images and encourage multiscale representations.Source: §4.2 · p. 3
• Each upsample block uses transpose convolution to increase spatial resolution, followed by a two-convolution residual block that enhances the features.Source: §4.2 · p. 4, Figure 2 · p. 4

4.2.1 Reconstruction at Two Scale Levels

• For two-scale reconstruction, 75% patch masking and general positional encodings feed an MAE at H×W; its decoded image is upsampled to 2H×2W and trained with higher-scale L1 loss.Source: §4.2.1 · p. 4

4.2.2 Reconstruction at Three Scale Levels

• For multispectral three-scale reconstruction, the 4H×4W image is downsampled to 2H×2W and H×W, then upsample blocks produce reconstructions at all three scales with L1 losses.Figure 3 shows fMoW-Sentinel RGB-channel examples at these resolutions and reports better SatMAE++ reconstruction than SatMAE at H×W.Source: §4.2.2 · p. 4, Figure 3 · p. 5

5.1. Pre-training Datasets

• SatMAE++ improves reconstruction on visible and masked patches, while fine-grained structural details remain difficult in some cases.Figure 4 compares reconstruction with SatMAE; fMoW-Sentinel includes 712,874 training, 84,939 validation, and 84,966 test images.Source: §5.1 · p. 5, Figure 4 · p. 6

5.3. Pre-training and Finetuning on fMoW-Sentinel

• 63.23 at epoch 12 is the reported peak for multi-scale pre-training, versus 61.61 at epoch 20 for single-scale pre-training.The passage reports faster convergence for multi-scale pre-training on the fMoW-Sentinel validation set.Source: Figure 5 · p. 6, §5.3 · p. 7

5.2. Pre-training and Finetuning on fMoW-RGB

• 78.14% on fMoW-RGB outperforms ScaleMAE and yields a 4.27% absolute gain over reproduced SatMAE results.The reported gain over the SatMAE number in its paper is 0.3%.Source: Table 2 · p. 5, §5.2 · p. 6

5.3. Pre-training and Finetuning on fMoW-Sentinel

• SatMAE++ improves fMoW-Sentinel accuracy by 1.75% over ViT-Large SatMAE; increasing pre-training scales adds 1.25% then 0.37%.The authors hypothesize that the effect of adding scales may become insignificant after a certain number.Source: §5.3 · p. 7, Table 4 · p. 7, Table 3 · p. 7

5.5. Transfer Learning on Downstream Datasets

• SatMAE++ provides consistent improvement across downstream land-cover and multi-label classification tasks.The evaluations include EuroSAT, RESISC-45, UC-Merced, and BigEarthNet.Source: §5.5 · p. 7, §5.4 · p. 7

5.5.1 Land Cover Classification

• 99.01% accuracy on EuroSAT surpasses SatMAE, while 97.48% on RESISC-45 surpasses ScaleMAE.The EuroSAT comparison is reported without using multi-spectral information.Source: §5.5.1 · p. 8, Table 5 · p. 8, Table 6 · p. 8
• ViT-Large reaches 94.05% accuracy on UC-Merced with SatMAE weights, and improves by approximately 3.6% with SatMAE++ weights.Source: §5.5.1 · p. 8, Table 7 · p. 8

5.5.2 Multi-Label Classification

• 85.11% average precision on BigEarthNet exceeds SeCo’s 82.62% and SatMAE’s 82.13%.The evaluation uses 10% of the training data and reports mean average precision.Source: §5.5.2 · p. 8, Table 8 · p. 8

6. Conclusion

• SatMAE++ incorporates multiscale information, improving model performance and speeding convergence during finetuning.The conclusion says existing state-of-the-art methods struggle to use multiscale information with multispectral data.Source: §6 · p. 8
• The framework is extensible to multiple scale levels and is not restricted to a single data modality.Source: §6 · p. 8
• Experiments on several downstream and mainstream datasets reveal the efficacy of the approach.Source: §6 · p. 8
• Future work includes extending multi-scale pre-training to dense prediction tasks.Source: §6 · p. 8