Source-linked AI summary
SpectralGPT: Spectral Remote Sensing Foundation Model
Danfeng Hong, Bing Zhang, Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, Hao Li, Pedram Ghamisi, Xiuping Jia, Antonio Plaza, Paolo Gamba, Jon Atli Benediktsson, Jocelyn Chanussot
TL;DR
SpectralGPT addresses the limited availability of foundation models designed for spectral remote-sensing data, which supports scene understanding beyond RGB. It introduces a 3D GPT-based architecture with progressive training and multi-target reconstruction, and reports superior downstream performance across scene classification, semantic segmentation, and change detection.
Problem
Existing foundation models mainly process RGB images, leaving a gap in models tailored to spectral remote-sensing data for scene understanding.
Method
SpectralGPT combines 3D masking, a spatial-spectral encoder, multi-target reconstruction, and progressive pretraining on diverse spectral data.
Results
SpectralGPT outperforms existing state-of-the-art competitors across scene classification, semantic segmentation, and change detection, while its reconstruction assessment spans 50% to 5% visible tokens.
Takeaways & Limitations
Trained on over one million spectral images with over 600 million parameters, SpectralGPT supports flexible spectral-data processing for large-scale Earth-observation applications.
Takeaways & Limitations
The change-detection setting can suffer from class imbalance and substantial ViT data requirements, increasing risks of false positives, overfitting, and reduced out-of-domain adaptability.
Abstract
from arXiv · showhide
The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.
1 INTRODUCTION
Spectral remote sensing provides rich information for scene understanding, but existing foundation models and labeled data are insufficient for fully exploiting large spectral datasets. SpectralGPT addresses these gaps with spectral-specific modeling, large-scale progressive pretraining, and strong downstream performance.
- Motivation: Spectral imaging captures information that supports object and scene recognition beyond RGB data alone.This supports applications including land-cover mapping, ecosystem monitoring, weather forecasting, and geological exploration.
- Research gaps: Existing RS foundation models often process RGB-like data and struggle to capture spatial-spectral representations in 3D tensor data.This limits their ability to fully characterize spectral information.
- Research gaps: Spectral foundation-model research remains scarce, while grouping adjacent bands can disrupt spectral continuity and weaken spatial-spectral coupling.Video-pretrained networks are also often poorly suited to spectral data because spectral frames contain different redundancy patterns from video frames.
- Proposed approach: SpectralGPT combines 3D masking, spatial-spectral token encoding, and multi-target reconstruction to preserve spectrally sequential characteristics.The model is purpose-built for spectral RS data within an MAE-based framework.
- Training and contributions: SpectralGPT is progressively pretrained across diverse datasets and inputs, including varying sizes, resolutions, time series, and geographic regions.The Sentinel-2 training effort uses over one million spectral images and produces Base, Large, and Huge models with approximately 100 million, 300 million, and 600 million parameters.
- Results: SpectralGPT outperforms existing state-of-the-art competitors across scene classification, semantic segmentation, and change detection.The work also introduces SegMunich, a 13-class urban semantic-segmentation benchmark dataset.
2.1 A Brief Recall of MAE
MAE learns representations from partially observed inputs and reconstructs the complete signal with an asymmetric encoder-decoder architecture. Its pipeline partitions an image, masks patches, encodes visible patches, decodes all tokens, and optimizes reconstruction error.
- MAE architecture: MAE reconstructs the original signal using an encoder that maps observed inputs to representations and a decoder that reconstructs the complete signal.Unlike classical autoencoders, its encoder operates only on partial observed inputs, while a lightweight decoder receives mask tokens.
- Pipeline: MAE partitions an input image into regular, non-overlapping p × p × C patches before masking.The input has H × W pixels and C dimensions.
- Pipeline: Only visible patches are sent to the learned encoder after masking identifies visible and masked patches.The encoder projects patches, adds positional embeddings, and processes them through transformer blocks.
- Pipeline: The decoder combines encoded visible patches with mask tokens and positional embeddings to reconstruct the image.A lightweight ViT decoder produces patch-sized outputs through a final linear projection.
- Optimization: MAE trains with mean squared error reconstruction loss over visible and masked patches.Patch normalization uses each patch’s pixel mean and standard deviation.
2.2 Methodological Overview of SpectralGPT
SpectralGPT combines 3D masking, an encoder for spectrally informed visual representations, and a decoder for multi-target reconstruction. Progressive training across diverse spectral data is intended to improve flexibility, robustness, and generalization.
- Methodological overview: SpectralGPT uses 3D masking, a spectrally visual encoder, and a multi-target reconstruction decoder as its three key components.The architecture is designed specifically for spectral data rather than RGB-only inputs.
- Methodological overview: Progressive training exposes SpectralGPT to diverse spectral data and enhances its flexibility, robustness, and generalization capabilities.The pretrained model is adapted to classification, segmentation, and change-detection tasks.
2.3 3D Masking on Spectral Data
SpectralGPT represents multi-band spectral images as 3D tensor data and masks non-overlapping spatial-spectral cubes. A 90% masking rate supports learning spatially and spectrally informed representations from the remaining tokens.
- 3D tokenization: SpectralGPT models multi-band spectral images as 3D tensors and partitions them into non-overlapping tokens along spatial and spectral dimensions.Each token has size p × p × k, where p and k denote spatial and spectral token sizes.
- 3D masking: The 3D cube masking strategy uses a 90% masking rate to learn spatially and spectrally visual representations.The approach is intended to enable efficient processing and more diverse knowledge extraction from spectral inputs.
- 3D masking: The masking operation produces visible and masked spectral representations from the tensor tokens.A token-wise binary mask indicates which tokens are masked, setting all pixels within those tokens to zero.
2.4 Encoder for Visible Tokens
The encoder transforms visible spatial-spectral tokens into feature embeddings and processes them with stacked self-attention blocks. These blocks compute attention-weighted representations that retain the input dimension for subsequent encoding.
- Visible tokens are projected into feature embeddings using a shared linear projection and positional encoding before entering the encoder.The encoder applies fθ to Esxi + Epos for visible tokens.
- The self-attention module linearly transforms each input embedding into query, key, and value representations using learnable projection matrices.
- Attention scores are computed from scaled query-key dot products and passed through softmax before weighting value embeddings.The embedding dimension d is used for scaling.
- The weighted value embeddings are summed to produce output features with the same dimension as the input tokens.These outputs can be processed by subsequent encoder layers.
2.5 Lightweight Decoder with Multi-Target Reconstruction
SpectralGPT trains a lightweight decoder to reconstruct original image tokens from encoder features. Its token-to-token and spectral-to-spectral reconstruction objectives jointly preserve spatial-spectral coupling and sequential spectral information.
- The lightweight decoder reconstructs original image tokens from encoder outputs representing visible and masked tokens.The encoder and decoder are trained end-to-end using reconstruction loss.
- The reconstruction loss combines token-to-token and spectral-to-spectral components to capture complementary spatial-spectral information.
- The overall reconstruction objective is defined using mean squared error in pixel space.
- The formulation distinguishes masked-token counts, spectral-token structure, and standard-token row-column locations.m and n denote masked and spectral token counts, while (r, c) identifies token position.
2.6 Progressive Pretraining
Progressive pretraining uses fixed-sized 3D tokens to process varied remote-sensing inputs and stitch their outputs into complete images. The strategy also introduces data with differing sizes, sensors, and quality levels progressively.
- Fixed-sized 3D tokens, such as 8 × 8 × 3, allow images with different sizes to pass through the encoder-decoder pipeline independently.Processed tokens are stitched back together to form the final output.
- Progressive feeding can move from 96 × 96 to 128 × 128 pixel images or from Sentinel-2 to Landsat-8 or Gaofen-2 data.
- The strategy is described as improving flexibility for input size and type while enhancing knowledge extraction from diverse data sources and model generalization.
- Sentinel-2 samples cover Earth’s inhabited areas and total 1,473,105 images.
- Training feeds fMoW-S2 before BigEarthNet-S2 because fMoW-S2 offers larger scale but relatively lower spectral-data quality, followed by higher-quality data.
2.7 Pretrained Dataset
The pretraining dataset contains more than one million Sentinel-2 spectral images across 12 bands, combining globally diverse fMoW-S2 data with regional BigEarthNet-S2 data. The sources provide different scales, geographic coverage, splits, and downstream fine-tuning usage.
- The dataset contains over one million Sentinel-2 spectral images across 12 bands from fMoW-S2 and BigEarthNet-S2.fMoW-S2 is globally diverse and labeled with 62 categories, whereas BigEarthNet is regional and covers more than ten European countries.
- The complete Sentinel-2 collection contains 1,473,105 images distributed across Earth’s inhabited areas.
- fMoW-S2 contains 882,779 images divided into 712,874 training, 84,939 validation, and 84,966 test images.Each image averages approximately 45 pixels in height and 60 pixels in width.
- BigEarthNet-S2 comprises 590,326 distinct, non-overlapping Sentinel-2 spectral image tokens.
- Pretraining uses 712,874 fMoW-S2 images and 354,196 BigEarthNet-S2 images, while 35,420 labeled BigEarthNet-S2 images are used for downstream fine-tuning.The fine-tuning subset represents 10% of labeled BigEarthNet-S2 images.
2.8 Implementation Details and Experimental Setup
The implementation harmonizes spectral inputs across datasets, partitions them into spatial-spectral tokens, and applies a high-masking pretraining regimen before downstream use.
- Sentinel-2 inputs retain 12 dominant spectral bands, excluding B10, with band-wise normalization to [0,1].
- Spectral images are partitioned into 8×8×3-pixel tokens, with separate learnable positional embeddings for spatial and spectral variation.
- Pretraining uses a 90% 3D masking ratio for 200 epochs on fMoW-S2 with AdamW and a 10^-4 learning rate.
- The downstream single-label classification architecture passes pretrained-model outputs through average pooling before prediction.
3 EXPERIMENTS
Experiments show that pretrained SpectralGPT models perform strongly across scene classification, semantic segmentation, and change detection, while ablations identify effects from model size, data, masking, targets, decoder depth, and token size.
- Single-label scene classification: 99.15% Top1 accuracy was achieved on EuroSAT, increasing to 99.21% when pretraining used both fMoW-S2 and BigEarthNet.
- Multi-label scene classification: 88.22% macro-mAP and 87.50% micro-mAP were achieved by SpectralGPT+ on BigEarthNet, surpassing fMoW-S2-only pretraining by 2.19% and 1.86%.
- Semantic segmentation: 1.1% and 2.3% higher mIoU than SatMAE were obtained by SpectralGPT and SpectralGPT+, respectively, with stronger recognition across many land-use categories.
- Change detection: 0.75% and 1.53% margins over SatMAE were reported for the highest F1 score in OSCD change detection, although precision was relatively lower than competing models.
- Ablation studies: A 90% masking ratio improved multispectral pretraining, while larger token sizes reduced performance and shallow decoders were unsuitable for spectral pretraining.
- Reconstruction targets: Raw-data pretraining may require longer schedules to match normalized or standardized inputs because spectral values are large and vary across bands.
- Model size: 91.39 mAP(micro) was achieved by ViT-H with SpectralGPT+ using 10% of downstream training data, exceeding models trained with the full training set.
- Visual comparison and geo-characteristic recoverability: SpectralGPT preserved critical structures above 90% masking and produced stronger spectral-band reconstructions than SatMAE with 10% visible patches.
4 CONCLUSION
SpectralGPT addresses challenges in processing and extracting knowledge from diverse spectral RS data by combining a 3D GPT architecture with progressive training and spectral reconstruction. The study reports stronger reconstruction capability than SatMAE and evaluates potential across spectral RS applications.
- SpectralGPT targets the challenges of efficiently processing diverse spectral RS big data and extracting knowledge from spatial-spectral information.
- SpectralGPT uses a 3D GPT architecture, progressive training, 3D masking, and multi-target reconstruction to model spectral RS data.The model is trained on over one million spectral images with over 600 million parameters.
- SpectralGPT shows stronger visual reconstruction capability than SatMAE at masking rates reaching over 90%.The comparison covers masking ratios from 50% to 95%.
- SpectralGPT is designed to make spectral RS big data more accessible and cost-effective for large-scale Earth observation applications.
- The study assesses MAE-based foundation models through spectral reconstruction experiments using inputs with 50% to 5% visible tokens.The analysis examines spectral-wise reconstruction and inference for geoscience applications.
- Future work will expand training-data diversity and volume and extend SpectralGPT to more downstream tasks to improve robustness and generalization.