Source-linked AI summary
Galileo: Learning Global & Local Features of Many Remote Sensing Modalities
Gabriel Tseng, Anthony Fuller, Marlena Reil, Henry Herzog, Patrick Beukema, Favyen Bastani, James R. Green, Evan Shelhamer, Hannah Kerner, David Rolnick
TL;DR
Remote sensing representation learning must accommodate diverse modalities and objects that vary greatly across space and time. Galileo uses a highly multimodal transformer with dual local-global masked self-supervision, and it outperforms specialist remote sensing models across eleven benchmarks and multiple tasks. The paper supports a generalist representation model while noting the need for collaboration with local communities and stakeholders because such models can be used to collect information about populations without their involvement.
Problem
Remote sensing representation learning must handle diverse modalities and objects spanning extreme spatial and temporal scales, while existing self-supervised methods are often specialized to particular modalities or input shapes.
Method
Galileo combines a highly multimodal transformer with masked modeling objectives that learn global features from deep targets and local features from shallow input projections.
Results
Galileo-Base ranks above larger image-specialized models and pixel-time-series-specialized models across an extensive suite covering eleven benchmarks and multiple tasks.
Takeaways & Limitations
A single Galileo model provides flexible multimodal representations for remote sensing data across different spatial and temporal scales and input shapes.
Takeaways & Limitations
The paper urges deploying Galileo in collaboration with local communities and stakeholders because such models can collect information about populations without their involvement.
Abstract
from arXiv · showhide
We introduce a highly multimodal transformer to represent many remote sensing modalities - multispectral optical, synthetic aperture radar, elevation, weather, pseudo-labels, and more - across space and time. These inputs are useful for diverse remote sensing tasks, such as crop mapping and flood detection. However, learning shared representations of remote sensing data is challenging, given the diversity of relevant data modalities, and because objects of interest vary massively in scale, from small boats (1-2 pixels and fast) to glaciers (thousands of pixels and slow). We present a novel self-supervised learning algorithm that extracts multi-scale features across a flexible set of input modalities through masked modeling. Our dual global and local contrastive losses differ in their targets (deep representations vs. shallow input projections) and masking strategies (structured vs. not). Our Galileo is a single generalist model that outperforms SoTA specialist models for satellite images and pixel time series across eleven benchmarks and multiple tasks.
1. Introduction
Remote sensing representation learning must handle both highly varied modalities and objects spanning extreme spatial and temporal scales. Galileo addresses these challenges with unified self-supervised learning and a multimodal model that performs strongly across diverse benchmarks.
- Galileo unifies self-supervised learning across multiple remote sensing modalities and input shapes, including image and pixel time series.
- Remote sensing objects range from single-pixel, short-lived boats to kilometer-scale glaciers tracked over decades.
- Existing remote sensing methods are often limited to particular modalities, input shapes, or temporal settings.
- Galileo uses masked modeling to learn both local and global features across multiple modalities, scales, and data shapes.
- Galileo-Base ranks above larger image-specialized models and pixel-time-series-specialized models across an extensive benchmark suite.
2. Global, Local, Multimodal Self-Supervision
Galileo combines multimodal tokenization with two masked self-supervised objectives to learn global and local representations across remote-sensing inputs, scales, and shapes. Its global and local pathways differ in masking structure and target depth, while both use within-input token contrastive learning.
- Method Intuition: Galileo is designed to learn rich representations from diverse remote-sensing modalities across different spatiotemporal scales.The model is intended to support pixel time series, image time series, and single images.
- Input Tokenization: Its transformer tokenizes spatial, temporal, and channel-group structure, then adds spatial, temporal, seasonal, and channel embeddings.The architecture supports variable input sequence lengths and channel groups.
- Making Predictions and Computing Losses: Both objectives predict masked target tokens from visible tokens and compare predictions with target encodings using a within-input contrastive loss.The target encoder is updated through an exponential moving average, while the online encoder receives gradient updates.
- Empirical Positioning: Galileo-Base is reported as the best-performing model against both image-specialized and pixel-time-series-specialized pretrained remote-sensing models while supporting more input types.The comparison uses rankings across image and pixel-timeseries tasks, with lower ranking numbers better.
- Global and Local Objectives: Global features use structured space-time masking and deeper target representations, whereas local features use unstructured masking and shallow input projections.Structured masking increases the distance between visible and target tokens; unstructured masking minimizes their average distance.
- Global and Local Objectives: The local objective applies PatchDisc to shallow pixel projections, making predictions similar to same-position pixels and dissimilar to pixels at other positions.This distinguishes the objective from reconstruction methods that use mean-squared error without repelling other pixels.
3. Experimental Framework
The experimental framework pretrains Galileo and evaluates it across diverse remote-sensing benchmarks, comparing it with existing pretrained models under controlled input and training conditions. The evaluation includes image, segmentation, and pixel-time-series tasks with multiple training-set sizes.
- Pretraining: Galileo models are pretrained for 500 epochs using the proposed algorithm.Three model sizes are pretrained.
- Downstream Tasks: Evaluation covers GeoBench Sentinel-2 classification and segmentation, MADOS marine-debris segmentation, Sen1Floods11 Sentinel-1 segmentation, PASTIS image-time-series segmentation, and optical pixel-time-series classification.The benchmark suite spans single-timestep images, image time series, and pixel time series.
- Comparisons: Comparisons rerun evaluations for all models while sweeping normalization methods and learning rates where appropriate.Images are resized to each model’s pretraining image size, and variable-patch-size models use patch size 4.
- Image Classification: Galileo-Base is the best model for image classification by kNN in the reported comparison.The table reports classification performance as percentages and shows the best architecture per method.
- Image Classification: Galileo-Base is the best model for image classification by finetuning in the reported comparison.The table reports classification performance as percentages and identifies the first- and second-best results across methods and architectures.
- Comparisons: Image classification and segmentation are evaluated at 100%, 20%, 5%, and 1% of the training data.These four settings are treated as training-set partitions.
4. Results
Galileo performs strongly across image, time-series, and multimodal remote-sensing benchmarks while balancing accuracy, computational cost, and feature scale through its dual global-local learning design.
- Image results: Galileo-Base outranks image-specialized models on image classification and segmentation, while smaller Galileo models often outperform much larger models.This is particularly valuable when computation is limited.
- Efficiency: Increasing patch size reduces MACs by splitting an input into fewer tokens, creating an accuracy–inference-cost trade-off.Figure 4 reports this trade-off for patch sizes 4, 8, and 16 on EuroSat.
- Generalist comparison: 10.8% higher accuracy on EuroSat distinguishes Galileo from AnySat among models supporting both single-timestep images and pixel time series.The comparison uses the two generalist models on standard benchmarks.
- Time series results: Galileo outranks Presto and far exceeds AnySat on pixel time-series benchmarks.The comparison covers the generalist AnySat and the pixel-time-series specialist Presto.
- Ablations: The combined global-local algorithm excels on both classification and segmentation, whereas either objective alone is strongest on only one task family.The dual-objective strategy also achieves successful training runs more consistently than single-objective alternatives.
- Ablations: 71.5% mIoU on MADOS is achieved by shallow local targets, compared with 58.9% for the global task on tiny-object segmentation.The result supports the specialization of local learning for fine-grained segmentation.
5. Related Work and Background
Galileo extends remote-sensing self-supervised learning beyond single-modality or single-shape settings by combining latent-space objectives with broad multimodal and spatiotemporal flexibility.
- Self-supervised learning: Galileo predicts and learns representations at different latent depths, from linear projections of the input space to the full encoder depth.This differs from methods that predict only pixels or representations at a single latent level.
- Contrastive learning: Galileo adopts contrastive patch learning but uses dual losses because PatchDisc alone remains prone to collapse.The dual losses stabilize pretraining for reliable improvement of the loss.
- Pretrained remote-sensing models: Earlier remote-sensing models generally process one timestep and limited modality combinations, whereas Galileo learns from more modalities.Galileo includes optical and SAR imagery together with topography, weather, population maps, night-lights, and land-cover maps.
- Pretrained remote-sensing models: Galileo flexibly models single-timestep imagery, multi-timestep imagery, and pixel time series across space and time.This supports the varied multimodal and multishape inputs used in remote-sensing workflows.
- Pretrained remote-sensing models: Compared with AnySat, Galileo processes many additional modalities that may matter for modeling diverse remote-sensing phenomena and benchmark performance.AnySat shares Galileo’s flexible treatment of space and time but lacks many of these modalities.
6. Conclusion
Galileo targets broad remote-sensing use by flexibly processing diverse modalities and input shapes while learning features at different scales. Extensive evaluations and ablations support its performance and transferability across domains and task types.
- Galileo is designed to process different remote-sensing modalities and input shapes while modeling phenomena occurring at different scales.
- Its architecture handles highly multimodal inputs that vary across both space and time.
- Its dual local-global self-supervised learning algorithm encourages representations of phenomena occurring at different scales.
- Hundreds of evaluations and thorough ablations demonstrate performance across domains and task types, while supporting the effectiveness and transferability of unified local, global, and multimodal self-supervision.
- Galileo is trained for modalities commonly used in practice beyond benchmark optical or radar inputs, including additional remote-sensing products.
- The model weights, pretraining code, pretraining data, and evaluation code are open sourced.
Impact Statement
Galileo is intended to support societally important remote-sensing applications, especially where few labels are available. The authors also identify a governance risk and encourage deployment with local communities and stakeholders.
- Remote-sensing machine learning spans applications including species distribution modelling and disaster management.
- Galileo is intended to support practitioners exploring and deploying these applications when few labels are available.
- The authors identify a risk that models could collect information about populations and enable decisions without their involvement.
- They encourage deploying Galileo in collaboration with local communities and stakeholders.
A. Methodology details
Galileo adapts latent prediction and masked modeling into separate global and local objectives for learning deep, abstract features and shallow, fine-grained features. The objectives alternate during pretraining, with global learning using batch-level negatives.
- General framework: Galileo adapts a latent prediction framework in which online and exponential-moving-average target encoders process two views of each sample.
- Learning global features: With a minibatch size of 1, PatchDiscB is equivalent to PatchDiscI.
- Learning global features: The global task uses varied channel groups and masking to learn abstract, lower-frequency features suited for classification.
- Learning global features: PatchDiscB samples negative patches from all instances within a batch, introducing a dependency on batch size.
- Learning local features: The local task samples 5% of tokens for one view and 50% for another, targeting low-level features for fine-grained segmentation.
- Combining local and global objectives: The combined method alternates between the local and global objectives during pretraining.
B.1. A globally sampled pretraining dataset
Galileo’s pretraining dataset is built by globally sampling geographically and thematically diverse tiles, then combining nine remote-sensing inputs spanning sensed imagery and semantically dense maps. An ablation evaluates the contribution of each data product.
- Dataset construction: The dataset construction splits WorldCover into 1000 × 1000-pixel tiles and uses k=150,000 k-means clusters to select representative training points.
- Dataset construction: The resulting sampling procedure constructs a rich pretraining dataset from globally distributed tiles.
- Dataset composition: The dataset contains 9 remote-sensing inputs, ranging from Sentinel-2 optical imagery to Dynamic World landcover maps.
- Dataset ablation: Table 11 ablates the dataset by omitting one data product at a time and measures downstream performance on MADOS, Sen1Floods11, CropHarvest, and EuroSat.
B.2. Implementation
The models use standardized self-supervised training and evaluation configurations, with training duration and hyperparameters varying between ablations and final Galileo models.
- Training setup: All models are trained on single H100 GPUs with an effective batch size of 512 and spatial data augmentations.Augmentations include random vertical and horizontal flips and 90-degree rotations.
- Optimization: Training uses bfloat16 precision, AdamW optimization, gradient clipping, learning-rate warmup, cosine cooldown, and an EMA target encoder.The target-encoder momentum starts at 0.996 and increases linearly to 1 during pretraining.
- Ablations: Ablations pretrain ViT-Tiny models for 200 epochs at a maximum learning rate of 2 × 10−3 and weight decay of 0.02.The weight-decay setting is specified in the continuation of the ablation-training description.
- Final models: Final Galileo models are pretrained for 500 epochs using architecture-specific sweeps over learning rates and weight decays.The ViT-Base sweep covers a wider learning-rate range than the ViT-Nano and ViT-Tiny sweeps.
- Evaluation protocol: Pretrained models are compared under identical evaluation conditions using representations as keys and values for kNN probing with k = 20.The evaluation setup follows the cited kNN implementation and applies the same conditions across pretrained models.
C.2. Evaluation Datasets
Evaluation spans GeoBench-modified image datasets, pixel time-series datasets, and segmentation settings, with standardized preprocessing and validation-based hyperparameter selection.
- Evaluation Datasets: The evaluation includes six GeoBench-modified datasets with shared train, validation, and test splits plus 1%, 5%, and 20% training partitions.The datasets are m-Eurosat, m-BigEarthNet, m-So2Sat, m-Brick-Kiln, m-Cashew-Plant, and m-SA-Crop-Type.
- Evaluation Datasets: The benchmark suite also includes Breizhcrops and CropHarvest pixel time-series tasks, alongside MADOS and PASTIS datasets.Breizhcrops uses regional train, validation, and test splits, while CropHarvest contains three crop-related classification tasks.
- Preprocessing: Input images are resized to each model’s pretraining size, and feature scaling is treated as an additional hyperparameter.The protocol follows evidence that input size and feature scaling can affect pretrained remote-sensing-model performance.
- Evaluation protocol: Reported results are obtained by sweeping the cross product of downstream hyperparameters using validation sets, with linear probes repeated five times and averaged.Feature scaling is selected during the first linear-probe run and fixed thereafter.
- Evaluation protocol: CropHarvest uses pretraining feature-scaling methods and default scikit-learn regression settings because its small training sets lack validation splits.This exception avoids validation-based hyperparameter selection for those tasks.
- Evaluation protocol: Feature normalization or standardization is evaluated using statistics from downstream data and several model-pretraining datasets.The candidate statistics include SatMAE, SSL4EO, Galileo, and Presto pretraining statistics.
D. Results
Results are reported across image classification, segmentation, and time-series evaluations, including analyses of patch-size effects, inference cost, and aggregate model rankings.
- D. Results: Patch-size analyses report Galileo performance on MADOS and m-Eurosat across different training-set percentages.The m-Eurosat analysis also records MACs per instance to expose performance–inference-cost trade-offs.
- D. Results: The reported result tables include full image-classification and segmentation results, with selected patch-size settings used in the main figures and tables.A patch size of 4 is used for the cited MADOS results in Tables 5 and 17.
- D. Results: Model rankings are computed by averaging each model’s rank across datasets and partitions.The aggregate rankings combine image-classification and image-segmentation results into an overall image ranking, alongside time-series rankings.
- D. Results: Image-classification results are reported through kNN and finetuning, while segmentation results are reported through linear probing.The tables report test performance in percentage units.