Source-linked AI summary
A Composition-Aware Pretraining Framework for Geospatial Foundation Models
Aryan Kashyap Naveen, Abhishek Srinivas, Pranav Moothedath, Shrutilipi Bhattacharjee
TL;DR
Existing geospatial foundation-model pretraining does not explicitly represent the fractional mixtures that characterize satellite scenes. This paper introduces composition-target histogram prediction with EMD-based distillation, improving region-level retrieval and classification while remaining competitive on fine-grained tasks. On ForestNet-12, mAP@10 rises from 0.279 to 0.434.
Problem
Existing pretraining methods process satellite imagery through a single-concept lens, despite cells containing heterogeneous land-cover mixtures.
Method
The framework maps each cell to a fractional visual-texture histogram and trains a Mamba-based backbone with EMD distillation and MIL auxiliary losses.
Results
The method substantially improves region-level retrieval and scene classification while remaining competitive on fine-grained prediction tasks; ForestNet-12 mAP@10 increases from 0.279 to 0.434.
Takeaways & Limitations
Explicit composition modeling produces discriminative embeddings for semantic similarity judgments involving nuanced land-cover mixtures.
Takeaways & Limitations
The approach currently processes only 3-channel RGB imagery and exhibits a resolution-semantics trade-off on pixel-level tasks.
Abstract
from arXiv · showhide
Geospatial foundation models have emerged as state-of-the-art methods for downstream Earth observation tasks. However, existing pretraining methodologies process imagery through a single-concept lens, failing to capture the highly compositional nature of complex satellite scenes. We propose a composition-aware pretraining framework that explicitly encodes fractional land-cover mixtures. Each satellite image cell is mapped to a histogram representing its fractional land-cover distribution, which we term the "composition target". These targets serve as the primary prediction objective and are distilled into the backbone using Earth Mover's Distance. Experimental evaluation shows that composition-aware pretraining yields substantial gains on region-level understanding tasks requiring semantic similarity judgment, including zero-shot image retrieval and scene classification, while remaining competitive on tasks requiring fine-grained spatial precision, such as segmentation and object detection. With a 36.8M-parameter backbone, our framework outperforms SatMAE and Prithvi-EO-2.0, which contain 303M and 600M parameters, respectively, in most retrieval and scene classification settings. On the fine-grained ForestNet-12 dataset, a rigorous testbed for compositional discrimination, our method boosts baseline mAP@10 from 0.279 to 0.434, a 55.6% relative improvement, providing direct evidence for the effectiveness of explicit composition modeling. The code implementation can be found at https://github.com/05kashyap/GFM_Composition_Pretraining
1 Introduction
Satellite scenes contain mixtures of land-cover concepts that existing single-concept pretraining paradigms do not explicitly model. The proposed framework predicts fractional visual-texture histograms to improve region-level understanding.
- Motivation: Satellite cells commonly mix multiple land-cover types rather than depicting one monolithic concept.Examples include coastal-port scenes combining water, docks, ships, roads, and roofs.
- Motivation: Patch reconstruction can overemphasize redundant background textures, while global contrastive learning can collapse heterogeneous mixtures into diluted holistic embeddings.Different scenes may therefore produce similar pooled vectors when their average spectral content aligns.
- Proposed Framework: The framework maps each image cell to a histogram over localized visual textures, explicitly representing their presence and proportions.The composition target is distilled into a Mamba-based backbone using EMD, with MIL auxiliary losses.
- Contributions: The method reports improvements of up to 56% in retrieval mAP@10 and 7% in scene-classification accuracy.These gains target tasks requiring region-level semantic understanding.
2 Background and Related Work
Geospatial foundation models have advanced through large-scale representation learning, but established reconstruction and global contrastive approaches remain misaligned with compositional satellite scenes. Related work motivates localized modeling that preserves fine-grained mixtures.
- Foundation Models: Remote-sensing foundation models learn generalizable representations from large volumes of imagery using transformer or state-space architectures.These models support a shift from task-specific remote-sensing systems toward generalized representations.
- Masked Image Modeling: Masked autoencoding methods reconstructing satellite patches can over-index on redundant homogeneous backgrounds instead of extracting semantic information.This creates a mismatch between pixel-level reconstruction and Earth-observation scene structure.
- Contrastive Learning: Global contrastive methods align image-level representations but can dilute localized details by collapsing multi-class spatial compositions into one embedding.The limitation is especially relevant when a cell contains several land-cover concepts.
3 Methodology
The methodology constructs patch-level pseudo-labels and composition targets before pretraining a geospatial foundation model to predict those targets.
- Methodology: The framework first creates a patch-level pseudo dataset, then generates quantitative composition targets, and finally performs foundation-model pretraining.This sequence defines the three main stages of the proposed methodology.
3.1 Pretraining Dataset Creation
The pretraining dataset is converted from satellite cells into localized, context-aware patch embeddings and normalized histograms over a learned visual vocabulary. These histograms serve as composition targets encoding fractional texture mixtures.
- Pretraining Dataset Creation: 512 × 512 satellite cells are divided into patches whose ViT embeddings are clustered and soft-assigned to generate composition targets.The pipeline shifts representation from whole images toward localized visual patches.
- Patch Embeddings: Each cell produces 1024 spatial tokens from 16 × 16 patches, with context-aware representations formed through transformer self-attention.The tokens retain surrounding semantic context, such as distinguishing open ocean from harbor water.
- Patch Embeddings: 2 × 2 average pooling compresses the 1024 tokens into 256 embeddings, each representing a 32 × 32 pixel region.The step reduces storage and clustering cost while preserving discriminative spatial semantics.
- Visual Vocabulary: K-Means clusters roughly 93 million patch embeddings into 512 visual-texture concepts using a uniformly sampled subset of 5 million normalized tokens.The resulting centroids form the global visual vocabulary.
- Composition Targets: Each patch distributes probability mass across centroids through soft assignment, reducing discretization errors between semantically adjacent clusters.Assignments are based on cosine similarity and an RBF kernel rather than a single nearest centroid.
- Composition Targets: The 256 patch assignments are summed per centroid and L1-normalized into a 512-dimensional composition target.Each histogram entry represents the total assignment mass received by one visual-texture centroid.
3.2 Foundation Model Training
Foundation model training combines composition-target prediction, bidirectional feature alignment, and detached auxiliary classification to teach DynamicVis compositional and semantic representations.
- The online objective uses raw images, offline histogram targets, and weak image-level annotations to train the backbone.
- DynamicVis processes high-resolution imagery with progressive downsampling and sparse routing to preserve spatial detail while focusing computation on salient tokens.
- The BoVW head predicts a 512-bin composition distribution and compares it with the target histogram using Earth Mover’s Distance over vocabulary-cluster distances.
- The MIL branch contrasts regional visual embeddings with categorical meta-embeddings using positive and negative pairs from weak region-level annotations.
- The auxiliary classification branch detaches backbone features so its loss updates only the auxiliary mapping and cannot dominate backbone training.
- The total loss weights compositional EMD, MIL, and classification terms, prioritizing fine-grained composition while constraining embedding topology.
4 Results and Analysis
The evaluation spans region-level understanding, instance-level perception, and pixel-level prediction while freezing backbone weights and training only lightweight task-specific heads.
- Downstream evaluation covers region-level understanding, instance-level perception, and pixel-level dense prediction tasks.
- Frozen backbone weights and lightweight task-specific heads make reported performance reflect representational quality rather than fine-tuning capacity.
4.1 Pretraining Pipeline Analysis
The pretraining pipeline uses large-scale fMoW imagery, converts cells into localized patches, and analyzes whether the learned vocabulary captures diverse geographic features and stable training behavior.
- 4.1 Pretraining Pipeline Analysis: The pipeline analysis validates the visual vocabulary and training stability before downstream evaluation.
- 4.1.1 Pretraining Dataset: fMoW supplies more than one million high-resolution satellite images from over 200 countries across 62 classes.
- 4.1.2 Visual Vocabulary: Figure 5 overlays diverse centroid activations, showing clusters focused on distinct localized geographical features.
- 4.1.1 Pretraining Dataset: Images are tiled into 512 × 512 cells and subdivided into 256 patches of 32 × 32 pixels to isolate localized geospatial textures.
- 4.1.2 Visual Vocabulary: A frozen DINOv3 encoder and K-Means clustering with K=512 create a visual vocabulary of localized texture concepts.
- 4.1.3 Training Stability: The multi-branch loss was necessary for stability, with Sinkhorn EMD converging smoothly alongside MIL and classification losses.
4.2 Evaluation on Region-Level Understanding
With frozen backbones and lightweight task-specific heads, the composition-aware model improves region-level retrieval and scene classification, especially on compositional imagery.
- Zero-Shot Image Retrieval: 20.5% relative improvement in AID mAP@10, from 0.670 to 0.808, demonstrates stronger zero-shot retrieval.Gains occur across retrieval cutoffs.
- Zero-Shot Image Retrieval: 0.279 to 0.434 mAP@10 on ForestNet-12 shows nearly doubled retrieval performance on fine-grained land-cover mixtures.ForestNet imagery contains mixtures such as forest, agriculture, infrastructure, and clearing.
- Zero-Shot Image Retrieval: The model retrieves compositionally relevant AID and ForestNet scenes, including Port, Park, and Oil Palm Plantation examples.Figure 7 marks correct matches with green frames and incorrect matches with red frames.
- Scene Classification: 0.928 Accuracy on UC Merced exceeds the DynamicVis-B baseline of 0.864 for whole-image scene classification.The evaluation uses frozen pooled features and a lightweight MLP head.
- Scene Classification: 81.60% accuracy on NWPU-RESISC45 exceeds DynamicVis-B's 0.692 but remains below SatMAE's 0.829.The benchmark contains 45 scene categories.
4.3 Evaluation on Instance-Level Perception and Pixel-Level Dense Prediction
Composition-aware pretraining remains competitive for dense and instance-level perception, but its cell-level mixture emphasis trades some boundary precision for region-level semantic representation.
- Change Detection: 0.806 F1-score and 0.680 IoU are achieved on LEVIR-CD change detection with frozen backbones.DynamicVis achieves slightly higher spatial precision on this task: F1 0.805 and IoU 0.690.
- Cross-Task Trade-off: Cell-level composition objectives favor semantically rich distributional summaries over sharp high-frequency boundaries, yielding region-level gains with modest pixel-level costs.This trade-off explains the competitive but slightly lower spatial precision on dense prediction tasks.
- Tiny Ship Detection: The composition-aware model achieves mAP@0.50 of 0.619 on LEVIR-Ship tiny object detection.It outperforms the DynamicVis baseline across both IoU thresholds but is consistently outperformed by the large variant.
- Semantic Segmentation: On WHU building segmentation, the DynamicVis baseline produces fewer false positives, consistent with stronger fine-grained spatial-topology preservation.The evaluation uses UperNet with multi-scale frozen-backbone features.
4.4 Architecture Analysis
The ablations show that vocabulary size controls composition-target granularity, while EMD distillation is the central contributor to representation quality.
- 4.4.1 Vocabulary Size: Small vocabularies collapse distinct textures, whereas excessively large vocabularies over-partition embeddings and create sparse composition targets.Vocabulary size therefore determines the granularity and strength of the training signal.
- 4.4.1 Vocabulary Size: k=512 achieves the highest AID mAP@10 of 0.761 ± 0.008 among the evaluated vocabulary sizes.The study evaluates k∈{32, 64, 128, 256, 512, 1024}; k=1024 reduces mAP@10 by −0.040 relative to k=512.
- 4.4.2 Loss Formulation Ablation: The loss ablation evaluates seven configurations combining EMD, MIL, and CLS terms on AID mAP@10 and ForestNet-4 mAP@10.The compositional distillation weight λ_emd is set at a 2:1 ratio to λ_mil, while λ_mil and λ_cls retain the DynamicVis ratio.
- 4.4.2 Loss Formulation Ablation: Removing EMD while retaining MIL and CLS causes a −0.148 drop in AID mAP@10.The reported gap between EMD-inclusive and EMD-absent configurations is substantial and consistent across both datasets.
- 4.4.2 Loss Formulation Ablation: EMD is identified as the primary driver of representation quality, and auxiliary MIL and CLS branches cannot substitute for it.This result supports histogram distillation as the core source of compositional discriminability.
5 Conclusions and Future Work
The framework explicitly represents heterogeneous land-cover mixtures, strongly benefiting region-level understanding while preserving competitive fine-grained performance. Future work targets broader architectures and modalities, learnable vocabularies, and multi-resolution targets to improve dense prediction.
- 5 Conclusions and Future Work: The composition-aware prior incurs only a minimal performance cost on change detection and semantic segmentation while preserving localized spatial features for instance- and pixel-level prediction.The framework favors holistic mixture representations over precise boundary information, creating a representational trade-off.
- 5 Conclusions and Future Work: Explicit fractional-composition modeling most benefits zero-shot image retrieval and other region-level understanding tasks.On ForestNet-12, mAP@10 rises from 0.279 to 0.434; on AID, it improves from 0.670 to 0.808, a 20.5% relative gain.
- 5 Conclusions and Future Work: Future evaluations should extend the method beyond the DynamicVis state-space model to standard Vision Transformers and modern CNNs.This would test whether the pretraining methodology generalizes across architectures.
- 5 Conclusions and Future Work: The current framework processes only 3-channel RGB imagery, motivating extensions to multispectral and hyperspectral data.Such extensions are proposed as a way to capture richer compositional information.
- 5 Conclusions and Future Work: Learnable vocabularies and multi-resolution histogram targets are proposed to replace static clustering and recover fine-grained spatial precision.Multi-resolution targets are intended to bridge dense-prediction performance gaps without compromising region-level representations.