Source-linked AI summary
Reuse, Don't Retrain: Efficient, Modality-Versatile Geospatial Foundation Models through Feature Distillation
Mario Hamon, Pierre Adorni, Minh-Tan Pham
TL;DR
GeoFMs can require repeated costly pretraining across backbone sizes and sensor modalities. This paper reuses frozen teacher representations through feature distillation, including an unpaired multi-expert student. The resulting students improve the released TerraMind accuracy–compute frontier, and TEDI-Large achieves the highest single-model average on PANGAEA-9.
Problem
GeoFM families often repeat costly pretraining across backbone sizes and sensor-specific models, and systematic comparisons of this scale-specific pretraining practice have been lacking.
Source: §1 · p. 1Method
The paper distills patch-level features from frozen CROMA and TerraMind teachers and extends the approach to TEDI, which routes unpaired modality inputs through adapters into a shared trunk.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Frozen GeoFM representations can be reused to derive efficient model families and consolidate independently pretrained modality experts in one backbone.
Source: §6 · p. 8, §7 · p. 8PANGAEA-9 is a fixed subset rather than the full benchmark, and equally weighted mean mIoU can hide task-specific regressions.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Geospatial foundation models (GeoFMs) are costly to pretrain, yet model families often repeat this process across backbone scales and sensing modalities. We investigate feature distillation as a more efficient way to reuse existing representations. First, we distill frozen CROMA and TerraMind teachers into smaller Vision Transformers through patch-level feature matching. Across nine PANGAEA tasks, distilled TerraMind Tiny, Small, and Base outperform their scale-matched released counterparts by 2.76, 3.02, and 1.87 mean IoU points, respectively; each also matches or exceeds the next-larger released model. Distilled CROMA-Base gains 2.05 points. TerraMind-Base required only 43 A6000 GPU-hours, approximately 0.5\% of the reported cost of independent pretraining. Second, we introduce TEDI, an unpaired multi-expert approach combining TerraMind for multispectral imagery and DINOv3 for high-resolution RGB. Separate patch embeddings route unpaired inputs into a shared Transformer trunk, removing the need for co-registered data. TEDI-Large achieves the highest single-model mean in our PANGAEA-9 comparison (62.51 mIoU), outperforming both teachers overall while retaining their modality-specific strengths. TEDI-Base reaches 61.75 mIoU with 72\% fewer parameters. These results show that distillation improves the GeoFM accuracy-compute frontier while consolidating complementary experts into a single versatile backbone.
1. Introduction
GeoFMs face redundant costs across backbone scales and sensor modalities: applying large encoders to continent-scale archives is costly, while heterogeneous sensors often require separate models. This paper studies reusing frozen pretrained representations through feature distillation, including a shared-trunk approach for independent modality inputs.
1. Introduction
- GeoFM families repeat costly pretraining across model sizes and sensor-specific models, despite the high cost of applying large encoders to continent-scale archives.Separate checkpoints may be needed for multispectral, radar, and high-resolution optical imagery.Source: §1 · p. 1
- Whether scale-specific GeoFM pretraining is necessary has not been established through a systematic comparison.CROMA and TerraMind release multiple sizes obtained through scale-specific pretraining.Source: §1 · p. 1
- The paper reuses frozen teacher representations through patch-level matching, extending from single-teacher students to modality-specific adapters around a shared Transformer trunk.The TEDI training images need not be paired or co-registered.Source: §1 · pp. 1–2
- Distilled TerraMind Tiny, Small, and Base outperform their scale-matched released counterparts and establish a stronger accuracy–compute frontier.The study also evaluates distilled CROMA models and combines TerraMind with DINOv3 in TEDI.Source: §1 · p. 2
2. Related Work
Prior GeoFM work builds versatile encoders through broad pretraining, while this paper asks whether existing specialist checkpoints can instead be compressed and consolidated. The related work situates that question within distillation methods and multi-task evaluation.
2.1. Geospatial foundation models
- GeoFM research includes multimodal pretrained families and flexible encoders, while this work asks whether existing specialist checkpoints can be compressed and consolidated without repeating their original objectives.The cited approaches include AnySat, SkySense V2, and Copernicus-FM.Source: §2.1 · p. 2
2.2. Knowledge distillation and model reuse
- Representation distillation transfers features from pretrained vision teachers, while TEDI combines independently pretrained modality experts using independent modality-specific datasets.Unlike the cited DEO approach, TEDI uses two frozen external experts and requires no paired cross-modal sample.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
2.3. GeoFM evaluation
- GeoFM rankings depend on evaluated capabilities and adaptation protocols, motivating heterogeneous multi-task benchmarks rather than reliance on one representative task.The paper uses a fixed nine-task PANGAEA subset and reports three downstream runs for reproduced models.Source: §2.3 · p. 3
3. GeoFM Reuse through Feature Distillation
The method matches student patch representations to frozen teacher features, using modality-specific input and projection adapters with a shared Transformer trunk. It applies this setup to both single-teacher compression and unpaired multi-expert distillation.
3. GeoFM Reuse through Feature Distillation
- The student processes each modality through its patch embedding and shared Transformer, with a training-only projection head matching the corresponding teacher features.Training minimizes mean patch-level cosine distance; projection heads are discarded afterward, leaving the trunk and required patch embeddings.Source: §3 · pp. 3–4
3.1. Deriving an efficient model family
- With one teacher, the objective reduces to single-teacher feature distillation, studied across CROMA and TerraMind teacher families.The comparison is intended to distinguish the general effect from a particular pretraining recipe.Source: §3.1 · p. 4
- CROMA-Large is distilled into the released Base architecture using the same SSL4EO-S12 corpus, creating an architecture- and corpus-matched comparison.TerraMind-Large is distilled into Base, Small, and Tiny students, but these use RoPE and GELU blocks rather than TerraMind’s native positional encoding and gated MLPs.Source: §3.1 · p. 4
3.2. Unpaired multi-expert distillation
- TEDI routes independent modality samples through separate patch embeddings and projection heads while sharing all Transformer blocks across student variants.Each sample is matched only to its corresponding frozen teacher, so cross-modal correspondence is unnecessary; the training batch contains equal amounts from each modality.Source: §3.2 · p. 4
- The reported comparison table covers mean test mIoU on PANGAEA-9, with three-run standard deviations for the paper’s entries and no uncertainty estimates for published baselines.A dagger marks fine-tuned entries; other backbones are frozen.Source: Table 1 · p. 4
4. Experiments
Experiments compare single-teacher distillation and unpaired multi-expert aggregation under a fixed PANGAEA-9 evaluation. TEDI-Large leads the single-model average, while TEDI-Base offers a smaller operating point.
4.1. Distillation protocol
- Students train for the same update budget, equivalent to 20 epochs over SSL4EO-S12, and downstream tests use checkpoints selected at 15 epoch-equivalents; teachers remain frozen.Single-teacher and TEDI runs use distinct image sources and batch compositions, with equal teacher weighting in TEDI.Source: §4.1 · pp. 4–5
4.2. Evaluation protocol
- PANGAEA-9 fixes five multispectral and four high-resolution tasks, excluding BioMassters for reproducibility and Crop Type Mapping–South Sudan for high inter-run variability.All encoders use the same subset; final comparisons report test scores after configurations are fixed.Source: §4.2 · p. 5
4.3. Efficient model-family derivation
- +1.87 for Base, +3.02 for Small, and +2.76 for Tiny: distilled TerraMind improves mean test IoU at every matched scale, with each student reaching or exceeding the next-larger released model.The Table 2 comparison reports mean test mIoU on PANGAEA-9; gains compare matched backbone sizes.Source: §4.3 · p. 5, Table 2 · p. 5
- 2.05 points: CROMA-KD-Base improves over released CROMA-Base after 15 distillation epoch-equivalents versus 300 pretraining epochs, using the same architecture and corpus.The passage describes this as a 20-times reduction in training length.Source: §4.3 · p. 5
4.4. Unpaired multi-expert aggregation
- 62.51 ± 0.83: TEDI-Large has the highest single-model average, matches TerraMind’s MS mean within 0.05 points, and is 0.14 below DINOv3 on HR.TEDI-Base scores 61.75 ± 0.45, 0.76 points lower with 72% fewer parameters.Source: §4.4 · p. 5, Table 3 · p. 6
4.5. System cost
- TEDI-Base uses 86M parameters and 24.6 GFLOPs, while TEDI stores 344MB—86% less model storage and 71% fewer routed FLOPs than deploying both Large experts.Projection heads are removed after distillation; both input embeddings remain stored, but only the selected embedding is evaluated for each image.Source: §4.5 · p. 6
- 63.75 hours on one RTX A6000 for TEDI-Base compares with TerraMind’s reported 9,216 A100 GPU-hours for pretraining TerraMind-Base, roughly 145 times fewer reported accelerator-hours.The passage qualifies this as a marginal-cost comparison, not a hardware-normalized training-efficiency experiment.Source: §4.5 · p. 6
5. Analysis and Ablations
The ablations show that widening TEDI’s shared token representation improves its performance, while adding a second expert entails a multispectral trade-off. Validation also indicates diminishing gains from longer distillation and no resolved difference between using 10% and 100% of SSL4EO-S12.
5.1. Training extent and data diversity
- Validation performance rises from approximately 61.1 at epoch 1 to 61.7 at epoch 15, then remains nearly unchanged at epoch 20.From epoch 5 onward, means differ by at most 0.2 points relative to comparable run-to-run variability; epoch 15 was selected as the highest validation mean before test results were accessed.Source: §5.1 · p. 6, Figure 3 · p. 7
- With updates fixed, validation rises from approximately 60.7 at 1% of SSL4EO-S12 to 61.6 at 10% and 61.7 at 100%.The runs indicate that 1% is insufficient but do not resolve a difference between 10% and 100%; SSL4EO-S12 is a geographically narrower subset of TerraMind’s pretraining data.Source: §5.1 · p. 6, Figure 3 · p. 7
5.2. What does the second expert contribute?
- Adding DINOv3 reduces TerraMind-KD’s MS mean by 1.29 points, while TEDI is 0.81 points above DINOv3-KD on HR tasks.Three runs do not establish a precise positive-transfer gain; independent students reach 62.11 overall, only 0.36 above TEDI, but require two 86M-parameter backbones. The ablation combines the second teacher with its corpus, so it cannot separate teacher knowledge from data diversity.Source: §5.2 · pp. 6–7, Table 5 · p. 7
5.3. Allocating capacity to token width
- 0.57 points: TEDI-B1024 improves on TEDI-B and comes within 0.19 points of TEDI-L with 70% fewer parameters and 58% fewer standardized GFLOPs.The wider-token variant adds 4M parameters and closes 75% of the TEDI-B–TEDI-L gap; corresponding single-expert B1024 controls show no significant score improvement.Source: Table 6 · p. 8, §5.3 · p. 8
6. Discussion and Limitations
The discussion presents distillation as a way to reuse GeoFM representations across model scales and modalities, while noting limits to the comparisons and evaluation scope.
Discussion
- CROMA provides a matched architecture-and-corpus comparison, whereas TerraMind students also change positional encoding, feed-forward design, and capacity allocation.The TerraMind gains are consistent with the CROMA result but cannot be attributed to distillation alone.Source: §6 · p. 8
Unpaired expert consolidation
- TEDI trains on independent Sentinel-2 and HR corpora, so it does not require paired, co-registered observations and can retain each expert’s native channels and resolution.The paper describes this as a route to consolidate experts even when their original pretraining corpora differ.Source: §6 · p. 8
Experimental scope
- The evaluation covers a fixed PANGAEA-9 subset and two experts across two input regimes; mean mIoU can hide task-specific regressions.The teacher-multiplicity ablation combines a second expert with a second image corpus, and three downstream runs limit conclusions about small differences.Source: §6 · p. 8
Data and release
- SSL4EO-S12 and FMoW do not represent every sensor, temporal regime, or sparsely observed region, and TEDI inherits its teachers’ biases.FMoW emphasizes functional sites and human infrastructure.Source: §6 · p. 8
7. Conclusion
The paper concludes that feature distillation can derive efficient GeoFM families and consolidate modality-specific experts. On PANGAEA-9, TEDI-Large achieves the highest single-model average, while TEDI-Base retains most of that performance with a smaller backbone.
Conclusion
- Feature distillation addresses two practical GeoFM problems: deriving efficient backbone families and consolidating modality-specific experts.The paper applies the approach to frozen teachers and a shared-backbone multi-expert extension.Source: §7 · p. 8
- On PANGAEA-9, TEDI-Large achieves the highest single-model average and matches both specialist domain means.TEDI-Base retains most of this performance with a substantially smaller backbone.Source: §7 · p. 8
- Reallocating Base capacity toward a 1024-dimensional token space further closes most of the remaining gap to TEDI-Large with a modest parameter increase.The paper frames the findings as support for pretraining strong experts once and reusing their representations across required scales and modalities.Source: §7 · p. 8