Source-linked AI summary
Cryo-Bench: Benchmarking Foundation Models for Cryosphere Applications
Saurabh Kaushik, Lalit Maurya, Beth Tellman, Valerio Marsocci
TL;DR
Cryosphere-focused GFM benchmarking has been limited by the lack of suitable evaluation datasets. The paper introduces Cryo-Bench and evaluates 14 GFMs against UNet and ViT baselines across cryosphere components, sensors, and regions. Results show varied model strengths, strong sparse-label performance, and substantial benefits from learning-rate tuning during fine-tuning.
Problem
Cryosphere evaluation of GFMs has been limited because existing suitable evaluation datasets were lacking.
Method
Cryo-Bench benchmarks 14 GFMs against UNet and ViT baselines across five cryosphere segmentation datasets spanning multiple components, modalities, and geographies.
Results
GFMs show varied cross-dataset performance, retain strong results with sparse labels, and improve substantially when learning-rate tuning accompanies fine-tuning.
Takeaways & Limitations
DOFA and TerraMind perform well across Cryo-Bench, supporting the use of tuned encoder fine-tuning for best performance and frozen encoders for quick results.
Takeaways & Limitations
The authors identify the need for cryosphere-specific foundation models and temporally rich cold-region training data.
Abstract
from arXiv · showhide
Geo-Foundation Models (GFMs) have been evaluated across diverse Earth observation task including multiple domains and have demonstrated strong potential of producing reliable maps even with sparse labels. However, benchmarking GFMs for Cryosphere applications has remained limited, primarily due to the lack of suitable evaluation datasets. To address this gap, we introduce \textbf{Cryo-Bench}, a benchmark compiled to evaluate GFM performance across key Cryospheric components. Cryo-Bench includes debris-covered glaciers, glacial lakes, sea ice, and calving fronts, spanning multiple sensors and broad geographic regions. We evaluate 14 GFMs alongside UNet and ViT baselines to assess their advantages, limitations, and optimal usage strategies. With a frozen encoder, UNet achieves the highest average mIoU of \textbf{66.38}, followed by TerraMind at \textbf{64.02} across five evluation dataset included in Cryo-Bench. In the few-shot setting (10\% input data), GFMs such as DOFA and TerraMind outperform UNet, achieving mIoU scores of \textbf{59.53}, \textbf{56.62}, and \textbf{56.60}, respectively, comapred to U-Net's 56.60. When fully finetuning GFMs, we observe inconsistent performance across datasets and models. However, tuning learning rate along with finetuning substantially improves GFM performance. For example, evaluation on two representative datasets (GLID and CaFFe) shows an average relative improvement of \textbf{12.77\%}. Despite having minimal Cryosphere representation in their pretraining data, GFMs exhibit notable domain adaptation capabilities and produce meaningful results across tasks. Based on our findings, We recommend encoder fine-tuning with hyperparameter optimization optimization to achieve the best possible performance, while using frozen encoders when users need quick results without extensive experimentation.(\href{https://github.com/Sk-2103/Cryo-Bench}{GitHub}).
1 Introduction
Cryosphere applications lacked suitable datasets for systematic GFM evaluation. Cryo-Bench addresses this gap by benchmarking GFMs against conventional baselines and examining representation quality, sparse-label performance, and tuning strategies.
- Cryosphere-focused evaluation of GFMs has remained unexplored because suitable datasets were lacking.
- The paper asks whether GFMs encode cryosphere-relevant features and how architecture, pretraining data, sparse labels, fine-tuning, and hyperparameter optimization affect performance.
- Cryo-Bench enables direct and systematic evaluation of GFMs for cryosphere applications.
- The benchmark evaluates 14 GFMs against UNet and ViT baselines using a common evaluation protocol.
- The study provides practical guidance on model selection under varying data-availability and computational-resource constraints.
2 Related Work
GFMs use self-supervised, task- and geography-agnostic representations across sensors and modalities, while existing benchmarks cover many Earth-observation domains but overlook the Cryosphere.
- GFMs shift Earth observation toward task- and geography-agnostic representations learned from large unlabeled remote-sensing collections.
- Contrastive learning and self-distillation support multimodal feature representations in models such as Galileo, CROMA, DINO, and DINO-MM.
- RAMEN allows users to adjust modality, spatial resolution, and temporal resolution according to desired detail and computational constraints.
- GEO-Bench, Pangaea, and SustainBench evaluate GFMs across broad domains, modalities, and sustainability-related applications.
- Existing evaluation datasets overlook the Cryosphere, whose components are difficult to map because of mixed and spectrally similar signals.
3 Cryo-Bench evaluation dataset
Cryo-Bench is a multimodal, geographically diverse benchmark for cryosphere segmentation, designed around five curation criteria and tasks that require domain adaptation.
- Cryo-Bench was curated using diverse components, geographies, sensors, peer-reviewed results, and open-access data as selection criteria.
- The benchmark includes debris-covered glaciers, glacial lakes, sea ice, and calving fronts across cryosphere applications.
- Most GFMs lack sea-ice and calving-front pretraining, so these tasks test adaptation to unseen domain-specific applications.
- Even models with limited snow or ice exposure require downstream domain adaptation because pretraining does not distinguish glacial from seasonal ice.
- Five semantic-segmentation datasets span RGB, multispectral, and single-band SAR imagery across global and regional locations.
4 Model Selection and Experiment design
The experiments follow the Pangaea protocol to compare GFMs under frozen-encoder, few-shot, cross-sensor, and learning-rate-tuned fine-tuning settings, while also measuring computational cost.
- The study evaluates GFMs across data-limited settings and cross-sensor scenarios using the Pangaea evaluation protocol.
- Frozen encoders extract features for a shared trainable UperNet decoder, with consistent optimization settings across GFMs.
- Few-shot experiments use 10% of stratified training samples with standardized 512×512 inputs.
- Learning rates of 1e-2, 1e-3, and 1e-5 are compared with the default 1e-4 on GLID and CaFFe.
- GFLOPs are compared with each model’s best mIoU to assess performance–computational-cost trade-offs.
5 Experimental Results
Cryo-Bench results show substantial variation across datasets, data regimes, fine-tuning strategies, and computational costs. Frozen UNet leads on average, whereas GFMs show advantages under limited labels and after learning-rate optimization.
- Frozen Encoder: 66.38 average mIoU makes UNet the strongest frozen-encoder model, followed by TerraMind at 64.02 across five datasets.
- Dataset Variation: 31.48 mIoU is the highest SICD result, while DOFA leads GLID and GLD and TerraMind leads SICD and GSDD.
- Few-Shot Setting: GFMs retain up to 94.2% of full-data performance with 10% labels, compared with 85.3% for UNet.
- Fine-Tuning: Full fine-tuning is non-monotonic: nine of 14 GFMs improve by 0.69–11.61%, while five decline by 2.3–25.19%.
- Learning-Rate Tuning: +7.08% average improvement on GLID and +32.71% across improving CaFFe models show that learning-rate tuning can substantially change fine-tuning outcomes.
- Efficiency: 93.58 mIoU at 61.42 GFLOPs makes DOFA the strongest GLID performance-efficiency result, while RemoteCLIP is the lightest model on both datasets.
6 Discussion
Discussion emphasizes that GFMs are especially useful with sparse labels and can adapt across sensors and domains, but their gains depend strongly on fine-tuning choices. The authors therefore distinguish quick frozen-encoder use from optimized fine-tuning and identify directions for cryosphere-specific models.
- Sparse Labels: GFMs retain up to 94.2% of full-data performance with 10% labels, versus 85.3% for UNet.
- Optimization: Learning-rate optimization improves 13 of 14 GFMs on GLID by an average of 5.37 percentage points and improves CaFFe by 7.3 points on average.
- Pretraining: Pretraining categories show monotonic fine-tuning improvements, with SatlasNet recording the largest gain and other models improving approximately 6–9 mIoU points.
- Interpretation: GFMs generally remain below frozen-encoder UNet performance, while showing cross-sensor and cross-domain adaptation despite limited cryosphere representation in pretraining.
- Future Directions: Future work should develop cryosphere-specific models, temporally rich cold-region data, fine-scale-preserving pretraining, and broader component coverage.
7 Conclusion
The paper introduces Cryo-Bench as a multi-component, multi-sensor benchmark and finds that GFMs can perform meaningfully despite limited cryosphere pretraining. Model choice should depend on sensing modality and computational priorities.
- Benchmark: Cryo-Bench covers debris-covered glaciers, sea ice, calving fronts, and glacial lakes across multiple sensors and broad geographic regions.
- Findings: GFMs demonstrate satisfactory representation learning and strong potential for cryosphere monitoring despite severe cryosphere underrepresentation in pretraining data.
- Model Selection: DOFA and TerraMind perform consistently well, although the best model varies by dataset and sensing modality.
- Practical Guidance: RemoteCLIP is recommended for three-band inputs including SAR, while DOFA offers the best performance-efficiency balance for multispectral inputs.
Supplementary Material
The supplementary material documents dataset geography and image-mask examples, then provides detailed results for fine-tuning, learning-rate tuning, and computational requirements.
- Supplementary Figures: Figures S1 and S2 provide dataset geographic distributions and example image–mask pairs.
- Supplementary Results: Tables S1 and S2 provide detailed full fine-tuning and learning-rate-tuning results.
- Efficiency Analysis: Tables S3 and S4 compare model performance and computational requirements using GFLOPs and latency for GLID and CaFFe.