Source-linked AI summary
Shortcut Learning in Geospatial Foundation Model Embeddings for Cross-Disaster Damage Assessment
Ali Shibli, Andrea Nascetti, Yifang Ban
TL;DR
Reliable disaster-damage prediction must transfer to events absent from training, a condition random tile splits do not test. The paper audits context and image-pair signals and evaluates source-only transformations; held-out-event performance drops under LOEO, while transformations improve it.
Problem
Random image-level splits can place tiles from the same disaster in training and test, so they do not establish whether geospatial foundation model predictions transfer to an unseen event.
Source: §1 · p. 1Method
Results
Under LOEO, full-pair BA falls from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT, while source-only feature transformations improve unseen-event performance.
Source: §4 · pp. 3–4Takeaways & Limitations
The results support evaluating damage models on held-out events and using image-free, temporal-view, and correspondence-breaking controls.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Geospatial foundation models can support disaster damage assessment, but high accuracy on familiar disasters may hide shortcut learning: prediction from event or scene characteristics rather than evidence that transfers to a new disaster. We study embeddings from TerraMind, Prithvi-EO-2.0, DOFA, and DINOv3 on the multi-disaster datasets xView2 and BRIGHT. Holding out an entire disaster, instead of randomly splitting image tiles, reduces mean balanced accuracy from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT. Four sets of tests examine what information supports prediction: event and acquisition context, before/after image inputs, building count, and correct image pairing. Pre-event images remain predictive, and a classifier given only the annotated number of buildings reaches balanced accuracy of 0.629 and 0.653. Combining before/after features performs best on average on xView2, but does not improve over pre-only features on BRIGHT. Shuffling image pairs within the same disaster and damage class causes modest losses. Motivated by these findings, we test whether feature transformations learned only from training disasters improve generalization to unseen disasters. These transformations improve performance, with supervised contrastive learning performing best on xView2 and PCA on BRIGHT. Our results highlight the importance of evaluation on unseen domains for assessing model robustness and show that simple feature-level transformations can improve generalization. Code and experiments can be found at https://github.com/alishibli97/gfm-shortcut-learning.
1 Introduction
The paper asks whether geospatial foundation model predictions transfer to disasters absent from training, since random tile splits can share event-specific cues across train and test. It audits context, temporal views, building counts, and image pairing, then tests source-only transfer methods.
1 Introduction
- The study compares random image-level splits, where events overlap, with leave-one-event-out evaluation and tests source-only transformations for unseen-event prediction.No held-out-event sample is used for training or feature transformation.Source: §1 · p. 2
- The event/context audit tests whether event prevalence, acquisition metadata, and building count predict damage.These are image-free controls that help identify available predictive signals.Source: §1 · p. 2
- The temporal-view and pairing audit examines pre-, post-, and change information and the effect of breaking image correspondence.The contribution description reports dataset-dependent use of these signals and only modest losses after correspondence is broken.Source: §1 · p. 2
- Source-only controls test event–class reweighting, compression, and supervised representation learning for held-out-disaster prediction.The contribution description reports little effect from reweighting and improved prediction from compression and supervised representation learning.Source: §1 · p. 2
2 Related Work
Prior work supplies reusable Earth-observation representations and studies shortcut learning and cross-disaster transfer. This paper adds controlled tests of context and image-pair information, emphasizing evaluation across unseen disasters.
2 Related Work
- Prithvi-EO-2.0, TerraMind, and DOFA offer reusable Earth-observation representations, while DINOv3 serves as a large self-supervised vision baseline [Szwarcman et al., 2026, Jakubik et al., 2025, Xiong et al., 2024, Siméoni et al., 2025].Robustness benchmarks nevertheless show sensitivity to real distribution shifts [Doerksen and Kerner, 2026, Li et al., 2025].Source: §2 · p. 2
- Prior work characterizes shortcuts as predictive but potentially unstable signals and studies cross-disaster adaptation [Geirhos et al., 2020, Ye et al., 2024, Mouradi and Kshirsagar, 2026].xBD established paired optical damage assessment, while BRIGHT adds globally distributed optical–SAR pairs [Gupta et al., 2019, Chen et al., 2025].Source: §2 · p. 2
- The paper complements existing work with controlled tests of context and image-pair information; its contribution is evaluation and audit rather than a new training loss.Its transfer baselines use supervised and cross-domain contrastive learning [Khosla et al., 2020, Wang et al., 2021].Source: §2 · p. 2
3 Shortcut Audit and Source-Only Mitigation
The audit evaluates damage prediction with event-held-out probes and isolates context, temporal-view, building-count, and pairing signals. It also tests transformations fitted using source events only.
3 Shortcut Audit and Source-Only Mitigation
- In leave-one-event-out evaluation, samples from one disaster are held out, and scaling, transformation, and probe fitting use only source events.A class-balanced linear probe predicts whether a tile contains at least one damaged building, with metrics averaged over events.Source: §3 · p. 2
- The audit tests event or acquisition context, separate temporal views, annotated building count, and whether correct same-location pre/post pairing matters.Pre-only features contain no disaster impact; post-only and difference features may contain direct damage evidence, and differences are computed between embeddings.Source: §3 · p. 2
- At test time, pre- or post-event features are shuffled within the same event and label to break location matching without reversing time.The probe remains trained on correctly paired source examples, and both shuffle directions are averaged.Source: §3 · p. 3
- Source-event transformations include PCA, a binary-cross-entropy MLP, ordinary SupCon, and cross-event SupCon.Cross-event SupCon allows same-label positive pairs only across different source events, testing whether that restriction adds transfer benefit.Source: §3 · p. 3
4 Experiments
Experiments on xView2 and BRIGHT show a substantial gap between familiar-event and unseen-event performance, alongside dataset-dependent image-input results. Source-only feature transformations improve held-out-event prediction, while audits identify predictive context, scene, and pairing signals.
Data and implementation
- The experiments use 9,168 xView2 pairs from 19 events and 3,029 BRIGHT pairs from 11 events; BRIGHT's repeated-channel optical/SAR input is a compatibility test, not a sensor-optimized pipeline.BRIGHT pairs pre-event RGB optical imagery with post-event SAR imagery.Source: §4 · p. 3
Event and context results
- Full-pair BA falls from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT under LOEO rather than random image-level splitting.The audit table averages results over four backbones.Source: §4 · p. 3, Table 1 · p. 4
- Without images, event-prior AUC reaches 0.821 and 0.715, while metadata LOEO BA is approximately chance at 0.498 and 0.500.The event prior is available only for the random-split setting because unseen events have no training labels.Source: §4 · p. 3, Table 1 · p. 4
Image inputs and pairing
- On xView2, post-only BA 0.617 exceeds pre-only 0.577 and full-pair BA is 0.638; on BRIGHT, pre-only BA 0.571 narrowly exceeds full-pair BA 0.568.Combined inputs therefore help on average in xView2 but not in the BRIGHT pipeline.Source: §4 · p. 3, Table 1 · p. 4
Count and pairing controls
- Count-only reaches BA 0.629 and 0.653; shuffling pairs lowers BA from 0.638 to 0.617 on xView2 and from 0.568 to 0.552 on BRIGHT.The pairing loss is modest when event and damage class are preserved.Source: §4 · p. 4, Table 1 · p. 4
Transfer improvements
- xView2 SupCon reaches BA 0.664/AUC 0.730 and BRIGHT PCA reaches BA 0.594/AUC 0.650; event–class weighting barely changes BA, and cross-event SupCon scores slightly lower than ordinary SupCon on both datasets.The table reports source-only LOEO transfer, averaging events and backbones; rankings are descriptive.Source: §4 · p. 4, Table 2 · p. 4
5 Conclusion
The conclusion cautions that familiar-disaster accuracy does not guarantee reliable prediction on a new disaster. The authors recommend held-out-event evaluation and controls, while identifying building-level severity and native sensor inputs as future work.
5 Conclusion
- High accuracy on familiar disasters does not guarantee reliable prediction on a new disaster.The conclusion recommends held-out-event evaluation and image-free, temporal-view, and correspondence-breaking controls.Source: §5 · p. 4
- Event statistics, scene information, and building count predict tile-level damage, while correct pre/post matching provides a modest benefit.The conclusion also reports that feature transformations improve unseen-event performance, but cross-event SupCon does not outperform ordinary SupCon or PCA.Source: §5 · p. 4
- Future work should evaluate building-level severity and native sensor inputs.Source: §5 · p. 4
6 Acknolwedgement
The work acknowledges computational support from NAISS, funded by the Swedish Research Council.
6 Acknolwedgement
- NAISS provided computational resources; it is funded by the Swedish Research Council.Source: §6 · p. 4