PaperlayerView exact revision on arXiv

Source-linked AI summary

PiCo: Physics-Informed Concept Probing for Geo-Foundation Models

Yihang She, Andrew Blake, Jianbo Qi, Srinivasan Keshav

arXiv:2610.20973v1cs.CV

TL;DR

Geo-foundation models are commonly evaluated by predictive performance, which gives limited insight into their physical sensitivities or how physical concepts contribute to predictions. PiCo learns concept vectors from controlled radiative-transfer simulations and probes embeddings; its results link physical sensitivities to downstream performance patterns and retain substantial classification accuracy using concept representations.

  • Problem

    Predictive accuracy alone reveals little about which physical features GFMs capture or how those features contribute to predictions, limiting performance hypothesis testing and model selection.

    Source: §I · p. 1

  • Method

    PiCo generates controlled synthetic observations with a 3D radiative-transfer model and game-engine scenes, learns physical concept vectors in GFM embeddings, and uses their projections to assess concept importance.

    Source: §I · p. 2

  • Results

    RGB restriction reduces dry-matter sensitivity and disproportionately degrades dry-vegetation and bare-ground estimation, while concept representations retain 70.0–93.8% of full-embedding accuracy in tree classification.

    Source: §I · p. 2, §VI · p. 11

  • Takeaways & Limitations

    PiCo provides a physical basis for testing domain-informed performance hypotheses and assessing the physical basis of model decisions.

    Source: §VI · pp. 11–12

Abstract

from arXiv · show

Geo-foundation models (GFMs) are typically evaluated by downstream task predictive performance, revealing little about which physical features they are sensitive to or use in prediction. This limits developers' ability to test hypotheses about model behaviour and users' ability to select models for new tasks. We propose Physics-Informed Concept Probing (PiCo), a post-hoc framework for explaining GFM predictions. PiCo identifies concept vectors in a GFM's embedding space by generating synthetic multispectral satellite observations from a 3D radiative transfer model coupled with a game engine. These vectors reveal physical sensitivities that inform domain-based hypotheses about downstream performance. Projecting embeddings onto these concept vectors produces low-dimensional representations for quantifying concept importance in making predictions. We evaluate PiCo on three GFMs with different input modalities using seven physical concepts covering forest leaf optics and canopy structure. In ground-cover estimation, RGB restriction reduces dry-matter sensitivity and disproportionately degrades predictions for dry vegetation and bare ground, supporting a hypothesis derived from sensitivity analysis. For hierarchical forest type and tree species classification, concept representations retain 70.0-93.8% of full-embedding accuracy. Models capture similar physical contrasts between classes, such as structural differences between broadleaf forest and cleared ground and leaf-water-content differences between oak and beech, yet differ in the strength of these contrasts and the concepts driving classification. PiCo thus provides a physical basis for hypothesis testing and model assessment, with potential applicability beyond Earth observation wherever forward models enable controlled simulation and pretrained embeddings are accessible for probing.

I. INTRODUCTION

Geo-foundation models are usually assessed by predictive accuracy, which does not reveal whether their embeddings capture physical features or how those features contribute to predictions. PiCo uses controlled physical simulations to probe model sensitivities and concept importance, and the experiments show that those sensitivities can inform downstream performance hypotheses.

I. INTRODUCTION

  • Accuracy-focused benchmarks leave developers limited means to test performance hypotheses and users limited ability to choose among GFMs using physical features.The paper notes that more than 150 GFMs are available, trained with different modalities and objectives.Source: §I · p. 1
  • PiCo connects physical states to learned representations by treating satellite observations as outputs of forward physical processes, including radiative transfer.Controlled simulations vary physical variables systematically, helping identify concept directions in embedding space.Source: §I · pp. 1–2, Figure 1 · p. 2
  • The study applies PiCo to three GFMs with different input modalities, using ground-cover estimation and hierarchical forest-cover and species classification.The evaluation uses SLATS and TreeSatAI datasets.Source: §I · p. 2
  • RGB restriction reduces dry-matter sensitivity and disproportionately degrades dry-vegetation and bare-ground cover estimates, supporting a sensitivity-based performance hypothesis.For tree classification, concept representations retain 70.0–93.8% of full-embedding accuracy, while models differ in which concepts drive classification.Source: §I · p. 2
  • PiCo learns physical concept vectors from controlled simulations and uses their projections to quantify concept importance in downstream predictions.The framework extends CAMP3D with 3D radiative transfer rendering to generate Sentinel-2-like observations from forest scenes with controlled physical variables.Source: §I · pp. 2–3
  • PiCo is not validated in other scientific domains; its potential there depends on forward models, controlled observation simulation, and accessible pretrained embeddings.The authors identify these conditions as shared with Earth observation.Source: §I · p. 3

II. RELATED WORK

Geo-foundation models offer reusable embeddings across downstream tasks, but current evaluations emphasize prediction accuracy and offer limited insight into physical behavior. Existing dimension-wise correlation analysis does not directly quantify how physical variables contribute to downstream predictions.

A. Geo-foundation models and current evaluations

  • The studied GFMs represent different observation modalities, but this work focuses on single-timestamp imagery rather than temporal information.DINOv3-Sat uses RGB; Prithvi-v2 adds near-infrared and short-wave infrared bands; OlmoEarth uses optical and microwave inputs.Source: A. Geo-foundation models and current… · p. 3
  • Geo-embeddings support reusable inputs for tasks such as land-cover mapping and crop monitoring, with the goal of enabling simple predictors from limited labels.The paper describes Earth observation archives as large, spatially extensive, and repeatedly observed over decades.Source: A. Geo-foundation models and current… · p. 3
  • Current GFM benchmarks largely emphasize predictive accuracy, leaving limited tools to test hypotheses about physical sensitivity and making model choice difficult for users.Prior analysis of AlphaEarth related embedding dimensions to environmental variables but did not establish their relation to downstream predictions or quantify their importance.Source: A. Geo-foundation models and current… · p. 3

B. Concept vectors and interpretability in representation learning

Concept vectors represent directions associated with concepts in neural representation spaces, and prior approaches use them to explain predictions or attribute concept importance. PiCo instead identifies physical concept directions from controlled radiative-transfer simulations, then projects real-observation embeddings onto them for downstream analysis.

B. Concept vectors and interpretability in representation learning

  • Concept vectors are unit directions associated with concepts in neural representations, often learned from images annotated with human-defined concepts.PiCo avoids relying on potentially arbitrary human-defined concepts or scarce physical-property annotations by using controlled simulations from predefined 3D radiative transfer models.Source: B. Concept vectors and… · p. 3
  • TCAV measures directional sensitivity but does not directly quantify contributions across concept combinations and requires gradients through the downstream predictor.PiCo uses SHAP to quantify concept marginal contributions across subsets, and its post-hoc design analyzes trained models without architectural changes or retraining.Source: B. Concept vectors and… · p. 3

C. Probing representations with synthetic data from controlled simulations

  • Controlled graphics simulations can isolate factors and provide ground-truth generative variables, although common synthetic vision datasets use simple 2D single-object images.Satellite imagery differs in image formation, spectral range, scale, and resolution; 3D radiative transfer models simulate vegetation structure, leaf optics, soil, and illumination-viewing geometry.Source: C. Probing representations with… · pp. 3–4
  • PiCo renders game-engine forest scenes into synthetic satellite observations and pairs embeddings with physical variables to learn linear concept vectors.Standardised projections of real-observation embeddings form a K-dimensional concept representation used to assess downstream decisions and quantify reliance with SHAP.Source: C. Probing representations with… · p. 4, Figure 2 · p. 4

A. Problem setup

PiCo links physical concepts to GFM predictions by simulating multispectral observations, deriving concepts from the same simulator configuration, and probing embeddings from real downstream observations.

A. Problem setup

  • PiCo uses a 3D radiative transfer model to render synthetic multispectral observations from forest scenes with known physical concept values.The RTM configuration includes scene geometry, surface optical properties, and illumination and viewing conditions.Source: A. Problem setup · p. 4
  • The same simulator configuration supplies user-defined physical concepts, including leaf-optical properties and canopy structure.These concepts provide physical variables to relate to GFM predictions.Source: A. Problem setup · p. 4
  • For real downstream observations, a GFM encoder produces embeddings that a downstream classifier uses to predict class labels.The prediction task can use a simple classifier such as kNN.Source: A. Problem setup · pp. 4–5

B. Physics-informed concept probing

PiCo learns physical concept directions from simulated observations, projects real-task embeddings onto those directions, and uses the resulting concept representation to assess prediction sufficiency and attribute class probabilities to concepts.

B. Physics-informed concept probing

  • A physically controlled pipeline pairs realistic 3D forest scenes with LESS radiative transfer simulations to generate multispectral observations and known concept values.These synthetic pairs provide the data for learning concept vectors.Source: B. Physics-informed concept probing · p. 5
  • For each physical concept, a ridge-regression probe predicts its calibrated value from embeddings of synthetic observations, and its weight direction defines a unit concept vector.The vector orientation follows the regression target; positive movement corresponds to an increase in the probe prediction.Source: B. Physics-informed concept probing · p. 5
  • The analysis focuses on position along each concept vector, not on transferring the synthetic probe’s absolute calibration to real downstream scenes.Differences between synthetic and real scene distributions may affect calibration scale and offset.Source: B. Physics-informed concept probing · p. 5
  • Each downstream embedding is represented by its centered and scaled projections onto the learned concept vectors.Centering removes dependence on a global embedding-space shift, while scaling makes class-discrimination information comparable across concepts.Source: B. Physics-informed concept probing · p. 5
  • Comparing the concept-only classifier’s held-out accuracy with the full-embedding classifier measures the proportion of full-embedding accuracy retained.Both classifiers use the same training samples and labels but are fitted independently on different representations.Source: B. Physics-informed concept probing · p. 5
  • SHAP attributes each class probability to concepts by averaging each concept’s marginal contribution across subsets of the other concepts.Positive contributions increase the probability assigned to a class; their magnitude indicates contribution strength, and absolute values are averaged across test samples for dataset-level importance.Source: B. Physics-informed concept probing · pp. 5–6

IV. EXPERIMENTAL PROTOCOL

The experiments use controlled forest simulations to probe seven optical and structural concepts across three GFMs, then test those sensitivities on ground-cover and hierarchical tree-classification tasks.

IV. EXPERIMENTAL PROTOCOL

  • Seven concepts span leaf chlorophyll, dry matter, and water content plus canopy height, fractional vegetation cover, leaf area index, and canopy density.The leaf-optical variables have distinct spectral signatures, while structural concepts are derived from 3D scene geometry.Source: §IV · p. 6
  • 21,000 synthetic satellite images were simulated from 10 structurally realistic forest scenes, comprising four coniferous and six broadleaf scenes.Table I summarizes simulation configurations for the LESS 3D radiative-transfer simulations.Source: §IV · p. 6, Table I · p. 6
  • The study compares OlmoEarth, Prithvi-v2, and RGB-only DINOv3-Sat across complementary ground-cover and tree-classification datasets.SLATS tests whether probe sensitivities inform ground-cover performance hypotheses, while TreeSatAI supports hierarchical classification analysis.Source: §IV · pp. 6–7
  • SLATS estimates field-measured fractions of green vegetation, dry vegetation, and bare ground from Sentinel-2 observations, using site-separated training and test data.TreeSatAI uses Sentinel-2 patches and a single-label subset requiring the dominant label to occupy at least 90% of labelled area; labels are organized into coarse forest-cover and finer categories.Source: §IV · pp. 7–8

A. Which physical concepts is each GFM sensitive to?

Concept vectors learned from controlled CAMP3D simulations are checked on independently reconstructed RAMI-V scenes. Their transferred sensitivities vary by concept and model, with RGB-only inputs especially limited for dry matter and water.

A. Which physical concepts is each GFM sensitive to?

  • CAMP3D embeddings show visible dry-matter gradients for OlmoEarth and Prithvi-v2, but not a comparably clear pattern for DINOv3-Sat.The visualizations pass the same simulated samples with varying leaf-optical and structural properties through each model.Source: A. Which physical concepts is each GFM… · p. 8
  • On RAMI-V, original OlmoEarth and Prithvi-v2 retain dry-matter and water-content sensitivity, with correlations of r = 0.67–0.82.Dry-matter correlations are approximately zero for DINOv3-Sat and both RGB variants; water-content correlations are weak (r = −0.03–0.18).Source: A. Which physical concepts is each GFM… · p. 8
  • Chlorophyll remains accessible across models and variants, while structural concepts retain positive correlations of r = 0.52–0.97.Chlorophyll correlation is lower for DINOv3-Sat (r = 0.59), and RGB restriction reduces transfer performance for several structural concepts.Source: A. Which physical concepts is each GFM… · p. 8
  • All three models place the sparse African savanna toward the low-density end of the canopy-density concept, while DINOv3-Sat shows greater within-scene dispersion.RAMI-V scenes vary leaf-optical properties while holding canopy structure fixed, so the dispersion suggests greater sensitivity to leaf-optics variation.Source: A. Which physical concepts is each GFM… · p. 8, Figure 7 · p. 9

B. Can a GFM’s physical sensitivity help anticipate its downstream performance?

PiCo’s physical sensitivity analysis motivates a prediction about ground-cover estimation: limited dry-matter sensitivity should be associated with poorer estimates of dry vegetation and bare ground. RGB ablations show larger R2 reductions for those targets than for green vegetation, supporting that hypothesis.

B. Can a GFM’s physical sensitivity help anticipate its downstream performance?

  • The hypothesis links limited dry-matter sensitivity to poorer estimates of dry vegetation and bare ground, whose plant material differs despite similar RGB appearance.Green vegetation was expected to be less affected because chlorophyll sensitivity is comparatively retained.Source: B. Can a GFM’s physical sensitivity… · p. 9
  • 20.6% and 18.6% R2 decreases for dry vegetation, and 20.4% and 20.1% for bare ground, exceed green vegetation decreases of 11.2% and 6.4% for OlmoEarth and Prithvi-v2, respectively.The RGB ablations replace non-RGB channels with corresponding mean values; the larger reductions support the anticipated failure mode from limited dry-matter sensitivity.Source: Figure 8 · p. 9, B. Can a GFM’s physical sensitivity… · p. 9
  • Strong performance on existing benchmarks does not establish sensitivity to physical properties needed for every downstream task.PiCo offers a way to investigate such limitations with physical simulations and formulate task-specific hypotheses before downstream evaluation.Source: B. Can a GFM’s physical sensitivity… · p. 9

C. How do physical concepts distinguish downstream classes

Seven-dimensional concept representations preserve much of the full embeddings’ classification accuracy, while concept projections and importance scores reveal physical class contrasts and differences in what models use for classification.

C. How do physical concepts distinguish downstream classes

  • 92.8–93.8% of full-embedding accuracy is retained for forest cover types and 70.0–83.0% for species-level classification.The results support using concept representations to interpret substantial aspects of the original downstream classification.Source: and contribute to predictions? · p. 10, Figure 9 · p. 10
  • Both models distinguish broadleaf forest from cleared areas through fractional-cover and leaf-area-index projections, but their dominant concepts differ.For cleared areas, leaf area index dominates for OlmoEarth, while chlorophyll dominates for Prithvi-v2.Source: and contribute to predictions? · p. 10
  • For oak and beech, both models place oak higher on the water-content axis, but OlmoEarth shows a stronger water contrast while Prithvi-v2 shows a stronger chlorophyll contrast.Water content is dominant for both classes in OlmoEarth; chlorophyll is dominant for both in Prithvi-v2.Source: and contribute to predictions? · p. 10
  • Structural contrasts are prominent between broadleaf forest and cleared areas, whereas leaf-optical contrasts can be more prominent in fine-grained classifications.The examples show physically meaningful contrasts in both models, with differences in contrast strength and dominant concepts.Source: and contribute to predictions? · p. 10

tion of concept representations?

Spruce’s low projection on the leaf-area-index concept is not straightforward evidence of low leaf area: its low NIR reflectance may reflect crown self-shadowing instead.

tion of concept representations?

  • Despite typically dense foliage, spruce has the lowest leaf-area-index projection in both models, contrary to the expected ordering relative to pine.Leaf area index is among OlmoEarth’s most important concepts for classifying both spruce and pine.Source: tion of concept representations? · p. 10
  • Spruce has the lowest mean NIR reflectance among conifer classes at 17.9%, which could help explain its low leaf-area-index projection.The learned concept associates low projections with low NIR reflectance in the CAMP3D scenes.Source: tion of concept representations? · p. 11, Table II · p. 11
  • Crown self-shadowing can outweigh the NIR-reflectance increase associated with higher leaf area, so spruce’s low projection could be misread as low leaf area index.The interpretation of concept representations may require species-specific context when other factors confound spectral observability.Source: tion of concept representations? · p. 11, Figure 10 · p. 11

VI. CONCLUSION

PiCo uses controlled synthetic observations to identify physical concept vectors in GFM embeddings and explain predictions. Across the evaluated settings, concept representations retain substantial classification accuracy, while the interpretation of physical concepts remains subject to confounding in observable signals.

VI. CONCLUSION

  • PiCo combines 3D radiative transfer modelling with a game engine to generate synthetic multispectral imagery with controlled physical variations and identify concept vectors in GFM embeddings.The evaluation covers three GFMs trained on distinct modalities and seven concepts spanning leaf optics and forest vegetation structure.Source: §VI · p. 11
  • Across the tree-species classification hierarchy, seven-concept representations retain 70.0–93.8% of full-embedding accuracy.Concept vectors learned from game-engine forest scenes transfer to independently reconstructed real-world canopies while retaining sensitivity to corresponding physical concepts.Source: §VI · p. 11
  • Models capture similar physical class contrasts yet differ in contrast strength and in the concepts that dominate their classifications.Examples include structural differences between broadleaf forest and cleared areas and water-related differences between oak and beech.Source: §VI · p. 11
  • Concept-based interpretations must account for other factors that can confound physical states’ spectral signatures and embedding responses.The paper identifies observability as a qualification on interpretation.Source: §VI · p. 12
  • PiCo may extend to scientific foundation models when forward physical observation models permit controlled simulations and pretrained embeddings are available for probing.The paper presents this as potential applicability beyond Earth observation.Source: §VI · p. 12

Related papers

Loading 2610.20973v1…