Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models

Motivation

• Global critical infrastructure mapping remains sparse and uneven, constraining risk assessment and motivating spatially explicit asset datasets.This problem is especially consequential where infrastructure mapping coverage is historically poorer and labeled data are limited.Source: §1 · pp. 2–3
• Foundation models may reduce labeled-data requirements by transferring priors learned from large satellite or natural-image datasets, but transfer to facility-scale infrastructure remains uncertain.Their pretraining emphasizes land-cover and other Earth-observation patterns, whereas facility-scale infrastructure classification involves substantially different object scales.Source: §1 · pp. 2–3

Research gap

• Existing evaluations do not address foundation models for cross-sector, facility-scale critical-infrastructure data; no benchmarks evaluate available models on facility-scale critical-infrastructure tasks, while specialized pipelines remain limited to one sector and non-global.Source: §1 · p. 3

Contribution

• Infra-Bench CLS benchmarks foundation-model classification on facility-scale critical infrastructure spanning energy, water, transportation, and telecommunications sectors represented consistently in OpenStreetMap.The dataset supports evaluation of foundation models and other deep-learning models on critical infrastructure.Source: §1 · pp. 3–4

Evaluation design

• Infra-Bench CLS compares foundation models with ResNet-18 baselines under linear probing and fine-tuning, including tests using 30 percent of training labels.The study asks how performance varies by protocol, label availability, infrastructure class, and sector.Source: §1 · p. 4

2.1 Dataset curation

• Hand-curated validation of 376 assets in Maine and New Hampshire identified substation mistagging and OSM omissions as predominant label-error modes.This regional characterization informs interpretation of foundation-model performance on substation subtyping but is not global validation.Source: §2.1 · pp. 5, 7
• The dataset comprises 18,756 globally sampled, facility-scale assets represented by weak OpenStreetMap labels and 600 m × 600 m Sentinel-1/Sentinel-2 tiles at 10 m resolution.Assets span energy, water, transportation, and telecommunications, with classes defined as spatially clustered facilities rather than individual sub-resolution components.Source: §2.1 · pp. 4–5, Figure 1 · p. 6
• Quality-controlled tiles are spatially deduplicated and partitioned into region-stratified train, validation, and test splits using 200 km × 200 km spatial blocks.The curation pipeline applies cloud, edge-artifact, valid-pixel-ratio, and size filters before producing the final verified dataset.Source: Figure 1 · p. 6

2.2 The evaluation protocols

• Seven backbone configurations are evaluated with frozen-feature linear probing and full fine-tuning, using per-model input bands and preprocessing matched to each model’s design.The evaluation includes SatlasPretrain S1/S2, CROMA, Prithvi-EO-2.0, AlphaEarth Foundations, OlmoEarth v1.1-Base, and DINOv3 ViT-L/16; AlphaEarth supports only linear probing.Source: §2.2 · pp. 7–8
• The reported evaluation covers 10 of 13 classes because wind farms and port terminals each have only n = 3 samples, insufficient for reliable per-class F1 estimates.Prithvi-EO-2.0 additionally has a resolution mismatch between its 30 m pretraining data and 10 m evaluation imagery.Source: §2.2 · p. 9
• F1 is reported instead of accuracy because class sizes range from n = 70 to 737, with per-class, macro, weighted, and sector-level results plus aggregate confusion matrices.The water works class is excluded because regional design variation provides no consistent visual signature at 10 m resolution.Source: §2.2 · p. 9
• Both linear probing and fine-tuning are repeated with 0.3× training data, retaining 30 percent of tiles per class while leaving validation and test sets unchanged.The fixed-seed subsample is reused across three training seeds to test foundation-model value under label-constrained or data-sparse conditions.Source: §2.2 · p. 10
• Relative to a random baseline, some tiles received different split assignments; the evaluation instead uses spatially blocked 200 km × 200 km Equal Earth blocks with region-stratified 70/15/15 splits.Blocks are assigned to train, validation, and test partitions using seed 42.Source: §2.2 · p. 10
• Resolution-matched evaluation is deferred because results can combine representation quality with transfer-cost effects from pretraining-distribution mismatch.This concern is noted most notably for Prithvi-EO-2.0 and cannot be isolated without evaluating each foundation model at matched resolution.Source: §2.2 · p. 10

2.3 Baselines

• Two randomly initialized ResNet-18 baselines establish pretraining-value references: a frozen-backbone Random Features model for linear probing and an end-to-end model for fine-tuning.The gap between foundation models and each corresponding baseline measures task-relevant information in pretrained features or the empirical value added by pretraining.Source: §2.3 · p. 11

3.1 Aggregate rankings

• DINOv3 ViT-L/16 leads under fine-tuning with 0.579 macro F1 and 0.627 weighted F1, outperforming EO-specialized models despite natural-image pretraining.Source: §3.1 · p. 12

3.4 Structural patterns across infrastructure classes

• 0.853 airport F1, 0.821 train-station F1, and 0.776 data-center F1 are the strongest class results, while energy subtypes remain difficult.The best fine-tuned scores for transmission and distribution substations are only 0.388 and 0.330, respectively, and ResNet-18 also struggles with these distinctions.Source: Figure 3 · p. 13, §3.4 · p. 19

3.2 Fine-tuning vs. linear probing

• Linear probing measures pretrained representation quality, whereas fine-tuning measures adaptation capacity when the backbone updates during training.Comparing these modes separates information already encoded in frozen features from gains produced by backbone adaptation.Source: §3.2 · p. 16

3.3 Training dataset size ablation (1.0× vs. 0.3×)

• 0.525 macro F1 for DINOv3 ViT-L/16 fine-tuned at 0.3× exceeds the fully trained ResNet-18 baseline’s 0.392, demonstrating label-efficient performance.DINOv3’s fine-tuned macro F1 declines 9 percent from 0.579 to 0.525, versus a 27 percent decline for ResNet-18 from 0.392 to 0.287.Source: §3.3 · pp. 16, 19, Figure 7 · p. 18

3.4 Structural patterns across infrastructure classes

• DINOv3 ViT-L/16 achieves the highest F1 in eight of ten infrastructure classes; OlmoEarth leads solar farms and SatlasPretrain S2 leads distribution substations.DINOv3’s eight wins include transmission substations, other distribution, power plants, wastewater plants, storage tanks, airports, train stations, and data centers.Source: §3.4 · p. 19

3.5 Performance by critical infrastructure sector

• Transportation > telecom > water > energy is the sector-performance ranking across all foundation models and both baselines.Transportation assets have multi-pixel, internally structured footprints, whereas energy subclasses require fine-grained distinctions between visually similar subtypes at 10 m resolution.Source: Figure 8 · p. 20, §3.5 · p. 20

3.6 Evaluation reliability

• DINOv3 ViT-L/16 FT at 1.0× is most stable (std = 0.002), while CROMA FT at 1.0× is most variable (std = 0.029).Under fine-tuning, foundation models cluster around a macro F1 std of 0.002–0.029.Source: Figure 9 · p. 21, §3.6 · p. 21
• AlphaEarth Foundations has the lowest linear-probing variance (macro F1 std = 0.001).Figure 9 reports three-seed mean validation macro F1 trajectories with ± 1 standard deviation bands at 1.0× training data.Source: Figure 9 · p. 21, §3.6 · p. 21
• Seed variance reflects class imbalance and stochastic head initialization rather than being foundation-model-specific.The Random Features baseline has LP macro F1 std = 0.005 at 1.0×, within the 0.001–0.023 range across foundation models.Source: §3.6 · p. 21

Fine-tuning versus linear probing

• Fine-tuning enables most foundation models to exceed the fully trained ResNet-18 baseline, while no linear probe does so.With labeled data, fine-tuned foundation models are preferred; when labels are scarce or only frozen features are feasible, DINOv3 ViT-L/16 provides the strongest representations but remains below ResNet-18.Source: §4 · p. 22

General-purpose transfer

• DINOv3 ViT-L/16 outperforms EO-pretrained foundation models on 10 m imagery, leading linear probing at both data scales and weighted-F1 fine-tuning.Its advantage is largest for train stations (0.821), airports (0.853), and data centers (0.776), suggesting general-purpose vision pretraining transfers effectively to EO classification.Source: §4 · p. 22

Label efficiency

• +0.133 macro F1 is the improvement of DINOv3 ViT-L/16 fine-tuned with 0.3× training data over the fully trained ResNet-18 baseline trained with 1.0× data.SatlasPretrain S2, Prithvi-EO-2.0, and OlmoEarth v1.1-Base also exceed that baseline by +0.107, +0.089, and +0.052, respectively; fine-tuned foundation models decline 9–18% under reduced labels versus 27% for the supervised baseline.Source: §4 · p. 22

Visual ambiguity and resolution

• Visual distinctiveness at 10 m constrains classification more than foundation-model choice for specific asset classes, with substation subclasses difficult across every model and ResNet-18.Their near-identical appearance and ontology ambiguity indicate that higher-resolution imagery or additional information is needed for reliable distinction.Source: §4 · p. 23

Transfer interpretation

• SatlasPretrain and OlmoEarth v1.1-Base included labels for eight of the ten reported classes during pretraining, so strong fine-tuned performance may combine representation quality with memorization.DINOv3 ViT-L/16 likely lacked aerial imagery of OSM-tagged facilities during pretraining, making its comparable fine-tuned performance evidence of transfer from natural web imagery.Source: §4 · p. 23

Implications

• Fine-tuned EO or general-purpose vision foundation models can support CI classification for visually distinctive assets, but ambiguous subclasses require higher-resolution imagery or additional information.Infra-Bench CLS’s per-class and per-sector results allow practitioners to assess this tradeoff for their applications.Source: §4 · p. 23

Ground-truth labels

• 42 percent of manually verified substations received unambiguous subtype descriptions, while 23 percent were unclassifiable and 24 percent had mixed designations.OSM labels were not manually verified globally, and subtype ambiguity is specific to the benchmark’s substation ontology; label errors may therefore produce conservative F1 scores, while the regional cross-validation may not generalize.Source: §5 · p. 24

Class coverage

• Wind farm and port terminal classes each had n = 3 test samples per seed, preventing reliable F1 estimation under spatial blocking.Water works was excluded because it was indistinguishable across geographic regions.Source: §5 · p. 24

Pretraining overlap

• 0.559 and 0.540 macro F1 for SatlasPretrain S2 and OlmoEarth v1.1-Base, respectively, may partly reflect memorization of CI-tagged patterns seen during pretraining.Their pretraining objectives included direct exposure to some evaluated infrastructure classes, complicating interpretation of apparent transfer.Source: §5 · p. 24

Training choices

• A 10× class-weight cap was chosen heuristically without systematic ablation, so alternative thresholds may change performance profiles for rare classes.Future work should verify the weighting choice through controlled ablation.Source: §5 · pp. 24–25

Evaluation protocol

• Fixed band sets, rather than each foundation model’s full expected configuration, limit protocol standardization; CROMA received only 6 of its expected inputs.Additional mismatches include SatlasPretrain S1 normalization differing from the benchmark’s dB convention and Prithvi-EO-2.0 receiving single-date inputs with T = 1 despite multi-temporal pretraining.Source: §5 · p. 25

Baseline interpretation

• At 1.0×, four fine-tuned foundation models exceeded the supervised ResNet-18 baseline by at least 0.139 macro F1, while SatlasPretrain S1 and CROMA fell slightly below it.This gap combines pretraining benefits with architecture size differences between the 11M-parameter ResNet-18 and 86M–300M foundation-model backbones; a matched-architecture baseline would isolate pretraining more cleanly.Source: §5 · p. 25

Evaluation conditions

• Native-resolution and full-band evaluations would separate representation quality from transfer cost, reveal resolution-coupling effects, and distinguish input-driven from architecture- or pretraining-driven variance.Using SatlasPretrain S1’s raw 16-bit SAR normalization could also provide a more accurate assessment of its F1 potential.Source: §6 · pp. 25–26

Robustness

• Stronger distribution-shift tests should assess whether fine-tuned models retain performance across held-out continents, unseen seasons, and different sensor modalities beyond 200 km spatial blocking.The benchmark currently provides geographic out-of-distribution evaluation at the 200 km block level but does not test these stronger shifts.Source: §6 · p. 26

Coverage and resolution

• Targeted data collection and higher-resolution imagery could add excluded classes, classify sub-pixel telecom assets, and improve identification of difficult substation sub-types.Priority gaps include wind farms, port terminals, water works, cell towers, transmission towers, and substation subtypes.Source: §6 · p. 26

Models and transfer

• Future iterations could evaluate excluded models—including Clay, TESSERA, DOFA, SatMAE, TerraMind, Galileo, and SMARTIES—and investigate why DINOv3 ViT-L/16 transfers effectively from natural imagery to EO.These studies could guide foundation-model selection and EO-specific pretraining objectives.Source: §6 · p. 26

Label quality

• Extending regional cross-validation with authoritative inventories would improve ground-truth reliability, especially for substation subtyping and transmission-versus-distribution labels.The Maine/New Hampshire cross-validation demonstrates the value of pairing OSM labels with HIFLD, ISO New England, and utility hosting-capacity maps.Source: §6 · p. 27

Pretraining and representations

• Pretraining-data ablations are needed to determine whether SatlasPretrain and OlmoEarth rankings reflect transfer capability rather than memorization of critical-infrastructure patterns.The benchmark also motivates infrastructure-specific foundation models that use infrastructure-rich imagery or network topologies for tasks such as damage assessment and interdependency analysis.Source: §6 · p. 27

Conclusion

• Infra-Bench CLS establishes a global foundation-model benchmark spanning 18,756 tiles across seven continents, 13 asset classes, and 30 experimental conditions.The benchmark provides a starting point for critical infrastructure classification through foundation models.Source: §7 · pp. 27–28
• Under 0.3× training data, DINOv3 ViT-L/16, SatlasPretrain S2, Prithvi-EO-2.0, and OlmoEarth v1.1-Base exceeded the full-data ResNet-18 baseline of 0.392 macro F1.Their macro F1 scores were 0.525, 0.499, 0.481, and 0.444, respectively.Source: §7 · p. 28
• Airports, train stations, and data centers were classified most reliably, with macro F1 scores of 0.853, 0.821, and 0.776, while power classes performed worst at F1 0.275–0.462.Source: §7 · p. 28
• Fine-tuning raised four of six fine-tunable foundation models above the fully trained ResNet-18 baseline of 0.392 macro F1, reaching up to 0.579.Under linear probing, no foundation model matched the ResNet-18 baseline.Source: §7 · pp. 27–28
• Transportation and telecommunications assets were classified most reliably overall, whereas energy assets were classified least reliably.Source: §7 · p. 28
• Foundation-model selection should follow adaptation protocol, class-level task requirements, and label availability rather than aggregate leaderboard position alone.This guidance qualifies the benefits of foundation models despite their performance advantages.Source: §7 · p. 28

• Claude models assisted with data curation, imagery retrieval, evaluation code, figure and table generation, and repository documentation.Source: Declaration of generative AI use · p. 28
• The authors executed all code, generated and verified numerical results from their own data, reviewed AI-assisted output, and accepted responsibility for content integrity and reference accuracy.Source: Declaration of generative AI use · pp. 28–29
• They also provided structural feedback, language refinement, framing discussion, secondary reference checking, and manuscript auditing against submission requirements.Source: Declaration of generative AI use · pp. 28–29

Data availability

• The benchmark dataset, including Sentinel-1 and Sentinel-2 tiles, labels, and spatial splits, is publicly available on Zenodo under the ODbL 1.0 license.The dataset derives from OpenStreetMap data (OpenStreetMap contributors, 2017).Source: Data Availability Statement · p. 29
• Code for data curation, model adaptation, and evaluation, together with the Maine and New Hampshire verification subset, is available on Zenodo under the MIT License.Source: Data Availability Statement · p. 29
• Sentinel-1 and Sentinel-2 imagery were accessed through the Microsoft Planetary Computer, while foundation-model weights are available from their respective original sources cited in Section 2.Source: Data Availability Statement · p. 29

Funding

• NSF supported the work through the National Center for Atmospheric Research under Cooperative Agreement No. 1852977.The paper was produced through supplemental funding from the NSF NCAR Faculty Innovator Program under the same agreement.Source: Funding · p. 29
• EO also received research support through the Integrated Hazard Institute under NASA Cooperative Agreement 80NSSC25M0077.Source: Funding · p. 29

Supplementary analyses

• Supplementary materials provide confusion matrices, training dynamics, per-class F1 scores, complete adaptation configurations, timing, ontology definitions, and accuracy, precision, and recall statistics.These materials cover Figures S1–S10 and Tables S1–S13.Source: Supporting Information · p. 35

Evaluation setup

• All models use identical 600 m × 600 m tiles at 10 m resolution, with model-specific recommended preprocessing, while results are aggregated across three seeds over 10 retained classes.The supporting information notes that row sums may be below 100% because predictions into wind farm, water works, and port terminal are omitted.Source: Supporting Information · p. 35

Confusion-matrix interpretation

• The largest confusion involves wastewater plants and the excluded water works class, with its magnitude varying across models.The three excluded classes are wind farm, water works, and port terminal.Source: Supporting Information · p. 35

Ontology and exclusions

• The ontology excludes sub-pixel infrastructure from 10 m imagery, while wind farm, port terminal, and water works are retained in the dataset but excluded from reported metrics.Wind farm and port terminal have insufficient test support, whereas water works has low signal at 10 m resolution.Source: Supporting Information · p. 56

Evaluation statistics

• Supplementary tables report per-class and aggregate F1, accuracy, macro and weighted precision, and macro recall across adaptation protocols and both training-data scales.The metrics are based on held-out test-set confusion matrices restricted to the 10 retained classes, with three-seed mean ± standard deviation reporting.Source: Table S8 · p. 57, Table S9 · p. 58, Table S10 · p. 59

• The analysis was sliced to the 10 retained classes.Source: Supporting Information · p. 62