PaperlayerView exact revision on arXiv

Source-linked AI summary

Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models

Justin Guthrie, Edward Oughton, Konrad Wessels, Matthew Rice, Isaac Corley

arXiv:2609.09482v1cs.CVcs.LG

TL;DR

Critical infrastructure mapping remains uneven, and existing deep-learning pipelines are limited. This paper introduces Infra-Bench CLS to evaluate foundation models for facility-scale infrastructure classification, finding that four models outperform the fully trained ResNet-18 baseline under fine-tuning, with data centers reaching F1 0.776.

  • Problem

    Critical infrastructure mapping remains uneven, while existing deep-learning pipelines for these assets remain limited.

  • Method

    Infra-Bench CLS evaluates foundation-model adaptation for facility-scale critical-infrastructure classification across 13 asset classes, seven models, and two adaptation settings.

  • Results

    Under fine-tuning, four foundation models outperform the fully trained ResNet-18 baseline, with data centers achieving F1 0.776.

  • Takeaways & Limitations

    Infra-Bench CLS establishes a global benchmark for evaluating foundation models on facility-scale critical-infrastructure classification.

  • Takeaways & Limitations

    The benchmark uses OpenStreetMap tags as ground-truth labels without manual verification.

Abstract

from arXiv · show

Critical infrastructure location data is often incomplete and unevenly distributed globally, especially in developing regions. Earth observation foundation models are proposed as a new step in enabling us to more efficiently understand the natural and built environment, raising questions as to their effectiveness in performing challenging downstream tasks. Yet, foundation models remain largely untested for detecting and classifying the facility-scale critical infrastructure that underpins a range of important societal and economic functions. Subsequently, Infra-Bench CLS is introduced as a benchmark to test foundation models on 18,756 Sentinel-1 SAR and Sentinel-2 multispectral facility-scale critical infrastructure asset images covering seven continents and 13 infrastructure classes, with results reported for the 10 retained classes. Using linear probing and fine-tuning for two training dataset levels (1.0x and 0.3x), seven foundation models are evaluated (SatlasPretrain S2, SatlasPretrain S1, CROMA, Prithvi-EO-2.0, AlphaEarth Foundations, OlmoEarth v1.1-Base, and DINOv3 ViT-L/16). When comparing macro F1 scores to a ResNet-18 supervised baseline of 39.2 percent, the best foundation model achieved 57.9 percent, a 48 percent improvement. Top performing classes were airports (F1 85.3 percent), train stations (F1 82.1 percent), and data centers (F1 77.6 percent). By contrast, many of the power sector classes perform poorly (F1 27.5-46.2 percent). These findings suggest foundation models can enable superior critical infrastructure classification, but future work should evaluate performance on higher-resolution imagery, particularly for poorly performing sectors, such as power.

1 Introduction

Global critical-infrastructure data remain uneven and sparse, while foundation models have not been systematically evaluated for facility-scale, cross-sector infrastructure classification. Infra-Bench CLS addresses this gap with a global benchmark comparing foundation models under full and reduced-label settings.

  • Uneven and sparse critical-infrastructure mapping constrains risk assessment and motivates globally spatially explicit infrastructure datasets.
  • Foundation models may reduce labeled-data requirements, but their land-cover-oriented representations may not transfer to sparsely labeled, facility-scale infrastructure classification.
  • No benchmark had evaluated available foundation models across global, facility-scale critical-infrastructure tasks, while existing pipelines remained sector-specific and non-global.
  • Infra-Bench CLS compares foundation-model classification across energy, water, transportation, and telecommunications using globally represented OpenStreetMap facility assets.

2 Methods

Infra-Bench CLS constructs a globally sampled, facility-scale critical-infrastructure benchmark from weak OpenStreetMap labels and co-located Sentinel-1/Sentinel-2 imagery. It evaluates seven backbone configurations using linear probing and fine-tuning, with model-specific inputs and preprocessing.

  • Dataset curation: Facility-level weak labels come from OpenStreetMap tags and identify clustered sites rather than individual components typically unresolved at 10 m.The benchmark includes energy, water, transport, and telecommunications facilities, with class definitions and inclusion criteria documented separately.
  • Dataset curation: Each asset is paired with Sentinel-1 VV/VH SAR and Sentinel-2 multispectral imagery at 10 m resolution, producing centered 600 m × 600 m tiles.Quality control filters and spatial deduplication are applied in the curation pipeline.
  • Foundation-model evaluation: Seven backbone configurations spanning six Earth-observation foundation models and DINOv3 ViT-L/16 are evaluated with linear probing and full fine-tuning.Linear probing freezes pretrained features, whereas fine-tuning updates all backbone parameters; AlphaEarth Foundations supports linear probing only.
  • Foundation-model evaluation: Inputs are matched to each model’s intended design through per-model band selection and preprocessing, while Prithvi-EO-2.0 features are mean-pooled across patch tokens.CROMA uses co-registered Sentinel-1/Sentinel-2 inputs, and AlphaEarth uses fixed precomputed embeddings.

3 Results

Foundation models outperform the supervised baseline after fine-tuning, with DINOv3 ViT-L/16 achieving the strongest aggregate performance and retaining an advantage under reduced labels. Performance is highest for transportation and telecom assets, while energy subtypes remain difficult across models.

  • Aggregate results: +0.316 macro F1 is SatlasPretrain S2’s fine-tuning gain over linear probing, the largest among foundation models, while CROMA gains only +0.050.Fine-tuning improves every foundation model at both data scales, with relative gains ranging from 17 percent for CROMA to 130 percent for SatlasPretrain S2.
  • Aggregate results: 0.579 macro F1 and 0.627 weighted F1 make fine-tuned DINOv3 ViT-L/16 the top-performing foundation model, exceeding the 0.392 macro-F1 ResNet-18 baseline.Under linear probing, every foundation model remains below the supervised baseline; fine-tuning places DINOv3, SatlasPretrain S2, OlmoEarth, and Prithvi-EO-2.0 above it.
  • Training-data ablation: 0.525 macro F1 for DINOv3 ViT-L/16 fine-tuning at 0.3× exceeds the fully trained ResNet-18 baseline’s 0.392 at 1.0×, demonstrating label efficiency.DINOv3’s relative decline is 9 percent, from 0.579 to 0.525, compared with 27 percent for ResNet-18, while four fine-tuned foundation models at 0.3× exceed the baseline’s 1.0× result.
  • Sector-level results: Transportation ranks above telecom, water, and energy across all seven foundation models and both baselines, with energy requiring fine-grained subtype discrimination.Transportation assets have larger, internally structured footprints, while energy subtypes are visually similar and constrained by 10 m resolution and label ambiguity.

4 Discussion

Fine-tuning makes foundation models substantially more effective than supervised training from scratch for facility-scale critical-infrastructure classification, while linear probing alone remains noncompetitive. Performance depends strongly on asset distinctiveness, imagery resolution, label availability, and possible pretraining overlap with downstream classes.

  • 4 Discussion: Four foundation models outperform the fully trained ResNet-18 baseline under fine-tuning, whereas no foundation model exceeds it under linear probing.Fine-tuning allows backbones to adapt to the benchmark’s infrastructure classes; DINOv3 ViT-L/16 provides the strongest frozen representations but remains below ResNet-18.
  • 4 Discussion: 0.821, 0.853, and 0.776 are DINOv3 ViT-L/16’s F1 scores for train stations, airports, and data centers, respectively, where its advantage is largest.General-purpose vision pretraining transfers effectively to 10 m Earth-observation imagery and can outperform EO-pretrained foundation models.
  • 4 Discussion: +0.133 macro F1 is achieved by DINOv3 ViT-L/16 fine-tuned with 0.3× training data over the fully trained ResNet-18 baseline trained with 1.0× data.SatlasPretrain S2, Prithvi-EO-2.0, and OlmoEarth v1.1-Base also exceed that baseline by +0.107, +0.089, and +0.052, respectively.
  • 4 Discussion: 27 percent is the supervised baseline’s relative decline from 1.0× to 0.3× data, versus 9–18 percent for top fine-tuned foundation models.This indicates stronger label efficiency for fine-tuned foundation models when labeled critical-infrastructure data is expensive or unavailable at scale.

5 Limitations

Infra-Bench CLS results are constrained by noisy and incomplete labels, limited test samples, possible pretraining exposure, heuristic weighting, protocol mismatches, and a baseline that conflates pretraining with architecture. These limitations can make errors appear model-generated and complicate interpretation of foundation-model gains.

  • Class coverage: Wind farm and port terminal classes each had only n = 3 test samples per seed, while water works was excluded because it was indistinguishable across geographic regions.Spatial blocking prevented enlarging the wind farm and port terminal test sets without violating the block structure.
  • Pretraining exposure: SatlasPretrain S2 and OlmoEarth v1.1-Base achieved FT 1.0× macro F1 scores of 0.559 and 0.540, respectively, but their pretraining may have exposed them to evaluated CI classes.Their apparent transfer may partly reflect memorization of CI-tagged patterns encountered during pretraining.
  • Protocol choices: The 10× class-weight cap was selected heuristically without systematic ablation, and alternative thresholds could produce different performance profiles, especially for rare classes.The benchmark also fixed band sets rather than each foundation model’s full expected input configuration and introduced model-specific normalization and temporal mismatches.
  • Baseline interpretation: At 1.0×, four best-performing fine-tuned foundation models exceeded the supervised ResNet-18 baseline by at least 0.139 macro F1, but SatlasPretrain S1 and CROMA fell slightly below it.The gap also combines pretraining benefits with architecture size differences: ResNet-18 has 11M parameters versus 86M–300M for foundation-model backbones.

6 Future work

Future work should test foundation models under native resolutions, full band configurations, stronger distribution shifts, and higher-resolution imagery. It should also improve ontology coverage and label reliability, examine transfer and memorization, and explore infrastructure-specific pretraining.

  • Model evaluation: Native-resolution, full-band, and SAR-normalization evaluations would separate representation quality from transfer and input effects and establish each model’s capability beyond the standardized comparison.The proposed tests include Prithvi-EO-2.0 at 30 m HLS, SatlasPretrain at native tile resolution, full expected band configurations, and raw 16-bit SAR scaling.
  • Robustness: Stronger distribution shifts—including held-out continents, unseen seasons, and different sensor modalities—remain untested beyond the existing 200 km geographic out-of-distribution evaluation.Future work should determine whether fine-tuned models retain in-distribution performance under these shifts.
  • Benchmark coverage: Higher-resolution imagery could classify sub-pixel telecom assets and make difficult substation subtypes easier to identify, while targeted data collection would improve evaluation of excluded classes.The excluded classes include wind farms, port terminals, and water works; cell towers and transmission towers are sub-pixel at 10 m resolution.
  • Label quality: Extending regional cross-validation and integrating authoritative inventories would improve ground-truth reliability, especially for ambiguous transmission/distribution substation subtyping.The Maine/New Hampshire cross-validation paired OSM labels with HIFLD, ISO New England, and utility hosting-capacity maps.
  • Pretraining analysis: Ablating OSM-derived pretraining data for SatlasPretrain and OlmoEarth would test whether benchmark rankings reflect transfer capability rather than memorization of critical-infrastructure patterns.Both models were exposed to OSM geolocated data during pretraining.
  • Future model development: Dedicated infrastructure foundation models using infrastructure-rich imagery or network topologies could support damage assessment, interdependency analysis, and topology-aware classification.Facility-scale critical infrastructure is proposed as a domain for dedicated foundation-model pretraining within the SETS framework.

7 Conclusion

Infra-Bench CLS establishes a global benchmark for facility-scale critical infrastructure classification with Earth observation foundation models across continents, asset classes, and experimental conditions. Its conclusions show that fine-tuning and reduced-label evaluation can improve or preserve performance, while results vary substantially by infrastructure class and adaptation strategy.

  • Conclusion: Infra-Bench CLS establishes the first global benchmark for facility-scale critical infrastructure classification, spanning 18,756 tiles, seven continents, 13 asset classes, and 30 experimental conditions.The benchmark evaluates seven foundation models and two adaptation approaches.
  • Conclusion: 0.525 macro F1 was achieved by DINOv3 ViT-L/16 with 0.3× training data, exceeding the 0.392 full-data ResNet-18 baseline alongside three other foundation models.SatlasPretrain S2 achieved 0.499, Prithvi-EO-2.0 0.481, and OlmoEarth v1.1-Base 0.444 under 0.3× training data.
  • Conclusion: 0.853 macro F1 for airports, 0.821 for train stations, and 0.776 for data centers were the strongest reported class results, while power classes scored F1 0.275–0.462.The conclusion also characterizes transportation and telecommunications as most reliable overall and energy as least reliable.
  • Conclusion: FM selection should be guided by adaptation protocol, class-level task requirements, and label availability rather than aggregate leaderboard position alone.The benchmark provides a starting point for critical infrastructure classification through foundation models.

Declaration of generative AI use

The authors used Anthropic’s Claude models to assist with code, data and imagery workflows, documentation, and writing, while retaining responsibility for execution, verification, originality, accuracy, and content integrity.

  • Declaration of generative AI use: Anthropic’s Claude models assisted with data curation, imagery retrieval, evaluation code, figure and table generation, repository documentation, and writing-related tasks.Writing assistance included structural feedback, language refinement, framing discussions, secondary reference checking, and auditing against submission requirements.
  • Declaration of generative AI use: All code was executed by the authors, and all numerical results were produced from and verified against the authors’ own data.The authors also reviewed AI-assisted output and verified the manuscript content.
  • Declaration of generative AI use: The authors confirm responsibility for the manuscript’s originality, accuracy, suitability for publication, and content integrity, including reference accuracy.They also checked the tools’ terms of use and reviewed all AI-assisted output.

Data Availability Statement

The Infra-Bench CLS dataset, code, and verification subset are publicly available under open licenses, while source imagery and foundation-model weights are accessible through their original providers.

  • Data Availability Statement: The benchmark’s Sentinel-1 and Sentinel-2 image tiles, labels, and spatial splits are publicly available on Zenodo under the Open Database License (ODbL) 1.0.The dataset derives from OpenStreetMap data.
  • Data Availability Statement: Code for data curation, model adaptation, and evaluation, plus the Maine and New Hampshire verification subset, is available on Zenodo under the MIT License.
  • Data Availability Statement: Sentinel-1 and Sentinel-2 imagery were accessed through the Microsoft Planetary Computer, while foundation-model weights are available from their respective original sources.

Funding

The work was supported by NSF through NCAR and supplemental funding from the NSF NCAR Faculty Innovator Program. Additional support came through the Integrated Hazard Institute under a NASA cooperative agreement.

  • Funding: NSF supported the work through the National Center for Atmospheric Research under Cooperative Agreement No. 1852977.NCAR is identified as a major facility sponsored by the U.S. National Science Foundation.
  • Funding: Supplemental funding from the NSF NCAR Faculty Innovator Program supported the project under the same cooperative agreement.
  • Funding: EO received research support through the Integrated Hazard Institute under NASA Cooperative Agreement 80NSSC25M0077.

Supporting Information

The supporting information provides supplementary confusion matrices, training dynamics, per-class and per-condition metrics, configurations, timing, and label-ontology details. It also documents shared preprocessing, aggregation across three seeds, and limitations from excluded classes and 10 m resolution.

  • Supporting Information: The supplement compiles aggregate confusion matrices, training dynamics, per-class F1 statistics, complete configurations, timing, ontology details, and per-condition evaluation metrics.These materials span Figures S1–S10 and Tables S1–S13.
  • Supporting Information: All models use identical 600 m × 600 m tiles from the Microsoft Planetary Computer at 10 m resolution, with model-specific recommended preprocessing protocols.Confusion matrices and per-class results are summed or averaged across three seeds and restricted to 10 retained evaluation classes.
  • Supporting Information: The largest confusion-matrix contribution is wastewater plant versus excluded water works, and its magnitude varies across models.Rows may fall slightly below 100% because predictions into wind farm, water works, and port terminal are omitted.
  • Supporting Information: The excluded classes reflect either limited test support or low signal at 10 m resolution, particularly for water works.The cited test-support limitation specifies wind farm and port terminal as having n = 3 per seed.
Loading 2609.09482v1…