Source-linked AI summary
DamageScope: Vision-Language Retrieval at Scale for Disaster Damage Assessment from Satellite Imagery
Ravi K. Rajendran, Biplob Debnath, Murugan Sankaradas, Srimat T. Chakradhar
TL;DR
Property damage assessment needs alternatives to costly, time-consuming, and hazardous on-site inspections, especially at large scale. DamageScope uses satellite imagery with VLMs and LLMs in a RAG framework, combining multi-vector clustering and dual-store retrieval. It reports improved clustering metrics, up to 14× faster indexing, and approximately 3× more cost-efficient and 2.9× faster question answering than baselines.
Problem
Traditional on-site inspections are costly, time-consuming, and hazardous, while large-scale VLM processing also presents computational, data-organization, and retrieval challenges.
Method
DamageScope combines satellite imagery, VLMs, and LLMs in a RAG framework with multi-vector clustering, dual-store organization, and natural-language querying.
Results
Up to 0.20 NMI, 0.26 FMS, and 0.26 ARI gains, up to 14× lower indexing time, and approximately 3× more cost-efficient and 2.9× faster question answering were reported against baselines.
Takeaways & Limitations
DamageScope provides a scalable framework for interactive property damage assessment that balances accuracy, scalability, and efficiency.
Takeaways & Limitations
Standard MaxSim is asymmetric, whereas clustering requires symmetric pairwise distances, motivating the proposed symmetric extension.
Abstract
from arXiv · showhide
Timely and accurate assessment of property damage is critical following natural disasters. Traditional on-site inspections are labor-intensive, costly, and often pose safety risks. Advances in satellite imagery and vision-language models (VLMs) enable scalable remote damage assessment; however, integrating VLMs into large-scale Earth observation pipelines presents challenges in computational efficiency, data organization, and information retrieval. To address these challenges, we present DamageScope, a retrieval-augmented framework that combines satellite imagery with Vision-Language Models (VLMs) and Large Language Models (LLMs) to automate property damage analysis. Built on a Retrieval-Augmented Generation (RAG) framework, DamageScope extracts structured visual representations from satellite imagery to support interactive natural language queries for damage assessment. To address scalability, we introduce a novel multi-vector embedding-based clustering algorithm that outperforms traditional single-vector embedding approaches while reducing indexing time by up to 14x. Furthermore, a dual-store data architecture minimizes LLM API calls, reducing both operational cost and response latency by up to approximately 3x. By effectively balancing scalability and operational efficiency, DamageScope provides a robust and practical solution for real-world damage assessment tasks.
1. Introduction
DamageScope combines satellite imagery, VLMs, and LLMs in a retrieval-augmented system for scalable property damage assessment. Its multi-vector clustering, dual-store organization, and selective retrieval address computational, data-organization, and query-processing challenges.
- Traditional on-site inspections are costly, time-consuming, and hazardous, motivating safer and more efficient damage assessment alternatives.
- A clustering-based strategy groups similar properties to reduce redundant VLM computation in large-scale disaster scenarios.
- DamageScope symmetrizes MaxSim by averaging directional scores, enabling the symmetric distance computations required for clustering.
- A multi-store architecture distributes damage data to optimize retrieval time and accuracy, while LLM-driven filtering limits context and API overhead.
- DamageScope integrates satellite imagery, VLMs, and LLMs to support natural-language querying for property damage insights.
- Up to 0.20 NMI, 0.26 FMS, and 0.26 ARI gains accompany a 14× indexing-time reduction; question answering is approximately 3× more cost-efficient and 2.9× faster than baselines.
2. DamageScope System
DamageScope combines multimodal structured representations, multi-vector clustering, and dual-store retrieval within a RAG pipeline for scalable property damage analysis. Its workflow reduces redundant VLM processing, supports semantic and structured queries, and produces stable clusters for efficient ingestion.
- System Architecture: DamageScope converts satellite imagery and property metadata into structured categorical predictions and natural-language reasoning for retrieval and question answering.Each property crop is processed by a VLM, while the RAG system retrieves evidence and synthesizes natural-language responses.
- Scalable Ingestion: ≈8.4 seconds per GPT-4o inference would require ∼23 hours to process 10,000 building footprints sequentially, motivating redundant-inference reduction.DamageScope addresses this overhead by clustering properties and analyzing only representative items.
- Scalable Ingestion: Four-class classification precedes intra-class clustering: “No Damage” properties require no further analysis, while other severity classes receive finer-grained clustering.The classes are No Damage, Minor, Major, and Destroyed; this ordering reduces heterogeneity and avoids unnecessary computation for undamaged properties.
- Multi-Vector Clustering: DamageScope uses multi-vector embeddings with hierarchical agglomerative clustering to preserve fine-grained visual cues and handle nonconvex, heterogeneous, uneven-sized damage groups.Single-vector embeddings can lose cues such as shingle loss, debris distribution, and partial collapse; HAC avoids k-means’ spherical-cluster assumption.
- Multi-Vector Clustering: A symmetric MaxSim variant averages directional scores, enabling valid pairwise distances for clustering and producing coherent, stable clusters with mutual visual similarity.Standard MaxSim is asymmetric, whereas the symmetric metric supports consistent distance computations and reduces redundant ingestion computations.
- Dual-Store Retrieval: DamageScope routes structured metadata to SQL and textual reasoning to a vector database, matching filtering and semantic retrieval operations to separate stores.This routing avoids unnecessary text processing and prevents large reasoning fields from inflating the LLM context window; an LLM query planner coordinates the queries.
3. Experiments
The experiments evaluate clustering quality, ingestion efficiency, and question-answering cost and latency. Multi-vector clustering improves damage-cluster alignment and substantially reduces ingestion and query-processing overhead.
- Evaluation setup: The experiments compare clustering quality, ingestion time, data organization, and LLM API usage cost against a baseline without clustering, single-store retrieval, or query-aware selection.The evaluation uses xBD imagery and property-level annotations, with clustering assessed using NMI, FMS, and ARI.
- Clustering performance: Multi-vector methods outperform single-vector baselines for RemoteCLIP across most cases, with the largest gains in fine-grained damage categories.For Image ID 31 Minor damage, Multi-vector + UPGMA reaches NMI = 0.561, FMS = 0.521, and ARI = 0.416, versus 0.428, 0.283, and 0.177.
- Clustering performance: RemoteCLIP benefits more consistently from multi-vector representations than SigLIP, although SigLIP achieves the best scores in a few settings.For Image ID 16 Minor damage, RemoteCLIP reaches NMI = 0.694 versus 0.526 and ARI = 0.565 versus 0.301; SigLIP’s advantage is inconsistent.
- Ingestion efficiency: 14.08× speedup reduces Image 31 ingestion time from 2,436.1 s to 172.9 s through clustering.Image 16 also improves from 1,335.6 s to 168.3 s, a 7.94× speedup; runtime depends on damage composition and scene complexity, not only instance count.
- Ingestion efficiency: Clustering consolidates 139 Image 31 instances into 20 clusters and 149 Image 16 instances into 22 clusters, reducing regions requiring individual processing.This lowers the instance-level ingestion workload and supports more scalable property-level damage analysis.
- Question answering: 2.9× reduction in average LLM token usage, from 7,050 to 2,420 tokens per query, is achieved through query-aware retrieval and compact context construction.End-to-end QA latency also decreases from 2.10 seconds to 0.69 seconds, approximately a 3× improvement, using optimized query plans and dual-store organization.
4. Related Work
Related work progresses from satellite and object-based damage detection to multimodal machine learning and vision-language methods. DamageScope extends this direction by combining multimodal AI, LLMs, and RAG for interactive satellite-based assessment.
- Satellite damage assessment: Earlier studies used high-resolution satellite imagery and object-based analysis to detect structural damage after earthquakes and floods.
- Machine learning and multimodal methods: Machine-learning approaches improved damage classification through CNNs, multitask learning, ensembles, dual-path spectral-spatial models, and multimodal fusion.Recent methods combine pre- and post-disaster imagery with metadata or fuse radar and optical data.
- DamageScope: DamageScope combines multimodal AI, LLMs, and retrieval-augmented generation to support fast, cost-efficient, interactive satellite-based damage assessment for insurers.
5. Conclusion
DamageScope targets costly, time-consuming, and hazardous physical inspections by using satellite imagery and AI-driven analysis for property damage assessment. The system is intended to support rapid, large-scale insurance assessments and more efficient claims services.
- DamageScope automates property damage evaluation using high-resolution satellite imagery, vision-language models, and large language models.
- The system enables rapid, large-scale assessments intended to streamline claims processing, enhance accuracy, and improve services for policyholders after natural disasters.
A.1. DamageScope Prompt
The DamageScope prompt elicits structured JSON damage assessments from GPT-4o. It specifies observable damage fields, concise reasons, and consistent output formatting for downstream retrieval and querying.
- The prompt asks GPT-4o to analyze a satellite image of a property and return a JSON structure for damage assessment.
- The structured output includes prediction and reason fields for overall, structural, lawn, water, and fire or smoke damage.
- The schema also records occupiability and damage level, including a categorical habitability status and a 0–3 damage-level prediction.
- Prompt guidelines restrict responses to visually observable content, prohibit guessing causes, require concise reasoning, and preserve consistent JSON keys and order.
A.2. Example Property-Level Output
Listing 5 presents an extended JSON record for property-level disaster damage, combining identifiers, categorical predictions, and explanatory reasoning strings.
- The extended record identifies property A12-487 and the disaster event as a wildfire.
- It records roof, structural, lawn, and fire-smoke damage predictions with accompanying reasons.
- The example marks water damage as false because no visible water pooling or related signs are reported.
- Occupiability is classified as uninhabitable, with structural and fire damage cited as the reason.
A.3. SQL and Vector Storage Examples
DamageScope separates structured attributes from descriptive reasoning in dual stores, allowing categorical and semantic queries to use different retrieval mechanisms.
- Structured and unstructured JSON fields are stored separately in DamageScope’s dual-store architecture.
- The structured SQL row stores property identifiers, Boolean damage indicators, occupiability, and damage level.
- The vector-store entry stores descriptive reasoning for roof, structural, lawn, fire-smoke, and occupiability attributes.
- SQL queries resolve numeric or categorical attributes, while vector queries retrieve semantic reasoning.