Source-linked AI summary
Learning from Multimodal and Multitemporal Earth Observation Data for Building Damage Mapping
Bruno Adriano, Naoto Yokoya, Junshi Xia, Hiroyuki Miura, Wen Liu, Masashi Matsuoka, Shunichi Koshimura
TL;DR
Rapid disaster response needs damage mapping despite uncertain availability of optical or SAR imagery and limitations in validation data. The paper builds a global multimodal, multitemporal dataset and evaluates a deep-learning framework across five modality scenarios. The framework consistently outperformed the baseline across scenarios, while the dataset and labels remain an approximation to visual interpretation and are slated for broader future expansion.
Problem
Damage mapping is constrained by unpredictable availability of optical and SAR imagery and by limited, costly validation data across disasters.
Method
The paper constructs a global dataset spanning earthquakes, tsunamis, and typhoons, then evaluates a deep-learning damage-mapping framework across single-mode, cross-modal, and fusion scenarios.
Results
The proposed Attention U-Net framework consistently outperformed the baseline model in all data modality scenarios.
Takeaways & Limitations
The dataset and framework supported acceptable building-damage predictions across the evaluated data-availability scenarios.
Takeaways & Limitations
The reference damage masks predominantly rely on visual interpretation, so results represent a relative approximation to human visual interpretation; future work will broaden disasters, resolutions, and disaster-wise evaluation.
Abstract
from arXiv · showhide
Earth observation technologies, such as optical imaging and synthetic aperture radar (SAR), provide excellent means to monitor ever-growing urban environments continuously. Notably, in the case of large-scale disasters (e.g., tsunamis and earthquakes), in which a response is highly time-critical, images from both data modalities can complement each other to accurately convey the full damage condition in the disaster's aftermath. However, due to several factors, such as weather and satellite coverage, it is often uncertain which data modality will be the first available for rapid disaster response efforts. Hence, novel methodologies that can utilize all accessible EO datasets are essential for disaster management. In this study, we have developed a global multisensor and multitemporal dataset for building damage mapping. We included building damage characteristics from three disaster types, namely, earthquakes, tsunamis, and typhoons, and considered three building damage categories. The global dataset contains high-resolution optical imagery and high-to-moderate-resolution multiband SAR data acquired before and after each disaster. Using this comprehensive dataset, we analyzed five data modality scenarios for damage mapping: single-mode (optical and SAR datasets), cross-modal (pre-disaster optical and post-disaster SAR datasets), and mode fusion scenarios. We defined a damage mapping framework for the semantic segmentation of damaged buildings based on a deep convolutional neural network algorithm. We compare our approach to another state-of-the-art baseline model for damage mapping. The results indicated that our dataset, together with a deep learning network, enabled acceptable predictions for all the data modality scenarios.
1. Introduction
Large-scale disasters require timely structural-damage information, but available EO modalities vary with weather, coverage, and acquisition conditions. The paper addresses this challenge with a global multimodal, multitemporal dataset and deep-learning framework evaluated across diverse data-availability scenarios.
- 1. Introduction: Optical and SAR imagery provide complementary damage information, while weather and acquisition conditions make the first available modality unpredictable.These constraints motivate evaluating single-mode, cross-modal, and fusion configurations.
- 1. Introduction: The study constructs a global multitemporal and multimodal EO dataset with labeled building footprints from large-scale earthquake and tsunami events.The broader dataset and experiments also include typhoon disasters and multiple sensor types.
- 1. Introduction: The proposed framework integrates remote sensing and deep learning to classify building-damage levels under several data-availability scenarios.The study evaluates the framework against state-of-the-art damage-recognition approaches.
- 1.1. Related work: Existing work spans optical and SAR damage analysis, while deep CNNs have expanded automated image-segmentation and damage-recognition capabilities.The xView2 and SpaceNet-6 challenges contributed large-scale optical and multimodal datasets, but addressed narrower modality or mapping settings.
- 1.1. Related work: Earlier damage-mapping methods depend strongly on validation quality and suitable pre- and post-disaster imagery, limiting applicability across datasets and affected areas.Field surveys remain the most precise reference source, but large EO datasets offer a resource for developing advanced mapping frameworks.
2. Materials
The materials comprise disaster imagery and building-damage references assembled across geographically diverse events using optical and SAR observations before and after each disaster. Labels combine field surveys and visual interpretation into a three-category structural-damage scale.
- 2.1. Disaster events: The dataset contains one typhoon, six earthquakes, and two tsunami disasters selected for complete multitemporal and multimodal coverage across diverse geographic settings.The event distribution and locations are summarized in Table 1 and Figure 1.
- 2.2. Optical imagery: Optical imagery includes VHR WorldView-2/3, QuickBird, and Pleiades data at approximately 0.5 m resolution and SPOT-6/7 data at approximately 1.5 m resolution.Only RGB bands available across all events were used, with pre- and post-event acquisitions processed for change analysis.
- 2.3. SAR imagery: SAR data complement optical observations because of their nearly all-weather acquisition capability, with pre- and post-event scenes collected across the events using HH polarization.The SAR data came from multiple commercial sensors and underwent radiometric correction, coregistration, multilooking, filtering, and terrain-related processing.
- 2.4. Generation of labels for damage categories: Building-damage references combine costly field surveys with visual interpretation, supplemented by OpenStreetMap building polygons when interpretation provides only damaged-building point locations.Point labels are assigned to polygons using majority voting for points within each building.
- 2.4. Generation of labels for damage categories: The labels use three structural-condition categories: Destroyed, Moderated, and Survived.Destroyed denotes collapse or wash-away, Survived denotes an undisturbed structure or roof, and Moderated is the intermediate category.
3. Method
The method combines an Attention U-Net-based semantic segmentation framework with a global multimodal, multitemporal damage dataset to handle varied data availability after disasters. It evaluates five single-mode, cross-modal, and fusion scenarios and adapts an xView2 winning model as a three-class baseline.
- 3.1. Convolutional neural network model: The framework performs multiclass semantic segmentation by extracting features from temporal datasets and mapping damaged-building classes from the resulting representations.It is designed for cases where building locations are not adequately known during immediate emergency response.
- 3.1. Convolutional neural network model: The network uses a two-stream U-Net encoder, concatenates pre- and post-disaster features, and adds attention gates that suppress trivial regions and focus on target shapes.Skip connections transfer encoder information to the decoder while transpose convolutions restore spatial resolution.
- 3.2. Training settings: The model uses a lightweight U-Net architecture, initializes its first-layer weights from ImageNet-pretrained ResNeXt, and fine-tunes them for the task.U-Net-based models are selected partly because they can work with relatively small training datasets.
- 3.2. Training settings: Training uses Adam with an initial learning rate of 1 × 10^-4, learning-rate decay, and categorical cross-entropy loss.The loss is optimized during network training, while the model is evaluated against an adapted xView2 winning Siamese U-Net for three-class segmentation.
- 3.4. Scenarios of data modality: Five scenarios cover fused optical-SAR inputs, optical-only, SAR-only, pre-optical/post-SAR, and broader pre-disaster multimodal inputs.The scenarios address uncertainty about which modality becomes available first after a disaster.
4. Results
The experiments evaluate the proposed mapping framework across data-modality scenarios using supervised CNN training and hold-out testing. Across these scenarios, the framework generally outperforms the xView2 baseline, especially when optical data are included, while SAR-only inputs remain difficult.
- Experimental evaluation: The reported scores are computed on a hold-out test set, with damage grades formed by ensembling three independently trained networks.The evaluation uses Fscore values and corresponding standard deviations summarized in Table 5.
- Error-matrix results: Mode 1 and Mode 2 achieve the strongest baseline performance, with approximately 0.70 accuracy for the Survived class.Mode 1 fuses multitemporal optical and SAR imagery, while Mode 2 uses optical images only.
- Error-matrix results: The proposed framework reduces misclassification across the three damage classes, although its Survived-class accuracy is slightly below the xView2 implementation.With multitemporal optical data, intermediate damage reaches approximately 0.63 accuracy, the highest class accuracy in that mode.
- Comparative results: In Mode 4, the framework produces a more balanced classification distribution and outperforms the xView2 solution.Mode 5 also improves substantially over SAR-only Mode 3 when pre-disaster optical data are added.
- Overall quantitative results: Our framework achieves superior results to the xView2 solution in almost all data-modality scenarios, with the highest average scores when optical datasets are involved.The cross-modal Mode 4 also shows strong performance, whereas SAR-only Mode 3 is unsatisfactory for both models.
5. Discussion
The discussion emphasizes the dataset’s global multimodal scope and the framework’s usefulness when disaster-response imagery is constrained. Optical-inclusive and cross-modal inputs perform well, whereas SAR-only mapping is limited by resolution, and adding pre-disaster optical data markedly improves SAR-based results.
- Dataset scope and limitations: The dataset spans earthquakes, tsunamis, and typhoons worldwide, but predominantly visual reference masks and an approximately 60% earthquake share constrain interpretation.The authors note that tsunami and typhoon sidewall damage is difficult to observe optically and may require SAR information.
- Overall findings: The framework satisfactorily classifies three building-damage levels across varied data-availability scenarios and outperforms the xView2 winning-solution baseline.The comparison is presented using the baseline in Figure 6(a) and the proposed Attention U-Net results in Figure 6(b).
- SAR-only limitations: SAR-only Mode 3 performs poorly for Moderated and Survived classes because the 5–10 m ground sampling distance limits accurate building-footprint delineation.Destroyed buildings remain partially locatable through detected pixel patterns, suggesting a possible tile-based alternative for severe-damage classification.
- Cross-modal mapping: Mode 4 notably outperforms the baseline and supports change detection between pre-disaster optical building features and post-disaster SAR features.The authors identify this cross-modal configuration as a strong option for rapid emergency response.
- SAR-assisted mapping: Mode 5 improves segmentation by approximately 50% over SAR-only Mode 3, reaching results similar to Modes 2 and 4.Pre-disaster optical data provide features that help recognize Moderated and Survived damage classes not reliably identified with SAR alone.
6. Conclusions
The study presents a multisensor, multitemporal building-damage dataset and Attention U-Net framework that consistently outperformed a baseline across data-modality scenarios. Optical imagery alone supported damage extraction and classification, while pre-disaster optical combined with post-disaster SAR produced acceptable results.
- The Attention U-Net framework consistently outperformed the modified xView2 Challenge baseline across all data-modality scenarios.The evaluated scenarios included single-mode, cross-modal, and optical–SAR fusion settings.
- The study created a building-damage dataset spanning three damage levels and multisensor optical and SAR imagery for emergency-response applications.The dataset was designed around realistic data-availability scenarios for disaster response.
- Optical-image training accurately extracted and classified building damage without requiring additional building-mask inputs.
- Pre-disaster optical imagery integrated with post-disaster SAR data yielded acceptable building-damage classification results.
- Future work will expand the dataset, add varied spatial resolutions, and evaluate disaster-wise training and testing splits toward an all-weather operational system.The authors describe the findings as an initial phase rather than a fully operational system.