Source-linked AI summary
Multi$^{\mathbf{3}}$Net: Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery
Tim G. J. Rudner, Marc Rußwurm, Jakub Fil, Ramona Pelich, Benjamin Bischke, Veronika Kopackova, Piotr Bilinski
TL;DR
Flood maps may take several hours to produce through semi-automated interpretation. Multi3Net fuses satellite imagery across sensors, resolutions, and time to segment buildings and flooded buildings. The authors report that it outperforms state-of-the-art models on both tasks and alternative fusion approaches for flooded-building segmentation.
Problem
Semi-automated interpretation can delay flood-map delivery by several hours after satellite imagery is provided.
Source: Piotr Bili´nski · p. 1Method
Multi3Net uses multiple CNN encoder-decoder streams to fuse satellite imagery across sensors, resolutions, and time for building and flooded-building segmentation.
Source: Piotr Bili´nski · p. 1Results
Multi3Net outperformed state-of-the-art models for building-footprint and flooded-building segmentation, and its building IoU was 75.3% versus 44.2% for the U-Net baseline.
Source: Conclusion · p. 7, Segmentation of Flooded Buildings with… · p. 7Takeaways & Limitations
The method can produce segmentation maps after a single successful satellite acquisition and improve them as additional imagery becomes available.
Source: Piotr Bili´nski · p. 1
Abstract
from arXiv · showhide
We propose a novel approach for rapid segmentation of flooded buildings by fusing multiresolution, multisensor, and multitemporal satellite imagery in a convolutional neural network. Our model significantly expedites the generation of satellite imagery-based flood maps, crucial for first responders and local authorities in the early stages of flood events. By incorporating multitemporal satellite imagery, our model allows for rapid and accurate post-disaster damage assessment and can be used by governments to better coordinate medium- and long-term financial assistance programs for affected areas. The network consists of multiple streams of encoder-decoder architectures that extract spatiotemporal information from medium-resolution images and spatial information from high-resolution images before fusing the resulting representations into a single medium-resolution segmentation map of flooded buildings. We compare our model to state-of-the-art methods for building footprint segmentation as well as to alternative fusion approaches for the segmentation of flooded buildings and find that our model performs best on both tasks. We also demonstrate that our model produces highly accurate segmentation maps of flooded buildings using only publicly available medium-resolution data instead of significantly more detailed but sparsely available very high-resolution data. We release the first open-source dataset of fully preprocessed and labeled multiresolution, multispectral, and multitemporal satellite images of disaster sites along with our source code.
Introduction
Flood maps can take hours to produce through semi-automated interpretation, while medium-resolution imagery is available more frequently than very high-resolution imagery. Multi3Net fuses satellite imagery across sensors, resolutions, and time to segment buildings and flooded buildings.
Introduction
- Flood-map delivery can take several hours after imagery arrives because map generation involves semi-automated interpretation..Source: Piotr Bili´nski · p. 1
Method
- The proposed approach targets rapid flood-damage segmentation by combining multiresolution and multisensor satellite imagery in a CNN.The paper also demonstrates the model on conventional building-footprint segmentation.Source: Piotr Bili´nski · p. 1
Piotr Bili´nski
- Multi3Net fuses multiresolution, multisensor, and multitemporal imagery in a CNN, and the paper releases a labeled disaster-imagery dataset.The network uses multiple encoder-decoder streams, each producing a sensor-specific output that can be combined into a joint prediction.Source: Piotr Bili´nski · p. 1
Background: Earth Observation
- Medium-resolution sensors provide weekly global acquisitions, whereas very high-resolution imagery is scheduled and becomes available only after a disaster.Radar acquisitions are unaffected by cloud coverage or lack of light, while conventional flood-mapping models perform poorly in urban and suburban areas.Source: Background: Earth Observation · p. 2
Related Work
- Building-footprint segmentation is an established satellite-imagery task, while damaged-building segmentation can be complicated by confounding damaged non-building structures.Prior approaches have used pre- and post-disaster imagery, including combinations of optical and radar data.Source: Related Work · p. 2
Segmentation Network Architecture
- Multi3Net uses a modified ResNet encoder with dilated convolutions to downsample by ×8 and a context module to enrich feature maps.Three bilinear upsampling blocks map latent features to labels, and the network is trained end-to-end.Source: Segmentation Network Architecture · p. 3
Multi3Net Image Fusion
Multi3Net processes imagery in separate streams and fuses their outputs, while matching feature-map resolutions across sensors. Pre- and post-disaster images are concatenated for temporal fusion, and training includes building-footprint pretraining before flooded-building segmentation.
Multi3Net Image Fusion
- Late fusion using additional convolutional layers performed best among the tested multisensor fusion approaches.The method concatenates stream predictions, then applies two 3×3 convolutions and a 1×1 convolution.Source: Multisensor Fusion · p. 3
- For Sentinel-1 and Sentinel-2 alone, feature maps are aligned at 96px × 96px and 10m resolution; with very high-resolution imagery, its feature maps are aligned to the Sentinel streams..Source: Multiresolution Fusion · p. 3
- Pre- and post-disaster images are concatenated into one input tensor for temporal fusion; two-stream shared-weight and feature-subtraction alternatives did not improve performance..Source: Multitemporal Fusion · p. 3
Network Training
- Flooded-building segmentation starts from weights obtained by pretraining Multi3Net on building footprints, then trains on flooded-building footprints.Training uses cross-entropy loss, random tile sampling, and rotation and flipping augmentation.Source: Network Training · p. 4
Data
The study uses two Houston districts affected by Hurricane Harvey and builds rasterized labels by combining building footprints with flooded-building annotations. Its inputs include processed Sentinel-1 radar, Sentinel-2 optical imagery, and post-event very high-resolution imagery.
Area of Interest
- The study selects two neighboring Houston districts affected by Hurricane Harvey, where extreme rainfall flooded neighborhoods after Addicks Reservoir overflowed..Source: Area of Interest · p. 4
Ground Truth
- OpenStreetMap building footprints are combined with DigitalGlobe crowdsourced flooded-building point annotations to create ground truth, rasterized at 2m or 10m according to imagery resolution.The East partition is used for training and the West partition for testing.Source: Ground Truth · p. 4
Data Preprocessing
- Radar preprocessing calibrates and terrain-corrects imagery, reduces speckle through spatial and temporal averaging, and calculates interferometric coherence between image pairs.Sentinel-2 top-of-atmosphere reflectances are used without further atmospheric correction.Source: Data Preprocessing · pp. 4–5
Network Inputs
- Inputs include 5m–10m medium-resolution imagery acquired before and after events and 0.5m post-event imagery; tiles cover 960m × 960m and overlap by 860m.Sentinel-1 inputs combine intensity, multitemporal filtered intensity, and coherence; Sentinel-2 bands are brought to 10m resolution.Source: Network Inputs · p. 5
Experiments & Results
Fusion-based approaches outperform single-sensor models, and combining Sentinel-1, Sentinel-2, and very high-resolution imagery yields the strongest reported building-footprint result. Multi3Net also exceeds a state-of-the-art single-sensor benchmark and the U-Net fusion baseline.
Experiments & Results
- Fusion-based approaches consistently outperformed models using data from single sensors across the reported building-footprint and flooded-building segmentation results.The section presents both quantitative and qualitative results.Source: Experiments & Results · p. 5
Evaluation Metrics
- The evaluation reports bIoU for building and flooded-building predictions, alongside background IoU, their mean (mIoU), and pixel accuracy.The passage defines IoU as the overlap of predicted and target pixels divided by their union; pixel accuracy is the percentage of correctly classified pixels.Source: Evaluation Metrics · pp. 5–6
Building Footprint Segmentation: Single Sensors
- 7.8% higher bIoU than Ohleyer (2018) was achieved using only very high-resolution imagery; among single-sensor inputs, very high-resolution imagery had the highest building IoU, followed by Sentinel-2.The reported single-sensor ranking suggests optical bands contain more relevant information for this task than radar images.Source: Building Footprint Segmentation:… · p. 6
Building Footprint Segmentation: Image Fusion
- 79.9% mIoU with Sentinel-1, Sentinel-2, and very high-resolution imagery exceeded 76.1% mIoU with Sentinel-1 and Sentinel-2; using all data sources achieved the highest accuracy.Multi3Net also outperformed the U-Net fusion baseline on building-footprint segmentation for all input types.Source: Building Footprint Segmentation: Image… · p. 6, Figure 5 · p. 6
Segmentation of Flooded Buildings with Multi3Net
Multi3Net fuses satellite imagery across resolutions, sensors, and time to segment flooded buildings. The reported results include gains over a U-Net fusion baseline and strong performance using publicly available medium-resolution imagery.
Segmentation of Flooded Buildings with Multi3Net
- Fusion eliminated most false positives and more accurately delineated detected structures compared with VHR-only predictions.Some buildings shown in magenta were detected only with multisensor input.Source: Figure 6 · p. 7, Segmentation of Flooded Buildings with… · p. 7
- Multi3Net significantly outperformed a U-Net fusion model, achieving a building IoU score of 75.3% compared with a bIoU score of 44.2% for the U-Net baseline.Source: Segmentation of Flooded Buildings with… · p. 7
- 75.3% mIoU was the best result among tested inputs for multiresolution, multisensor, multitemporal imagery; medium-resolution Sentinel-1 and Sentinel-2 imagery reached 59.7% mean IoU.Table 3 reports flooded-building segmentation results for different Multi3Net input data.Source: Segmentation of Flooded Buildings with… · pp. 6–7, Table 3 · p. 7
Conclusion
- For building-footprint segmentation, Multi3Net with publicly available medium-resolution radar and optical imagery performed almost on par with VHR-based models, while full fusion outperformed alternative fusion approaches for flooded buildings.The authors also report outperforming fully convolutional networks and Mask-RCNNs on VHR building-footprint segmentation, using pyramid sampling pooling to aggregate spatial context.Source: Conclusion · p. 7
- The authors describe the method as applicable to different flood events and easy to deploy, and release a labeled multiresolution, multispectral, multitemporal disaster-imagery dataset and source code.They state that the method substantially reduces the time needed to produce highly accurate flood maps.Source: Conclusion · p. 7
A1. Training & Model Evaluation Details
The area of interest was divided into spatially separate partitions for training and testing, with the East partition split between training and validation.
A1. Training & Model Evaluation Details
- The East partition was randomly divided into training and validation sets at a 4:1 split, while the spatially separate West partition was used for testing.Hyperparameters were optimized on the validation set.Source: A1. Training & Model Evaluation Details · p. 9, Figure 7 · p. 9
- Figure 7 marks the East and West training and testing partitions and highlights flooded buildings in red.Source: Figure 7 · p. 9
A2. Additional Experiments
Multi3Net outperformed a U-Net fusion baseline on building-footprint segmentation across input types and on flooded-building segmentation.
A2. Additional Experiments
- 75.3% bIoU for Multi3Net versus 44.2% bIoU for the U-Net baseline on flooded-building segmentation.Source: A2. Additional Experiments · p. 9
- Multi3Net outperformed the U-Net fusion baseline on building-footprint segmentation for all tested input types.Table 4 presents the building-footprint segmentation results for Multi3Net and the U-Net baseline.Source: A2. Additional Experiments · p. 9, Table 4 · p. 9