Source-linked AI summary
SpaceNet 6: Multi-Sensor All Weather Mapping Dataset
Jacob Shermeyer, Daniel Hogan, Jason Brown, Adam Van Etten, Nicholas Weir, Fabio Pacifici, Ronny Haensch, Alexei Bastidas, Scott Soenen, Todd Bacastow, Ryan Lewis
TL;DR
MSAW addresses limited open, high-resolution SAR data for building-footprint extraction by providing a multi-modal dataset, challenge, and baseline. The benchmark finds that optical pre-training and transfer learning improve SAR performance over SAR-only training.
Problem
Most open high-resolution overhead-imagery datasets focus on optical data, while building-footprint extraction with SAR remains insufficiently explored.
Method
The paper introduces the MSAW dataset and challenge, combining SAR and optical imagery over approximately 120 km^2 with 48,000 building-footprint labels, and presents extraction benchmarks.
Results
0.21±.02 with optical transfer learning versus 0.135±.002 without it, showing a performance benefit from pre-training on optical data before SAR training.
Takeaways & Limitations
The results support using optical pre-training and consistent SAR viewing-direction treatment when developing building-footprint extraction models.
Takeaways & Limitations
SAR-domain transfer and domain-adaptation practices remain nascent, so further research is required to establish best practices.
Abstract
from arXiv · showhide
Within the remote sensing domain, a diverse set of acquisition modalities exist, each with their own unique strengths and weaknesses. Yet, most of the current literature and open datasets only deal with electro-optical (optical) data for different detection and segmentation tasks at high spatial resolutions. optical data is often the preferred choice for geospatial applications, but requires clear skies and little cloud cover to work well. Conversely, Synthetic Aperture Radar (SAR) sensors have the unique capability to penetrate clouds and collect during all weather, day and night conditions. Consequently, SAR data are particularly valuable in the quest to aid disaster response, when weather and cloud cover can obstruct traditional optical sensors. Despite all of these advantages, there is little open data available to researchers to explore the effectiveness of SAR for such applications, particularly at very-high spatial resolutions, i.e. <1m Ground Sample Distance (GSD). To address this problem, we present an open Multi-Sensor All Weather Mapping (MSAW) dataset and challenge, which features two collection modalities (both SAR and optical). The dataset and challenge focus on mapping and building footprint extraction using a combination of these data sources. MSAW covers 120 km^2 over multiple overlapping collects and is annotated with over 48,000 unique building footprints labels, enabling the creation and evaluation of mapping algorithms for multi-modal data. We present a baseline and benchmark for building footprint extraction with SAR data and find that state-of-the-art segmentation models pre-trained on optical data, and then trained on SAR (F1 score of 0.21) outperform those trained on SAR data alone (F1 score of 0.135).
1. Introduction and Related Work
High-resolution remote-sensing datasets have focused mainly on optical imagery, leaving SAR underrepresented despite its all-weather, day-and-night capabilities. MSAW addresses this gap with co-collected SAR and optical data, building annotations, and a public challenge for overhead mapping.
- Research gap: Most open high-resolution overhead datasets focus on optical imagery and omit other remote-sensing modalities such as SAR.Existing SAR datasets are few, and available alternatives are often small, non-georeferenced, or too coarse for high-resolution multimodal mapping.
- SAR characteristics: SAR actively illuminates the ground with radio waves, enabling collection without sunlight and through cloud cover.Radar intensity reflects returned energy related to material properties and physical shape rather than visible color.
- SAR characteristics: SAR imagery introduces interpretation and computer-vision challenges through non-literal appearance and geometric distortions such as foreshortening and layover.Layover can place returns from an object’s top nearer the sensor and obscure its bottom.
- MSAW contribution: MSAW combines half-meter quad-polarized X-band SAR and half-meter optical imagery over 120km2 in Rotterdam.The dataset is designed for very-high-resolution multimodal mapping and extends the SpaceNet corpus geographically.
- MSAW contribution: MSAW provides 48,000 unique building footprints with height information and includes a public challenge for developing mapping algorithms.Challenge results are intended to provide a baseline and reference benchmark for future SAR and optical research.
2. Dataset
MSAW combines high-resolution SAR and optical imagery over Rotterdam with building annotations, preprocessing, and a train/test structure designed for SAR-focused evaluation.
- Dataset extent and content: MSAW covers approximately 120 km^2 of Rotterdam with heterogeneous urban, rural, suburban, industrial, and port environments.The imagery is intended to support multi-modal data-fusion experiments across varied building sizes, densities, contexts, and appearances.
- Dataset split and evaluation: Training contains SAR and optical imagery, whereas testing and final scoring contain SAR only to mimic settings where historical optical data exist but concurrent optical collection is unavailable.The tiled data are split into 50% training, 30% testing, and 20% final scoring.
- Sensor collections: The dataset includes repeated aerial X-band SAR collects with four polarizations and a cloud-free WorldView-2 optical collection.SAR strips were collected over three days, while the optical imagery was collected on August 31, 2019.
- Sensor processing: SAR imagery is co-registered, multilooked, intensity-transformed, geo-registered, and ortho-rectified before use.Processing includes subpixel polarization alignment, noise reduction, logarithmic scaling, and correction for topography.
- Sensor processing: Optical data comprise 0.5 m panchromatic, 2.0 m multispectral, and 0.5 m pan-sharpened bands.The optical products are atmospherically compensated and ortho-rectified before tiling.
- Annotations and tiling: Building labels derive from 3DBAG polygons, undergo manual quality control, and are refined by dissolving co-located address polygons.Areas with substantial annotation inaccuracies are excluded, and labels are clipped to valid imagery during tiling.
3. Baseline Building Extraction Experiments
The MSAW baseline extracts building footprints from SAR imagery using a U-Net-based segmentation pipeline with optical-to-SAR transfer learning and viewing-direction normalization. Performance varies with modality preparation and building geometry, while building properties help explain IOU variation.
- Baseline Model: The baseline uses a TernausNet architecture to produce building-versus-background segmentation masks, which are converted into vector building footprints.It combines a U-Net with a VGG-11 encoder and a weighted Dice and focal loss.
- Baseline Model: Optical pretraining initializes SAR training, while SAR rotations standardize layover direction and small buildings under 20m2 are excluded from SAR training.The rotation addresses varying acquisition directions, and excluding tiny structures is intended to improve performance on larger buildings.
- Metrics: The SpaceNet Metric evaluates footprint extraction using precision and recall at an IOU threshold of 0.5, emphasizing instance-level delineation over pixel-wise agreement.The metric supports detailed counts of building instances and sizes.
- Results and Discussion: 0.21±.02 F1 is achieved by the baseline on MSAW, comparable to earlier overhead-optical building-extraction baselines and recent off-nadir optical models.The reported comparison concerns the SpaceNet building-extraction metric.
- Results and Discussion: 0.135±.002 F1 without optical transfer learning and 0.12±.03 with random rotations show benefits from optical initialization and consistent SAR viewing direction.These values are compared across baseline variants.
- Results and Discussion: Recall is approximately 0.1 near 100m2, rises toward 0.5 for much larger buildings, peaks near 0.5 around 20m height, then declines above 20m.The authors associate the decline for taller buildings with stronger layover or foreshortening, while buildings over 30m average recall below 0.2.
4. Conclusion
MSAW contributes a multi-modal SAR–optical dataset, baseline, and challenge for building-footprint extraction, while its results motivate further research into high-resolution SAR feature extraction and multi-modal analysis.
- MSAW provides SAR and optical imagery with 48,000 building-footprint labels and height estimates from LiDAR.
- The paper establishes a baseline and quantitative benchmarks for building-footprint extraction using SAR and optical data.
- Optical pre-training followed by transfer learning improves performance by 55% over training on SAR data alone.
- The overall F1 score of 0.21 remains relatively low, leaving scope for future research on high-resolution SAR feature extraction.
- The dataset is intended to support multi-modal analysis, data fusion, small-object detection, and cross-modal domain-adaptation research.