Source-linked AI summary
SpaceNet: A Remote Sensing Dataset and Challenge Series
Adam Van Etten, Dave Lindenbaum, Todd M. Bacastow
TL;DR
Foundational mapping remains labor-intensive, especially when disaster response requires timely updates. SpaceNet addresses this need with a large labeled satellite-imagery corpus and public challenges spanning building and road extraction, reporting strong building-footprint performance and a graph-based road-network result.
Problem
Foundational mapping relies heavily on manual creation and validation of roads and building footprints, despite its importance for humanitarian and disaster response.
Method
SpaceNet releases labeled satellite imagery and runs public challenges for automated building-footprint and road-network extraction, using IoU-based building evaluation and the graph-theoretic APLS metric for roads.
Results
A building-footprint challenge achieved an F1 score of 0.69 on multiple cities, while the road-network competition produced a winning APLS score of 0.66.
Takeaways & Limitations
The dataset and challenge series provide validated training data and competition outputs aimed at improving automated foundational mapping, including routable road-network extraction.
Takeaways & Limitations
Pixel-based metrics inadequately incentivize connected road networks, and APLS, although designed for this purpose, is not differentiable.
Abstract
from arXiv · showhide
Foundational mapping remains a challenge in many parts of the world, particularly in dynamic scenarios such as natural disasters when timely updates are critical. Updating maps is currently a highly manual process requiring a large number of human labelers to either create features or rigorously validate automated outputs. We propose that the frequent revisits of earth imaging satellite constellations may accelerate existing efforts to quickly update foundational maps when combined with advanced machine learning techniques. Accordingly, the SpaceNet partners (CosmiQ Works, Radiant Solutions, and NVIDIA), released a large corpus of labeled satellite imagery on Amazon Web Services (AWS) called SpaceNet. The SpaceNet partners also launched a series of public prize competitions to encourage improvement of remote sensing machine learning algorithms. The first two of these competitions focused on automated building footprint extraction, and the most recent challenge focused on road network extraction. In this paper we discuss the SpaceNet imagery, labels, evaluation metrics, prize challenge results to date, and future plans for the SpaceNet challenge series.
1 Background
SpaceNet addresses the manual burden of foundational mapping by releasing labeled satellite imagery and organizing public challenges for automated building and road extraction. These capabilities are framed as useful for humanitarian response and broader remote-sensing applications.
- Manual mapping currently creates roads, building footprints, and points of interest, motivating automated feature extraction for downstream humanitarian and disaster-response uses.
- The first two SpaceNet challenges targeted building footprint extraction, while the third targeted road network extraction.
- Automated building and road extraction could support aid-station placement during epidemics and logistics in disaster-stricken regions.
- The SpaceNet team released a large labeled satellite-imagery corpus on AWS alongside public challenges designed to increase satellite imagery’s utility.
- SpaceNet’s future plan is to expand automated capabilities through high-quality labeled datasets and targeted challenges addressing increasingly complex geospatial problems.
2 Source Imagery and Labels
SpaceNet combines high-resolution imagery with validated building and road labels across multiple challenge datasets and geographic areas. The corpus is publicly distributed and supports varied remote-sensing machine-learning research.
- 2.1 Existing Datasets: Existing labeled overhead datasets are often small or have lower-than-desired labeling fidelity for foundational mapping.
- 2.3 Challenge 1 - Rio De Janeiro Building Footprints: SpaceNet Challenge 1 used WorldView 2 imagery with 50 cm pan-sharpened RGB resolution, 8-band multispectral imagery, and 2,544 km2 of coverage.
- 2.3 Challenge 1 - Rio De Janeiro Building Footprints: Challenge 1 labeled over 300,000 Rio de Janeiro building footprints, with adjoining buildings marked as separate structures.
- 2.4 Challenge 2 - Las Vegas, Paris, Shanghai, Khartoum Building Footprints: Challenge 2 covered Las Vegas, Paris, Shanghai, and Khartoum in 24,586 200 m × 200 m scenes containing 302,701 building footprints.
- 2.5 Challenge 3 - Road Networks: Challenge 3 reused Challenge 2 imagery in 400 m chips and labeled road centerlines and attributes including road type, surface type, bridges, and lane number.
- 2.6 Additional Data: SpaceNet also hosts datasets from the IARPA Functional Map of the World and Urban 3D Challenge to diversify its corpus and encourage new machine-learning research.
3 The SpaceNet Buildings Metric
The SpaceNet Buildings Metric evaluates building-footprint detection by measuring object overlap, applying a one-to-one matching rule, and averaging city-level F1 scores.
- Building footprint extraction is evaluated as an object detection problem using scale-invariant intersection over union (IoU), which measures overlap between predicted and labeled objects.IoU divides intersection area by union area.
- An IoU of ≥0.5 denotes a true positive detection.
- Each labeled region can have at most one true positive, selected through a decreasing-IoU sequential search.Matched label–proposal pairs are removed before the search continues.
- For each scene, the metric counts true positives, false positives, and false negatives, then aggregates them within each city to calculate precision, recall, and F1.False positives are proposals with IoU < 0.5, while false negatives are ground-truth buildings without valid proposals.
- The total SpaceNet Buildings Metric is the arithmetic mean of the F1 scores for each city.
4 APLS: The SpaceNet Roads Metric
The paper develops APLS, a graph-based metric for road-network extraction that evaluates physical routes and connectivity rather than relying only on pixel overlap. It addresses cases where legacy metrics reward visually accurate but unroutable predictions.
- 4.1 Pixel-Based Metrics: Pixel-based IoU, F1, and relaxed F1 metrics can penalize small road-width errors more heavily than brief breaks that disrupt network connectivity.Figure 3 illustrates this mismatch: the prediction with a road gap receives higher legacy scores despite being unsuitable for routing.
- 4.2 APLS Metric: APLS measures similarity by symmetrically comparing optimal path lengths between corresponding nodes in ground-truth and proposal road graphs.The metric uses Dijkstra’s shortest-path algorithm and sums proportional path-length differences across graph paths.
- 4.2 APLS Metric: APLS assigns a maximum proportional difference of 1.0 when a corresponding path is missing, heavily penalizing disconnected routes.This makes missing roads consequential for the metric rather than treating them as limited pixel-level errors.
- 4.2 APLS Metric: The metric incorporates betweenness centrality, penalizing missing high-centrality nodes more than missing low-centrality nodes because intersections matter more for routing.Missing edges, including breaks caused by overhanging trees, are therefore heavily penalized.
- 4.2.1 Graph Augmentation: APLS uses control nodes at intersections, endpoints, and 50-meter edge midpoints to capture both logical and physical road topology.For imperfect alignment, ground-truth control points are snapped to proposal nodes within a default 4-meter buffer; a reverse snapping operation penalizes spurious proposals.
- 4.2.4 Total APLS Metric: Total APLS combines two directional scores through their harmonic mean and scales from 0 (poor) to 1 (perfect).One direction snaps ground-truth nodes to the proposal graph, while the reverse snaps proposal nodes to the ground-truth graph.
5 SpaceNet Competition Results
SpaceNet competitions advanced automated building and road extraction, with performance improving across challenges and strong results from deep-learning-based approaches.
- Building footprint challenges: The first building challenge’s YOLT baseline achieved an F1 score of 0.21, while the winning random-forest polygonization algorithm reached F1 = 0.26.
- Building footprint challenges: The second building challenge achieved F1 = 0.69 with an ensemble of three U-Net segmentation masks, thresholding, and filtering.YOLT and modified MNC baselines achieved F1 = 0.60 and F1 = 0.57, respectively.
- Building footprint challenges: The second building challenge outperformed the first, likely reflecting higher resolution, improved labels, and/or improved competitor algorithms.
- Road network challenge: Road-challenge entries were surprisingly good despite the novel metric and requirement for geospatial vector submissions.
- Road network challenge: The road baseline achieved APLS = 0.49, while top performers used related segmentation, skeletonization, and post-processing approaches.The winning implementation used an ensemble of deep-learning segmentation encoders and decoders to train a global road model.
6 Conclusions
SpaceNet positions satellite imagery and machine learning as tools for updating foundational maps, especially where disasters or other dynamic events demand broad, timely coverage.
- Satellite imagery may provide faster, broader updates for foundational mapping during natural disasters and other dynamic events than terrestrial or airborne methods.
- SpaceNet provides high-resolution multi-band imagery with validated building-footprint and road-network labels.
- Building-footprint challenges produced algorithms with F1 scores of 0.69 across multiple cities and showed vast improvement from Challenge 1 to Challenge 2.
- The paper introduces APLS because pixel-based F1 is suboptimal for routing, and reports a winning road-network score of 0.66.APLS is based on shortest paths and rewards correct road centerlines and intersections more strongly than pixel-based metrics.
- Future SpaceNet plans include more imagery, labels, and challenges, including off-nadir building-footprint localization for disaster-response scenarios.
The SpaceNet Roads Dataset labeling guidelines
The roads dataset defines road geometry, connectivity, classification, and attributes to produce centerline-based networks suitable for routing.
- Road vectors must be centerlines within 2m, or 7 pixels, of the observed road.Even- and odd-lane roads use specified centerline conventions, while divided highways receive one centerline per traffic direction.
- Road vectors must form connected networks for routing, while non-connected overpasses must not share intersection points.
- Roads must not bisect building footprints, and bridges or overpasses must carry a Boolean bridge flag.
- A divided highway is defined as a road with a physical median or barrier preventing turns across traffic.Listed median types include concrete, asphalt, green space, and dirt or unpaved road; yellow hatched pavement lines do not qualify.
- Road types follow the OpenStreetMap highway taxonomy, including motorway, primary, secondary, tertiary, residential, and unclassified categories.
- The schema records geometry, road identifier, road type, paving status, bridge type, and lane number.
Appendix C. APLS Metric
APLS compares ground-truth and proposal road networks through shortest-path differences between selected control nodes, using node injection and symmetric snapping to account for imperfect alignment and spurious roads.
- APLS sums path-length differences across possible routes between graph nodes.
- Control nodes are placed at intersections, endpoints, and edge midpoints, with nodes injected every 50 meters for SpaceNet’s 400m tiles.
- The metric augments the proposal graph by snapping ground-truth control points to nearby proposal edges within a default 4-meter buffer.
- The inverse snapping operation maps proposal control nodes onto the ground-truth graph to penalize spurious road proposals.
Appendix D. Building Detection Results
This section presents snapshots from the winning implementation of XD_XD for Buildings Round 2, including results across four cities and visual distinctions between prediction outcomes and ground truth.
- The section provides snapshots of the winning algorithm rather than a quantitative comparison.
- The winning XD_XD implementation is shown on Buildings Round 2 test imagery from Vegas, Khartoum, and Paris.The figure lists Vegas twice among the four clockwise snapshots.
- Blue outlines mark ground truth, green outlines mark true positives, and red outlines mark false positives.
Appendix E. Road Baseline Algorithm
The road baseline converts SpaceNet imagery and GeoJSON labels into road-network graphs through CNN segmentation, mask refinement, skeletonization, and graph rendering. The APLS metric is presented as better suited than pixel-based F1 for evaluating route inference because it rewards node placement and connectivity.
- APLS rewards correct node placement and connectivity, making it a proposed improvement over pixel-based F1 for automated route inference evaluation.
- The baseline begins by inferring road segmentation masks from SpaceNet imagery with convolutional neural networks.Examples include PSPNet and U-Net.
- Training masks are created from raw imagery and SpaceNet GeoJSON road labels using a 2m road centerline halfwidth assumption.
- The paper illustrates the pipeline with a GeoJSON label, an orange road-network overlay, a road-centerline segmentation mask, and a sample test-data output.
- Predicted masks are refined with thresholding, opening, closing, and smoothing before downstream graph construction.
- The refined mask is skeletonized and then rendered into a graph structure for road-network representation.The figure cites scikit-image skeletonize and the sknw package as example tools.