A comparative assessment of global building and settlement datasets across geographic and settlement contexts

1 Introduction

• Buildings and settlement layers support population mapping, hazard-exposure analysis, urban-growth monitoring, and fine-scale study of settlement form and infrastructure.These uses are especially important where local information is limited despite rapid settlement growth.Source: §1 · p. 1
• A product can match aggregate building area while missing or misplacing individual buildings, so no single accuracy metric captures every operational question.Omission, false detections, and positional or delineation errors affect completeness, quantities, and geometric agreement differently.Source: §1 · pp. 1–2
• Accuracy varies with settlement conditions and reference-data properties, including building size, density, mapping source, and temporal alignment.Aggregate statistics can conceal differences across geographic and urban–rural settings.Source: §1 · p. 2
• The study benchmarks seven vector and raster products against harmonized footprints across 135 study areas, using complementary accuracy measures and stratified and diagnostic analyses.Its questions address product rankings within data structures, variation across contexts and raster resolutions, and relationships among detection, geometry, and quantity accuracy.Source: §1 · p. 2

2 Related Work

• Reported building-dataset accuracy depends on the reference, geographic setting, unit of analysis, spatial scale, and metric, limiting direct comparison across studies.Even similarly named measures may be calculated over buildings, pixels, grid cells, tiles, or cities.Source: §2 · p. 2
• Vector studies have measured interproduct overlap or product-specific instance detection, while raster studies have assessed classification agreement and aggregated built-up quantities.These evaluation approaches do not measure interchangeable properties.Source: §2 · p. 3
• Earlier contextual analyses examined selected products or settings, but had not jointly compared detection, geometric, and quantity accuracy across multiple vector and raster products.Prior stratification considered factors such as density, region, wealth, urban–rural setting, or building size, often within narrower scopes.Source: §2 · p. 3
• Reference provenance and temporal mismatch can affect measured errors, while the literature had few common-protocol evaluations combining these diagnostics with multiple accuracy dimensions.In rapidly growing settlements, buildings added after reference imagery may appear as false positives against an older reference.Source: §2 · pp. 3–4

3 Datasets

• The benchmark includes three vector products and four raster products, evaluated against reference footprints across 135 cities, communities, and municipalities.The study areas span multiple continents, with particular emphasis on the Global South.Source: §3 · p. 4

3.1 Vector Building-Footprint Products

• Overture Maps, Global Building Atlas, and 3D-GloBFP provide vector building footprints, with the latter two also reporting building-height information.Overture is an open global vector dataset distributed in GeoParquet; GBA combines PlanetScope extraction with fused footprint sources, while 3D-GloBFP provides building polygons and height estimates for approximately 2020.Source: §3.1 · p. 4

3.2 Raster Settlement and Building-Density Products

• The raster products differ in what they represent and in their temporal and spatial characteristics, spanning building presence, density, height, built-up surface, and settlement extent.OBT provides annual estimates from 2016–2023; TEMPO provides quarterly estimates from 2018–2025; GHSL includes five-year built-up-surface estimates; WSF Tracker provides settlement-extent time series at approximately six-month intervals.Source: §3.2 · p. 4

3.3 Reference Datasets

• Reference footprints were compiled from institutional and community sources and screened for geometric quality, temporal suitability, and consistency with visible buildings.Sources include HOTOSM campaigns, World Bank-supported projects, sampled SpaceNet 7 urban areas, and University of Edinburgh mapping of informal-settlement dwellings in Cape Town; Table 1 describes product temporal coverage, imagery epochs, and updates.Source: §3.3 · pp. 4, 6, Table 1 · p. 5

4 Methodology

• The framework matches each product’s representation to its evaluation: object-based detection and geometry for vectors, grid-based built-up presence and area for rasters, plus stratified and diagnostic analyses.Candidate rasters are binarized at native resolution before aggregation to a common grid.Source: §4 · p. 6

4.2 Vector Building-Footprint Evaluation

• Vector assessment separates object detection from geometric agreement, using F1, matched IoU, relative area error, and boundary F-score with a 2 m positional tolerance.IoU captures geometry-sensitive segmentation quality, while relative area error tests footprint-area bias.Source: §4.2 · p. 6, §4.2.2 · p. 8

4.2.1 Object Matching and Detection

• Within fixed 1 km × 1 km tiles, vector footprints are matched one-to-one by IoU; matches define true positives, while unmatched candidates and references define false positives and negatives.The IoU threshold is 0.25 for SpaceNet 7 references and 0.50 for other reference datasets.Source: §4.2.1 · p. 6, Figure 2 · p. 7

4.3.1 Raster Accuracy Metrics

• Raster metrics combine pixel-level precision, recall, and F1 with built-up-area bias and quantity and allocation disagreement.Quantity disagreement captures differences in total mapped area; allocation disagreement captures spatially paired errors despite the correct total amount.Source: §4.3.1 · p. 8

4.4 Building Count and Density Assessment

• For raster products, approximate building counts are derived from predicted built-up area and mean reference building area, while vector counts come directly from polygons.Count differences assess whether gridded products can support approximate count or density estimates.Source: §4.4 · pp. 8, 10

4.3 Raster Settlement-Layer Evaluation

• Raster evaluation compares binarized reference and candidate built-up masks at 10 m for OBT and WSF Tracker and at 100 m for all raster products.Reference cells use a fixed 20 m² built-up threshold; candidate masks are aggregated and re-thresholded on the common evaluation grid.Source: §4.3 · p. 8, Figure 3 · p. 9

4.5 Stratified evaluation and diagnostic experiments

• Stratified analyses examine reference-data source, building density, and mean building size using fixed classes and Spearman rank correlations for tile-level F1 associations.The source stratification distinguishes references sharing OpenStreetMap provenance with a candidate product from those that do not.Source: §4.5 · p. 10
• Two diagnostics assess whether small buildings concentrate vector errors and whether matching WSF Tracker dates to reference imagery changes city-level F1.The temporal experiment uses the WSF Tracker time step closest to each reference-imagery acquisition date.Source: §4.5 · p. 10

5.1.1 Vector Building-Footprint Products

• Overture led vector products with median city-level F1 0.786 and was top in 58% of common-coverage areas; city-by-city switching raised median F1 only 0.004.Source: §5.1.1 · p. 10, Figure 4 · p. 11

5.1.2 Raster Settlement Products

• At 100 m, WSF Tracker had the highest median F1 (0.862), ahead of TEMPO (0.768), GHSL (0.752), and OBT (0.718); 10 m and 100 m evaluate different spatial-granularity tasks.Source: §5.1.2 · p. 11, Figure 5 · p. 12

5.3.1 Reference-Data Source

• WSF Tracker at 100 m had the highest median F1 across reference groups, while the roughly 0.13 income-group gap in 52 SpaceNet 7 cities was exploratory.The benchmark was not designed as a regionally representative sample.Source: §5.3.1 · pp. 12, 14

5.2.1 Vector Area and Building Counts

• Overture and GBA overestimated vector quantities, while 3D-GloBFP was closest to aggregate agreement despite having the lowest detection F1.Median area bias and count ratio were 0.093 and 1.155 for Overture, and 0.124 and 1.167 for GBA; 3D-GloBFP's were −0.045 and 0.905. Near-zero median bias did not ensure consistent city-level performance.Source: §5.2 · p. 11, §5.2.1 · p. 12, Figure 6 · p. 13

5.2.2 Raster Built-Up-Area Agreement

• OBT had the smallest median relative area errors (0.125 at 10 m; 0.176 at 100 m), while WSF Tracker's were largest (2.146 at 10 m; 2.285 at 100 m).At 100 m, quantity disagreement accounted for 83%–99% of total disagreement. Estimated-to-reference count ratios were 1.09 for TEMPO, 1.10 at 100 m and 1.11 at 10 m for OBT, 1.48 for GHSL, and 2.6 for WSF Tracker; product-specific calibration could improve count estimates.Source: §5.2.2 · p. 12, Figure 7 · p. 13

5.3.2 Building Density and Size

• Raster tile F1 rose with reference density (Spearman ρ = 0.58–0.75), whereas vector density effects were weaker and building-size associations varied by product.Dense cells reduce the influence of positional errors; GBA and 3D-GloBFP performed less well in tiles dominated by small buildings, while Overture was largely insensitive to mean building size.Source: §5.3.2 · p. 14

5.3.3 High false positive rates for smaller buildings

• For all three vector products, median false-positive footprint area was 58–72 m2 versus 87–116 m2 for true positives, and 71–81% of 20–40 m2 candidates were false positives.Source: §5.3.3 · p. 14, Table 5 · p. 14

5.3.4 Temporal mismatch

• Aligning WSF Tracker to the closest reference-imagery date increased mean F1 by +0.060 (median +0.037), suggesting reported accuracy may be conservative in fast-growing cities.Extending the observation cutoff beyond the reference date increased recall but lowered precision and F1 because newly detected settlements counted as false-positive area against older references.Source: §5.3.4 · p. 14

6.1 Interpreting Comparative Product Performance

• Overture was the strongest vector product for detection and geometric agreement, while raster rankings varied by resolution and aggregate quantity agreement did not always track F1.At 10 m, OBT had higher F1 and closer reference-area agreement than WSF Tracker; at 100 m, WSF Tracker had the highest F1 but the largest positive area bias.Source: §6.1 · pp. 14–15
• Coarser aggregation raises overlap scores by reducing sensitivity to local positional, omission, and delineation errors, so 10 m and 100 m scores represent different evaluation emphases.At 100 m, quantity disagreement accounted for most raster disagreement, and positive area bias can reflect mapped roads, parking areas, and other built surfaces as well as classification error.Source: §6.1 · pp. 15–16

6.2 Variation across Geographic and Settlement Contexts

• Overture and GBA agreed especially well with HOTOSM references, but shared OpenStreetMap provenance may contribute to that result; Overture's advantage was smaller against independent sources.Reference-source comparisons are confounded with geography, acquisition date, settlement form, and annotation protocol, and the reference sample was not designed to be globally representative.Source: §6.2 · p. 16
• Raster F1 rose with settlement density and was lowest and most variable in sparse rural and peri-urban areas; vector performance was less density-sensitive, particularly for Overture.Building size had a secondary effect: 10 m raster products improved modestly for larger buildings, while GBA and 3D-GloBFP were less consistent in tiles dominated by small or dispersed buildings.Source: §6.2 · p. 16

6.3 Implications for future evaluation

• The benchmark combines representation-specific evaluation, complementary detection, geometry, area, and count metrics, stratification by city areas, and temporal-alignment diagnostics; its procedures can test future products and releases.The discussion recommends reporting object-, grid-, and area-based metrics alongside reference provenance, temporal alignment, resolution, and the physical feature represented by each product.Source: §6.3 · pp. 16–17
• Global products offer broad coverage and consistent processing but may not match locally produced high-resolution datasets where regional precision is needed.Local pipelines may trade global consistency for regional precision, and the same validation framework can compare them with global products.Source: §6.3 · p. 17

6.4 Limitations

• Product and reference dates differ, so construction, demolition, and redevelopment can appear as errors; reported agreement may therefore be conservative, especially in rapidly urbanising locations.Overture and HOTOSM references also share OSM provenance, which may inflate agreement in those locations.Source: §6.4 · p. 17
• Pooled statistics combine different SpaceNet 7 and other-source matching thresholds, and city-scale results do not certify individual building polygons or support building-level decisions without independent verification.City-level averages may conceal weaker performance in sparse peripheral settlements.Source: §6.4 · p. 17

7 Conclusion

• No global building dataset is uniformly superior across representations, metrics, and settlement conditions; Overture led vector detection, while raster leaders differed by evaluation resolution.Overture's median city-level F1 was 0.786; OBT's median F1 was 0.642 at 10 m, while WSF Tracker's was 0.862 at 100 m.Source: §7 · p. 17
• Raster F1 increased with building density and was least reliable in sparse areas, while coarser 100 m evaluation raised F1 without indicating better fine-scale delineation.The 10 m products performed somewhat better where buildings were larger, whereas vector products were less sensitive to these conditions, particularly Overture Maps.Source: §7 · pp. 17–18
• Global products are useful where authoritative local footprints are unavailable, but parcel-level reliability or detailed geometry may require locally validated high-resolution mapping.Their uncertainty, resolution, area bias, and geographic variation also need explicit consideration.Source: §7 · p. 18

Declaration of generative AI use

• Generative AI assisted with language refinement, editing, and manuscript clarity; authors reviewed the content and retained responsibility, and AI was not used on study data or results.Source: Declaration of generative AI use · p. 18

Supplementary materials

• The procedure rasterises vector references, binarises each candidate at native resolution, then area-weights aggregation to the common evaluation grid.This preserves native spatial support while applying a common built-up-area definition at the evaluation resolution.Source: Raster harmonisation and binarisation · p. 21

• Reference footprints were rasterised with integer oversampling and block-averaged to preserve sub-pixel coverage.Source: §1 · p. 21
• 20 m2 was the minimum reference area for classifying a pixel as built-up.Source: §1 · p. 21
• The same Mref threshold was applied across candidate datasets to keep the reference definition of built-up constant.Source: §1 · p. 21

• Continuous candidates used native pixel values and areas to estimate predicted built-up area before binarisation.Source: §2 · p. 21
• Native pixels were classified using each dataset's minimum detectable building area, Md.Source: §2 · p. 21
• Categorical products such as WSF Tracker used valid built-up codes rather than an area threshold.Source: §2 · p. 21

3. Aggregation to the evaluation grid

• Native binary masks were aggregated by overlap area, allowing integer and non-integer resolution ratios while excluding nodata and pixels without valid observations.Source: §3 · p. 21
• The aggregated prediction and reference used the same evaluation-grid threshold, while area metrics retained continuous predicted area and sub-pixel information.Source: §3 · pp. 21–22

Temporal coverage of multitemporal datasets

• WSF Tracker encodes settlement timing per pixel, and benchmarking used global cutoff as_of_code=19, the most recent available period at acquisition.Source: Temporal coverage of multitemporal… · p. 22