PaperlayerView exact revision on arXiv

Source-linked AI summary

Building Extraction at Scale using Convolutional Neural Network: Mapping of the United States

Hsiuhan Lexie Yang, Jiangye Yuan, Dalton Lunga, Melanie Laverdiere, Amy Rose, Budhendra Bhaduri

arXiv:1805.08946v1cs.CV

TL;DR

Large-scale, up-to-date building maps are needed for understanding urban dynamics, but reliable extraction across extensive remote-sensing imagery remains challenging. The paper compares four CNN architectures across the continental United States, then combines SegNet with signed-distance labels and additional near-infrared information. The resulting framework produces seamless United States building maps without further post-processing and supports building extraction at scale.

  • Problem

    Reliable large-scale building-footprint mapping remains challenging, despite the importance of building maps for understanding urban dynamics and estimating population and planning cities.

  • Method

    The paper evaluates four state-of-the-art CNNs, combines SegNet with signed-distance labels for instance-level extraction, and fuses models using additional spectral bands.

  • Results

    The improved CNN framework generates seamless building maps for the contiguous United States without further post-processing.

  • Takeaways & Limitations

    The reported building quality and processing time demonstrate that the CNN-based framework fits large-scale building extraction.

  • Takeaways & Limitations

    GPU memory capacity and latency between GPU nodes constrain batch size and CNN architecture size, while limiting experiments with more than three spectral bands.

Abstract

from arXiv · show

Establishing up-to-date large scale building maps is essential to understand urban dynamics, such as estimating population, urban planning and many other applications. Although many computer vision tasks has been successfully carried out with deep convolutional neural networks, there is a growing need to understand their large scale impact on building mapping with remote sensing imagery. Taking advantage of the scalability of CNNs and using only few areas with the abundance of building footprints, for the first time we conduct a comparative analysis of four state-of-the-art CNNs for extracting building footprints across the entire continental United States. The four CNN architectures namely: branch-out CNN, fully convolutional neural network (FCN), conditional random field as recurrent neural network (CRFasRNN), and SegNet, support semantic pixel-wise labeling and focus on capturing textural information at multi-scale. We use 1-meter resolution aerial images from National Agriculture Imagery Program (NAIP) as the test-bed, and compare the extraction results across the four methods. In addition, we propose to combine signed-distance labels with SegNet, the preferred CNN architecture identified by our extensive evaluations, to advance building extraction results to instance level. We further demonstrate the usefulness of fusing additional near IR information into the building extraction framework. Large scale experimental evaluations are conducted and reported using metrics that include: precision, recall rate, intersection over union, and the number of buildings extracted. With the improved CNN model and no requirement of further post-processing, we have generated building maps for the United States. The quality of extracted buildings and processing time demonstrated the proposed CNN-based framework fits the need of building extraction at scale.

I. INTRODUCTION

The paper addresses the difficulty of creating reliable large-scale building-footprint databases by evaluating state-of-the-art CNNs and developing a scalable extraction framework. It extends extraction toward precise instance-level outlines, spectral-band fusion, and seamless mapping across the contiguous United States.

  • Reliable building-footprint databases remain challenging to establish at scale from remote sensing imagery because existing methods rely on unscalable assumptions or limited building hypotheses.
  • The study evaluates several state-of-the-art CNNs across testing sites in the United States to assess their suitability for efficient large-scale building extraction.
  • Signed-distance labels are incorporated to improve building-outline precision and move extraction from semantic segmentation toward instance-level labeling.
  • Fusing CNN models trained with additional spectral bands yields a desirable performance boost while using a pretrained model for initialization.
  • A single selected CNN generates the first seamless building maps for the contiguous United States using a GPU cluster, followed by quality checks and minimal retraining.

II. RELATED WORK

The related work establishes why building extraction requires dense pixel-wise segmentation rather than patch-level classification. CNN-based approaches improve feature learning and scalability, while FCN-style methods address resolution and dense-labeling challenges.

  • Building maps support urban planning, population modeling, socioeconomic studies, and other geospatial applications, making remote sensing a primary data resource for their generation.
  • Patch-level CNNs assign one class to each image patch, whereas building extraction requires pixel-wise labels to recover complete building outlines.
  • Pooling in classification CNNs produces feature maps smaller than the input, but same-resolution pixel-wise outputs are critical for interpreting remote sensing imagery.
  • FCN replaces fully connected layers with convolutional layers and uses upsampling to generate dense prediction maps matching the input image size.
  • The study prefers end-to-end training without post-processing because it enables more repeatable and scalable evaluations for large-scale building mapping.

C. Pre-trained Models

The paper builds on pretrained CNNs and dense-labeling architectures to improve building extraction while emphasizing boundary accuracy. FCN provides multiscale dense predictions, but its boundary detail remains limited for small buildings.

  • Deep CNNs require sufficient training samples and often long training times, while pretrained models can improve results and shorten fine-tuning time.
  • The paper combines pretrained-model benefits with pixel-level segmentation, improved building-outline delineation, and exploration of an additional spectral band.
  • FCN generates input-sized prediction maps through transposed-convolution upsampling and fuses shallow and deep feature maps to capture local and global information.
  • FCN lacks boundary details for small objects such as small buildings, limiting the precision of footprint outlines.
  • CRFasRNN and SegNet were developed as FCN variants to improve dense semantic labeling, especially around object boundaries.

B. Conditional Random Field as Recurrent Neural Networks

The paper frames building extraction as instance-aware pixel labeling and combines CNN segmentation with boundary-sensitive labels and near IR fusion. CRF and SegNet address segmentation detail through different architectural strategies, while softmax fusion combines RGB and CIR predictions.

  • B. Conditional Random Field as Recurrent Neural Networks: CRF refines coarse pixel predictions to produce sharper boundaries and finer-grained segmentation results.The paper describes CRF as a graphical-model approach for improving pixel-level labeling.
  • B. Conditional Random Field as Recurrent Neural Networks: SegNet uses max-pooling indices during upsampling to address imperfect boundary delineation in semantic segmentation.Its upsampling strategy differs from FCN by incorporating stored indices from max-pooled feature maps.
  • B. Conditional Random Field as Recurrent Neural Networks: Signed-distance labels recast extraction as 128-class segmentation, with class 64 representing boundaries and labels above 64 representing building interiors.Positive distances indicate building interiors, while negative values represent non-buildings.
  • B. Conditional Random Field as Recurrent Neural Networks: The signed-distance formulation supplies surrounding semantics to help distinguish nearby buildings and reduce errors from shadows or roads.This targets precise individual-building extent, especially in dense urban areas.
  • B. Conditional Random Field as Recurrent Neural Networks: Near IR information is incorporated through RGB and CIR CNNs whose softmax likelihoods are averaged with equal weights at inference.The fusion strategy addresses vegetation occlusion while retaining pre-trained RGB-model parameters.

V. EXPERIMENTS

The experiments evaluate CNN building extraction on public 1-meter NAIP imagery using geographically distributed training data and independent continental United States test sites. LiDAR-derived footprints provide training and evaluation references for assessing generalization and reliability.

  • V. EXPERIMENTS: The study uses cross-validation metrics to analyze multiple CNN models and selects an optimal model for large-scale mapping across the United States.The evaluation compares pixel-labeling performance before deployment.
  • V. EXPERIMENTS: 1-meter NAIP imagery with four bands covers the contiguous United States and serves as the imagery source for building mapping.The available bands are red, green, blue, and near infrared.
  • V. EXPERIMENTS: Training-data characteristics are treated as important because CNN performance is sensitive to differences between training data and the target imagery.The paper questions whether open-source training datasets alone are sufficient for large-scale extraction.
  • V. EXPERIMENTS: 5,173 500-by-500-pixel samples from nine geographically distributed cities were divided into 4,000 training and 1,173 validation images.The samples were generated from a LiDAR building footprints database.
  • V. EXPERIMENTS: 78 independent image tiles, each approximately 6000-by-7000 pixels, formed the test set across sites spanning the continental United States.Each test tile had a LiDAR building-footprint map as ground truth.
  • V. EXPERIMENTS: LiDAR footprints support reliable training-data quality and assessment of the CNN results.The test-site maps provide reference footprints for evaluation.

B. Network Configurations

The network configurations compare FCN variants at 8-stride and 4-stride resolutions, using pre-trained VGGNet initialization and SGD training. The finer FCN-4s setting is included because FCN-8s can be too coarse for some buildings.

  • B. Network Configurations: FCN-8s and FCN-4s experiments compare two output-resolution settings for building extraction.The 4-stride model fuses detailed feature maps to provide finer resolution.
  • B. Network Configurations: FCN-8s was considered too coarse because some buildings became unidentifiable after equivalent eightfold downsampling.This motivated inclusion of the 4-stride configuration.
  • B. Network Configurations: Both FCN configurations used pre-trained VGGNet initialization and batch size 1 during training.The experiments used stochastic gradient descent with a learning rate of 10^-10 and weight decay of 5×10^-4.

2) CRFasRNN:

The CRFasRNN models connect CRF processing to trained FCN variants, while the SegNet setup uses pre-trained VGGNet initialization and GPU-limited batch size. Signed-distance experiments additionally use frequency-based weighted loss, and all evaluations run on a single K80 GPU.

  • 2) CRFasRNN:: CRFasRNN was connected to trained FCN-8s and FCN-4s models, producing FCN-8s-CRF and FCN-4s-CRF.The CRF parameters followed the default settings from the referenced method.
  • 2) CRFasRNN:: The CRFasRNN optimization used SGD with learning rate 10^-11 and momentum 0.99.
  • 2) CRFasRNN:: SegNet used pre-trained VGGNet initialization and batch size 3, the maximum permitted by the available GPU capacity.Its SGD learning rate was 10^-3, with weight decay 5×10^-4 and momentum 0.9.
  • 2) CRFasRNN:: Signed-distance-label experiments applied a class-frequency-weighted loss function.
  • 2) CRFasRNN:: All CNN evaluations used one NVIDIA Tesla K80 GPU, while nationwide inference was parallelized across eight K80 GPUs.The models were implemented with Caffe and trained for 120,000 iterations.

VI. RESULTS DISCUSSION

The study evaluates building extraction with precision, recall, F-score, IoU, and overall accuracy. These metrics use counts of correctly and incorrectly labeled building and non-building pixels.

  • Precision, recall, F-score, and IoU evaluate pixel-based building extraction, while overall accuracy provides an additional completeness measure.IoU is included as a segmentation metric alongside standard building-extraction measures.
  • Overall accuracy measures the proportion of correctly labeled pixels among true positives, false positives, false negatives, and true negatives.
  • TP denotes correctly extracted building pixels, FP denotes non-building pixels mislabeled as buildings, TN denotes correctly identified non-building pixels, and FN denotes missed building pixels.

A. Training CNN Models with Binary Labels

Binary-label experiments show that SegNet performs best overall, while FCN and CRFasRNN face limitations when buildings are nearly connected or initial boundaries are poor. SegNet’s max-pooling indices preserve building-edge features and support individual-building extraction.

  • A. Training CNN Models with Binary Labels: FCN-4s-Bin and FCN-8s-Bin produce comparable binary-label metrics, indicating limited benefit from higher-resolution feature maps.
  • A. Training CNN Models with Binary Labels: Nearly connected buildings are often identified as one large object because convolution and pooling reduce spatial detail, which FCN-4s does not fully recover.
  • A. Training CNN Models with Binary Labels: CRF improves boundaries for some FCN-8s results, but its performance depends on initial FCN extractions and offers minimal improvement when buildings are closely spaced.Further improvement may require more CRF iterations and parameter tuning, increasing training effort.
  • A. Training CNN Models with Binary Labels: SegNet-Bin achieves the best binary-label F-score (0.68) and IoU (0.52) among the evaluated models.The result is attributed to max-pooling indices that capture strong building-edge features.
  • A. Training CNN Models with Binary Labels: SegNet’s sparse max-pooling indices capture building edges and corners, while Seg-Bin can extract individual buildings in the illustrated example.

B. With Signed-distance Labels and Model Fusion

Using SegNet as the base architecture, the study evaluates signed-distance labels and model fusion to improve instance-level building extraction. The combined approach improves separation of individual buildings, including small structures, while fused models perform best across the reported metrics.

  • Signed-distance labels: SegNet-Dist improves on SegNet-Bin, and accurately differentiates individual small buildings in complex scenes.SegNet was selected because SegNet-Bin outperformed other binary-label CNNs; signed-distance labels further improve its results.
  • Signed-distance labels: 3Conv-Dist detects a higher percentage of buildings than deeper CNNs trained with binary labels, but fewer than SegNet-Dist.This comparison supports combining signed-distance labels with SegNet’s max-pooling indices for more accurate building extraction.
  • Model fusion: SegNet-Dist-Fused achieves the best precision, recall, F-score, IoU, and building-detection results among the nine tested models.Both fused models outperform their RGB-only counterparts, while SegNet-Dist-Fused performs best overall.
  • Model fusion: With binary labels and CIR input, the model obtains precision 0.75 and recall 0.65; signed-distance labels with CIR obtain precision 0.74 and recall 0.68.The study compares CIR-based results with RGB-input results to examine the contribution of additional spectral information.
  • Evaluation: The evaluation covers high-density, complex, and sparse-building scenes, while site-level variation is examined across 78 testing sites.These examples represent common scenarios in the testbed, and consistency across sites is treated as important for large-scale extraction.

VII. GENERATING COUNTRY SCALE BUILDING MAPS AND IDENTIFYING CHALLENGES

The preferred model produced contiguous-U.S. building maps without post-processing, but performance varied with imagery quality and landscape, requiring targeted retraining and further domain-adaptation work.

  • Country-scale mapping: The model processed all NAIP images into contiguous-U.S. building maps in under one minute per approximately 56 km^2 on each of 8 NVIDIA Tesla K80 GPUs, without post-processing.A Pennsylvania example remained satisfactory despite no training data from that state.
  • Challenges: Image quality and radiometric variation across states produced poorer extraction in some areas, especially where imagery was blurry, low-contrast, or associated with unusual terrain.NAIP imagery was collected by multiple contractors using potentially different cameras, contributing to state-level variation.
  • Targeted retraining: Adding 141 Wyoming samples, including 108 negative and 13 positive samples, reduced false positives without sacrificing building-extraction performance.The retrained model also reduced false positives in states with landscapes and RGB means similar to Wyoming.
  • Challenges: Inconsistent image quality and varied terrain across extensive areas remain major challenges requiring further work on CNN-based building extraction.The paper identifies transfer learning, domain adaptation, and representative training-sample selection as future directions.

VIII. CONCLUSION

The study addressed large-scale building extraction by evaluating state-of-the-art segmentation approaches, using diverse training data and pretrained models, and extending extraction toward instance-level accuracy. It produced seamless contiguous-U.S. building maps while identifying computational and domain-variation challenges for future work.

  • VIII. CONCLUSION: The study addressed training-data and computational challenges using geographically diverse samples, additional negative samples, and pretrained models.Pretrained models were used to reduce training time.
  • VIII. CONCLUSION: The authors evaluated four semantic segmentation approaches and combined max-pooled indices with signed-distance labels to support accurate instance-level building extraction.They also proposed a fusing strategy to improve building-extraction results.
  • VIII. CONCLUSION: The proposed architecture established high-resolution, seamless building maps for the contiguous United States, with model outputs manually quality-checked for verification.The authors report that the results informed decision making during Hurricanes Irma and Harvey.
  • Future work: GPU memory, inter-node latency, and limited spectral-band use constrain current training and motivate future multi-GPU, larger-architecture, and domain-adaptation studies.The paper also identifies representative-sample selection as a future research topic.
Loading 1805.08946v1…