Source-linked AI summary
Building Instance Classification Using Street View Images
Jian Kang, Marco Körner, Yuanyuan Wang, Hannes Taubenböck, Xiao Xiang Zhu
TL;DR
Individual-building land-use classification is more informative for some urban applications than patch- or pixel-wise mapping, but remains difficult because remote-sensing images mainly show roofs and building footprints are challenging to retrieve. The paper combines CNN classification of facade structures in street-view images with geographic building footprints, constructs a benchmark dataset, and generates region- and city-scale maps. The resulting maps achieve relatively high individual-building classification accuracies and have potential for several urban analyses.
Problem
Individual-building land-use maps are needed for applications such as urban population-density mapping and utility planning, but roof-only imagery and difficult footprint extraction limit this task.
Method
The framework uses geographic building footprints to associate individual buildings with street-view images, classifies facade structures with CNNs, and trains and evaluates models on a benchmark dataset.
Results
The method produced building classification maps at region and city scales, with relatively high accuracies reported for individual-building land-use classification.
Takeaways & Limitations
Building-instance classification maps can provide insight into urban areas and support analyses of population density, social structure, city economy, and urban planning.
Takeaways & Limitations
Performance is constrained by street-view coverage and quality, GIS footprint availability, and labeling errors among visually similar building classes.
Abstract
from arXiv · showhide
Land-use classification based on spaceborne or aerial remote sensing images has been extensively studied over the past decades. Such classification is usually a patch-wise or pixel-wise labeling over the whole image. But for many applications, such as urban population density mapping or urban utility planning, a classification map based on individual buildings is much more informative. However, such semantic classification still poses some fundamental challenges, for example, how to retrieve fine boundaries of individual buildings. In this paper, we proposed a general framework for classifying the functionality of individual buildings. The proposed method is based on Convolutional Neural Networks (CNNs) which classify facade structures from street view images, such as Google StreetView, in addition to remote sensing images which usually only show roof structures. Geographic information was utilized to mask out individual buildings, and to associate the corresponding street view images. We created a benchmark dataset which was used for training and evaluating CNNs. In addition, the method was applied to generate building classification maps on both region and city scales of several cities in Canada and the US. Keywords: CNN, Building instance classification, Street view images, OpenStreetMap
1. Introduction
Existing EO-based land-use classification operates mainly on patches or pixels, while individual-building classification requires finer spatial and semantic information. The paper addresses this gap by combining facade-rich street-view imagery, geographic building footprints, CNNs, and a benchmark dataset to produce building-level maps.
- Motivation: EO imagery supports large-scale urban land-cover monitoring, but transferring from land cover to individual-building land use is difficult.Physical indicators such as building volume, density, alignment, and grouped patterns are insufficient for directly assigning labels at the individual-building scale.
- Challenges: Roof structures in nadir-looking remote-sensing images can provide subtle visual differences between building classes, while building-footprint extraction remains challenging.Clear footprint segmentation usually requires additional height information.
- Proposed framework: Street-view facade structures provide richer information for distinguishing building functionality than corresponding roof structures.The framework uses freely available street-view images together with online geographic maps such as OpenStreetMap or Google Maps.
- Contributions: The paper proposes a general framework for land-use classification at the level of individual buildings.Geographic information is used to identify building instances and associate them with street-view imagery for CNN-based classification.
- Contributions: The authors build a street-view benchmark dataset for training CNN classifiers based on facade structures.The dataset supports training and evaluation of building-instance classification models.
- Applications: Building classification maps are presented as potentially useful for very high-resolution population mapping, urban social-structure understanding, city-economy analysis, and urban planning.The paper reports applications at both region and city scales.
2. Related work
Earlier remote-sensing work developed handcrafted and learned features for land-use and scene classification, but did not explicitly address classification at the individual-building level. The paper positions its workflow as addressing this unresolved scale-specific problem.
- Feature extraction: Remote-sensing feature extraction has used SIFT with bag of visual words, textural features, 3D digital-surface-model features, and sparse-coding features.These approaches represent longstanding handcrafted and learned feature-engineering directions.
- Deep learning: Deep-learning approaches have applied stacked autoencoders, deep belief networks, and CNN transfer learning to remote-sensing classification.CNN features learned from ImageNet were investigated through fine-tuning procedures.
- Research gap: The reviewed literature had not explicitly addressed land-use classification at the level of individual buildings.This scale-specific gap motivates the paper’s building-instance classification framework.
3. Overall workflow
The workflow links mapped building footprints to street-view imagery, removes unusable views, and classifies each building by aggregating CNN predictions across retained images.
- Building footprints and associated GPS locations are retrieved from GIS sources, then used to download street-view images showing individual building facades.
- Retrieved street-view images undergo outlier removal because interiors, vehicles, and trees can obscure the building facade needed for classification.
- The benchmark contains 19,658 geo-tagged images spanning eight building classes, with roughly 2,500 images per class.
- A pretrained CNN is fine-tuned on the benchmark rather than trained from scratch because the dataset is too small for a CNN with millions of parameters.
- For each building instance, the final class is obtained at decision level by aggregating CNN outputs from its retained street-view images.
4. Experiments
Experiments evaluate CNNs trained on a multi-class street-view benchmark and use the selected model to produce building-instance maps at regional and city scales.
- Training: Four CNN architectures—AlexNet, VGG16, ResNet18, and ResNet34—were trained, using ImageNet-pretrained convolutional layers and randomly initialized fully connected layers.Training used cross-entropy loss, stochastic gradient descent, learning-rate decay, dropout, and data augmentation.
- Training: The benchmark contains 19,658 street-view images spanning eight building classes, with geographically separate training and testing cities.The dataset includes 17,600 training images and 2,058 test images.
- Training: Training losses decreased with epochs; AlexNet reached about 65% top-1 precision, while the other networks reached about 70%, with stronger overfitting in ResNet34.AlexNet validation convergence continued to 80 epochs, whereas the other networks converged within 60 epochs.
- Testing: VGG16 achieved the highest F1 scores for apartment, church, garage, industrial, and office-building classes, while ResNet34 led the remaining classes.AlexNet performed worse than the other three networks overall, so VGG16 was selected for map generation.
- Regional maps: The selected models generated building classification maps for Vancouver and Fort Worth study areas, identifying predominantly residential and office-building districts, respectively.The Vancouver area contained 196 building instances, while the Fort Worth area contained 316.
- City-scale maps: City-scale maps classified 6,124 buildings in Calgary, 64,389 in Boston, and 45,978 in Toronto, with overall accuracy around 0.7, around 0.55, and above 75%, respectively.The evaluations used 1,000 randomly selected buildings per city with OSM building tags as reference.
- City-scale maps: The city-scale maps revealed spatial patterns including Calgary industrial districts, Boston residential and central business areas, and Toronto central apartments and offices with surrounding industry.These distributions were used to characterize urban structure and population-density patterns within the mapped cities.
5. Discussion
The discussion attributes classification difficulty to initialization, ambiguous labels and imagery, while city-scale maps reveal distinct urban building distributions.
- Classification performance: Different fully connected-layer initializations eventually converge to comparable classification accuracies.This applies to the reported AlexNet and VGG16 training experiments.
- Classification performance: Churches are relatively easier to recognize, whereas retail and industrial buildings are harder to identify.The paper relates this difference to the uniqueness of church structures and ambiguity among other classes.
- Error sources: OSM labeling errors, especially between industrial and office buildings, limit the reliability of benchmark ground truth.The cited example depicts a visually office-like building labeled industrial in OSM.
- City-scale mapping: Boston maps show office and apartment densities declining outward from the center, while house density increases outward.The city-scale visualization also characterizes Boston as having one central business district without large industrial districts.
- City-scale mapping: Toronto maps place most apartments and offices centrally, while industrial buildings are concentrated in surrounding regions.The associated distribution maps and class-proportion chart provide city-scale summaries of these predicted classes.
- Error sources: Multiple buildings in one street-view image and views dominated by side faces can produce misclassification or low confidence.A house beside a garage is misclassified as a garage, while a church lacking its typical facade has reduced confidence.
6. Conclusion and future work
The paper presents a CNN-based framework and benchmark for classifying individual buildings, producing more informative maps for urban analysis. Future work targets richer inputs and denser coverage where street-view imagery or building footprints are unavailable.
- Conclusion: The framework classifies individual buildings using a street-view benchmark with eight categories and produces region- and city-scale classification maps.Among four CNN architectures, VGG16 was selected for generating the classification maps.
- Future work: Future performance improvements may come from fusing social-media descriptions and text visible in images, such as brand names.These additional inputs are proposed as future work rather than evaluated results in the cited passage.
- Future work: Denser maps require remote-sensing or GIS information for buildings without street-view images, plus building detection when GIS footprints are unavailable.The proposed extension addresses both missing imagery and missing footprint data.