Source-linked AI summary
Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution
Shivangi Srivastava, John E. Vargas-Muñoz, Devis Tuia
TL;DR
Urban landuse mapping needs automation because traditional field surveys and manual interpretation are time-consuming and labor-intensive, while overhead imagery alone can be insufficient. The paper develops an end-to-end multimodal model combining overhead imagery with variable-sized ensembles of ground pictures, and reports improved landuse accuracy, including when ground pictures are unavailable.
Problem
Overhead imagery alone may not sufficiently distinguish urban landuse because remotely sensed information can be insufficient for describing landuse.
Method
The paper develops a multimodal deep-learning model combining overhead imagery with ensembles of ground-based pictures and a common representation for missing ground data.
Results
The multimodal CNN improves accuracy for many landuse classes and can predict labels when ground-based pictures are unavailable.
Takeaways & Limitations
The approach supports urban-object landuse classification using widely available imagery and can be applied across urban environments where OSM annotations are available.
Takeaways & Limitations
The model could be further improved with a better-quality dataset.
Abstract
from arXiv · showhide
Landuse characterization is important for urban planning. It is traditionally performed with field surveys or manual photo interpretation, two practices that are time-consuming and labor-intensive. Therefore, we aim to automate landuse mapping at the urban-object level with a deep learning approach based on data from multiple sources (or modalities). We consider two image modalities: overhead imagery from Google Maps and ensembles of ground-based pictures (side-views) per urban-object from Google Street View (GSV). These modalities bring complementary visual information pertaining to the urban-objects. We propose an end-to-end trainable model, which uses OpenStreetMap annotations as labels. The model can accommodate a variable number of GSV pictures for the ground-based branch and can also function in the absence of ground pictures at prediction time. We test the effectiveness of our model over the area of Île-de-France, France, and test its generalization abilities on a set of urban-objects from the city of Nantes, France. Our proposed multimodal Convolutional Neural Network achieves considerably higher accuracies than methods that use a single image modality, making it suitable for automatic landuse map updates. Additionally, our approach could be easily scaled to multiple cities, because it is based on data sources available for many cities worldwide.
1. Introduction and Related Work
Urban landuse mapping is difficult because overhead imagery often cannot distinguish classes that share materials or structures, while traditional surveys are costly to update. The paper addresses this with a multimodal deep-learning system combining overhead and ground views, designed for scalable urban-object classification.
- Field surveys and manual interpretation are time-consuming, labor-intensive, and impractical for frequent map updates.
- Overhead imagery alone is challenging because landuse classes combine multiple landcover types, while the same landcover types recur across different uses.A university may contain buildings, trees, grass, water, and roads, whereas similar buildings may represent a government office or school.
- Urban landuse describes how delimited spaces are utilized, including schools, hospitals, museums, parks, and other human uses.
- Ground imagery can reveal object functions hidden from above, such as school entrances, but crowd-sourced repositories suffer from geolocation, content, and coverage problems.Google Street View offers objective, accurately geolocated, densely available pictures across many cities, with regular updates.
- The proposed model combines overhead and ground views to predict landuse at the urban-object level using widely available data and OpenStreetMap annotations.It accepts a variable number of street-level images and includes an embedding module for objects lacking ground pictures at test time.
- The multimodal model obtains accuracies above 70% for most landuse categories and outperforms unimodal architectures.The results support complementarity between aerial and ground information, while the approach is intended to transfer to cities with available OSM annotations.
2. Methods
The method predicts landuse categories for OpenStreetMap urban-object polygons using complementary overhead imagery and variable-sized sets of ground-based pictures. It combines modality-specific CNN features and can retrieve missing ground-view features when no street-level image is available at test time.
- Multimodal architecture: The proposed CNN uses separate overhead-imagery and ground-based-picture streams whose learned features are combined to predict the final landuse category.The model extracts discriminative features from both modalities before classification.
- Task definition: Landuse classification predicts a category for each OpenStreetMap urban-object polygon rather than performing pixel-level semantic segmentation.Each object has an OSM footprint and label, with one overhead image and a variable collection of ground-based pictures.
- Ground-based stream: The ground-based branch uses a VIS-CNN that produces one representation from an arbitrary number of GSV pictures for each urban-object.The same pretrained VGG16-based feature extractor is applied to each picture, after which the resulting vectors are aggregated.
- Ground-based stream: Average aggregation performed better than max aggregation in experiments, so the model uses the average aggregator for the ground-based features.Max pooling selects the strongest element-wise representation, whereas averaging emphasizes repeated attributes across pictures.
- Missing-modality handling: When GSV images are unavailable, the method projects overhead and GSV CNN features into a joint embedding and retrieves the nearest training GSV feature as a proxy.The retrieved neighbor’s GSV pictures are supplied to the multimodal model together with the test urban-object’s overhead imagery.
3. Dataset
The study builds urban-object datasets from OpenStreetMap footprints and labels, Google Maps overhead imagery, and Google Street View ground-based pictures for Île-de-France and Nantes.
- Each urban-object is represented by one overhead image and an ensemble of side-view pictures, with OSM providing the corresponding landuse label.
- The study groups OSM landuse categories into 16 classes according to usage similarity and excludes rare classes with insufficient footprints or GSV pictures.
- The Île-de-France dataset contains 5,941 urban-objects, while the Nantes dataset contains 1,835 urban-objects.
- Both datasets use the same landuse classes except Nantes omits Marina because only one corresponding urban-object was retrieved.
- The collection includes 44,957 GSV pictures for Île-de-France and 9,908 for Nantes, alongside overhead imagery covering each footprint.
4. Experimental Setup
The experiments train end-to-end CNN models on overhead and ground-based imagery, evaluate them with repeated train/test splits, and test retrieval-based prediction when GSV images are unavailable.
- VGG16 extracts features from overhead imagery and VIS-CNN extracts features from GSV pictures for the multimodal and unimodal models.
- Training uses stochastic gradient descent with momentum for 50 epochs, beginning at learning rate 0.001 and reducing it tenfold every 10 epochs.
- The dataset is divided into five mutually exclusive splits, with 80% of urban-objects per landuse class used for training and the remainder for testing.
- Performance is measured using overall accuracy and average accuracy per class, averaged across the five splits.
5. Results and Discussion
The multimodal CNN generally outperforms overhead-only and GSV-only baselines, with complementary views improving many classes and retrieved GSV images enabling prediction without direct ground pictures.
- Around 6% higher OA and more than 7% higher AA are observed than for the VGG16 overhead-only model.
- The multimodal CNN outperforms unimodal baselines for almost all landuse classes, with gains exceeding 9% for educational, hospital, post-office, and fuel.
- Complementary cues from overhead and ground views support correct predictions, including playgrounds and school features, cemetery layouts and walls, and post-office vehicles and signage.
- Heritage sites remain difficult because overhead imagery lacks discriminative information, producing 15.8% accuracy for the overhead model and degrading the multimodal result.
- In Nantes, the multimodal model is 5% more accurate in OA and 10% in AA than overhead-only imagery, and improves on VIS-CNN by more than 16% in OA and 11% in AA.
- When GSV pictures are unavailable, CCA-based retrieval improves results over direct nearest-neighbor labels and nearly closes the gap with the full multimodal model.
6. Conclusions and Outlook
The paper concludes that an end-to-end multimodal model combines overhead and ground views for landuse classification and can operate when ground pictures are missing.
- The model jointly learns feature extraction from single modalities and their fusion for multimodal landuse classification.
- Complementary visual information from either modality improves accuracy for many classes in Île-de-France.
- The model can predict landuse labels without ground-based pictures by retrieving a plausible set of GSV pictures from the training set.
- Using Google Street View, Google Maps, and OpenStreetMap supports scaling the approach to other cities.
- The authors identify better labels and refined GSV-download heuristics as routes to improving dataset quality and model accuracy.