Learning Aerial Image Segmentation from Online Maps

Pascal Kaiser; Jan Dirk Wegner; Aurelien Lucchi; Martin Jaggi; Thomas Hofmann; Konrad Schindler

Source-linked AI summary

Learning Aerial Image Segmentation from Online Maps

Pascal Kaiser, Jan Dirk Wegner, Aurelien Lucchi, Martin Jaggi, Thomas Hofmann, Konrad Schindler

arXiv:1707.06879v1Jul 21, 2017cs.CV

TL;DR

The paper addresses whether noisy, abundant online map labels can reduce the manual annotation bottleneck in aerial-image segmentation. It trains and compares CNN-based segmenters using manually labeled data and large-scale OSM-derived labels across cities and image sets. The experiments show that noisy labels can provide acceptable performance and improve results when used at scale, while the study remains limited to OSM-supported classes and RGB Google Maps imagery.

Problem
Deep CNNs require large annotated datasets, motivating the question of whether abundant but noisy public map labels can replace substantial manual labeling.
Method
The study compares FCN-based aerial segmentation trained with manual ground truth and automatically generated OSM labels across multiple city datasets and transfer settings.
Results
Large-scale noisy OSM training achieves acceptable segmentation, can completely replace small amounts of highly accurate labels, and improves performance when augmenting manual data.
Takeaways & Limitations
Noisy large-scale online map data can substantially reduce manual labeling effort, although the best overall results come from augmenting manually labeled data at very large scale.
Takeaways & Limitations
The study is limited to RGB orthophotos with unknown radiometric calibration and street-map data intended for navigation purposes.

Abstract

from arXiv · show

This study deals with semantic segmentation of high-resolution (aerial) images where a semantic class label is assigned to each pixel via supervised classification as a basis for automatic map generation. Recently, deep convolutional neural networks (CNNs) have shown impressive performance and have quickly become the de-facto standard for semantic segmentation, with the added benefit that task-specific feature design is no longer necessary. However, a major downside of deep learning methods is that they are extremely data-hungry, thus aggravating the perennial bottleneck of supervised classification, to obtain enough annotated training data. On the other hand, it has been observed that they are rather robust against noise in the training labels. This opens up the intriguing possibility to avoid annotating huge amounts of training data, and instead train the classifier from existing legacy data or crowd-sourced maps which can exhibit high levels of noise. The question addressed in this paper is: can training with large-scale, publicly available labels replace a substantial part of the manual labeling effort and still achieve sufficient performance? Such data will inevitably contain a significant portion of errors, but in return virtually unlimited quantities of it are available in larger parts of the world. We adapt a state-of-the-art CNN architecture for semantic segmentation of buildings and roads in aerial images, and compare its performance when using different training data sets, ranging from manually labeled, pixel-accurate ground truth of the same city to automatic training data derived from OpenStreetMap data from distant locations. We report our results that indicate that satisfying performance can be obtained with significantly less manual annotation effort, by exploiting noisy large-scale training data.

I. INTRODUCTION

The paper asks whether abundant but noisy online map labels can reduce the costly manual annotation needed for aerial-image segmentation. It studies deep CNNs trained on OpenStreetMap-derived data and finds that large noisy datasets can support performance while substantially reducing manual labeling.

Manual pixel-accurate labels are costly and limited, while less accurate reference data can be available in arbitrary quantities at no cost.
The study tests whether training-data volume and variety can compensate for lower label accuracy and improve generalization.
Deep CNNs are used because they perform strongly on high-resolution semantic labeling, tolerate label noise, and avoid task-specific feature design.
Noisy labels work well only with substantially larger training sets; small sets of about 2 km2 underperform hand-labeled data.
Massive OSM labels further improve classifiers even when high-quality training data is available, reducing manual labeling effort.
The study is limited to buildings and roads from OSM and RGB Google Maps imagery affected by unknown radiometric manipulations.

II. RELATED WORK

Prior aerial-image mapping methods use engineered local features, probabilistic object models, or CNNs. The paper identifies a lack of large-scale, systematic quantitative studies of noisy map labels for high-resolution aerial segmentation.

Very-high-resolution aerial imagery supports detailed urban mapping, but rule-based systems generalize poorly across different city layouts.
Conventional approaches combine local image features with classifiers or probabilistic models that encode object-level structure.
Deep CNNs learn discriminative features and long-range contextual representations directly from raw images without a separate feature-definition step.
Fully convolutional networks return spatially explicit label images and upsample predictions to the original image resolution.
Only two prior works examined fine-tuning CNNs trained on large-scale OSM labels, leaving systematic quantitative evidence limited.

III. METHODS

The methods section presents an automatic OSM-based training-data pipeline and an FCN-based semantic-segmentation procedure. It frames the approach as a way to train on large quantities of weakly labeled aerial data.

The paper automatically generates training data from OSM and then trains a fully convolutional network for semantic segmentation.
The method combines automatic dataset generation with technical specifications for the FCN architecture.
The training procedure is described after the OSM data-generation approach as part of the overall pipeline.

A. Generation of Training Data

Training images come from Google Maps and labels from OSM, which are transformed into pixel-wise maps for buildings, roads, and background. The resulting labels are useful but inherit alignment, format, and temporal limitations.

Very-high-resolution RGB images are downloaded from Google Maps, while building and road coordinates are obtained from OSM.
OSM vector data is preferred because object types, metadata, and identifiers support filtering and manipulation.
OSM and Google Maps align relatively well despite independent acquisition and processing.
Label quality is constrained by co-registration effects, unknown road widths, perspective-related displacement, and non-true-orthophoto artifacts.
Pixel-wise labels are produced by transforming OSM coordinates into image pixels and rasterizing building polygons and road representations.

B. Neural network architecture

The network transforms aerial pixels through convolutional and pooling layers into spatially explicit class scores, then upsamples them to full resolution. Skip connections restore intermediate detail lost during downsampling.

Convolutional layers use localized, shared filters to transform image pixels into learned feature representations.Shared weights reflect shift invariance and reduce the number of parameters.
Max-pooling downsamples feature maps, enlarging receptive fields and retaining local translation invariance.Each pooling operation retains the maximum value within a 2×2 neighborhood.
The network converts final convolutional features into low-resolution score maps for the three target classes, then upsamples them and applies softmax probabilities.Intermediate feature maps are merged during upsampling to recover spatial detail.
Three skip connections connect intermediate layers to the upsampling path, improving preservation of fine image details.The added connection complements the original two connections in the FCN variant.

C. Implementation Details

The implementation modifies the original FCN by adding a higher-resolution skip connection to sharpen segmentation boundaries without increasing model size.

The FCN adds a third skip connection after Pool 2 because the original two produced insufficiently sharp edges.The additional higher-resolution connection consistently improved results for the authors’ data.
The added skip connection does not increase parameters: the modified model has 134′276′540 versus 134′277′737 in the original.The small reduction comes from decomposing the final upsampling kernel into two smaller kernels.

D. Training

Training minimizes pixel-wise multinomial logistic loss on mean-centered image patches using stochastic gradient descent, with optional VOC or OSM pre-training. Training stops when validation F1-score ceases improving.

The model minimizes multinomial logistic loss summed over each 500×500 pixel input patch.RGB channel means are subtracted independently for every patch before training and inference.
Optimization uses stochastic gradient descent with momentum 0.9, minibatches of one image, scheduled learning-rate reductions, weight decay, and dropout.Learning rates start at 5× 10−9 and are reduced twice by a factor of 10 when progress stops.
Training runs for 45000–140000 iterations, or 3.5–6.5 epochs, until average validation F1-score stops improving.This validation criterion determines when optimization ends.
The experiments compare no pre-training with weights pre-trained on Pascal VOC or with OSM data.The pre-training source is specified for each experiment.

IV. EXPERIMENTS

Experiments on multiple city datasets test whether noisy OSM labels can substitute for, augment, or reduce manual annotation in aerial-image segmentation.

Experiments use four large city datasets and address complete substitution, augmentation, and partial substitution of manual labeling.The scenarios evaluate training solely from noisy OSM labels, pre-training project-specific models, and adapting generic OSM models with little dedicated data.
Complete substitution asks how well semantic segmentation can be learned without manual labels using only noisy OSM-derived labels.
Augmentation tests whether large-scale OSM pre-training and publicly available images improve segmentation on independently acquired project data.
Partial substitution tests whether generic OSM pre-training can adapt to a specific location and data source with little dedicated training data.
The paper states that the experiments investigate and validate the hypotheses introduced earlier, with explicit answers summarized at the section’s end.

A. Datasets

The study uses multi-city Google Maps/OSM datasets alongside the ISPRS Potsdam benchmark, with standardized tiling and region-based splits for evaluation.

Dataset composition: Four large datasets cover Chicago, Paris, Zurich, and Berlin, while Potsdam provides a smaller comparison dataset with high-accuracy ISPRS ground truth.The Potsdam region was constrained by the benchmark’s available imagery and ground truth, making it an order of magnitude smaller than the other cities’ large-scale datasets.
Preprocessing: 500 × 500-pixel non-overlapping tiles are used for all aerial images and corresponding label maps.The tile size was selected to provide geographical context while keeping FCN training and prediction efficient.
Resolution matching: Potsdam ISPRS imagery is down-sampled from 5 cm to 9.1 cm GSD to match the Potsdam-Google data.This resolution matching is applied in all experiments.
Dataset splits: Each dataset is divided into mutually exclusive training, validation, and test regions.Validation loss is monitored during training to prevent overfitting.

B. Results and discussion

The experiments show that large-scale noisy OSM training data can replace much manual labeling, while combining it with limited accurate labels provides a strong performance–annotation trade-off. Diverse city-scale pre-training improves transfer to unseen locations and can outperform generic pre-training.

Augmentation with open data: 0.797 F1-score after adding Berlin to noisy OSM training data improved performance by 2 percentage points over the smaller setup.The result was only slightly below Zurich’s 0.810 and indicates that city-scale noisy training data is crucial.
Complete substitution of manual labels: 0.779 F1-score from OSM labels on the ISPRS training images beat baseline Ia by 1.5 percentage points but remained 3 points below baseline Ib.Compared with equally large quantities of pixel-accurate labels, performance dropped by 10 percentage points.
Augmentation with open data: 0.825 road-class F1-score after open-data pre-training improved on baseline II’s 0.764, while buildings showed no further improvement.Visual comparisons showed small improvements for both roads and buildings, but the quantitative gain was concentrated in roads.
Partial substitution of manual labels: 0.837 versus 0.874 showed that partial substitution with large-scale OSM pre-training remained about 4 percentage points below the gold standard.Fine-tuning with limited problem-specific accurate labels compensated for approximately 65% of the gap.

V. CONCLUSIONS AND OUTLOOK

The study shows that large-scale weakly labeled data can substantially reduce manual annotation needs while improving segmentation and generalization, though its evidence remains bounded by available imagery and map data.

85% of manually annotated high-quality data can be replaced by large-scale low-accuracy training data in the reported experiments.
Large-scale weakly labeled data improves segmentation performance and the models’ ability to generalize.
Training only on open data achieves reasonable, though far from optimal, results when training and test images come from the same source.
The proposed workflow combines OSM-labeled pre-training with domain adaptation to images at hand, when suitable public images and map data exist.
The study is limited to RGB orthophotos with unknown radiometric calibration and online street-map data, while broader datasets and object classes require further research.