PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery

Jamie Sherrah

arXiv:1606.02585v1cs.CV

TL;DR

High-resolution aerial imagery supports object-level semantic labelling, but fine boundaries and non-spectral distinctions require dense methods and richer image features. The paper adapts FCNs to aerial imagery, introduces no-downsampling full-resolution inference, and combines pre-trained image features with DSM features. Applied to the ISPRS Vaihingen and Potsdam benchmarks, the methods achieve state-of-the-art accuracy, while remaining limited by reduced spatial feature resolution and generalisation demands.

  • Problem

    High-resolution aerial imagery makes individual objects distinguishable, but semantic labelling requires fine boundaries and appearance-based features beyond spectral information alone.

  • Method

    The paper adapts fully convolutional networks to aerial imagery, uses no downsampling for full-resolution outputs, and combines pre-trained image features with DSM features in a hybrid network.

  • Results

    State-of-the-art semantic-labelling accuracy was achieved on the ISPRS Vaihingen and Potsdam benchmark datasets.

  • Takeaways & Limitations

    FCNs, no-downsampling architectures, and pre-trained image features benefit semantic labelling of high-resolution aerial imagery within the evaluated benchmarks.

  • Takeaways & Limitations

    Although outputs are dense, spatial feature resolution still decreases with layer depth, motivating future multi-scale feature fusion.

Abstract

from arXiv · show

The trend towards higher resolution remote sensing imagery facilitates a transition from land-use classification to object-level scene understanding. Rather than relying purely on spectral content, appearance-based image features come into play. In this work, deep convolutional neural networks (CNNs) are applied to semantic labelling of high-resolution remote sensing data. Recent advances in fully convolutional networks (FCNs) are adapted to overhead data and shown to be as effective as in other domains. A full-resolution labelling is inferred using a deep FCN with no downsampling, obviating the need for deconvolution or interpolation. To make better use of image features, a pre-trained CNN is fine-tuned on remote sensing data in a hybrid network context, resulting in superior results compared to a network trained from scratch. The proposed approach is applied to the problem of labelling high-resolution aerial imagery, where fine boundary detail is important. The dense labelling yields state-of-the-art accuracy for the ISPRS Vaihingen and Potsdam benchmark data sets.

1 Introduction

High-resolution aerial imagery enables object-level semantic labelling, but requires appearance-based features and dense methods that preserve fine boundary detail. This work adapts FCNs with no downsampling and hybrid use of pre-trained image features and elevation data.

  • Motivation: 5–10 cm imagery distinguishes individual objects, shifting remote sensing from coarse land-use classification toward pixel-level semantic labelling.Spectral properties alone may not distinguish objects such as grass and trees or roads and roofs.
  • Fully convolutional networks: FCNs apply CNN classifiers convolutionally to aerial imagery, improving accuracy and computational efficiency over patch-based training.The approach processes all overlapping patches through convolutional operations during training and classification.
  • No-downsampling FCNs: No-downsampling FCNs preserve full input resolution at every layer, avoiding interpolation while improving accuracy over downsampling FCNs.The approach is especially beneficial when the network’s downsampling factor is large.
  • Hybrid architecture: A hybrid FCN combines pre-trained convolutional image features with DSM features trained from scratch to use both appearance and elevation information.The hybrid architecture addresses the three-channel input restriction of pre-trained networks.
  • Results: The methods report first published results on the ISPRS Potsdam benchmark and state-of-the-art semantic-labelling accuracy for high-resolution aerial imagery.The paper evaluates its approach on benchmark aerial-imagery datasets.

2 Related Work

Prior remote-sensing work used handcrafted features, unsupervised learning, or pre-trained CNNs mainly for classification and detection. FCNs had achieved strong semantic-labelling results in computer vision, but had not been used for fully convolutional training on remote-sensing data.

  • Remote-sensing methods: Earlier very-high-resolution remote-sensing methods densely extracted colour, edge, or object-level features before classification.These approaches addressed scene parsing and labelling across satellite and aerial imagery.
  • CNN transfer learning: Pre-trained CNN features transferred successfully across aerial and satellite imagery, while prior work also fine-tuned CNNs for overhead image-tile classification.CNNs learned reusable image features from large datasets such as ImageNet.
  • Remote-sensing methods: Other studies applied unsupervised convolutional auto-encoders or CNNs with noisy labels to detect targets such as planes, dry docks, roads, and buildings.These methods focused on target detection rather than the fully convolutional training studied here.
  • Research gap: None of the reviewed remote-sensing approaches employed fully convolutional training, making FCN adaptation to aerial imagery the paper’s main related-work gap.
  • Fully convolutional networks: FCNs trained on overlapping image patches had recently produced leading semantic-labelling results in computer vision, but their outputs were lower resolution because of downsampling.Earlier FCN work compensated through interpolation methods such as bilinear upsampling and shift-and-stitch.

3 Overhead Imagery and Data Sets

The paper applies semantic labelling to large, high-resolution aerial imagery, where objects and extended regions must be distinguished across broad spatial contexts. Experiments use ISPRS Vaihingen and Potsdam imagery with elevation data, ground-truth labels, and separate training and validation partitions.

  • 3 Overhead Imagery and Data Sets: Overhead images are much larger and have less bounded spatial context than typical multimedia images, while semantic labelling must cover both objects and extended background regions.Sliding windows provide spatial context, but some interpretations may require a much larger context.
  • 3 Overhead Imagery and Data Sets: The benchmarks provide near-infrared, red, and green ortho-rectified imagery with corresponding digital surface models; Potsdam additionally provides blue-channel imagery.Ground-truth classes include impervious surface, building, tree, low vegetation, car, and unknown.
  • 3 Overhead Imagery and Data Sets: The experiments use the ISPRS Vaihingen and Potsdam 2D Semantic Labelling Challenge data sets.The task is to generate labels for unlabelled test imagery from these benchmarks.
  • 3 Overhead Imagery and Data Sets: Labelled training images are divided into separate training and validation subsets, with validation results used for experimental comparisons and excluded from training.Test-set accuracies, when reported, are generated by the challenge website.

4 FCN Versus Patch-Based Methods

The paper replaces redundant patch-wise CNN processing with fully convolutional training and inference, while addressing FCN resolution loss and boundary detail through architectural choices. FCN training improves efficiency and accuracy relative to patch-based training, and broader rotation augmentation adds only an incremental gain.

  • 4 FCN Versus Patch-Based Methods: Patch-based processing repeats operations on overlapping patches, whereas FCNs process all overlapping patches together as an image filter and produce a 2D label matrix.This convolutional formulation also extends to training and can use more data efficiently.
  • 4.1 The CNN as an Image Filter: The CNN classifier is converted into a CNN filter by applying convolutional operations across larger images instead of producing one label for one fixed-size image.The filter’s spatial support remains the classifier’s original n × n input size.
  • 4.1 The CNN as an Image Filter: Padding introduces adjacent image features at filter boundaries rather than zeros, creating an architectural consideration when a classifier was trained to expect zero padding.Figure 2 illustrates the effect of convolutional padding on FCN outputs.
  • 4 FCN Versus Patch-Based Methods: Downsampling makes FCN outputs lower resolution than inputs; the overall downsampling factor is the product of layer strides, such as s = 16 for four stride-2 pooling layers.Lower output resolution is undesirable when semantic labelling requires fine boundary detail.
  • 4 FCN Versus Patch-Based Methods: Because full remote-sensing images exceed GPU memory, training and classification divide inputs into non-overlapping tiles processed separately.The tiles are treated as separate images during minibatch training.
  • 4 FCN Versus Patch-Based Methods: FCN training is more memory efficient because each pixel appears once in a minibatch, and it effectively trains on more data than randomly sampled patch-based training.The paper presents these as two reasons FCN training should be more accurate.
  • 4.3 Experiments: FCN training produces a significant improvement over patch-based training, while increasing rotation augmentation from 90-degree to 10-degree increments yields only an incremental accuracy improvement.The 10-degree rotation data is retained for subsequent experiments, and Figure 3 provides qualitative label examples.

5 Downsampling and Upsampling in CNNs

The paper preserves full-resolution outputs by pooling without downsampling and using atrous filters, avoiding interpolation while retaining pooling’s benefits. This design is computationally efficient and improves labelling, especially for fine structures and deeper networks.

  • Motivation: Full-resolution labelling is important because overhead imagery contains objects and regions whose boundary accuracy matters.Conventional FCNs produce lower-resolution label maps and typically restore resolution through interpolation or learned deconvolution.
  • No-downsampling design: The no-downsampling FCN retains pooling but removes downsampling, using atrous filters to control parameter growth as layer depth increases.Filter and pooling support expand with depth, while holes in the filters avoid severely increasing model parameters.
  • Efficiency: At test time, no-downsampling produces the same output as shift-and-stitch while computing each convolution only once instead of s^2 times.At training time, it also increases the effective training set size by using shifted-and-interleaved versions during backpropagation.
  • Experiments: The no-downsampling approach improves Vaihingen overall accuracy by 0.53% and car accuracy by 10.23% compared with a pool-and-downsample FCN.The car-class gain is attributed to finer sampling and increased representation of a class comprising about 1% of the data.
  • Experiments: Potsdam accuracy improves by 1.98%, with fine structures benefiting most from the higher-resolution network.The Potsdam network uses stride s = 32 because the first convolutional layer has stride 2.
  • Experiments: No-downsample training is generally more beneficial for deeper networks with larger downsampling factors, although the improvement is not monotonic.The method achieved higher accuracy than downsampling versions across the compared network depths.

6 Pre-Trained CNNs

The paper fine-tunes pre-trained image features for aerial semantic labelling and combines them with DSM features in a hybrid network. On Potsdam, pre-training and near-infrared inputs improve accuracy, while late DSM fusion adds little and can be sub-optimal.

  • Motivation: Pre-trained CNN features from ImageNet are used to exploit discriminative texture and appearance information in high-resolution aerial imagery.The motivation is to improve over pixel-level spectral and elevation cues, which achieve around 75% accuracy.
  • Hybrid architecture: The hybrid network feeds three spectral bands through pre-trained convolutional layers and DSM through randomly initialized layers, then concatenates their features.The concatenated features are passed to two randomly initialized fully connected layers.
  • Potsdam experiments: Fine-tuned VGG-16 features improve Potsdam overall accuracy by almost 1.91% over a randomly initialized five-channel network.The pre-trained convolutional layers use a learning rate of 0.0001, while fully connected layers are randomly initialized.
  • Potsdam experiments: CIR input increases accuracy by 1.13% over RGB and improves low-vegetation accuracy by 1.82%, indicating the value of near-infrared information for vegetation delineation.The labelling examples show improved vegetation delineation with CIR data.
  • Hybrid fusion: Adding DSM to pre-trained CIR features yields only a small improvement, and the all-five-channel hybrid network performs worse than CIR-only input.These results indicate that late fusion does not fully exploit height information and is sub-optimal in this setting.
  • Potsdam experiments: The no-downsampling hybrid network improves car accuracy by 6.31%, while CRF post-processing improves accuracy for each tested approach.The no-downsampling network also better labels finely detailed scene parts in the qualitative comparison.
  • Benchmark results: The FCN approach reaches a best Potsdam challenge accuracy of 90.3%, while pre-trained features do not improve Vaihingen over the standard FCN.The paper attributes the dataset difference to Potsdam’s higher resolution and stronger texture cues; Vaihingen benefits more from early DSM fusion.

7 Conclusion

The paper reports that FCNs, including a no-downsampling variant and pre-trained feature fusion, achieve strong semantic labelling results on high-resolution aerial imagery, while identifying remaining resolution and generalisation challenges.

  • 7 Conclusion: No-downsampling FCNs preserve full input resolution and significantly improve labelling accuracy for detailed high-resolution imagery.Training is computationally demanding, while test-time cost is approximately 4× versus approximately 20–70× for shift-and-stitch interpolation.
  • 7 Conclusion: The no-downsampling network still loses spatial feature resolution with increasing layer depth, motivating future multi-scale feature fusion.The paper identifies combining features from multiple scales as a way to improve labelling detail.
  • 7 Conclusion: Pre-trained CNN features benefit Potsdam semantic labelling, while DSM fusion produces dataset-dependent gains.Late fusion modestly improves Potsdam, whereas input-level DSM fusion gives a larger boost on Vaihingen.
  • 7 Conclusion: The methods achieve state-of-the-art accuracy on the publicly available Vaihingen and Potsdam benchmark datasets.The paper also reports 89.7% accuracy on Potsdam using image features without elevation data.
  • 7 Conclusion: Practical deployment is constrained by generalisation across scenes, the need for larger labelled datasets, and varying availability of input bands.The authors suggest automatic or pseudo-label generation and handling noisy labels as directions for broader generalisation.

Appendix A Experimental Setup

The experiments use tiled aerial-image inputs with fixed channel configurations, standard optimisation settings, GPU-constrained training, and benchmark-specific accuracy assessment with optional CRF smoothing.

  • Appendix A Experimental Setup: FCN filter support is 64×64 pixels for Vaihingen, 128×128 for Potsdam, and 224×224 for pre-trained networks.Most experiments use 256×256 tiles, while 64×64 inputs use non-overlapping 128×128 crops.
  • Appendix A Experimental Setup: Inputs contain four channels for Vaihingen and five for Potsdam, with each channel mean-subtracted before training.Vaihingen uses infrared, red, green, and normalised DSM; Potsdam adds blue.
  • Appendix A Experimental Setup: Large images are processed by overlapping tiles that are classified separately and reassembled, with tile size reduced when GPU memory requires it.Tiles generally measure 512×512 pixels and overlap by half the FCN input size to reduce boundary artefacts.
  • Appendix A Experimental Setup: Accuracy assessment follows the Vaihingen and Potsdam challenge protocol, reporting overall accuracy and class-wise F1 scores.The unknown class is excluded from Vaihingen validation metrics but included for Potsdam.
  • Appendix A Experimental Setup: Optional pixel-wise 4-connected CRFs smooth CNN outputs using probabilities as unary potentials and Canny-derived edges as binary potentials.For Potsdam, the edge mask combines CIR, RGB, and NDVI edges.
Loading 1606.02585v1…