PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks

Michele Volpi, Devis Tuia

arXiv:1608.00775v2cs.CV

TL;DR

Ultra-high-resolution imagery demands models that can learn high-level concepts while preserving dense spatial labeling. The paper proposes a downsample-then-upsample CNN, which achieves accurate, geometrically improved, and efficient semantic labeling on sub-decimeter aerial imagery.

  • Problem

    Handcrafted-feature pipelines depend on feature choices and parameters, while patch classification is inefficient and does not explicitly model spatial label configurations.

  • Method

    The method encodes high-level representations into a coarse spatial map, then learns deconvolutions to upsample them to the original resolution for dense pixel labeling.

  • Results

    The full patch labeling CNN is the most accurate across evaluated metrics, outperforms the superpixel baseline, and requires 31s for five test images compared with 13841s for naive patch classification inference.

  • Takeaways & Limitations

    Full-patch prediction provides efficiency and semantic/geometric accuracy advantages without post-processing or external classifiers and handcrafted features.

Abstract

from arXiv · show

Semantic labeling (or pixel-level land-cover classification) in ultra-high resolution imagery (< 10cm) requires statistical models able to learn high level concepts from spatial data, with large appearance variations. Convolutional Neural Networks (CNNs) achieve this goal by learning discriminatively a hierarchy of representations of increasing abstraction. In this paper we present a CNN-based system relying on an downsample-then-upsample architecture. Specifically, it first learns a rough spatial map of high-level representations by means of convolutions and then learns to upsample them back to the original resolution by deconvolutions. By doing so, the CNN learns to densely label every pixel at the original resolution of the image. This results in many advantages, including i) state-of-the-art numerical accuracy, ii) improved geometric accuracy of predictions and iii) high efficiency at inference time. We test the proposed system on the Vaihingen and Potsdam sub-decimeter resolution datasets, involving semantic labeling of aerial images of 9cm and 5cm resolution, respectively. These datasets are composed by many large and fully annotated tiles allowing an unbiased evaluation of models making use of spatial information. We do so by comparing two standard CNN architectures to the proposed one: standard patch classification, prediction of local label patches by employing only convolutions and full patch labeling by employing deconvolutions. All the systems compare favorably or outperform a state-of-the-art baseline relying on superpixels and powerful appearance descriptors. The proposed full patch labeling CNN outperforms these models by a large margin, also showing a very appealing inference time.

I. INTRODUCTION

Ultra-high-resolution semantic labeling requires models that learn spatial relationships and representations directly from data rather than relying on potentially sub-optimal handcrafted features. The paper proposes a fully supervised CNN architecture that downsamples and then upsamples spatial representations for dense labeling.

  • Motivation: Semantic labeling assigns a land-cover or land-use class to every pixel, unlike patch classification, which predicts one label for an entire patch.Patch classification commonly assigns the patch label to its central pixel, while semantic labeling predicts labels at each location.
  • Motivation: Handcrafted-feature methods depend on predefined extraction procedures and parameters whose performance is unknown for the specific data.Their filters are not learned end-to-end, potentially making the resulting representations sub-optimal.
  • Deep learning: CNNs learn task-specific representations end-to-end by optimizing a differentiable loss through trainable network weights.This provides a data-driven alternative to separately designed appearance descriptors.
  • Deep learning in remote sensing: Patch classification is inefficient and can oversmooth object boundaries because it does not explicitly learn spatial configurations of labels.A patch may contain semantically valid competing classes, making a single central-pixel label ambiguous.
  • Contribution: The proposed network encodes land-cover concepts into a coarse spatial bottleneck and learns deconvolutions to recover the original patch resolution.This architecture models global relationships on a coarser signal and local class structure during upsampling.
  • Contribution: The paper evaluates dense-labeling CNNs against standard CNN strategies on the Vaihingen and Potsdam aerial-image semantic-labeling datasets.The proposed approach is trained specifically for dense labeling rather than classification.

II. CONVOLUTIONAL NEURAL NETWORKS

The paper introduces CNNs as hierarchical feed-forward networks composed of trainable processing layers. It motivates the architecture review and the use of convolutional and deconvolutional operators for spatial processing.

  • CNN structure: A CNN is a sequential hierarchy of processing layers that transforms an input through trainable units toward an output.The network can be viewed as a concatenation of functions applied from the input onward.
  • CNN structure: CNN layers apply learnable convolutions shared across image locations rather than using a separate neuron for each location.This weight sharing distinguishes CNNs from densely connected multilayer perceptrons.
  • Section organization: The section reviews CNN structure and strategies for reducing overfitting before presenting the paper’s specific network architectures.This review establishes the components used in the proposed system.

A. CNN building blocks

The paper’s CNN architectures combine convolutions, nonlinearities, pooling, deconvolutions, and differentiable classification losses to produce semantic labels. The proposed full patch labeling model uniquely uses deconvolutions to upsample spatial representations for dense prediction.

  • CNNs use sequential trainable layers that transform inputs into task-specific outputs through learned representations.
  • Convolutions: Convolutional layers apply shared learnable filters across spatial locations, mapping K-dimensional inputs to K′-dimensional activations.
  • Nonlinear activations: ReLU-family nonlinearities improve gradient propagation and efficient computation, while leaky ReLU allows gradients to pass for negative activations.Leaky ReLU addresses the possibility that standard ReLU permanently disables neurons with negative activations.
  • Spatial Pooling: Pooling summarizes activations over local windows, downsampling signals while promoting translation invariance.Max pooling propagates the strongest activation in each window but may overfit because gradients flow through maximum values.
  • Deconvolutions: Deconvolutions, implemented as transposed convolutions, upsample spatial signals while learning channel-wise filters.The proposed CNN-FPL uses these operators to relocate activations onto an upsampled grid.
  • Classification layer and loss: The classification layer uses softmax scores and cross-entropy loss, averaging loss over spatial predictions in dense labeling.Inference assigns each location the class with the highest predicted probability.

B. Mitigating overfitting

The training procedure mitigates overfitting and improves generalization through regularization, normalization, augmentation, balanced sampling, transformations, and noise injection. These measures address redundancy, class imbalance, and variation in spatial and spectral appearance.

  • Patch-level prediction can be suboptimal because ultra-high-resolution data are redundant and semantic variation is concentrated in spatial arrangements.
  • Dropout reduces neuron co-adaptation by randomly disabling neurons and their connections during training.
  • Batch normalization stabilizes activation distributions, supports larger learning rates and mini-batches, and reduces sensitivity to initialization.
  • Data augmentation creates label-preserving synthetic examples to increase training diversity, regularize the model, and reduce patch correlation.The paper uses rotations, flips, and additive noise as augmentation strategies.
  • Random spatial sampling accounts for uneven class frequencies, including rare cars and ubiquitous roads in the Vaihingen training data.The cited frequencies are 1.2% for cars and 27.94% for roads.
  • Noise injection encourages spectrally smooth decision rules by reducing correlations among similar mini-batch patches.

A. Common strategies

All architectures share common training procedures involving sampled super-batches, Xavier-style initialization, unit stride, and momentum-based stochastic gradient descent. Validation patches are sampled separately and are not used for training.

  • Training uses super-batches of sampled training patches, with each epoch defined as 500 passes over the super-batch set.
  • Convolutional and deconvolutional filters use improved Xavier initialization and stride 1.
  • Optimization uses stochastic gradient descent with momentum multiplier 0.9.
  • Validation error is monitored using uniformly class-sampled validation patches that remain excluded from training.

B. Architecture 1: patch classification (CNN-PC)

CNN-PC is a standard patch-classification architecture that predicts the central-pixel label from each input patch. It uses convolutional layers followed by pooling and a fully connected classifier for single-class scores per patch.

  • CNN-PC predicts one label per input patch, specifically the label of the patch’s central pixel.
  • The architecture uses four convolutional stages with 64, 64, 128, and 256 neurons before mapping activations to class scores.
  • CNN-PC applies max pooling, leaky ReLUs, and 50% dropout during feature extraction.
  • Training: The network trains with scheduled learning rates for 400 epochs using mini-batches of 128 and weight decay 0.01.

C. Architecture 2: subpatch labeling (CNN-SPL)

CNN-FPL extends subpatch labeling with learned upsampling, using a downsampling bottleneck followed by deconvolutions to produce dense, structured predictions at input resolution.

  • CNN-FPL architecture: CNN-FPL combines a CNN-SPL downsampling block with three deconvolution layers that compensate for three max-pooling levels.The deconvolution layers replace the 1 × 1 convolutions and double activation-map size during upsampling.
  • CNN-FPL architecture: The 65 × 65 input patch is reduced to a 9×9 activation map, forcing a rough spatial representation before learned upsampling.The authors describe this bottleneck as beneficial for representing class structure within the patch.
  • Inference: Each CNN-FPL forward pass produces a segmentation the same size as the input patch, enabling dense labeling with stride one.The approach can feed forward an entire image and directly obtain a dense labeling or class-conditional score map.
  • Training: CNN-FPL training uses a smaller mini-batch of 32 and an extended schedule, while CNN-PC and CNN-SPL use batches of 128.The smaller CNN-FPL batches slow convergence but ease overfitting; weights from CNN-PC initialize common downsampling blocks.
  • Architecture selection: The final CNN-FPL design uses three downsampling layers because two were too shallow and four were too deep to optimize.Validation experiments selected the intermediate design as the best trade-off.

IV. DATA AND EXPERIMENTAL SETUP

The experiments evaluate CNN models on fully annotated Vaihingen and Potsdam aerial-image tiles using spectral and elevation information, with multiple validation metrics and a handcrafted-feature baseline.

  • Datasets: Vaihingen contains 33 tiles at 9cm resolution, including 16 fully annotated tiles used for training and validation.The networks use stacked NIR-R-G and normalized DSM inputs, giving four input dimensions.
  • Datasets: Potsdam provides 38 tiles at 5cm resolution, with 24 densely annotated tiles and five input dimensions from spectral channels plus DSM.Eighteen annotated tiles are used for training and six for validation.
  • Compared methods: The comparison includes CNN-PC, CNN-SPL, CNN-FPL, and a superpixel baseline using handcrafted spectral, morphology, texture, and elevation features.The handcrafted baseline also includes vegetation and water indices derived from the available channels.
  • Evaluation: Evaluation uses full, background-excluded, eroded-full, and eroded-background-excluded strategies on the corresponding validation sets.The eroded settings remove three-pixel boundary regions to tolerate small edge errors.
  • Evaluation: Reported metrics are overall accuracy, Kappa, average class accuracy, and class-averaged F1 score.Overall accuracy and Kappa are global measures, whereas average class accuracy and F1 are class-specific.

1) Numerical results:

CNN-FPL is the most accurate CNN configuration across evaluation settings and preserves structures qualitatively, while also offering much faster inference than naive patch classification.

  • Numerical results: CNN-FPL is the most accurate tested CNN under all evaluation settings and metrics on the Vaihingen validation set.Removing clutter improves CNN-PC by 11–12 points in AA and F1, while label-patch methods gain 4–5 points.
  • Numerical results: Eroded-boundary evaluation produces significantly higher accuracies, indicating that prediction boundaries are often blurred within the three-pixel erosion radius.CNN-SPL remains between CNN-PC and CNN-FPL, while all CNN models outperform the handcrafted superpixel baseline.
  • Inference time: 31s total inference time, or 6.2 s/image, is reported for CNN-FPL on five validation images, compared with 13841s for naive CNN-PC inference.Stride-two inference roughly halves time while losing less than 1% OA; CNN-SPL predicts the validation maps in 9s total.
  • Qualitative results: CNN-FPL produces the best qualitative segmentations, preserving thin elongated structures, object shapes, and gaps between buildings.The learned upsampling also reduces spurious small building predictions through class co-occurrences.
  • Qualitative results: CNN-based methods are generally more accurate for difficult car segmentation, while CNN-FPL balances detecting individual cars with segmenting them.The background class is detected with varying success because it combines semantically and visually different regions.
  • Input information: Combining elevation and spectral information eases detection of elevated building and tree classes, including visually ambiguous rooftops.The elevation distinction helps prevent vegetated rooftops from being classified as grass.

3) Submission to challenge:

On the Vaihingen and Potsdam challenge submissions, CNN-FPL achieved competitive test accuracy without post-processing while requiring far less inference time than patch classification.

  • Vaihingen submission: 87.3 points in overall accuracy placed CNN-FPL fifth on the Vaihingen challenge leaderboard, tied with ADL 2.The comparison used predictions for 17 unlabeled tiles and the independent Vaihingen evaluation criteria.
  • Submission design: The reported CNN-FPL results used only CNN classification scores, without external nonlinear classifiers or conditional-random-field post-processing.The authors note that adding handcrafted features or random-field smoothing might gain a few accuracy points but would hinder the paper’s contribution.
  • Potsdam validation: CNN-FPL improved all considered accuracy metrics over CNN-PC, CNN-SPL, and the SP-MSF baseline on the Potsdam validation set.CNN-PC had higher OA and K but lower AA and F1 than the superpixel baseline, while CNN-FPL offered better accuracies across the metrics.

2) Qualitative results:

Qualitative predictions show that CNN-FPL preserves object geometry and resolves ambiguous spatial arrangements more effectively than patch-based alternatives, while remaining competitive overall without post-processing.

  • Geometric accuracy: CNN-FPS segmented cars and buildings with geometrically accurate whole-object shapes, unlike the multiple-part or undersegmented predictions of CNN-PC and CNN-SPL.The reported benefit is attributed to learned deconvolutions that upsample predictions to full image resolution.
  • Mechanism: The downsample-then-upsample architecture encoded class relationships and co-occurrences through learned deconvolutions, producing interdependent pixel predictions.The authors interpret learned upsamplings as activation-specific interpolation filters that encode spatial dependence.
  • Overall assessment: Across two challenging datasets, the approach reached state-of-the-art-aligned results without CRF or MRF post-processing or handcrafted features.The conclusion frames full-patch segmentation as advantageous for both efficiency and semantic or geometric accuracy.
Loading 1608.00775v2…