PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Resolution Semantic Labeling with Convolutional Neural Networks

Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, Pierre Alliez

arXiv:1611.01962v1cs.CV

TL;DR

Dense semantic labeling requires pixel-level spatial precision, which standard categorization CNNs do not directly provide. The paper analyzes existing designs and proposes MLP, which learns to combine features at different resolutions; it reports competitive performance, including second place among 29 methods with 89.5 overall accuracy on Vaihingen.

  • Problem

    Dense semantic labeling requires high spatial accuracy for assigning labels to every pixel, while categorization CNNs trade spatial precision for robustness.

  • Method

    The paper analyzes high-resolution labeling architectures and proposes MLP, which extracts multi-resolution features and learns how to combine them.

  • Results

    89.5 overall accuracy placed MLP second out of 29 methods in the ISPRS Vaihingen challenge.

  • Takeaways & Limitations

    Learning to combine features across resolutions provides a flexible semantic-labeling architecture that improves classification accuracy.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have received increasing attention over the last few years. They were initially conceived for image categorization, i.e., the problem of assigning a semantic label to an entire input image. In this paper we address the problem of dense semantic labeling, which consists in assigning a semantic label to every pixel in an image. Since this requires a high spatial accuracy to determine where labels are assigned, categorization CNNs, intended to be highly robust to local deformations, are not directly applicable. By adapting categorization networks, many semantic labeling CNNs have been recently proposed. Our first contribution is an in-depth analysis of these architectures. We establish the desired properties of an ideal semantic labeling CNN, and assess how those methods stand with regard to these properties. We observe that even though they provide competitive results, these CNNs often underexploit properties of semantic labeling that could lead to more effective and efficient architectures. Out of these observations, we then derive a CNN framework specifically adapted to the semantic labeling problem. In addition to learning features at different resolutions, it learns how to combine these features. By integrating local and global information in an efficient and flexible manner, it outperforms previous techniques. We evaluate the proposed framework and compare it with state-of-the-art architectures on public benchmarks of high-resolution aerial image labeling.

I. INTRODUCTION

Dense semantic labeling assigns a class to every pixel, requiring precise object boundaries that categorization CNNs are not designed to preserve. The paper analyzes existing architectures and introduces MLP, which learns to combine multi-resolution features for accurate, efficient aerial-image labeling.

  • Dense semantic labeling assigns a semantic class to every individual pixel, with high-resolution aerial analysis requiring precisely located object boundaries.
  • Categorization CNNs cannot be directly transferred to pixel labeling because their recipe sacrifices spatial precision for robust object recognition.
  • The paper analyzes dilation, deconvolution, and skip architectures according to desired properties of semantic labeling networks.
  • MLP extracts features at different resolutions and learns how to combine them into fine-grained classification maps.
  • MLP achieves better aerial-labeling performance than many recent techniques while being simpler and smaller.
  • Experiments use the Vaihingen and Potsdam benchmarks, which require outlining small objects with high spatial precision.

II. CONVOLUTIONAL NEURAL NETWORKS

The paper presents neural networks as layered, trainable function approximators and frames pixel labeling as central-pixel classification. CNN training uses a loss function optimized through gradient descent and backpropagation.

  • Feed-forward neural networks pass messages through an acyclic graph, with stacked layers transforming earlier outputs into hierarchical features.
  • A neuron applies an activation function to a weighted sum of inputs, with weights and biases learned to optimize task performance.
  • ReLU activations are popular in image analysis because of practical training advantages.
  • Pixelwise semantic labeling can be formulated as classifying the central pixel of an image patch.
  • Training minimizes a loss comparing predicted and target labels, using gradient descent, backpropagation, and mini-batch estimates.

A. Convolutional Layers

CNN convolutional layers use local, shared filters to learn spatial features efficiently. Downsampling enlarges receptive fields and improves robustness, but loses the spatial precision required for dense labeling.

  • A. Convolutional Layers: CNN convolutional layers impose local connectivity and associate neurons with spatial image locations.
  • A. Convolutional Layers: A convolutional layer applies learned kernels to its input, with the same filter reused across locations to provide translation invariance.
  • A. Convolutional Layers: These constraints reduce parameters relative to fully connected layers while enabling CNNs to learn relevant spatial features directly from images.
  • B. Increasing the Receptive Field: Receptive fields specify the input extent influencing a neuron, and semantic labeling often requires broad spatial context.
  • B. Increasing the Receptive Field: Downsampling through pooling enlarges receptive fields with fewer parameters, but trades localization capability for recognition robustness.

C. Fully Convolutional Networks (FCNs)

Fully convolutional networks produce outputs for variable-sized inputs and reduce the cost of dense prediction. Their downsampled outputs motivate learned upsampling, while semantic labeling benefits from retaining context without uniform full-resolution processing.

  • C. Fully Convolutional Networks (FCNs): Dense labeling classifies patch centers, whereas fully convolutional networks apply convolutions across many locations to produce spatially organized outputs.
  • C. Fully Convolutional Networks (FCNs): FCNs reduce computational complexity and allow training and testing with different patch sizes.
  • C. Fully Convolutional Networks (FCNs): Classifying a central pixel requires surrounding spatial context, but outer regions can be represented at lower resolution without preventing identification.
  • C. Fully Convolutional Networks (FCNs): Downsampling makes FCN outputs smaller than inputs, motivating learned deconvolutional layers that upsample feature maps.

III. ANALYSIS OF HIGH-RESOLUTION LABELING CNNS

High-resolution labeling must preserve fine spatial detail while incorporating broad context, a tension that standard upsampling does not fully resolve. The analysis compares dilation and deconvolution approaches, highlighting their benefits and computational or localization limits.

  • Dense labeling requires detailed high-resolution outputs that retain object boundaries while using substantial contextual information.
  • Dilation Networks: Dilation networks generate high-resolution predictions through interleaved shifted outputs or operations on non-contiguous feature elements.
  • Dilation Networks: Dilation enlarges the receptive field without increasing trainable parameters, but pooling still limits spatial precision and naive upsampling has no theoretical disadvantage.
  • Dilation Networks: Dilated convolutions do not inherently provide robustness to spatial variation, and dilation networks can require impractical computation and memory for moderately large architectures.
  • Dilation Networks: Dilation can improve aerial labeling over FCN upsampling, especially for tiny minority-class objects such as cars, because training compares interleaved outputs directly with individual pixels.

B. Deconvolution Networks (unpooling)

Deconvolution networks learn complex upsampling by mirroring an FCN with deconvolution and unpooling layers. They recover localization through pooling indices, but their effectiveness depends on max pooling and their greater depth complicates optimization.

  • Deconvolution Networks: Deconvolution networks replace single-layer upsampling with a learned multi-layer upsampling function that mirrors convolution and pooling operations.
  • Deconvolution Networks: Max unpooling recalls each pooled maximum’s location, helping the decoder recover precise object boundaries after higher-level recognition.
  • Deconvolution Networks: The encoder–decoder uses a shared middle representation to translate between images and classification maps.
  • Deconvolution Networks: The unpooling scheme cannot inherently recover lost spatial resolution when downsampling is not performed with max pooling.
  • Deconvolution Networks: Deconvolution networks are roughly twice as deep as associated FCNs, increasing trainable parameters and often making optimization slower and more difficult.

C. Skip Networks

Skip networks combine classification maps derived from CNN features at multiple resolutions to balance localization and recognition. The paper identifies this strategy as efficient but argues that learning to combine features is more flexible than adding classification maps.

  • Skip network operation: Skip networks generate class-specific score maps from intermediate CNN features at different resolutions, then combine them progressively from coarse to fine.Lower-resolution maps are upsampled to match higher-resolution maps before element-wise addition.
  • Motivation: This multi-resolution fusion addresses the localization/recognition trade-off by mixing contextual and spatially precise observations without unnecessarily increasing network depth or width.The approach is described as combining “what and where.”
  • Limitations: The paper argues that combining classification verdicts is inflexible because shallow high-resolution classifiers may not detect objects recognized only after deeper contextual reasoning.The authors specifically question refining coarse building predictions with higher-resolution building classifiers.
  • Proposed alternative: The proposed alternative combines multi-resolution features rather than classification maps, allowing high-resolution edge information to refine coarse object boundaries.This design replaces high-resolution object detectors with feature cues such as edge detectors when refining buildings.
  • MLP network: The MLP architecture upsamples intermediate features, concatenates them into a feature pool, and learns their combination through a neural network applied pixel by pixel.This learned combination generalizes the skip network’s element-wise addition and supports more complex relations between resolutions.
  • MLP network: The MLP is implemented with convolutional layers using 1 × 1 kernels, applying the same learned feature-combination function at every spatial location.The paper presents a one-hidden-layer MLP as a minimal flexible system for combining the pooled features.

A. Datasets and Evaluation Metrics

The experiments use Vaihingen and Potsdam high-resolution aerial-image benchmarks, with closely specified training, validation, input-channel, and class configurations. Performance is measured using overall accuracy and class-sensitive F1 metrics.

  • Datasets: The Vaihingen benchmark contains 33 tiles averaging 2494 × 2064 pixels at 9 cm spatial resolution, with 16 fully annotated tiles.It provides NIR, red, green, and normalized DSM data; five images are selected for validation.
  • Datasets: The Potsdam benchmark contains 38 tiles of 6000 × 6000 pixels at 5 cm resolution, including 24 annotated tiles and an additional blue channel.Seven tiles are used for validation and 17 for training.
  • Datasets: Both datasets use six semantic classes: impervious surface, building, low vegetation, tree, car, and clutter/background.The Vaihingen experiments predict five classes by excluding clutter because of insufficient training data, whereas Potsdam uses all six.
  • Evaluation metrics: Evaluation uses overall accuracy for total pixel correctness and F1-score for class-specific performance, including mean F1 across classes.Ground-truth boundaries are eroded to account for labeling mistakes before accuracy is measured.
  • Network setup: Experiments derive skip, MLP, and deconvolution architectures from a base fully convolutional network, whose final predictions are upsampled to the original resolution.The base FCN downsamples by a factor of 16 and uses a 1 × 1 convolution for class scores.
  • Network setup: All experiments use DSM, NIR, red, and green channels, while Potsdam input is downsampled and its output linearly upsampled by a factor of 2.This adjustment aims to provide similar receptive-field coverage in meters across the two datasets.

C. Training

The evaluation compares base FCN derivatives and other semantic-labeling methods on Vaihingen and Potsdam. MLP generally provides the strongest accuracy, including second place in the Vaihingen ISPRS challenge.

  • Comparison of derived architectures: MLP achieves the best performance in almost every comparison against the base FCN, unpooling, and skip architectures.Skip improves over the base network, whereas unpooling does not necessarily improve it.
  • Comparison of derived architectures: Injecting lower-resolution features significantly improves classification accuracy, with MLP learning how to combine those features.
  • Comparison with other methods: MLP outperforms dilation networks on both Vaihingen and Potsdam, making it the most competitive presented architecture category.
  • Comparison with other methods: MLP also outperforms deconvolution, CNN+RF, CNN+RF+CRF, and Dilation+CRF on Vaihingen.
  • Comparison with other methods: On Potsdam, VGG+Dilation has the best overall performance, although MLP outperforms the simpler non-pretrained dilation network.The VGG+Dilation result is not best on every individual class.
  • ISPRS challenge submission: 89.5 overall accuracy places MLP second among 29 methods in the Vaihingen ISPRS challenge.The submitted MLP was simpler than several competing leaderboard methods.

E. Visual Results

Visual comparisons show that multi-resolution architectures produce sharper boundaries and recover small objects better than the base FCN. They also reveal artifacts from unpooling and inaccuracies in the ground truth.

  • Closeup comparisons: The base FCN produces blobby objects, while the derived methods generate sharper boundaries and recover small objects through lower-resolution reasoning.The improvement is especially visible for cars and a thin road.
  • Closeup comparisons: Unpooling often produces tiny artifacts, including isolated pixels, potentially because max unpooling creates sparse upsampled maps.
  • Closeup comparisons: MLP outlines object boundaries more accurately than skip, including the staircase-shaped edge of a building.
  • Ground-truth quality: Ground-truth imprecision limits fine-labeling assessment, including an inaccurately labeled car and an omitted lightwell.
  • Recognition behavior: CNNs distinguish visually similar small objects by labeling roof objects as buildings and comparable objects outside as clutter.
  • Entire-tile comparisons: MLP recognizes a large white building missed or partially recovered by other methods, while Dilation produces building holes absent from MLP.The comparison concerns entire Vaihingen test tiles.

F. Running Times

The running-time evaluation shows that derived architectures add overhead to the base FCN, with MLP being the slowest derivative. Despite this, MLP combines the highest accuracy with substantially faster validation inference than dilation.

  • Execution-time comparison: MLP is the slowest derived network, followed by unpooling and skip, and all three add overhead to the base FCN.
  • Execution-time comparison: MLP classifies the Vaihingen validation set in about 30 seconds and the Potsdam validation set in 2 minutes.
  • Accuracy–efficiency trade-off: MLP provides the highest accuracy while running much faster than the dilation network.
  • Accuracy–efficiency trade-off: The architecture allocates computational resources across resolutions rather than spending them uniformly at high resolution.
Loading 1611.01962v1…