PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks

Nicolas Audebert, Bertrand Le Saux, Sébastien Lefèvre

arXiv:1609.06846v1cs.CVcs.NE

TL;DR

Semantic segmentation of Earth Observation imagery requires dense prediction despite EO’s distinct viewpoint and mixed object–surface content. The paper adapts SegNet-based fully convolutional networks, adds multi-scale aggregation, and fuses heterogeneous sensor predictions with residual correction. The resulting framework reaches state-of-the-art performance on the ISPRS Vaihingen dataset, including 89.8% after data fusion.

  • Problem

    EO imagery differs from everyday images, while earlier patch-based CNN methods relied on an unsupervised segmentation stage that constrained dense semantic labeling.

    Source: §1 · p. 2

  • Method

    The paper transfers an ImageNet-initialized SegNet encoder-decoder to EO data, adds multi-kernel multi-scale aggregation, and learns residual correction for complementary-sensor fusion.

    Source: §1 · p. 2, SegNet · p. 5, Multi-kernel Convolutional Layer · p. 7

  • Results

    89.8% accuracy is reached on the ISPRS Vaihingen 2D Labeling Challenge, while the method improves the FCN-only DLR 1 result by 1.4% from 88.4%.

    Source: §4.2 · p. 11, Data Fusion and Residual Correction · p. 14

  • Takeaways & Limitations

    Deep features and visual filters learned from generic images can be built upon for remote sensing tasks, while multi-scale context and heterogeneous data fusion further improve segmentation.

    Source: §5 · p. 14, Data Fusion and Residual Correction · p. 14

    The authors identify computational overhead from several long parallel fusion streams and leave earlier data-oriented fusion for future work.

    Source: §5 · p. 14

Abstract

from arXiv · show

This work investigates the use of deep fully convolutional neural networks (DFCNN) for pixel-wise scene labeling of Earth Observation images. Especially, we train a variant of the SegNet architecture on remote sensing data over an urban area and study different strategies for performing accurate semantic segmentation. Our contributions are the following: 1) we transfer efficiently a DFCNN from generic everyday images to remote sensing images; 2) we introduce a multi-kernel convolutional layer for fast aggregation of predictions at multiple scales; 3) we perform data fusion from heterogeneous sensors (optical and laser) using residual correction. Our framework improves state-of-the-art accuracy on the ISPRS Vaihingen 2D Semantic Labeling dataset.

1 Introduction

Semantic segmentation enables pixel-wise land-cover and object labeling in Earth Observation imagery, but EO data differs substantially from everyday images. This work adapts fully convolutional networks to multimodal EO data and introduces multi-scale prediction aggregation and learned sensor fusion.

  • EO semantic segmentation assigns each pixel a thematic land-cover or object class, supporting applications such as land-cover mapping.Source: §1 · p. 1
  • Bird’s-eye acquisition places EO objects in a flat 2D plane and reduces depth-related cues such as projected shadows.Source: §1 · p. 2
  • Patch-based CNN approaches used an unsupervised segmentation stage, creating a bottleneck because higher accuracy required strong oversegmentation.Source: §1 · p. 2
  • The paper adapts fully convolutional networks to multimodal EO data for supervised dense segmentation and introduces multi-kernel aggregation with residual data fusion.The multi-kernel layer aggregates predictions from different spatial scales, while residual correction learns a corrective term over averaged complementary-sensor predictions.Source: §1 · p. 2

2 Related Work

Semantic segmentation has progressed from region-based classification toward dense prediction with deep networks. In EO imagery, CNN and fully convolutional approaches have improved land-cover understanding and spatial labeling beyond traditional methods.

  • Semantic segmentation assigns a semantic class to each coherent image region using pixel-wise dense prediction models.Source: §2.1 · p. 2
  • Fully Convolutional Networks replace a classification CNN’s probability vector with a probability map for dense semantic prediction.Source: §2.1 · p. 3
  • CNN-based deep features have outperformed hand-crafted features and Support Vector Machines for EO land-cover classification.Source: §2.2 · p. 3
  • Fully convolutional architectures can learn both pixel categories and spatial structures, including building shapes and curves.Source: §2.2 · p. 3

3 Proposed Method

The proposed method adapts SegNet to large Earth Observation images through patch-based processing, transfer learning, multi-scale prediction aggregation, and heterogeneous-data fusion.

Patch-based processing

  • Large EO tiles are split into smaller sliding-window patches, with overlapping predictions averaged at test time for arbitrary-sized images.The ISPRS Vaihingen tile dimensions exceed typical CNN input resolutions, motivating patch-based processing.Source: §3.1 · pp. 3–4

Input representation

  • IRRG imagery is processed as a three-channel input, while DSM, NDSM, and NDVI form a complementary composite image for a second stream.DSM height information helps distinguish classes such as roofs from roads and bushes from trees.Source: §3.1 · p. 4

SegNet architecture

  • SegNet uses a VGG-16-based encoder-decoder with pooling indices reused for decoder unpooling to relocalize features at full resolution.Encoder weights are initialized from VGG-16, while decoder weights are initialized randomly.Source: SegNet · pp. 4–5

Transfer learning

  • The encoder learning rate is varied relative to the decoder, including transfer-learning settings and a from-scratch baseline, to assess adaptation of generic filters to EO imagery.The experiments compare lre/lrd values of 1, 0.5, 0.1, and 0, plus random initialization of both parts.Source: SegNet · pp. 5–6

Multi-scale aggregation

  • The multi-kernel decoder applies parallel 3 × 3, 5 × 5, and 7 × 7 convolutions to aggregate predictions across receptive-field scales.The branches share the network topology and weights except for the final layer, approximating an ensemble of three models.Source: Multi-kernel Convolutional Layer · pp. 6–7, Figure 2 · p. 6

Heterogeneous data fusion

  • Heterogeneous-data fusion retains the two SegNet outputs and intermediate feature maps, then uses a three-convolution correction network to learn residual prediction adjustments.Simple post-softmax averaging and direct six-channel concatenation are also considered, while residual correction uses complementary streams to address small errors.Source: §3.3 · pp. 8–9

4 Experiments

Experiments evaluate the method on the ISPRS Vaihingen benchmark, examining transfer learning, overlapping inference, multi-kernel aggregation, and multimodal fusion. The combined strategies achieve state-of-the-art performance, with reported gains in global accuracy and class-specific F1 scores.

Experimental setup

  • The model was trained on the full training and validation sets using the same strategy for comparison with state-of-the-art methods.Benchmark evaluation was performed by the organizers.Source: §4.1 · p. 10, §4.2 · p. 10

Sliding-window overlap

  • A 32px stride with 75% overlap improves accuracy by +1% over no overlap while processing a tile in 4 minutes on a Tesla K20c.A 128px stride takes less than 20 seconds, illustrating the accuracy–speed trade-off.Source: Sliding Window Overlap · p. 12

Transfer learning

  • A low encoder learning rate gives the highest validation accuracy, while rates that are too low or too large can respectively limit specialization or increase overfitting risk.The experiments compare encoder learning-rate settings during transfer learning.Source: Transfer Learning · p. 12

Multi-kernel convolutional layer

  • The multi-kernel convolutional layer adds 0.4% accuracy by averaging multi-scale predictions and removing isolated artifacts in homogeneous regions.The layer can smooth predictions without retraining the network from scratch.Source: Transfer Learning · p. 12, Figure 6 · p. 12
  • 89.4% global accuracy is reached with the multi-kernel method, including best F1 gains for impervious surfaces (+1.0%), buildings (+0.8%), and cars (+3.7%).The method remains competitive on low vegetation and tree classes despite cars representing only 1.2% of the dataset.Source: Transfer Learning · p. 13

Data fusion and residual correction

  • 89.8% global accuracy is reached after data fusion and residual correction, with improved building and vegetation F1 scores but lower-than-expected car F1.Naive averaging provides a 0.3–0.4% gain, while the residual correction network uses source-specific information to refine predictions.Source: Data Fusion and Residual Correction · pp. 13–14

5 Conclusion and Future Work

The work transfers SegNet-style DFCNs to Earth Observation imagery, adds multi-scale prediction aggregation and residual correction for heterogeneous-sensor fusion, and improves the ISPRS Vaihingen state of the art by 1%. Future work targets broader architectures, earlier fusion, and earlier multi-scale integration.

Conclusion

  • The study shows that ImageNet-initialized encoder-decoder architectures, notably SegNet, can be transferred to remote sensing data.This supports building remote-sensing models on deep features and visual filters learned from generic images.Source: §5 · p. 14
  • A multi-kernel convolutional layer aggregates predictions across different spatial-context sizes, improving accuracy through model averaging.The layer performs parallel convolutions with several filter sizes.Source: §5 · p. 14
  • Residual correction identifies and corrects small errors left by naive averaging of predictions from heterogeneous inputs.The approach uses a dual-stream architecture for prediction-oriented data fusion.Source: §5 · p. 14

Future Work

  • Future work will test residual correction across network topologies and investigate earlier data fusion and multi-scale integration.Earlier fusion is intended to reduce the computational overhead of several long parallel streams.Source: §5 · p. 14

Related papers

Loading 1609.06846v1…