Source-linked AI summary
Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks
Nicolas Audebert, Bertrand Le Saux, Sébastien Lefèvre
TL;DR
Semantic segmentation of Earth Observation imagery requires dense prediction despite EO’s distinct viewpoint and mixed object–surface content. The paper adapts SegNet-based fully convolutional networks, adds multi-scale aggregation, and fuses heterogeneous sensor predictions with residual correction. The resulting framework reaches state-of-the-art performance on the ISPRS Vaihingen dataset, including 89.8% after data fusion.
Problem
EO imagery differs from everyday images, while earlier patch-based CNN methods relied on an unsupervised segmentation stage that constrained dense semantic labeling.
Method
The paper transfers an ImageNet-initialized SegNet encoder-decoder to EO data, adds multi-kernel multi-scale aggregation, and learns residual correction for complementary-sensor fusion.
Results
89.8% accuracy is reached on the ISPRS Vaihingen 2D Labeling Challenge, while the method improves the FCN-only DLR 1 result by 1.4% from 88.4%.
Takeaways & Limitations
Deep features and visual filters learned from generic images can be built upon for remote sensing tasks, while multi-scale context and heterogeneous data fusion further improve segmentation.
Takeaways & Limitations
The authors identify computational overhead from several long parallel fusion streams and leave earlier data-oriented fusion for future work.
Abstract
from arXiv · showhide
This work investigates the use of deep fully convolutional neural networks (DFCNN) for pixel-wise scene labeling of Earth Observation images. Especially, we train a variant of the SegNet architecture on remote sensing data over an urban area and study different strategies for performing accurate semantic segmentation. Our contributions are the following: 1) we transfer efficiently a DFCNN from generic everyday images to remote sensing images; 2) we introduce a multi-kernel convolutional layer for fast aggregation of predictions at multiple scales; 3) we perform data fusion from heterogeneous sensors (optical and laser) using residual correction. Our framework improves state-of-the-art accuracy on the ISPRS Vaihingen 2D Semantic Labeling dataset.
1 Introduction
Semantic segmentation enables pixel-wise land-cover and object labeling in Earth Observation imagery, but EO data differs substantially from everyday images. This work adapts fully convolutional networks to multimodal EO data and introduces multi-scale prediction aggregation and learned sensor fusion.
- EO semantic segmentation assigns each pixel a thematic land-cover or object class, supporting applications such as land-cover mapping.
- Bird’s-eye acquisition places EO objects in a flat 2D plane and reduces depth-related cues such as projected shadows.
- Patch-based CNN approaches used an unsupervised segmentation stage, creating a bottleneck because higher accuracy required strong oversegmentation.
- The paper adapts fully convolutional networks to multimodal EO data for supervised dense segmentation and introduces multi-kernel aggregation with residual data fusion.The multi-kernel layer aggregates predictions from different spatial scales, while residual correction learns a corrective term over averaged complementary-sensor predictions.
2 Related Work
Semantic segmentation has progressed from region-based classification toward dense prediction with deep networks. In EO imagery, CNN and fully convolutional approaches have improved land-cover understanding and spatial labeling beyond traditional methods.
- Semantic segmentation assigns a semantic class to each coherent image region using pixel-wise dense prediction models.
- Fully Convolutional Networks replace a classification CNN’s probability vector with a probability map for dense semantic prediction.
- CNN-based deep features have outperformed hand-crafted features and Support Vector Machines for EO land-cover classification.
- Fully convolutional architectures can learn both pixel categories and spatial structures, including building shapes and curves.
3 Proposed Method
The proposed method adapts SegNet to large Earth Observation images through patch-based processing, transfer learning, multi-scale prediction aggregation, and heterogeneous-data fusion.
- Patch-based processing: Large EO tiles are split into smaller sliding-window patches, with overlapping predictions averaged at test time for arbitrary-sized images.The ISPRS Vaihingen tile dimensions exceed typical CNN input resolutions, motivating patch-based processing.
- Input representation: IRRG imagery is processed as a three-channel input, while DSM, NDSM, and NDVI form a complementary composite image for a second stream.DSM height information helps distinguish classes such as roofs from roads and bushes from trees.
- SegNet architecture: SegNet uses a VGG-16-based encoder-decoder with pooling indices reused for decoder unpooling to relocalize features at full resolution.Encoder weights are initialized from VGG-16, while decoder weights are initialized randomly.
- Transfer learning: The encoder learning rate is varied relative to the decoder, including transfer-learning settings and a from-scratch baseline, to assess adaptation of generic filters to EO imagery.The experiments compare lre/lrd values of 1, 0.5, 0.1, and 0, plus random initialization of both parts.
- Multi-scale aggregation: The multi-kernel decoder applies parallel 3 × 3, 5 × 5, and 7 × 7 convolutions to aggregate predictions across receptive-field scales.The branches share the network topology and weights except for the final layer, approximating an ensemble of three models.
- Heterogeneous data fusion: Heterogeneous-data fusion retains the two SegNet outputs and intermediate feature maps, then uses a three-convolution correction network to learn residual prediction adjustments.Simple post-softmax averaging and direct six-channel concatenation are also considered, while residual correction uses complementary streams to address small errors.
4 Experiments
Experiments evaluate the method on the ISPRS Vaihingen benchmark, examining transfer learning, overlapping inference, multi-kernel aggregation, and multimodal fusion. The combined strategies achieve state-of-the-art performance, with reported gains in global accuracy and class-specific F1 scores.
- Experimental setup: The model was trained on the full training and validation sets using the same strategy for comparison with state-of-the-art methods.Benchmark evaluation was performed by the organizers.
- Sliding-window overlap: A 32px stride with 75% overlap improves accuracy by +1% over no overlap while processing a tile in 4 minutes on a Tesla K20c.A 128px stride takes less than 20 seconds, illustrating the accuracy–speed trade-off.
- Transfer learning: A low encoder learning rate gives the highest validation accuracy, while rates that are too low or too large can respectively limit specialization or increase overfitting risk.The experiments compare encoder learning-rate settings during transfer learning.
- Multi-kernel convolutional layer: The multi-kernel convolutional layer adds 0.4% accuracy by averaging multi-scale predictions and removing isolated artifacts in homogeneous regions.The layer can smooth predictions without retraining the network from scratch.
- Multi-kernel convolutional layer: 89.4% global accuracy is reached with the multi-kernel method, including best F1 gains for impervious surfaces (+1.0%), buildings (+0.8%), and cars (+3.7%).The method remains competitive on low vegetation and tree classes despite cars representing only 1.2% of the dataset.
- Data fusion and residual correction: 89.8% global accuracy is reached after data fusion and residual correction, with improved building and vegetation F1 scores but lower-than-expected car F1.Naive averaging provides a 0.3–0.4% gain, while the residual correction network uses source-specific information to refine predictions.
5 Conclusion and Future Work
The work transfers SegNet-style DFCNs to Earth Observation imagery, adds multi-scale prediction aggregation and residual correction for heterogeneous-sensor fusion, and improves the ISPRS Vaihingen state of the art by 1%. Future work targets broader architectures, earlier fusion, and earlier multi-scale integration.
- Conclusion: The study shows that ImageNet-initialized encoder-decoder architectures, notably SegNet, can be transferred to remote sensing data.This supports building remote-sensing models on deep features and visual filters learned from generic images.
- Conclusion: A multi-kernel convolutional layer aggregates predictions across different spatial-context sizes, improving accuracy through model averaging.The layer performs parallel convolutions with several filter sizes.
- Conclusion: Residual correction identifies and corrects small errors left by naive averaging of predictions from heterogeneous inputs.The approach uses a dual-stream architecture for prediction-oriented data fusion.
- Future Work: Future work will test residual correction across network topologies and investigate earlier data fusion and multi-scale integration.Earlier fusion is intended to reduce the computational overhead of several long parallel streams.