Source-linked AI summary
Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networks
Benjamin Bischke, Patrick Helber, Joachim Folz, Damian Borth, Andreas Dengel
TL;DR
High-resolution satellite segmentation often produces building predictions with poor boundaries, limiting precise footprint extraction. The paper introduces an uncertainty-weighted cascaded multi-task loss using distance-transform boundary information and evaluates it on the Inria dataset, reporting an 8.3% improvement over recent methods without post-processing.
Problem
Building-footprint segmentation in high-resolution satellite imagery remains difficult because predictions often fail to preserve building boundaries across varied urban landscapes.
Method
The paper trains semantic and geometric tasks jointly with an uncertainty-weighted cascaded multi-task loss based on truncated distance-transform representations.
Results
8.3% higher validation-set Intersection over Union (IoU) than current state-of-the-art accuracy is reported without post-processing.
Takeaways & Limitations
Boundary information can be leveraged through multi-task learning to improve building-footprint segmentation on the Inria Aerial Image Labeling Dataset.
Abstract
from arXiv · showhide
The increased availability of high resolution satellite imagery allows to sense very detailed structures on the surface of our planet. Access to such information opens up new directions in the analysis of remote sensing imagery. However, at the same time this raises a set of new challenges for existing pixel-based prediction methods, such as semantic segmentation approaches. While deep neural networks have achieved significant advances in the semantic segmentation of high resolution images in the past, most of the existing approaches tend to produce predictions with poor boundaries. In this paper, we address the problem of preserving semantic segmentation boundaries in high resolution satellite imagery by introducing a new cascaded multi-task loss. We evaluate our approach on Inria Aerial Image Labeling Dataset which contains large-scale and high resolution images. Our results show that we are able to outperform state-of-the-art methods by 8.3\% without any additional post-processing step.
I. INTRODUCTION
The paper targets building-footprint segmentation in high-resolution satellite imagery, where predictions often overlap ground truth yet lose building boundaries. It proposes a cascaded multi-task loss and evaluates deeper-network and decoder choices, reporting improved accuracy without post-processing.
- Building-footprint segmentation is challenging globally because urban density, building shapes, and illumination vary with geographic location.
- FCN predictions can overlap ground-truth masks while failing to preserve their boundaries, producing rounded or “blobby” building shapes.
- Existing boundary-improvement approaches increase model complexity, parameter count, or inference memory through additional networks and multi-resolution feature maps.
- The proposed approach combines a cascaded multi-task loss with a deeper network architecture to incorporate building boundary information while using less inference memory than prior approaches.
- VGG16 encoder features and decoder architectures are evaluated on the Inria Aerial Image Labeling Dataset through detailed experiments.
- 3.1% improvement is reported from the uncertainty-weighted cascaded multi-task loss without major architectural changes.
- 8.3% higher validation-set Intersection over Union (IoU) than current state-of-the-art accuracy is reported without post-processing.
II. RELATED WORK
Related work spans traditional feature-based building extraction and deep semantic-segmentation methods. These approaches improve overall predictions but often require post-processing or still fail to represent precise semantic boundaries.
- Semantic segmentation methods use skip-connections, dilated convolutions, or conditional random fields to recover details lost through pooling.
- Despite improving overall segmentation, existing approaches often poorly segment boundaries between semantic classes.
- Traditional building-footprint extraction uses vegetation, texture, and color features with classifiers such as AdaBoost, SVMs, and random forests.
- Traditional and newer CNN-based methods commonly apply additional post-processing to refine segmentation or remove false buildings.
III. CASCADED MULTI-TASK NETWORK
The network uses multi-task learning to jointly represent semantic labels and geometric boundary information. Boundary distances are truncated, signed, quantized, and converted into pixel-wise classification targets.
- The method trains a segmentation network on input images, ground-truth semantic masks, and corresponding truncated distance class labels.
- A. Output Representation: Geometric information is derived from each pixel’s distance to building boundaries, allowing boundary location to be learned alongside semantic classes.
- A. Output Representation: Distance transforms are truncated at a threshold so the representation emphasizes pixels nearest to object borders.
- A. Output Representation: Signed distances distinguish pixels inside and outside building masks, while continuous values are uniformly quantized for training.
- A. Output Representation: The quantized distance map is one-hot encoded into binary pixel-wise maps that act as classification maps for border-distance bins.
B. Enoder-Decoder Network Architecture
The network uses a SegNet-style VGG16 encoder-decoder, with pooling indices transferred from encoder to decoder for feature-map upsampling. A distance-prediction layer feeds its features, together with decoder features, into the final segmentation layer.
- The network is based on SegNet, using a VGG16 encoder with 13 convolutional layers and five max-pooling layers.
- The decoder mirrors the encoder and uses transferred pooling indices to upsample feature maps.
- Hdist predicts building-border distances from the decoder output, while Hseg computes the final building-footprint segmentation mask.
- Hseg receives concatenated feature maps from Hdist and the last decoder layer.
C. Uncertainty Based Multi-Task Loss
The method combines distance-class and segmentation losses in a cascaded multi-task objective. Task weights are learned from prediction uncertainty rather than fixed or selected through grid search, with numerical-stability adjustments during training.
- The multi-task objective combines pixel-wise classification losses for distance classes and segmentation masks.
- Each task loss is weighted by a scalar representing its importance in the combined loss.
- Relative task weights are learned from the uncertainty of each task’s predictions instead of being equated or grid-searched.
- The model represents classification likelihood using uncertainty-scaled outputs passed through a softmax function.
- The uncertainty-weighted classification formulation is expressed through negative log likelihood and simplified for both tasks.
- For numerical stability, training predicts log(σ2_i), and network parameters and uncertainty weights are optimized with SGD.
A. Inria Aerial Image Labeling Dataset
The Inria Aerial Image Labeling Dataset contains large, high-resolution aerial imagery spanning ten cities and varied urban settlements, with building and nonbuilding ground-truth classes.
- The dataset contains 360 ortho-rectified aerial RGB images with 0.3 m spatial resolution.
- Each image tile is 5000 x 5000 pixels and covers 1500 x 1500 m, for an overall area of 810 sq. km.
- The imagery covers ten cities with settlements ranging from dense metropolitan districts to alpine towns.
- Ground truth distinguishes building and nonbuilding semantic classes.
B. Evaluation Metrics
The experiments evaluate building-footprint segmentation using Intersection over Union for buildings and pixel accuracy.
- Intersection over Union measures the overlap between predicted and ground-truth building pixels divided by their union.
- Accuracy reports the percentage of correctly classified pixels.
C. Importance of a Deeper Encoder-Decoder-Architecture
This section compares encoder-decoder choices using a shared VGG16 encoder and evaluates how architecture affects building-footprint segmentation accuracy. VGG16-based models outperform prior state-of-the-art methods, with SegNet improving IoU by more than 5%.
- Architecture comparison: The experiment compares FCN, SegNet, and FCN + MLP decoder architectures using a VGG16 encoder.The FCN uses up-sampling and convolution, SegNet uses a reversed VGG16 decoder, and FCN + MLP concatenates encoder feature maps before classification.
- Validation results: VGG16-based models outperform state-of-the-art methods on the validation set.All compared networks use the same encoder while differing in decoder type.
- Validation results: SegNet improves the IoU by more than 5% over the referenced state-of-the-art performance.The table caption identifies SegNet as the strongest improvement among the decoder comparisons.
- Architecture comparison: The FCN + MLP approach is reported to achieve the highest accuracy on the dataset.Its decoder upsamples and concatenates all feature maps from the FCN encoder, then uses an MLP to reduce them to class predictions.
- Training setup: Encoder weights are initialized from a VGG16 model pretrained on ImageNet when applicable.Training uses SGD with learning rate 0.01, weight decay 0.0005, and momentum 0.9.
D. Importance of Distance Prediction
This experiment tests whether predicting distance classes to building boundaries improves segmentation compared with a semantic-only baseline. It uses SegNet with the best preceding semantic-segmentation result as the baseline.
- Baseline: The distance-prediction experiment uses SegNet trained with semantic negative log likelihood as its baseline.The baseline achieved 70.14% IoU for the building class in the previous experiment.
- Distance prediction: The baseline network is modified by removing Hseg and attaching Hdist to predict truncated, quantized distance classes.The distance representation is based on a truncated and quantized distance mask.
E. Importance of Uncertainty Based Multi-Task Learning
This section evaluates uncertainty-weighted multi-task learning that combines semantic segmentation with boundary-related distance information. The uncertainty-weighted loss performs best on both reported evaluation metrics, while predictions show sharper edges than FCN outputs.
- Multi-task learning: The multi-task loss combines boundary and semantic information to improve segmentation results.The experiment retrains the network using an uncertainty-based loss after initializing it with weights from the distance-prediction experiment.
- Uncertainty weighting: Uncertainty task weighting achieves the best per-location and overall results on both evaluation metrics.Equal task weighting improves overall accuracy over both single-task losses but remains below uncertainty weighting.
- Contribution: The method incorporates geometric information into deep neural networks through a multi-task loss for building-footprint segmentation.The paper focuses on leveraging segmentation-mask boundary information in high-resolution satellite imagery.
- Qualitative results: The proposed multi-task network produces less blobby building-footprint predictions with sharper edges than an FCN.The figure compares RGB imagery, ground-truth masks, proposed predictions, FCN predictions, and distance-class masks.
- Overall result: The approach outperforms recent methods on the Inria Aerial Image Labeling Dataset by 8.3%.The authors report this result as evidence of the effectiveness of incorporating boundary information through multi-task learning.