Source-linked AI summary
TernausNetV2: Fully Convolutional Network for Instance Segmentation
Vladimir I. Iglovikov, Selim Seferbekov, Alexander V. Buslaev, Alexey Shvets
TL;DR
Instance segmentation methods are often complex, while semantic segmentation cannot reliably separate touching objects of the same class. TernausNetV2 extends a fully convolutional encoder-decoder network with multispectral transfer learning and an additional border channel, achieving a public leaderboard score of 0.74 on the DeepGlobe building-detection challenge.
Problem
Semantic segmentation can merge closely packed buildings into one connected component, motivating a simpler approach to instance-level extraction from satellite imagery.
Method
TernausNetV2 uses a U-Net-style fully convolutional encoder-decoder with a pretrained WideResNet-38 encoder, 11-channel multispectral input, and a second output channel for touching building borders.
Results
0.74 public leaderboard score was achieved on the DeepGlobe-CVPR 2018 building-detection sub-challenge, reported as state-of-the-art.
Takeaways & Limitations
The approach adds instance-segmentation functionality to a fully convolutional semantic-segmentation architecture while enabling transfer learning from RGB to multispectral inputs.
Abstract
from arXiv · showhide
The most common approaches to instance segmentation are complex and use two-stage networks with object proposals, conditional random-fields, template matching or recurrent neural networks. In this work we present TernausNetV2 - a simple fully convolutional network that allows extracting objects from a high-resolution satellite imagery on an instance level. The network has popular encoder-decoder type of architecture with skip connections but has a few essential modifications that allows using for semantic as well as for instance segmentation tasks. This approach is universal and allows to extend any network that has been successfully applied for semantic segmentation to perform instance segmentation task. In addition, we generalize network encoder that was pre-trained for RGB images to use additional input channels. It makes possible to use transfer learning from visual to a wider spectral range. For DeepGlobe-CVPR 2018 building detection sub-challenge, based on public leaderboard score, our approach shows superior performance in comparison to other methods. The source code corresponding pre-trained weights are publicly available at https://github.com/ternaus/TernausNetV2
1. Introduction
High-resolution satellite imagery can support building extraction for broad geographic coverage, but separating touching instances remains difficult for semantic segmentation. TernausNetV2 addresses this by extending a fully convolutional encoder-decoder with multispectral inputs and a touching-border output.
- Motivation: Automatic building extraction could support urban planning, population monitoring, and topographic databases covering nearly any region.Satellite imagery is widely available and includes optical, infrared, and extra channels, although its resolution is lower than aerial imagery.
- Related work: Fully convolutional networks produce dense pixel-wise segmentation maps in a single forward pass, while skip connections improve localization.Their encoder extracts high-level features and the decoder upsamples them to the input resolution.
- Problem: Semantic segmentation cannot reliably separate closely packed objects because imperfect boundaries can merge instances into one connected component.Even perfect predictions may merge objects when no distance exists between them.
- Contribution: TernausNetV2 extends a TernausNet encoder-decoder with an ImageNet-pretrained WideResNet-38 encoder and 11-channel RGB-plus-multispectral input.This generalizes transfer learning from RGB images to multispectral data.
- Contribution: An additional output channel predicts touching or nearby object areas, which post-processing uses to partition binary masks into separate instances.This provides a simpler alternative to the stacked-network approach described for instance separation.
2. Dataset
The dataset consists of high-resolution multispectral satellite imagery from four cities, with polygon annotations for building instances and imperfect labels. Predictions are evaluated through the DeepGlobe website.
- Dataset: 30 cm WorldView-3 satellite imagery provides 650x650-pixel images covering 195x195 m2 of the Earth’s surface.The training data originate from the SpaceNet dataset.
- Dataset: Each region includes RGB, panchromatic, and 8-channel low-resolution multispectral imagery.The combined data provide the multispectral inputs used by the model.
- Dataset: The dataset covers Vegas, Paris, Shanghai, and Khartoum, with city-specific train and test image counts.Train counts are 3831, 1148, 4582, and 1012; test counts are 1282, 381, 1528, and 336, respectively.
- Annotations: Training images have polygon lists describing building instances, but annotations are imperfect, especially in high-density areas.The paper attributes this imperfection to the cost of mask annotation.
- Evaluation: Figure 1 presents the RGB input, predicted binary mask and touching borders, and building instances after watershed transformation.The sequence illustrates the prediction and instance-separation stages.
3. Model
TernausNetV2 uses a U-Net-style encoder-decoder with skip connections, an ImageNet-pretrained WideResNet-38 encoder, and 11-channel input. Its two-channel output represents building footprints and touching borders.
- Architecture: The model uses an encoder-decoder architecture with skip connections between corresponding encoder and decoder blocks.The contracting path captures context, while the expanding path upsamples feature maps for pixel-level localization.
- Encoder: WideResNet-38 with in-place activated batch normalization replaces the original U-Net encoder and was pretrained on ImageNet.In-place activated batch normalization reduces memory use, enabling larger batches and input images.
- Output: The decoder ends with a 1x1 convolution producing two channels: a combined building mask and touching-instance borders.The touching-border channel supports subsequent separation of adjacent building instances.
- Input: The network accepts RGB plus eight multispectral channels by replacing the first convolution with an 11-channel layer.Original weights are copied to the RGB channels, while the additional channels are initialized with zeros.
4. Training
Training combines binary cross entropy with a differentiable soft Jaccard loss for two independent mask outputs, while staged training adapts RGB-pretrained weights to multispectral inputs. Inference separates instances with watershed post-processing.
- Multispectral transfer: The RGB-to-multispectral training schedule freezes the encoder for the first epoch, then unfreezes all layers for end-to-end training.Zero-initialized extra-channel weights initially leave the RGB pathway active before multispectral weights are learned.
- Objective: The two output channels independently predict building footprints and touching borders.Both outputs are binary masks trained jointly.
- Objective: The loss combines binary cross entropy with soft Jaccard loss so training optimizes pixel classification and mask intersection over union.The experiments use α = 0.7 for the combined loss.
- Augmentation: Data augmentation uses random rescaling, full-range rotations, 384x384 crops, and color transformations.The stated spatial scale range is 0.5 to 1.5 of the input image.
- Inference: Inference pads images to 672x672, predicts the two masks, crops the padding, and applies watershed transformation to separate instances.Touching borders are subtracted from the binary mask to form seeds for watershed processing.
5. Conclusions
The model combines fully convolutional semantic features with instance-segmentation functionality for satellite-image building detection, achieving a public leaderboard score of 0.74.
- The model uses a fully convolutional neural network traditionally applied to semantic segmentation, augmented with an additional output for instance segmentation.
- WideResnet-38 pretrained on ImageNet serves as the encoder, providing semantic features while remaining memory efficient.
- 0.74 public leaderboard score was achieved for satellite imagery building detection, described as state of the art.