Source-linked AI summary
Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network
Lichao Mou, Xiao Xiang Zhu
TL;DR
The paper addresses vehicle instance segmentation, which requires both pixel-wise vehicle labeling and assignment of pixels to individual vehicles in challenging aerial imagery. It proposes an end-to-end residual fully convolutional network that jointly learns vehicle regions and semantic boundaries, and introduces a challenging UAV video dataset. The proposed B-ResFCN achieves the best reported segmentation performance on both evaluated datasets.
Problem
Vehicle instance segmentation must identify vehicle pixels and associate them with physical instances, especially when vehicles are extremely close together.
Method
The paper proposes an end-to-end residual fully convolutional network that jointly learns vehicle semantic segmentation and semantic boundary detection.
Results
B-ResFCN achieves the best segmentation performance on both the Potsdam scene and Busy Parking Lot dataset.
Takeaways & Limitations
The work provides a boundary-aware end-to-end approach and a challenging dataset for benchmarking vehicle instance segmentation.
Abstract
from arXiv · showhide
Object detection and semantic segmentation are two main themes in object retrieval from high-resolution remote sensing images, which have recently achieved remarkable performance by surfing the wave of deep learning and, more notably, convolutional neural networks (CNNs). In this paper, we are interested in a novel, more challenging problem of vehicle instance segmentation, which entails identifying, at a pixel-level, where the vehicles appear as well as associating each pixel with a physical instance of a vehicle. In contrast, vehicle detection and semantic segmentation each only concern one of the two. We propose to tackle this problem with a semantic boundary-aware multi-task learning network. More specifically, we utilize the philosophy of residual learning (ResNet) to construct a fully convolutional network that is capable of harnessing multi-level contextual feature representations learned from different residual blocks. We theoretically analyze and discuss why residual networks can produce better probability maps for pixel-wise segmentation tasks. Then, based on this network architecture, we propose a unified multi-task learning network that can simultaneously learn two complementary tasks, namely, segmenting vehicle regions and detecting semantic boundaries. The latter subproblem is helpful for differentiating closely spaced vehicles, which are usually not correctly separated into instances. Currently, datasets with pixel-wise annotation for vehicle extraction are ISPRS dataset and IEEE GRSS DFC2015 dataset over Zeebrugge, which specializes in semantic segmentation. Therefore, we built a new, more challenging dataset for vehicle instance segmentation, called the Busy Parking Lot UAV Video dataset, and we make our dataset available at http://www.sipeo.bgu.tum.de/download so that it can be used to benchmark future vehicle instance segmentation algorithms.
I. INTRODUCTION
Vehicle extraction from high-resolution remote-sensing imagery remains difficult because appearance, shadows, illumination, and clutter affect performance. Existing approaches span detection and semantic segmentation, but instance segmentation must also separate individual vehicles.
- High-resolution remote-sensing vehicle extraction is challenging under appearance variation, shadows, illumination changes, and cluttered backgrounds.
- Vehicle detection localizes vehicle instances, typically with bounding boxes and confidence scores.
- CNN-based methods became prominent because they automatically extract mid- and high-level abstract features from raw images for pattern recognition.
- Vehicle semantic segmentation labels each pixel as vehicle or another category and can provide more accurate pixel-wise extraction than detection.
C. Is Semantic Segmentation Good Enough for Vehicle Extraction?
Vehicle instance segmentation combines pixel-wise vehicle classification with assignment of pixels to physical vehicle instances. The paper addresses this under challenging aerial scenes by proposing an end-to-end boundary-aware network and a new benchmark dataset.
- Touching vehicles make semantic segmentation insufficient when applications also require the exact number of vehicles.
- Vehicle instance segmentation must classify each pixel and associate it with a physical vehicle instance.
- The Busy Parking Lot UAV Video dataset provides challenging variation in vehicle appearance, shadows, clutter, and extremely close vehicle distances for quantitative comparison.
- The paper proposes an end-to-end semantic boundary-aware multi-task fully convolutional network using residual features.
- The network jointly optimizes vehicle semantic segmentation and semantic boundary detection to differentiate extremely close vehicles.
A. Fully Convolutional Network for Semantic Segmentation
Fully convolutional networks provide an efficient architecture for semantic segmentation and have motivated extensions aimed at improving segmentation performance.
- FCNs were introduced as efficient and effective architectures for semantic segmentation.
- Subsequent FCN extensions modified pooling, used dilated convolutions, or added separate CRF post-processing to improve segmentation.
B. Residual Fully Convolutional Network (ResFCN)
ResFCN combines residual learning with multi-level contextual features and upsampling to produce full-resolution segmentation maps. Its direct information paths preserve low-level detail useful for small vehicles, while the boundary-aware extension jointly separates vehicle regions and touching instances.
- Residual design: ResFCN uses ResNet features and theoretically analyzes why residual architectures can outperform traditional feedforward FCNs.
- Multi-level features: The architecture combines feature representations from 32 × 32, 16 × 16, and 8 × 8 residual layers, then upsamples and sums predictions for full-resolution output.
- Residual design: Residual shortcuts bypass convolutional layers through identity mappings, allowing blocks to learn residuals instead of complete outputs.
- Residual design: ResFCN creates direct forward and backward paths that shuttle low-level visual information across the network, supporting small-object instance segmentation.
- Boundary-aware extension: The boundary-aware extension uses two identical branches to jointly predict vehicle regions and semantic boundaries for differentiating touching vehicles.
C. Semantic Boundary-Aware ResFCN
The proposed network jointly learns vehicle-region segmentation and semantic-boundary detection to separate closely spaced vehicles. It uses boundary cues as complementary information without post-processing fusion.
- Semantic boundaries provide complementary cues for separating vehicles whose regions are ambiguous where objects touch.
- The method uses a unified multi-task network to learn vehicle regions and corresponding semantic boundaries simultaneously.
- The joint objective combines vehicle-region and boundary losses, weighted by λ for the boundary term.
- At test time, the first branch produces the segmentation map, and connected regions define individual vehicle instances.
- The network is trained end to end, with boundary information intended to improve differentiation of clustered vehicles and instance-level segmentation.
A. Datasets
The experiments use the ISPRS Potsdam dataset and a newly proposed Busy Parking Lot UAV Video dataset. The latter targets difficult scenes with closely parked vehicles and provides sparse-in-time, dense-in-space instance annotations.
- The ISPRS Potsdam dataset contains 38 6000×6000 ortho-rectified aerial images at 5 cm spatial resolution, with pixel-wise masks for 24 tiles.
- Five of the 24 annotated Potsdam training tiles were randomly selected as the experimental test set.
- Potsdam imagery was downsampled to 15 cm/pixel, and experiments used only red, green, and blue channels.
- The Busy Parking Lot dataset addresses the lack of a compelling benchmark for quantitatively evaluating vehicle instance segmentation.
- The UAV video has 1920×1080 pixels, approximately 15 cm-per-pixel resolution, 24 frames per second, and 60 seconds of duration.
- Five manually annotated frames provide dense spatial and sparse temporal instance masks for evaluating the long sequence.
B. Training Details
Training uses augmented image patches, Nesterov Adam, early stopping, and small mini-batches. The supplied passages also identify dataset tables, sensitivity analysis, and qualitative evaluation figures used in the experiments.
- Nesterov Adam was selected because it converged faster than standard momentum SGD or Adam for this task.
- Tables report vehicle counts and vehicle-pixel counts for the ISPRS Potsdam and Busy Parking Lot datasets.
- The learning rate was set to 2e−04, with β1 = 0.9, β2 = 0.999, ϵ = 1e−08, and schedule decay 0.004.
- The experiments include a λ sensitivity analysis on ISPRS Potsdam and qualitative results for the Busy Parking Lot dataset.
- Data augmentation expanded 931 unique 256 × 256 training patches to 14,896 samples using horizontal and vertical flips.
C. Qualitative Evaluation
The qualitative evaluation compares FCN backbones on Potsdam and examines generalization to an unseen Busy Parking Lot scene. Residual FCNs improve segmentation quality, but close vehicle spacing and shadows remain difficult.
- Figure 7 compares ground truth with VGG-FCN, Inception-FCN, Xception-FCN, ResFCN, and B-ResFCN using different colors for vehicle instances.
- ResFCN qualitatively improves instance segmentation quality over VGG-FCN, Inception-FCN, and Xception-FCN on ISPRS Potsdam.The comparison fuses outputs from the last three convolutional blocks for each FCN variant.
- ResFCN shows fairly strong generalization to an unseen scene outside the training dataset on the Busy Parking Lot dataset.
- Extremely close vehicles cannot always be separated into individual instances, and shadows further deteriorate the segmentation results.
D. Quantitative Evaluation
The quantitative evaluation uses instance-level detection and segmentation metrics alongside pixel-level baselines. B-ResFCN performs best across Potsdam and Busy Parking Lot, while the latter remains more challenging.
- B-ResFCN obtains a 95.87% pixel-wise F1 score using RGB information on the authors’ Potsdam test set, competitive with the reported CASIA2 score of 96.2%.Pixel-wise OA and F1 evaluate segmentation at pixel level rather than instance level.
- Instance-level evaluation uses F1, precision, recall, and Dice similarity coefficient, with the first three assessing detection and Dice assessing segmentation.
- Busy Parking Lot produces lower detection and segmentation performance than Potsdam because of high density, strong light, shadows, and slightly blurry imagery.
- The instance-level Dice coefficient evaluates individual-object segmentation using matched segmented vehicles and ground-truth objects with area-based coefficients.
- B-ResFCN achieves the best instance-segmentation performance on both Potsdam and Busy Parking Lot.
- 1.16% improvement in instance-level Dice similarity coefficient occurs on Potsdam, while the improvement reaches 7.31% on Busy Parking Lot versus ResFCN.
- Boundary-component networks generally produce better instance segmentations than corresponding networks without boundary components.The comparison supports the usefulness of multi-task learning across CNN variants.
IV. CONCLUSION
The paper proposes a semantic boundary-aware residual fully convolutional network for vehicle instance segmentation and reports that visual and quantitative experiments demonstrate its effectiveness.
- The proposed network addresses vehicle instance segmentation using semantic boundary-aware unified multi-task learning.
- Residual blocks provide multi-level contextual features for producing pixel-wise vehicle likelihood maps.
- Separate branches jointly predict vehicle segmentation masks and semantic boundaries to help distinguish touching vehicles.
- The approach is evaluated on the ISPRS Potsdam dataset and the Busy Parking Lot UAV Video dataset using instance-level metrics.
- Visual and quantitative experimental analyses demonstrate the effectiveness of the proposed approach.