Source-linked AI summary
Recent progress in semantic image segmentation
Xiaolong Liu, Zhidong Deng, Yuhan Yang
TL;DR
Semantic image segmentation is widely used, but its methods, datasets, and evaluation practices span a broad and evolving field. This paper surveys traditional approaches and recent DNN-based progress across eight methodological aspects, reporting improved accuracy and representative benchmark records. It concludes by organizing the field’s advances in accuracy and speed.
Problem
Semantic image segmentation supports applications across computer vision, medicine, and intelligent transportation, motivating systematic coverage of its methods, datasets, and evaluation metrics.
Method
The paper reviews traditional segmentation methods and comprehensively surveys recent DNN-based methods across eight aspects, alongside datasets and evaluation metrics.
Results
85.4% mIoU on PASCAL VOC 2012 and 80.2% on Cityscapes were reported for a single PSPNet with pyramid pooling.
Takeaways & Limitations
The review organizes progress toward semantic segmentation that is more accurate, faster, or both.
Abstract
from arXiv · showhide
Semantic image segmentation, which becomes one of the key applications in image processing and computer vision domain, has been used in multiple domains such as medical area and intelligent transportation. Lots of benchmark datasets are released for researchers to verify their algorithms. Semantic segmentation has been studied for many years. Since the emergence of Deep Neural Network (DNN), segmentation has made a tremendous progress. In this paper, we divide semantic image segmentation methods into two categories: traditional and recent DNN method. Firstly, we briefly summarize the traditional method as well as datasets released for segmentation, then we comprehensively investigate recent methods based on DNN which are described in the eight aspects: fully convolutional network, upsample ways, FCN joint with CRF methods, dilated convolution approaches, progresses in backbone network, pyramid methods, Multi-level feature and multi-stage method, supervised, weakly-supervised and unsupervised methods. Finally, a conclusion in this area is drawn.
1 Introduction
Semantic image segmentation classifies image pixels by object category and supports applications including medicine and intelligent transportation. This paper briefly reviews traditional methods while organizing recent DNN progress and surveying datasets and evaluation metrics.
- Semantic image segmentation classifies each pixel into its category, also called pixel-level classification.
- Segmentation supports applications including road-sign detection, medical imaging, instrument tracking, and land-use classification.
- The paper distinguishes traditional methods from methods developed after DNN adoption.
- The review focuses on recent DNN-based progress and surveys segmentation datasets and evaluation metrics.
2 Datasets and evaluation metrics
The paper reviews benchmark datasets for semantic segmentation and the metrics used to evaluate performance. These resources span general, scene-parsing, and autonomous-driving settings.
- Datasets: Common segmentation datasets include PASCAL VOC, MS COCO, ADE20K, Cityscapes, and KITTI.
- Datasets: PASCAL VOC 2012 contains 20 classes, 11,530 train/validation images, 27,450 ROI-annotated objects, and 6929 segmentations.
- Datasets: ADE20K contains 150 object and stuff classes with both object-segmentation and parts-segmentation masks.
- Datasets: Cityscapes focuses on urban street scenes and includes 30 classes in 5000 finely annotated images collected from 50 cities.
- Datasets: KITTI is an autonomous-driving dataset covering road detection, stereo reconstruction, optical flow, visual odometry, and 3D object and tracking tasks.
- Evaluation metrics: Regular evaluation metrics include pixel accuracy, mean accuracy, mean intersection-over-union, and frequency-weighted intersection-over-union.
3 Traditional methods
Before DNN methods, semantic segmentation relied on hand-designed features, clustering, energy models, edge and region-based approaches, and classifiers. The paper also describes SVMs and graphical models used in traditional segmentation.
- Features and approaches: Traditional segmentation used features including pixel color, HOG, SIFT, LBP, SURF, corner detectors, bag-of-visual-words, and textons.
- Unsupervised methods: K-means provides an unsupervised approach that assigns data points to the nearest of k predefined centroids and iteratively updates them.
- Segmentation formulations: Other traditional branches formulate segmentation as an energy model or use edge detection and region-growing methods.
- Classifiers: SVMs classify feature vectors with binary labels using a weight vector and bias, formulated as an optimization problem.
- Classifiers: Slack variables address linearly inseparable problems, while kernel methods map features into higher dimensions.
- Graphical models: MRFs model P(y, x), whereas CRFs model conditional probabilities P(Y|X) as structured extensions of logistic regression.
4 Recent DNN in segmentation
Artificial neural networks build on weighted artificial neurons, while CNNs use shared-weight, spatially restricted connections for image tasks. FCN adapts CNNs to dense segmentation through fully convolutional layers, interpolation, skips, and end-to-end training.
- Artificial neurons sum weighted inputs and apply an activation function to produce an output value.
- CNNs use shared-weight architectures with spatially restricted connectivity and translation-invariant characteristics.
- Fully convolutional network: FCN replaces fully connected layers with fully convolutional layers and uses interpolation to restore output dimensions to the input size.
- Fully convolutional network: FCN adds skip connections, accepts arbitrary-sized inputs, produces correspondingly sized outputs, and supports end-to-end training with efficient inference and learning.
- Fully convolutional network: 20% relative improvement to 62.2% mean IU on PASCAL VOC was reported for VGG-based FCN, with inference taking less than one fifth of a second per typical image.
4.2 Up-sample method: interpolation versus deconvolution
Semantic segmentation uses up-sampling to recover feature-map resolution. Interpolation is computationally efficient, while deconvolution and un-pooling can identify pixel-wise labels and predict masks.
- Deconvolution: Noh et al.’s deconvolution network combines deconvolution and un-pooling layers to identify pixel-wise class labels and predict segmentation masks.
- Deconvolution: Unlike FCN, Noh et al.’s network processes individual object proposals and combines the resulting instance-wise segmentations for final semantic segmentation.
- Interpolation: Bi-linear interpolation is broadly used for up-sampling because of its computational efficiency and good recovery of the original image.
- Deconvolution: Deconvolution reverses convolution and can recover feature-map size to the original input dimensions.
4.3 FCN joint with CRF and other traditional methods
FCN-based segmentation can be combined with CRFs to improve localization, while domain-transform filtering offers a faster alternative to dense CRF inference. Related approaches also use contextual graphical models and dilated convolutions.
- FCN joint with CRF: DeepLab combines a fully connected CRF with the final DCNN layer because final-layer responses are insufficiently localized for accurate object segmentation.
- FCN joint with CRF: 71.6% IOU accuracy was reported on the PASCAL VOC-2012 test set for the DCNN–CRF method.
- FCN joint with CRF: Domain-transform filtering was combined with DCNN because dense CRF inference is computationally expensive, and it produced comparable results while capturing object boundaries accurately.
- Other graphical-model methods: MRF-based segmentation can combine label-context mixtures and high-order relations as additional information.
- Dilated convolution: Dilated convolution aggregates multi-scale contextual information using dilation parameters that expand the receptive field without the passage describing a standard down-sampling operation.
4.5 Progress in backbone network
Semantic-segmentation backbones originate largely from image-classification networks, progressing from VGG, AlexNet, and GoogLeNet toward residual and newer multi-branch architectures. Pyramid methods combine pooled features across scales.
- Backbone networks: The backbone is the network’s main structure, and segmentation backbones are derived from image-classification architectures.
- Backbone networks: FCN adopts VGG-16, while related work also considers AlexNet and GoogLeNet as classification backbones.
- Backbone networks: ResNet enabled a new semantic-segmentation breakthrough, prompting evaluations of feature-map resolution, layer count, and field-of-view size.
- Backbone networks: ResNeXt was introduced as a next generation of ResNet, while Inception variants were subsequently adopted in semantic-segmentation work.
4.6 Pyramid method in segmentation
Pyramid strategies incorporate information across image, feature, sampling, or pooling scales to improve semantic segmentation’s handling of multi-scale context. The review describes image pyramids, atrous spatial pyramid pooling, and pyramid pooling as representative approaches.
- Image pyramid: Image pyramids successively downsample images, with Gaussian pyramids supporting downsampling and Laplacian pyramids supporting reconstruction to higher resolution.
- Image pyramid: Multi-scale inputs and sliding pyramid pooling capture patch-background context, while Deeplab merges features from resized inputs for pixel-wise classification.
- Atrous spatial pyramid pooling: ASPP uses filters at multiple sampling rates to probe effective fields of view and capture image context at multiple scales.
- Pooling pyramid: 85.4% mIoU on PASCAL VOC 2012 and 80.2% on Cityscapes were reported for a single PSPNet with pyramid pooling.
- Pooling pyramid: Pyramid pooling aggregates context from different region sizes, upsamples outputs to the original size, and concatenates them into a mixed feature representation.
4.7 Multi-level feature and multi-stage method
Multi-level and multi-stage methods combine information from different network depths or route computation through stages. These designs address the tension between coarse semantic information, precise localization, and efficient processing.
- Multi-level feature: Hypercolumns combine activations from CNN units across layers because final-layer features are coarse while earlier layers localize precisely but lack semantics.
- Multi-level feature: FCN uses skip connections as a multi-level strategy to improve segmentation evidence.
- Multi-stage method: Unlike conventional cascades of independent models, deep layer cascade treats one deep model as several sub-models.
- Multi-stage method: Deep layer cascade classifies easy regions in shallow stages and reserves deeper stages for a few hard regions.
- Multi-stage method: Deep layer cascade is reported to improve segmentation performance while accelerating both training and testing of the deep network.
4.8 Supervised, weakly-supervised and unsupervised methods
Although most semantic segmentation progress has used supervised learning, the review also identifies semi-supervised and non-supervised research.
- Most semantic segmentation progress has been conducted under supervised learning, alongside research on semi-supervised and non-supervised methods.
5 Conclusion
The paper concludes by organizing recent semantic segmentation progress, especially DCNN-based work, across major architectural and learning dimensions. It presents this review as a resource for researchers working in the area.
- The review covers FCNs, upsampling, CRF combinations, dilated convolutions, backbone networks, pyramid methods, multi-level and multi-stage methods, and learning-supervision settings.
- Recent methods aim to make semantic image segmentation more accurate, faster, or both.
- The paper is distributed under a Creative Commons Attribution 4.0 International License.