PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images

Irem Ulku, Erdem Akagunduz

arXiv:1912.10230v5cs.CV

TL;DR

2D semantic segmentation must recover class labels at original pixel resolution while addressing localisation and broader architectural challenges. This survey synthesises datasets, evaluation measures, and a chronological taxonomy of deep learning methods across three eras. It identifies fine-grained localisation and global context as central concerns and highlights public challenges as an influence on field development.

  • Problem

    Semantic segmentation must assign and localise class labels at every original image pixel, while post-FCN architectures still face localisation, global-context, and multiscale limitations.

  • Method

    The survey reviews 2D deep learning methods, public image sets, leaderboards, evaluation measures, and their technical evolution across three chronological eras.

  • Results

    The review identifies fine-grained pixel-label localisation as the definitive challenge and observes that global context determines method performance.

  • Takeaways & Limitations

    Public challenges and leaderboards have accelerated development, supporting continued creation of image sets and challenges for specific segmentation subjects.

  • Takeaways & Limitations

    The field remains constrained by time-consuming pixel-level annotations, limited generalisation to new domains and classes, and demands for real-time accuracy.

Abstract

from arXiv · show

Semantic segmentation is the pixel-wise labelling of an image. Since the problem is defined at the pixel level, determining image class labels only is not acceptable, but localising them at the original image pixel resolution is necessary. Boosted by the extraordinary ability of convolutional neural networks (CNN) in creating semantic, high level and hierarchical image features; several deep learning-based 2D semantic segmentation approaches have been proposed within the last decade. In this survey, we mainly focus on the recent scientific developments in semantic segmentation, specifically on deep learning-based methods using 2D images. We started with an analysis of the public image sets and leaderboards for 2D semantic segmentation, with an overview of the techniques employed in performance evaluation. In examining the evolution of the field, we chronologically categorised the approaches into three main periods, namely pre-and early deep learning era, the fully convolutional era, and the post-FCN era. We technically analysed the solutions put forward in terms of solving the fundamental problems of the field, such as fine-grained localisation and scale invariance. Before drawing our conclusions, we present a table of methods from all mentioned eras, with a summary of each approach that explains their contribution to the field. We conclude the survey by discussing the current challenges of the field and to what extent they have been solved.

1 Introduction

Semantic segmentation assigns labels at each original image pixel, and CNNs have driven recent progress toward practical solutions. This survey focuses on deep learning methods for 2D visible imagery, reviewing datasets, evaluation, and the field’s chronological evolution.

  • Problem definition: Semantic segmentation requires assigning class labels to, and localising them at, each original image pixel.The relevant classes depend on the application, including road scenes and medical imagery.
  • Motivation: CNNs’ success in producing semantic, hierarchical image features has accelerated semantic segmentation research.The paper links this progress to the emergence of practical solutions after a long period of study.
  • Scope: The survey restricts its detailed analysis to deep learning-based semantic segmentation methods using 2D visible imagery.The authors exclude other modalities because the literature is too broad to analyse comprehensively across all of them.
  • Scope: It reviews public image sets, leaderboards, and performance measures used to evaluate 2D semantic segmentation.The survey considers both classification and localisation aspects of segmentation performance.
  • Organisation: The paper proposes a taxonomy and chronological analysis spanning pre- and early deep learning, fully convolutional, and post-FCN methods.Its organisation follows these periods, with the fully convolutional era treated as a milestone.
  • Related surveys: Earlier surveys are described as either problem-specific, broad but technically coarse, or insufficiently focused on deep learning and chronological evolution.The authors position their taxonomy as addressing these limitations for 2D semantic segmentation.

2 Image Sets, Challenges and Performance Evaluation

This section surveys the 2D image sets, evaluation criteria, and computational-efficiency considerations used in semantic segmentation. It emphasizes dataset scale and balance, pixel-level accuracy and localisation, and practical benchmarking constraints.

  • Image Sets: Deep learning performance depends strongly on training-data quality and depth, motivating examination of large-scale 2D semantic-segmentation image sets.The survey distinguishes general-purpose datasets from urban street datasets used for driverless-car systems.
  • Image Sets: PASCAL VOC and COCO are prominent general-purpose benchmarks, while ADE20K provides scene-centric images with object and part annotations.PASCAL VOC has 20 foreground classes and one background class; COCO contains 200K labelled images, 1.5 million instances, and 80 categories.
  • Image Sets: Cityscapes is a large-scale urban-street benchmark with high-resolution images from 50 cities and fine or coarse annotation quality levels.It provides fine annotations for 5,000 images and coarse annotations for 20,000 images.
  • Image Sets: Datasets with insufficient scale or strong class imbalance may not directly yield high segmentation accuracy, so the survey focuses on large-scale sets.The authors leave techniques for handling small-scale or imbalanced datasets beyond the survey’s scope.
  • Performance Evaluation: Segmentation evaluation separates classification accuracy from localisation, using metrics such as IoU and precision-recall measures alongside computational complexity.IoU accounts for false positives and false negatives but does not measure boundary accuracy or distinguish their relative significance.
  • Computational Complexity: Computational efficiency is assessed inductively because benchmarking every state-of-the-art method with optimal hyperparameters is impractical.The survey groups methods into four computational-efficiency levels and relates the categorisation to architectural design.

3 Before Fully Convolutional Networks

Before FCNs, semantic segmentation used graphical models, layered detectors, and early CNN adaptations, but these approaches faced limitations in semantic representation, efficiency, and segmentation quality.

  • Pre-FCN methods included graphical models such as MRFs and CRFs for pixel-level scene labelling.
  • Layered models combined separately pretrained object detectors to extract semantic information.Their effectiveness was limited by detector failures and finite hand-selected class banks.
  • CRFs later remained useful as post-processing refinement layers because they model relationships between adjacent pixel predictions.
  • Early deep approaches converted classification networks such as AlexNet and VGG into segmentation networks by fine-tuning fully connected layers.
  • These early CNN approaches suffered from overfitting, time-consuming fully connected layers, and insufficient network depth for abstract semantic features.
  • Because early segmentation results were unsatisfactory, researchers added refinement processes or replaced fully connected decision layers with recurrent or separately computed segmentation structures.

4 Fully Convolutional Networks for Semantic Segmentation

Fully Convolutional Networks removed fully connected layers from classification CNNs and enabled efficient dense prediction at arbitrary image resolutions through upsampling and skip connections.

  • FCNs adapted classification networks such as AlexNet, VGG, and GoogLeNet into fully convolutional segmentation networks using transfer learning and fine-tuning.The commonly used FCN-32s, FCN-16s, and FCN-8s architectures were transfer-learned from VGG.
  • Removing fully connected layers made per-image inference considerably faster because convolutional layers contain far fewer weights.
  • Deconvolutional layers enabled FCNs to upsample coarse convolutional outputs into dense segmentation maps at any desired image resolution.
  • FCNs introduced skip architectures that connect nonadjacent layers by summing or concatenating their outputs.These connections allow information to flow across architectural stages where pooling or dropout might otherwise discard it.
  • Skip connections subsequently developed into encoder-decoder structures for semantic segmentation.

5 Post-FCN Approaches

Post-FCN research largely followed the fully convolutional paradigm while targeting its main weaknesses: localisation loss, limited global context, and absent multiscale processing.

  • Most subsequent semantic segmentation approaches followed the FCN design after its introduction.
  • The principal FCN drawbacks were inefficient loss of label localisation, inability to process global context, and lack of multiscale processing.
  • Post-FCN techniques were categorised according to how they addressed localisation, context, multiscale processing, scale invariance, and joint object detection.

5.1 Techniques for Fine-grained Localisation

Fine-grained localisation methods combine hierarchical semantics with higher-resolution information, broader receptive fields, contextual refinement, feature fusion, or recurrent processing, often with explicit computational trade-offs.

  • 5.1.1 Encoder-Decoder Architecture: Encoder-decoder architectures use an encoder to reduce spatial dimensions and a decoder to recover object details through same-level skip connections.This combines abstract hierarchical features with fine localisation, although information can decay while passing through intermediate layers.
  • 5.1.2 Spatial Pyramid Pooling: Spatial Pyramid Pooling creates multilevel feature relations and can accept CNN inputs of different sizes, but it is not inherently scale-invariant.Scale invariance requires training images at different scales, with or without SPP.
  • 5.1.3 Feature Fusion: Feature-fusion methods combine representations from different sources to provide refinement or global context, but their hybrid structures are relatively difficult to train.
  • 5.1.4 Dilated Convolution: Dilated convolutions enlarge receptive fields more rapidly than contiguous convolutions while preserving feature-map resolution without pooling or subsampling.Their drawback is higher GPU storage and computation because feature maps do not shrink through the hierarchy.
  • 5.1.5 Graphical Models: CRF refinement layers use low-level interactions between neighbouring pixels or feature maps to build context knowledge and finer label localisation.
  • 5.1.6 Recurrent Approaches: Recurrent approaches use sequential, graph-structured, or multiscale interactions to improve pixel-level segmentation in still images and image sequences.Examples include Conv-LSTM, Graph LSTM, Graph-RNN, LSTM chains, and CNN-RNN hybrids.

5.2 Scale-Invariance

Scale invariance means processing inputs independently of object-to-scene scale or image resolution, a property distinct from merely using multiscale information. Training on multiple scales can improve segmentation but does not inherently provide scale invariance.

  • Scale invariance is the ability to process an input independently of relative object scale or image resolution.
  • Multiscale information can improve pixel-wise segmentation while leaving a method dependent on scale or resolution.
  • Training with multiple image scales can produce scale-robust behavior, but these methods lack inherent scale invariance.

5.3 Object Detection-based Methods

Object detection-based methods first locate objects and then segment them, using architectures designed for variable numbers of objects. The field has progressed from region proposals and two-stage systems toward one-stage detectors and global-area feature assembly.

  • Object detection architectures must handle images containing an unknown number of objects, unlike conventional fixed-output classification or segmentation models.
  • RCNN introduced region proposals followed by separate CNN-based object detection for each image region.
  • Mask-RCNN extended the detection pipeline to instance segmentation by detecting objects and producing masks for them.
  • YOLO and SSD replaced cumbersome RCNN-style structures with single-shot convolutional prediction of bounding boxes and class probabilities.
  • Recent approaches incorporate segmentation into one-stage detectors or assemble masks from intermediate fully convolutional feature maps.
  • Bounding-box and image-level supervision are being used to reduce reliance on pixel-wise annotations in weakly supervised instance segmentation.

5.4 Evolution of Methods

The survey organizes representative methods and their computational efficiency to clarify the field’s evolution. Across eight years, segmentation results improved from pioneering FCN architectures to more advanced DeepLab and CRF-as-RNN systems, while refinement modules increasingly gave way to faster designs.

  • Table 1 summarizes representative methods, leaderboard positions, core ideas, and categorical computational efficiency.
  • The table covers 34 methods across an eight-year period, from early deep learning approaches to recent state-of-the-art techniques.
  • Figure 6 depicts progressively improving results from FCN architectures to DeepLab and CRF-as-RNN methods on a PASCAL VOC example.
  • Graphical-model refinement modules are being abandoned because of their slow nature, and recent studies increasingly explore object detection or NAS-based approaches.
  • Recent studies show no significant performance leap, encouraging experimentation with object detection and Neural Architecture Search rather than only hand-crafted architectures.

6 Future Scope and Potential Research Directions

Future research targets the unresolved demands of semantic segmentation: expensive pixel-level annotation, limited generalization, and real-time accuracy. Proposed directions include alternative supervision, efficient architectures, multiscale processing, refinement mechanisms, and architectural search.

  • Future challenges: Pixel-level annotation remains time-consuming, while generalization to new domains and classes and real-time high accuracy remain open challenges.
  • Alternative supervision: Unsupervised, semi-supervised, and weakly supervised methods seek alternatives to pixel-level annotation.
  • Architecture design: Multiscale architectures combine features across resolutions, while encoder-decoder designs and skip connections support localized pixel predictions.
  • Refinement and context: CRF, recurrent, attention, and refinement modules can improve contextual or boundary modeling but often limit computational efficiency.
  • Real-time segmentation: Fully convolutional designs without fully connected layers can provide relatively fast computation, including architectures reporting near real-time performance.
  • Computational efficiency: Efficient segmentation research uses compact or shallow models and low-resolution inputs to reduce the computational burden of large-scale processing.

7 Conclusions

The survey reviews the ten-year evolution of deep learning-based 2D semantic segmentation through public datasets, evaluation methods, chronological analysis, and technical comparison. It identifies fine-grained pixel localisation and the local-global semantics gap as continuing challenges, while highlighting public challenges as drivers of development.

  • Survey scope: The survey analyses public image sets, leaderboards, evaluation techniques, and deep learning-based 2D semantic segmentation methods across three chronological eras.The eras are the pre- and early deep learning era, the fully convolutional era, and the post-FCN era.
  • Technical challenges: Fine-grained pixel-label localisation remains the definitive challenge, motivating methods that connect local information with global context.The reviewed approaches include graphical models, context-aggregating networks, recurrent methods, and attention-based modules.
  • Field development: Public challenges strongly affect the field by encouraging academic and industrial groups to compete on leaderboards, accelerating development.The survey recommends similar datasets and challenges for more specific areas, including 2D medical images.
  • Future directions: The survey is positioned as an update intended to measure the field’s progress and indicate possible future directions amid rapid, continuing development.The authors describe an ongoing need for updated surveys of semantic segmentation.
Loading 1912.10230v5…