PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey

Farhana Sultana, Abu Sufian, Paramartha Dutta

arXiv:2001.04074v3cs.CV

TL;DR

Image segmentation is widely needed but challenging because it must preserve low-level spatial information. This survey reviews CNN-based semantic and instance segmentation, compares architectures and training details, and surveys panoptic models. It concludes that model architecture, optimization, hyper-parameters, preprocessing, and loss functions all contribute to state-of-the-art success.

  • Problem

    Image segmentation is an indispensable but challenging computer-vision task because semantic labeling requires spatially variant features that preserve pixel context.

  • Method

    The paper surveys CNN-based semantic and instance segmentation, explores state-of-the-art architectures, compares their training details and hyper-parameter tuning, and presents tabular model features.

  • Results

    The survey covers the evolution of CNN-based semantic and instance segmentation models, compares their architectures and training aspects, and provides a glimpse of state-of-the-art panoptic models.

  • Takeaways & Limitations

    Understanding successful segmentation models requires considering network architecture together with optimization, hyper-parameters, preprocessing, and loss functions.

  • Takeaways & Limitations

    Some earlier instance-segmentation approaches were not end-to-end trainable or required substantial post-processing, while bounding-box proposals incurred costly alignment and separate mask generation.

Abstract

from arXiv · show

From the autonomous car driving to medical diagnosis, the requirement of the task of image segmentation is everywhere. Segmentation of an image is one of the indispensable tasks in computer vision. This task is comparatively complicated than other vision tasks as it needs low-level spatial information. Basically, image segmentation can be of two types: semantic segmentation and instance segmentation. The combined version of these two basic tasks is known as panoptic segmentation. In the recent era, the success of deep convolutional neural networks (CNN) has influenced the field of segmentation greatly and gave us various successful models to date. In this survey, we are going to take a glance at the evolution of both semantic and instance segmentation work based on CNN. We have also specified comparative architectural details of some state-of-the-art models and discuss their training details to present a lucid understanding of hyper-parameter tuning of those models. We have also drawn a comparison among the performance of those models on different datasets. Lastly, we have given a glimpse of some state-of-the-art panoptic segmentation models.

1. Introduction

The paper surveys CNN-based image segmentation because the task is widely needed and requires pixel-level spatial information. It organizes notable models, training details, dataset performance, and panoptic segmentation developments.

  • Image segmentation supports applications from autonomous driving to medical diagnosis, motivating a survey of recent CNN-based models.
  • The survey covers semantic and instance segmentation, with a brief overview of panoptic segmentation.
  • It provides comparative architectural details of popular state-of-the-art segmentation models.
  • Training details are compared to clarify hyper-parameter tuning across the surveyed models.
  • Performance metrics are compared across different datasets.

2. Background Details

The background defines image segmentation as pixel-based partitioning that preserves spatial information and distinguishes semantic, instance, and panoptic forms. It also situates CNNs within segmentation and notes that video object segmentation is outside the paper’s detailed scope.

  • Image segmentation partitions an image into regions according to pixel properties such as color, intensity, and texture.
  • Semantic segmentation assigns class labels to pixels, whereas instance segmentation separates individual object instances.
  • Panoptic segmentation unifies semantic and instance segmentation.
  • CNN-based methods are surveyed for image segmentation, alongside examples of CNN use in video object segmentation.
  • Video object segmentation is not covered in detail because of the article’s scope and size.

3. Semantic Segmentation

Semantic segmentation associates each image pixel with a class label. The paper surveys the evolution of CNN-based semantic segmentation models and includes a process view of the task.

  • Semantic segmentation associates each pixel of an image with a class label.
  • The section surveys the evolution of successful CNN-based semantic segmentation models after AlexNet’s success in 2012.
  • Figure 3 presents the process of semantic segmentation as a black-box view.

3.1. Evolution of CNN based Semantic Segmentation Models

CNN-based semantic segmentation evolved through diverse architectures that preserve spatial information, incorporate context, and target specialized settings such as real-time or domain-adaptive operation.

  • FCN-based models removed fully connected layers from traditional CNNs to retain image spatial information.
  • DeepLab progressed from atrous convolution and CRF integration to ASPP and cascaded deep ASPP for incorporating multiple contexts.
  • DeconvNet, U-Net, and SegNet use related encoder-decoder or mirrored architectural intuitions with model-specific modifications.
  • PSPNet added a Pyramid Pooling Module to incorporate global contextual information for better segmentation.
  • ENet and ICNet were used as real-time semantic segmentation models for autonomous vehicles.
  • Researchers also combined CNNs with adversarial networks, attention modules, recurrent criss-cross attention, and global attention for semantic segmentation.

3.2. Some popular state-of-the-art semantic segmentation models

This section surveys prominent CNN-based semantic segmentation architectures, organized by their use of fully convolutional, dilated/atrous, top-down/bottom-up, and global-context designs. These models progressively address dense prediction, spatial resolution, fine-grained detail, and contextual information.

  • Based on Fully Convolutional Network: FCN converts pretrained classifiers into dense predictors by replacing fully connected layers with 1×1 convolutions and adding skip connections for fine-grained segmentation.FCN-VGG16 achieved the highest accuracy among the compared FCN variants on the PASCAL VOC 2011 validation dataset, while bilinear interpolation upsampled coarse outputs 32×.
  • Based on Fully Convolutional Network: FCN’s reliance on local information loses global semantic context, making semantic segmentation ambiguous.The survey identifies contextual information from the whole image as useful for reducing this ambiguity.
  • Based on Dialtation/Atrous convolution: DialatedNet uses dilated convolutions to accumulate multi-scale contextual information and expand receptive fields without losing spatial information.Its receptive fields expand from 3×3 to 7×7 and 15×15 in the illustrated architectures.
  • Based on Dialtation/Atrous convolution: DeepLab combines atrous computation with a fully connected pairwise CRF to address downsampling and spatial-invariance difficulties in semantic segmentation.The atrous algorithm supports efficient dense computation, while the CRF captures fine details.
  • Based on Top-down/Bottom-up approach: FCN’s high final upsampling rate produces coarse outputs, whereas gradual upsampling improves accuracy but does not incorporate global contextual information.The survey presents this as a trade-off between fine-grained segmentation and contextual coverage.

3.3. Discussion

The discussion emphasizes that semantic segmentation performance depends on both architectural design and optimization choices. It summarizes model evolution, training configurations, feature and loss choices, and chronological comparisons using mAP at IoU thresholds.

  • Architectural evolution: FCN established a path for semantic segmentation, while later models added contextual features, atrous processing, encoder-decoder designs, and related architectural refinements.DeepLab variants use atrous methods, while SegNet, DeconvNet, and U-Net use hierarchically opposed pathways.
  • Optimization and training: Most surveyed models use stochastic gradient descent, but mini-batch sizes differ and depend partly on the number of GPUs used for training.The comparison indicates that optimization details contribute to model success beyond architecture alone.
  • Preprocessing and loss functions: Pretrained base networks, data augmentation, and loss functions are important design choices, with cross-entropy loss used in most surveyed models.Base networks evolve alongside classification models, while more complex designs may use different loss functions to obtain higher accuracy.
  • Comparative evaluation: The survey lists important features of each state-of-the-art model and compares their semantic-segmentation performance across datasets.The comparative metric is mean average precision (mAP) at Intersection over Union (IoU) thresholds, presented chronologically.

4. Instance Segmentation

This section introduces CNN-based instance segmentation as a task that independently masks each object instance, extending object detection beyond labeled bounding boxes. It surveys the evolution and selected state-of-the-art models for this task.

  • Task definition: Instance segmentation independently produces a mask for each object instance, unlike semantic segmentation, which does not distinguish separate instances.The task is closely related to object detection, where bounding boxes identify and classify object instances.
  • Research evolution: CNN-based instance segmentation research emerged alongside semantic segmentation research to improve segmentation accuracy.The survey explores the evolution and selected state-of-the-art CNN-based models for instance segmentation.

4.1. Evolution of CNN based Instance Segmentation Models:

CNN-based instance segmentation evolved from proposal-driven pipelines toward shared-feature, end-to-end, position-sensitive, and dense segmentation approaches. Key developments addressed proposal cost, mask coarseness, instance separation, and missing sliding-window formulations.

  • Detector-based approaches: Instance segmentation models commonly adapted CNN-based object detectors to predict segmentation masks, often improving accuracy and reducing test time.The task adds a segmentation mask to object detection outputs.
  • Proposal-driven approaches: SDS introduced a four-step instance segmentation architecture based on proposal generation, feature extraction, region classification, and region refinement.Its architecture followed the R-CNN object-detector design.
  • Intermediate features: Hypercolumn incorporated intermediate feature maps into SDS, improving segmentation accuracy over using only the final layer feature map.Earlier models primarily relied on the last-layer feature map for classification, detection, and segmentation.
  • End-to-end cascades: Multi-task Network Cascades replaced external mask proposals with cascaded stages that share convolutional features across instance differentiation, mask estimation, and categorization.Feature sharing also led to faster segmentation models.
  • Position-sensitive methods: Position-sensitive methods addressed the need for translation-variant predictions so identical pixels from different object instances could be segmented separately.InstanceFCN integrated relative position into fully convolutional networks for instance segmentation.
  • Dense segmentation: TensorMask introduced dense instance segmentation to fill the gap left by sliding-window object detection approaches that had not been used for instance segmentation.The approach extended a successful object-detection formulation into dense instance segmentation.

4.2. Some State-of-the-art Instance Segmentation Models:

The survey organizes state-of-the-art instance segmentation models by their principal architectural features, including proposal generation, multi-scale information, relative position, and feature propagation. It describes representative pipelines from SDS through PANet and discusses their associated limitations.

  • Proposal generation: SDS generates region proposals with MCG, processes them through sibling CNN branches, concatenates box and mask features, and predicts class scores with an SVM.The four steps are proposal generation, feature extraction, region classification, and region refinement.
  • Cascaded architectures: MNC decomposes instance segmentation into instance differentiation, mask estimation, and object categorization across three cascaded stages with shared convolutional features.An RPN produces class-agnostic bounding boxes before mask regression and category scoring.
  • Bounding-box proposals: Mask R-CNN extends Faster R-CNN with class, bounding-box, and mask branches, using RoIAlign to preserve spatial location during RoI processing.Its second stage simultaneously predicts a class label, bounding-box offset, and binary mask for each RoI.
  • Segmentation proposals: DeepMask generates class-agnostic segmentation proposals with shared CNN branches that predict masks and estimate whether patches are centered on full objects.It uses segmentation proposals instead of less informative bounding-box proposals.
  • Mask refinement: SharpMask refines DeepMask’s coarse masks through a top-down refinement network, improving alignment with actual object boundaries.Its architecture combines a bottom-up feed-forward network with a refinement module.
  • Multi-scale features: MultiPath combines skip connections, a foveal structure, and integral loss to integrate multi-scale and contextual features for detection and segmentation.The model connects multiple VGG16 convolutional layers to foveal regions for object localization.
  • Position-sensitive methods: InstanceFCN, FCIs, and MaskLab use relative-position or position-sensitive score maps, with FCIs jointly and concurrently performing detection and segmentation in one network.MaskLab combines semantic segmentation, object detection, and direction score maps to handle instances.
  • Feature propagation: PANet builds on Mask R-CNN and FPN to propagate information between low-level localization-rich and high-level semantic-rich feature maps.Its design emphasizes feature propagation across network layers.

4.3. Discussion:

The survey compares instance segmentation architectures, optimization settings, preprocessing, losses, model features, and average precision across datasets. It presents these comparisons chronologically and emphasizes that training choices vary across models.

  • Architectural comparison: The reviewed models span detector-based, semantic-segmentation-based, proposal-generation, position-sensitive, and feature-propagation architectures.The survey groups models according to their principal architectural features.
  • Training and optimization: Most state-of-the-art models use stochastic gradient descent with differing mini-batch sizes, learning rates, weight decay, momentum, and initialization settings.Table 5 compares optimization hyperparameters across instance segmentation models.
  • Backbones and preprocessing: Researchers choose different CNN classification, object-detection, or semantic-segmentation backbones according to model availability and application domain.Backbones may be pretrained on a dataset.
  • Backbones and preprocessing: Data augmentation is common, while loss-function differences follow variations in model architecture.The survey summarizes preprocessing and losses in Table 6.
  • Performance comparison: Table 8 compares instance segmentation models by average precision at IoU thresholds on datasets including PASCAL VOC and MS COCO in chronological order.CNN-based instance segmentation models began achieving better accuracy on these datasets around 2014.

5. Panoptic Segmentation

Panoptic segmentation unifies semantic and instance segmentation by assigning every pixel both a semantic label and an instance identity. The survey reviews models that combine branches or tasks using shared multi-scale backbones and specialized fusion mechanisms.

  • Definition: Panoptic segmentation associates every image pixel with a semantic class and identifies instances within particular classes.Its output contains semantic-label and instance-prediction channels.
  • Definition: The first panoptic segmentation formulation unified semantic and instance segmentation to cover both stuff and thing classes and introduced panoptic quality as its metric.The model was evaluated on Cityscapes, ADE20K, and Mapillary Vista.
  • Panoptic architectures: OANet uses an FPN backbone, separate semantic and Mask R-CNN instance branches, and a Spatial Ranking Module for the final panoptic output.The model is described as end-to-end.
  • Panoptic architectures: UPSNet uses a ResNet-FPN Mask R-CNN backbone and three subnetworks for semantic segmentation, instance segmentation, and panoptic segmentation.Its semantic branch uses deformable convolution for stuff classes.
  • Panoptic architectures: A multitask panoptic network uses a ResNet-FPN backbone and four task-specific subnetworks, with capability for object occlusion and scene depth ordering.The architecture is end-to-end trainable.

6. Conclusion

The survey presents CNN-based image segmentation as a task shaped by architecture, optimization, hyper-parameters, preprocessing, and loss design. It compares state-of-the-art models across datasets and outlines panoptic segmentation developments.

  • Image segmentation requires spatially variant features to preserve pixel context for semantic labeling.
  • Semantic segmentation assigns each pixel a semantic label, whereas instance segmentation separates individual object instances.
  • State-of-the-art segmentation performance depends on network architecture as well as optimization, hyper-parameters, preprocessing, and loss-function choices.
  • The survey categorizes the evolution of CNN-based segmentation models and details optimization strategies for selected state-of-the-art approaches.
  • It compares model performance across datasets and surveys recent state-of-the-art panoptic segmentation models.
  • Models can be selected according to application requirements and adapted through domain-specific fine-tuning.
Loading 2001.04074v3…