PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Segmentation Using Deep Learning: A Survey

Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, Demetri Terzopoulos

arXiv:2001.05566v5cs.CVcs.LG

TL;DR

Image segmentation supports many visual understanding applications, but deep-learning approaches had not been comprehensively synthesized. This survey reviews over 100 methods, compares their performance across major benchmarks, and identifies open challenges and research directions.

  • Problem

    Image segmentation underpins visual understanding applications, motivating a comprehensive synthesis of the rapidly growing deep-learning literature.

  • Method

    The survey reviews more than 100 deep-learning segmentation methods, comparing architectures, training choices, datasets, losses, strategies, and performance.

  • Results

    Deep-learning segmentation models show substantial accuracy gains across benchmarks, including about 23% relative improvement over initial FCN models on Cityscapes and 25%-42% relative mIoU improvement across datasets.

  • Takeaways & Limitations

    The survey organizes the field’s progress while highlighting the need for more challenging datasets, interpretable models, reproducible evaluations, and faster real-time systems.

  • Takeaways & Limitations

    The survey notes that the underlying behavior and learned representations of deep segmentation models remain insufficiently understood.

Abstract

from arXiv · show

Image segmentation is a key topic in image processing and computer vision with applications such as scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, and image compression, among many others. Various algorithms for image segmentation have been developed in the literature. Recently, due to the success of deep learning models in a wide range of vision applications, there has been a substantial amount of works aimed at developing image segmentation approaches using deep learning models. In this survey, we provide a comprehensive review of the literature at the time of this writing, covering a broad spectrum of pioneering works for semantic and instance-level segmentation, including fully convolutional pixel-labeling networks, encoder-decoder architectures, multi-scale and pyramid based approaches, recurrent networks, visual attention models, and generative models in adversarial settings. We investigate the similarity, strengths and challenges of these deep learning models, examine the most widely used datasets, report performances, and discuss promising future research directions in this area.

1 INTRODUCTION

Image segmentation partitions images or video frames into semantic labels or individual objects and underpins applications including medical analysis, autonomous vehicles, surveillance, and augmented reality. This survey reviews more than 100 deep learning-based methods, datasets, performance comparisons, challenges, and future directions through 2019.

  • Motivation: Image segmentation partitions images or video frames into multiple segments or objects for visual understanding applications.Applications include tumor boundary extraction, tissue-volume measurement, navigable-surface and pedestrian detection, video surveillance, and augmented reality.
  • Problem formulation: Semantic segmentation assigns object-category labels to every pixel, whereas instance segmentation partitions individual objects.Semantic segmentation is generally harder than image classification because classification predicts one label for an entire image.
  • Survey scope: The review analyzes training data, network architectures, loss functions, training strategies, and key methodological contributions.It also provides comparative summaries of reviewed methods’ properties and performance on popular benchmarks.
  • Resources and outlook: The survey covers around 20 popular segmentation datasets grouped into 2D, 2.5D RGBD, and 3D images, and identifies challenges and future directions.These datasets and directions complement the survey’s comparative analysis of deep learning segmentation methods.

2 OVERVIEW OF DEEP NEURAL NETWORKS

This section surveys prominent deep neural architectures used in computer vision, including CNNs, RNNs/LSTMs, encoder-decoder models, and GANs. It also outlines their core mechanisms, limitations, and use of transfer learning when labeled segmentation data are limited.

  • Transfer Learning: When labeled data are insufficient for training from scratch, image-segmentation models can use transfer learning from models pretrained on ImageNet and retrain the encoder from those initial weights.The approach assumes pretrained models capture semantic image information, enabling training with fewer labeled samples.
  • Convolutional Neural Networks: CNNs are widely used in computer vision and combine convolutional, nonlinear, and pooling layers to extract features, model nonlinear functions, and reduce spatial resolution.Convolutional layers apply learned filters, nonlinear layers apply activations, and pooling layers summarize local neighborhoods.
  • Recurrent Neural Networks: RNNs process sequential data using current inputs and previous hidden states, but long sequences can cause weak long-term dependency capture and vanishing or exploding gradients.LSTMs address these issues with input, output, and forget gates regulating information flow through a memory cell.
  • Encoder-Decoder Models: Encoder-decoder models map inputs to outputs by compressing data into a latent representation and decoding it into predictions such as enhanced images or segmentation maps.Auto-encoders are a special case in which the input and output are the same.
  • Generative Adversarial Networks: GANs use a generator to map noise to target-like samples and a discriminator to distinguish generated samples from real ones.Subsequent variants include convolutional GANs, conditional GANs, and Wasserstein-based objectives.

3 DL-BASED IMAGE SEGMENTATION MODELS · 3.1 Fully Convolutional Networks · 3.2 Convolutional Models With Graphical Models

The survey reviews more than 100 deep learning segmentation methods through architectural categories, emphasizing fully convolutional networks and graphical-model integrations. FCNs enable end-to-end dense prediction on variable-sized images, while later methods address their context and localization limitations with CRFs and MRFs.

  • 3 DL-BASED IMAGE SEGMENTATION MODELS: The survey organizes more than 100 deep learning segmentation methods proposed through 2019 into 10 architectural categories rather than grouping them primarily by segmentation goal.Common components include encoder-decoder structures, skip-connections, and multi-scale processing.
  • 3.1 Fully Convolutional Networks: Long et al. introduced an early semantic segmentation FCN containing only convolutional layers, allowing arbitrary-sized inputs and same-sized segmentation maps.They adapted VGG16 and GoogLeNet by replacing fully connected layers with fully convolutional layers.
  • 3.1 Fully Convolutional Networks: FCN skip-connections fuse deep coarse semantic features with shallow fine appearance features to produce accurate, detailed segmentations.The model achieved state-of-the-art performance on PASCAL VOC, NYUDv2, and SIFT Flow.
  • 3.1 Fully Convolutional Networks: Conventional FCNs became a milestone for end-to-end semantic segmentation but remain unsuitable for real-time inference, efficient global context modeling, and easy transfer to 3D images.These limitations motivated subsequent efforts to improve FCN-based segmentation.
  • 3.1 Fully Convolutional Networks: ParseNet addresses FCN’s missing global context by pooling each layer over the whole image, normalizing and unpooling the resulting context vector, and concatenating the new feature maps.The module replaces convolutional layers and produces smoother segmentation than an FCN.
  • 3.1 Fully Convolutional Networks: FCNs have been applied to brain tumor, instance-aware semantic, skin lesion, and iris segmentation.These applications span medical and instance-aware segmentation problems.
  • 3.2 Convolutional Models With Graphical Models: Because FCNs can ignore useful scene-level context and deep CNN responses can be poorly localized, researchers integrated probabilistic graphical models, especially CRFs and MRFs, into deep learning architectures.Fully connected CRFs refine coarse CNN score maps and improve object-boundary localization.
  • 3.2 Convolutional Models With Graphical Models: Graphical-model approaches jointly train CNNs with fully connected CRFs, exploit patch-patch and patch-background context, or encode high-order label relations in deterministic CNN-based MRF inference.Reported work includes encouraging results on PASCAL VOC 2012 and single-forward-pass Parsing Networks that replace iterative MRF optimization.

3.3 Encoder-Decoder Based Models

Encoder-decoder architectures are a dominant family of deep-learning segmentation models, spanning general and medical image segmentation. These models combine encoding for feature extraction with decoding for pixel-wise prediction, but can lose fine-grained information during encoding.

  • General encoder-decoder models: Encoder-decoder segmentation works are grouped into general segmentation and medical image segmentation models.The survey identifies encoder-decoder architectures as a popular family used by most deep-learning segmentation approaches.
  • General encoder-decoder models: Deconvolutional semantic segmentation combines a VGG16-based convolutional encoder with deconvolution and unpooling layers that generate pixel-wise class probabilities.The model achieved 72.5% accuracy on PASCAL VOC 2012, the best result among methods trained without external data at the time.
  • General encoder-decoder models: SegNet uses a VGG16-shaped encoder and decoder with transferred pool indices to produce dense feature maps, while reducing trainable parameters relative to competing architectures.A Bayesian SegNet variant models uncertainty in the encoder-decoder network for scene segmentation.
  • General encoder-decoder models: HRNet preserves high-resolution representations during encoding by connecting parallel high-to-low-resolution streams and repeatedly exchanging information across resolutions.Other related approaches include SDN, LinkNet, W-Net, and locality-sensitive deconvolution networks for RGBD segmentation.
  • Medical image segmentation: U-Net uses data augmentation, a contracting path for context, and a symmetric expanding path for precise localization, and won the ISBI cell tracking challenge 2015 by a large margin.It was trained on 30 transmitted light microscopy images and later extended to 3D, nested, and road-segmentation applications.

3.4 Multi-Scale and Pyramid Network Based Models · 3.5 R-CNN Based Models (for Instance Segmentation)

Multi-scale and pyramid architectures improve segmentation by combining representations across resolutions and scales, while R-CNN-based models extend object detection pipelines to instance-level masks. These approaches include feature pyramids, pyramid pooling, multi-resolution reconstruction, multi-branch instance prediction, and dense 4D-tensor formulations.

  • 3.4 Multi-Scale and Pyramid Network Based Models: 3.4 Multi-Scale and Pyramid Network Based Models: FPN combines bottom-up and top-down pathways with lateral connections to merge low- and high-resolution features for segmentation masks.Its pyramidal hierarchy incurs marginal extra cost, and two multilayer perceptrons generate masks.
  • 3.4 Multi-Scale and Pyramid Network Based Models: 3.4 Multi-Scale and Pyramid Network Based Models: PSPN uses ResNet and dilated features with pyramid pooling at four scales to learn global scene context.The pooled representations distinguish patterns at different scales before forming the final pixel-wise prediction.
  • 3.4 Multi-Scale and Pyramid Network Based Models: 3.4 Multi-Scale and Pyramid Network Based Models: A Laplacian-pyramid reconstruction network refines segment boundaries using high-resolution skip connections and multiplicative gating.The authors report that low apparent spatial resolution does not eliminate sub-pixel localization information in high-dimensional convolutional features.
  • 3.4 Multi-Scale and Pyramid Network Based Models: 3.4 Multi-Scale and Pyramid Network Based Models: Other multi-scale segmentation models include DM-Net, CCN, APC-Net, MSCI, and salient object segmentation.These models extend multi-scale analysis through dynamic filters, contrasted context, gated aggregation, adaptive pyramid context, and context intertwining.
  • 3.5 R-CNN Based Models (for Instance Segmentation): 3.5 R-CNN Based Models (for Instance Segmentation): Mask R-CNN extends Faster R-CNN with branches for bounding boxes, classes, and binary masks, simultaneously detecting objects and generating instance masks.Faster R-CNN uses a region proposal network and RoIPool to extract proposal features for object localization and classification.
  • 3.5 R-CNN Based Models (for Instance Segmentation): 3.5 R-CNN Based Models (for Instance Segmentation): PANet builds on Mask R-CNN and FPN by adding an augmented bottom-up pathway that improves low-layer feature propagation.Its architecture also includes adaptive feature pooling and a fully-connected fusion component.
  • 3.5 R-CNN Based Models (for Instance Segmentation): 3.5 R-CNN Based Models (for Instance Segmentation): Instance-aware models use cascaded networks for instance differentiation, mask estimation, and object categorization, while MaskLab adds semantic and direction predictions to Faster R-CNN.MaskLab produces box detection, semantic segmentation, and direction prediction outputs, using predicted boxes to localize instances.
  • 3.5 R-CNN Based Models (for Instance Segmentation): 3.5 R-CNN Based Models (for Instance Segmentation): TensorMask formulates dense instance segmentation as prediction over 4D tensors, achieving large gains over baselines and results comparable to Mask R-CNN.Other directions include R-CNN-derived mask-proposal models and bottom-up grouping methods such as Deep Watershed Transform and real-time instance segmentation.

3.6 Dilated Convolutional Models and DeepLab Family

Dilated convolutions enlarge receptive fields without increasing computational cost and became central to real-time segmentation. The DeepLab family progressively combined dilation with ASPP, cascaded and parallel modules, and encoder-decoder designs such as DeepLabv3+.

  • Dilated convolutions: A 3 × 3 kernel with dilation rate 2 matches the receptive field of a 5 × 5 kernel using only 9 parameters, without increasing computational cost.The dilation rate determines spacing between kernel weights and enlarges the receptive field.
  • DeepLab family: DeepLabv2 uses dilated convolution to preserve resolution and Atrous Spatial Pyramid Pooling to capture objects at multiple sampling rates.Dilated convolution addresses resolution loss caused by max-pooling and striding.
  • DeepLab family: DeepLabv3 combines cascaded and parallel dilated-convolution modules, adding 1 × 1 convolution and batch normalization within ASPP before producing per-pixel logits.ASPP outputs are concatenated and processed by another 1 × 1 convolution.
  • DeepLab family: DeepLabv3+ adopts an encoder-decoder architecture with atrous separable convolutions, combining depthwise spatial convolution with pointwise 1 × 1 convolution.Its encoder uses the DeepLabv3 framework and a modified, deeper Xception backbone.

3.7 Recurrent Neural Network Based Models

RNN-based segmentation models link pixels sequentially to capture short- and long-term dependencies and global context, addressing semantic segmentation beyond CNNs. Their 2D structure and difficult-to-parallelize sequential computation remain key challenges.

  • Recurrent Neural Network Based Models: RNNs can link and process pixels sequentially to model global context and potentially improve semantic segmentation, but natural image geometry poses a challenge.RNNs are described as useful for short- and long-term pixel dependencies, while images retain a natural 2D structure.
  • Recurrent Neural Network Based Models: ReSeg stacks ReNet layers over pretrained VGG-16 convolutional features, then uses up-sampling layers for image segmentation.ReNet layers sweep images horizontally and vertically in both directions to encode activations and provide global information.
  • Recurrent Neural Network Based Models: 2D LSTM networks jointly perform classification, segmentation, and context integration while learning texture and spatial model parameters in one model.The approach targets pixel-level segmentation and classification of natural scene images with complex spatial label dependencies.
  • Recurrent Neural Network Based Models: Graph LSTM represents arbitrary-shaped superpixels as semantically consistent graph nodes instead of evenly divided pixels or patches for semantic segmentation.It generalizes LSTM from sequential or multidimensional data to general graph-structured data and adaptively constructs an undirected graph.
  • Recurrent Neural Network Based Models: RNN models also support semantic labeling in RGB-D scene mapping and segmentation from natural-language expressions through recurrent and convolutional architectures.DA-RNNs integrate semantic outputs with mapping techniques such as Kinect-Fusion, while CNN+LSTM models jointly process visual and linguistic information.
  • Recurrent Neural Network Based Models: Sequential RNN computation is slower than CNN processing because it cannot be easily parallelized.This sequential nature is identified as a limitation of RNN-based models.

3.8 Attention-Based Models

Attention-based models improve semantic segmentation by weighting features across positions, scales, and contextual dependencies. The surveyed approaches include soft multi-scale attention, reverse attention, pyramid attention, and dual spatial-channel attention.

  • Attention-Based Models: Chen et al. learn soft weights for multi-scale features at each pixel, outperforming average and max pooling.The mechanism assesses feature importance at different positions and scales while jointly training with multi-scale images.
  • Attention-Based Models: Huang et al.’s Reverse Attention Network learns features not associated with a target class alongside representative semantic features.RAN uses a three-branch architecture for direct and reverse attention.
  • Attention-Based Models: The Pyramid Attention Network combines attention mechanisms with spatial pyramids to extract precise dense features using global contextual information.This approach avoids complicated dilated convolutions and artificially designed decoder networks.
  • Attention-Based Models: Fu et al.’s dual attention network captures contextual dependencies through self-attention in spatial and channel dimensions.Its two attention modules are appended to a dilated FCN; the position module aggregates features from all positions using a weighted sum.
  • Attention-Based Models: Other surveyed methods include object-context pooling, expectation-maximization attention, criss-cross attention, recurrent attention, pointwise spatial attention, and discriminative feature networks.These methods extend attention mechanisms to semantic, scene, and instance segmentation tasks.

3.9 Generative Models and Adversarial Training · 3.10 CNN Models With Active Contour Models · 3.11 Other Models

The surveyed extensions broaden deep-learning segmentation through adversarial and semi-supervised training, FCN–active-contour integration, and diverse architectures for semantic, instance, and panoptic segmentation. These methods introduce mechanisms for improving accuracy, modeling spatial structure, incorporating contour principles, and capturing contextual or multi-scale information.

  • 3.9 Generative Models and Adversarial Training: Adversarial training pairs a semantic segmentation network with a discriminator that distinguishes ground-truth maps from generated maps, improving accuracy on Stanford Background and PASCAL VOC 2012.Luc et al. trained both a convolutional semantic segmentation network and an adversarial network for this purpose.
  • 3.9 Generative Models and Adversarial Training: GAN-based methods also support semi-weakly and semi-supervised segmentation by generating extra examples or discriminating predicted probability maps from the ground-truth distribution.The semi-supervised framework uses an FCN discriminator and combines cross-entropy, adversarial, and semi-supervised losses.
  • 3.9 Generative Models and Adversarial Training: An adversarial critic with multi-scale L1 loss encourages medical-image segmentors to learn global and local features representing long- and short-range pixel relationships.The segmentor is an FCN that generates segmentation label maps.
  • 3.10 CNN Models With Active Contour Models: FCNs and Active Contour Models can be combined through loss functions incorporating area and size information from predicted masks during cardiac-MRI ventricle segmentation.This approach was inspired by a global energy formulation associated with active contour principles.
  • 3.10 CNN Models With Active Contour Models: Other FCN–active-contour strategies use ACMs as post-processors or pursue co-learning, including level-set ACMs implemented as RNNs for natural-image semantic segmentation.Deep Active Contours is another cited example, while medical-image segmentation work is also described.
  • 3.11 Other Models: Additional segmentation architectures include EncNet, RefineNet, OCR, BoxSup, graph convolutional networks, Wide ResNet, Exfuse, FoveaNet, and Ladder DenseNet.OCR learns object regions under ground-truth supervision and augments pixel representations with object-contextual representations.
  • 3.11 Other Models: Panoptic segmentation is an increasingly popular problem addressed by architectures including Panoptic Feature Pyramid Network, attention-guided networks, Panoptic-DeepLab, and efficient panoptic segmentation.The survey timeline presents representative deep-learning works for semantic and instance segmentation since 2014.

4 IMAGE SEGMENTATION DATASETS

The survey organizes pixel-wise image segmentation datasets into 2D, 2.5D RGB-D, and 3D categories for evaluating model performance. It highlights widely used benchmarks and notes that data augmentation can expand labeled training samples, especially for small datasets.

  • Dataset categories: Segmentation datasets are grouped into 2D, 2.5D RGB-D, and 3D images, with pixel-wise labels available for evaluating model performance.The survey also provides dataset characteristics within each category.
  • Data augmentation: Data augmentation increases training samples by applying transformations to both input images and segmentation maps, particularly for small datasets such as those in medical applications.Examples include translation, reflection, and rotation, with transformations applied in data space, feature space, or both.
  • 2.5D RGB-D datasets: RGB-D benchmarks support indoor and scene-understanding research through aligned color-depth images, annotated video sequences, semantic labels, and geometric information.NYU-D V2 contains 1,449 densely labeled RGB-depth pairs; SUN-3D contains 415 sequences across 254 spaces; SUN RGB-D contains 10,000 densely annotated RGB-D images.

5 PERFORMANCE REVIEW

The section reviews common segmentation-accuracy metrics and tabulates deep-learning performance across semantic, instance, panoptic, and RGB-D benchmarks. It reports substantial accuracy gains while noting that inconsistent benchmarks and limited reproducibility hinder comparisons.

  • Performance metrics: Pixel accuracy measures the fraction of correctly classified pixels, while Mean Pixel Accuracy averages per-class correctness across classes.These metrics are defined for K + 1 classes, comprising K foreground classes and the background.
  • Performance metrics: IoU measures predicted-ground-truth overlap divided by their union, and Mean-IoU averages IoU across classes.IoU ranges from 0 to 1 and is widely used for reporting modern segmentation performance.
  • Performance metrics: Precision, recall, F1 score, and Dice coefficient provide additional accuracy measures, with Dice essentially identical to F1 for boolean foreground maps.Dice is more commonly used in medical image analysis and is closely related to IoU.
  • Limitations: Across-the-board comparisons remain difficult because some models use non-standard benchmarks or subsets, and few publications reproducibly report execution time and memory footprint.Most studies emphasize accuracy, although ideal evaluation also considers inference speed and storage requirements.
  • Benchmark performance: 23% relative gain over the initial FCN model is reported for the latest models on the Cityscapes test dataset.The review also reports improved accuracy on the PASCAL VOC test set since FCN, the first deep-learning segmentation model.

6 CHALLENGES AND OPPORTUNITIES

Deep learning has advanced image segmentation, but progress depends on addressing dataset limitations, model understanding, label scarcity, real-time and memory constraints, and challenges in 3D point-cloud segmentation.

  • Datasets: More challenging datasets with many overlapping objects are needed to train models for dense, realistic still-image segmentation scenarios.Large-scale datasets for semantic and instance segmentation exist, but coverage remains insufficient for dense object arrangements and substantial overlaps.
  • Datasets: Large-scale, challenging 3D image datasets are especially needed for medical image analysis because existing datasets are often too small or synthetic.Creating 3D datasets is more difficult than creating lower-dimensional datasets.
  • Model Understanding: Open questions remain about what deep segmentation models learn, how to interpret their features, and what minimal architecture achieves a target accuracy.Existing kernel-visualization techniques do not yet provide a concrete account of the models’ underlying behavior.
  • Learning with Limited Labels: Weakly supervised, few-shot, unsupervised, and transfer-learning methods are promising because labeled segmentation data are difficult to collect, especially in medical imaging.Transfer learning uses a generic segmentation model trained on many labeled samples before adapting it to a target setting.
  • Efficiency: Near-real-time segmentation requires at least 25 frames per second, yet most current models fall short; FCN-8 takes roughly 100 ms on a low-resolution image.This capability is important for systems such as autonomous vehicles.
  • 3D Point-Cloud Segmentation: 3D point-cloud segmentation is increasingly important but challenging because point clouds are unordered and unstructured, complicating the application of CNNs and classical deep-learning architectures.Applications include 3D modeling, self-driving cars, robotics, and building modeling.

7 CONCLUSIONS

The survey reviews more than 100 deep-learning-based image segmentation algorithms, organizing them into ten categories and analyzing their performance across major benchmarks.

  • 7 CONCLUSIONS: More than 100 recent deep-learning image segmentation algorithms are surveyed and grouped into ten categories.Categories include CNN and FCN, RNN, R-CNN, dilated CNN, attention-based, and generative/adversarial models.
  • 7 CONCLUSIONS: Quantitative performance analyses cover popular benchmarks including PASCAL VOC, MS COCO, Cityscapes, and ADE20k.
Loading 2001.05566v5…