PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Deep Learning Techniques for Image Segmentation

Swarnendu Ghosh, Nibaran Das, Ishita Das, Ujjwal Maulik

arXiv:1907.06119v1cs.CVcs.LGcs.NE

TL;DR

Deep learning has produced many image-segmentation techniques across application domains, creating a need for analytical understanding. This paper surveys traditional and deep learning approaches, categorizes major algorithms, and explains their key contributions, advantages, and limitations intuitively.

  • Problem

    Image segmentation spans many applications and increasingly diverse deep learning techniques, motivating a need for analytical understanding of these approaches.

  • Method

    The paper surveys segmentation methods from traditional approaches through deep learning, categorizes major algorithms, and uses intuitive explanations and unified diagrams to compare their contributions.

  • Results

    The survey explains the advantages, disadvantages, similarities, and differences of major deep learning image-segmentation algorithms.

  • Takeaways & Limitations

    Choosing an image-segmentation approach requires analyzing factors such as datasets, annotations, supervision type, architecture, and efficiency requirements.

  • Takeaways & Limitations

    Pre-training and clustering can be inefficient when available samples have a unique distribution or differ from the test-data distribution.

Abstract

from arXiv · show

The machine learning community has been overwhelmed by a plethora of deep learning based approaches. Many challenging computer vision tasks such as detection, localization, recognition and segmentation of objects in unconstrained environment are being efficiently addressed by various types of deep neural networks like convolutional neural networks, recurrent networks, adversarial networks, autoencoders and so on. While there have been plenty of analytical studies regarding the object detection or recognition domain, many new deep learning techniques have surfaced with respect to image segmentation techniques. This paper approaches these various deep learning techniques of image segmentation from an analytical perspective. The main goal of this work is to provide an intuitive understanding of the major techniques that has made significant contribution to the image segmentation domain. Starting from some of the traditional image segmentation approaches, the paper progresses describing the effect deep learning had on the image segmentation domain. Thereafter, most of the major segmentation algorithms have been logically categorized with paragraphs dedicated to their unique contribution. With an ample amount of intuitive explanations, the reader is expected to have an improved ability to visualize the internal dynamics of these processes.

1 Introduction

Image segmentation assigns pixels to meaningful, connected semantic regions and encompasses several related computer-vision problems. Its applications span medical imaging, surveillance, satellite imagery, defense, and forensic recognition.

  • Image segmentation divides an image into meaningful regions by assigning labels to pixels with shared visual or semantic properties.The labeled pixels are expected to be connected with respect to the relevant property.
  • Semantic segmentation classifies each pixel into predefined classes, but the appropriate semantics depend on the task.A pedestrian system may group an entire person, whereas action recognition may require separate body-part classes.
  • Saliency detection focuses segmentation on the most important object, while foreground-background separation isolates scene regions.
  • Instance-specific segmentation separates multiple instances of the same object and is often coupled with object detection.It supports applications such as counting objects in image retrieval or visual question answering.
  • Segmentation supports medical abnormality and organ localization, surveillance of people and cars, satellite imagery, defense guidance, and forensic recognition.

2 Motivation

The paper responds to the emergence of many deep-learning segmentation methods by analyzing why and when they work or fail. It presents these methods in an intuitive, organized format intended to help readers understand and choose among them.

  • Earlier surveys covered traditional segmentation, application areas, or specific algorithm types, while newer deep-learning methods created additional classes requiring analysis.
  • The paper analyzes each method's successes and failures so readers can understand when and how techniques perform across challenges.
  • Traditional techniques are introduced before major deep-learning algorithms are categorized into families and explained with limited mathematics.
  • A shared diagram format helps readers understand fundamental similarities and differences between network architectures.
  • The paper also discusses major application areas to help new researchers pursue a field of interest.

3 Impact of Deep Learning on Image Segmentation

Deep learning expanded image segmentation beyond traditional approaches by applying learned representations and enabling broader datasets, challenges, and competitions. Convolutional architectures preserve spatial information for pixel-level outputs, while activation maps can support segmentation.

  • Deep learning affected image segmentation alongside object classification, detection, localization, and tracking.
  • Convolutional kernels learn feature-specific activation maps that can be interpreted as segmentation-relevant representations.
  • Deep learning's growth brought many datasets, challenges, and competitions that encouraged segmentation research across domains.

4 Image Segmentation using Deep Learning

Deep learning segmentation methods reuse convolutional feature activations while adding architectures and refinement mechanisms to preserve pixel-level detail. The section surveys fully convolutional, instance-oriented, dilated-convolution, and CRF-based approaches.

  • Convolutional activations contain semantic components that can be used to segment images, motivating many deep learning segmentation methods.
  • Fully convolutional layers: Fully convolutional networks directly classify pixels from convolutional feature maps, achieving 90.3% pixel accuracy and 62.7% mean IOU on PASCAL VOC 2011.
  • Fully convolutional layers: DeepMask predicts a central-object mask and recognition score, while SharpMask refines masks through top-down convolutional processing.
  • Fully convolutional layers: SharpMask achieved 39.3 average recall versus DeepMask's 36.6 on the MS COCO Segmentation Dataset.
  • Context and pixel correlation: Atrous or dilated convolution expands the sensory region without relying solely on larger kernels or pooling, while DeepLab also uses CRF refinement.
  • Context and pixel correlation: CRF pairwise potentials encourage nearby pixels with similar RGB intensities to receive the same class, boosting DeepLab mean IOU by up to 4% on PASCAL 2012.

4.2 Convolutional autoencoders

Convolutional autoencoder segmentation uses an encoder-decoder structure to reconstruct pixel-level class distributions. Skip connections, transposed convolutions, and pooling indices help recover high-resolution boundaries and spatial information.

  • Segmentation autoencoders replace input reconstruction with reconstruction of the desired pixel-level class distribution.
  • Encoder-decoder down-sampling and up-sampling can produce pixel-level probabilities at the same resolution as the input image.
  • Decoder operations: Transposed convolution, also called convolution with fractional strides, is introduced to reverse the size-changing effects of traditional convolution.
  • Skip connections: Skip connections combine abstraction levels from different layers to generate crisp segmentation maps and improve gradient flow through deep networks.
  • U-Net: U-Net uses a convolutional and max-pooling encoder with a mirrored decoder containing convolutions and transposed convolutions.
  • SegNet: SegNet stores max-pooling indices in five VGG-16-based encoding blocks and uses them during decoding to support spatially informed unpooling.

4.3 Adversarial Models

Adversarial segmentation treats the segmentation network as a generator and trains a discriminator to distinguish predicted masks from ground-truth masks. The section also situates this approach among instance-level, weakly supervised, and unsupervised alternatives.

  • Adversarial Models: GAN training poses generator and discriminator learning as opposing objectives: the generator improves outputs by fooling the discriminator, while the discriminator improves at detecting generated images.The optimization is expressed as a min-max problem.
  • Adversarial Models: Adversarial segmentation uses a generator to produce class masks and a discriminator to distinguish generated masks from ground-truth masks.Conditional GANs can also frame segmentation as image-to-image translation using semantic boundaries and output segmentations.
  • Instance-Level Segmentation: Instance segmentation differs from semantic segmentation because separate instances of the same object receive different labels, motivating sequential models.Convolutional LSTMs replace fully connected weights with convolutional layers to retain spatial information and reduce image-related computation costs.
  • Attention Models: Attention-based instance segmentation can use external memory, a box network, a segmentation module, and a scoring module that stops when scores fall below a user-defined threshold.The external memory stores boundary details, while the box network selects the next image sub-region for segmentation.
  • Unsupervised Models: Unsupervised segmentation methods address missing ground truth through multiple objectives, clustering-based supervisory signals, adversarial collaboration, superpixel constraints, or reconstruction with normalized-cut losses.W-Net cascades two U-Nets: one segments the image and the other reconstructs it, while mean-square-error and normalized-cut losses are minimized simultaneously.
  • Weakly Supervised Models: Weak supervision can use cheaper and more abundant bounding-box annotations to generate pixel-level segmentation maps, as in BoxSup.BoxSup combines region proposals and multi-scale grouping, then uses the resulting map to tune an FCN; it attained an mIOU of 75.1 in the cited setting.

4.6 Interactive Segmentation

Interactive segmentation incorporates user guidance to improve segmentation in complex, noisy, or poorly illuminated images, while efficient architectures address deployment constraints.

  • Interactive approaches: Interactive segmentation uses image and interaction streams, such as positive and negative point maps, which are fused for prediction.Interaction maps can be generated from point intensities based on Euclidean distance.
  • Interactive approaches: Deep extreme cut uses four user-provided boundary-extreme points to create heatmaps for a DenseNet101 and pyramid scene parsing pipeline.The points indicate the leftmost, rightmost, topmost, and bottommost object regions.
  • Interactive approaches: Polygon-RNN extracts multiscale VGG features and uses a recurrent network to output contour points for interactive image annotation.Users provide a tight bounding box as part of the interaction.
  • Efficient segmentation: Efficient segmentation must balance output quality with hardware cost and real-time response requirements in practical systems.The paper identifies efficiency as important for consumer-level systems.
  • Efficient segmentation: ENet reduces segmentation-model size through a shallow asymmetric encoder-decoder, parallel pooling, PReLU activations, and factorized filters.The network has 0.37 Million parameters.
  • Efficient segmentation: Deep Layer Cascade classifies easier pixels earlier and forwards uncertain pixels to later stages, reducing computation to unresolved regions.On VOC12, it reached 82.7 mIOU and processed 23.6 frames per second, compared with 14.6 fps for SegNet and 7.1 fps for DeepLab-V2.
  • Efficient segmentation: SegFast combines depth-wise separable convolutions with SqueezeNet fire modules and achieves a 0.6 Million-parameter model with a 0.38-second CPU forward pass.Its performance was reported as comparable to networks such as SegNet.

5 Applications

Image segmentation supports applications spanning information retrieval, healthcare, autonomy, post-production, biometrics, and surveillance by isolating meaningful objects or regions.

  • Information and visual systems: Segmentation represents spatial relations among objects and supports content-based image retrieval and related visual information tasks.The paper also connects it with visual question answering, interactive query-based image processing, and description generation.
  • Healthcare: Healthcare applications segment vessels, tissues, nerves, and other organic elements, while localizing tumors and aneurysms.These tasks use images from different imaging sources and body regions.
  • Autonomy and robotics: Deep-learning-based object analysis supports robotic maneuverability, autonomous driving, motion detection, tracking, and exploration of remote environments.Examples include deep-sea and space exploration with intelligent robots.
  • Post-production: Post-production uses segmentation for image matting, compositing, and rotoscoping.
  • Biometrics: Biometric verification segments informative regions in iris, fingerprint, finger-vein, and dental-record systems.
  • Surveillance: Surveillance segmentation separates objects of interest from clutter despite occlusion, lighting, and weather issues.Applications include tracking, searching, anomaly detection, threat detection, and traffic control.

6 Discussion and Future Scope

The discussion frames algorithm selection around data, annotations, architecture, refinement, accuracy, speed, and deployment constraints, while identifying weakly supervised and unsupervised learning as open challenges.

  • Method selection: Choosing a segmentation algorithm requires analyzing the variables that affect performance because many methods offer different contributions, advantages, and disadvantages.
  • Data and annotations: Dataset availability and annotation quality strongly influence method choice, with larger related datasets often used for pre-training on small datasets.Pre-training on classification or localization tasks can also help learn better weights when pixel labels are unavailable.
  • Learning supervision: Supervised methods are numerous, whereas unsupervised and weakly supervised segmentation remain insufficiently mature for fully scalable pixel-level learning.The paper highlights modeling data distributions, selecting class counts, and creating accurate pixel maps as unsupervised challenges.
  • Architecture and refinement: Architecture choices include fully convolutional models, multiscale feature fusion, and pre-trained VGGNet, ResNet, or DenseNet encoders.
  • Architecture and refinement: Performance assessment should consider both speed and accuracy, with CRFs, superpixels, and human interaction serving as refinement strategies.CRFs can be simulated as RNNs to create end-to-end trainable refinement modules.
  • Future scope: Future progress depends on obtaining abundant, high-quality data, especially accurate pixel-level annotations that often require manual intervention.The paper describes learning meaningful segments from data distributions as especially challenging for large unstructured datasets.

7 Conclusion

The paper surveys the rapid growth of deep-learning image-segmentation algorithms and explains their advantages, disadvantages, similarities, and differences through intuitive descriptions and unified diagrams.

  • The survey explains the pros and cons of state-of-the-art deep-learning segmentation algorithms using accessible concepts and a unified representation scheme.The authors suggest that future work could accompany the theoretical survey with empirical analysis.

Supplementary Information

Traditional image segmentation methods use intensity, clustering, histogram, edge, region, graph, watershed, and handcrafted feature cues to assign pixels or regions to segments.

  • Thresholding: Thresholding assigns class labels using image-intensity values and can select a threshold through maximum variance, fuzzy logic, or nonlinear rules.
  • Clustering methods: Clustering methods such as K-means divide images into multiple classes and work well when objects contrast strongly with the background.
  • Histogram-based methods: Histogram methods analyze peaks and troughs to determine an optimal number of segments without requiring the number of clusters beforehand.
  • Edge detection and region-growing methods: Edge and region-growing methods use semantic boundaries or local similarity, respectively, with region growing expanding from seeds while remaining within boundaries.
  • Graph based approaches: Graph-based approaches represent pixels or pixel groups as weighted graph nodes, using graph cuts or probabilistic models for labeling and energy minimization.
  • Watershed transformations and feature based techniques: Watershed methods interpret image gradients as a topographic surface, using high-gradient pixels as segmentation contours; feature-based methods classify pixels from visual descriptors.Fully connected networks used for feature-based segmentation can require O(n^2) trainable weights for larger activation maps.

A brief history of neural networks

Neural networks progressed from linear learnable models toward deeper architectures as researchers confronted nonlinear limitations and gained access to stronger computation and larger datasets.

  • A brief history of neural networks: Perceptrons and artificial neurons modeled linear relationships with learnable weights but could not learn dependent-variable cases such as XOR.
  • Advancement in Hardwares: The availability of GPUs and CUDA reduced the computational bottleneck created by the large floating-point workloads of deep learning.
  • Larger Datasets: New datasets and challenges, including MNIST and CIFAR, helped expand neural-network research after parallel computation became available.

Broad categories of typical deep learning models

The paper groups deep learning models for computer vision into sequential, autoencoding, convolutional, and generative categories, each organized around a distinct processing or learning mechanism.

  • Sequential Models: Sequential models process input sequences while carrying learned information across time steps, and LSTM architectures preserve gradients across long distances through an additive highway channel.These models are applied in computer vision tasks including video processing and instance segmentation.
  • Autoencoders: Autoencoders encode raw inputs into latent representations and use decoders to reconstruct them, with loss minimization limiting information loss.Figure 18 depicts a generic autoencoder using fully connected linear layers.
  • Convolutional Neural Networks: Convolutional neural networks learn image-feature kernels through backpropagation instead of requiring manually designed kernels.Figure 19 presents a typical convolutional neural network.
  • Generative Models: Generative models address the spatial complexity of image generation through approaches including variational autoencoders and adversarial learning.Figure 20 shows a block diagram of a generative adversarial network.
Loading 1907.06119v1…