PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Semantic Segmentation

Martin Thoma

arXiv:1602.06541v2cs.CV

TL;DR

Pixel-level semantic segmentation spans diverse algorithmic choices, but the literature lacks a unified overview of their taxonomy, evaluation, methods, and problematic cases. This survey organizes segmentation algorithms, reviews measures and datasets alongside traditional and neural approaches, and identifies directions for comparison and analysis. It reports representative results from Random Forests, SVMs, and AlexNet while emphasizing that systematic comparative studies remain open.

  • Problem

    The survey addresses the need to organize diverse semantic segmentation algorithms, evaluation resources, traditional and neural methods, and problematic data situations.

  • Method

    The paper presents a taxonomy and brief overview of automatic, passive semantic segmentation algorithms, covering clustering, random walks, Random Forests, SVMs, and neural networks.

  • Results

    The survey reports representative prior results including 66.9% per-pixel accuracy on MSC, 42% average per-pixel accuracy on Pascal VOC 2007, 7th rank by mean accuracy in the 2010 PASCAL segmentation challenge, and 17.0% error on ILSVRC-2010.

  • Takeaways & Limitations

    Comparative studies on public datasets and systematic examination of segmentation problems remain open directions, alongside ensembles and neural-network–CRF combinations.

Abstract

from arXiv · show

This survey gives an overview over different techniques used for pixel-level semantic segmentation. Metrics and datasets for the evaluation of segmentation algorithms and traditional approaches for segmentation such as unsupervised methods, Decision Forests and SVMs are described and pointers to the relevant papers are given. Recently published approaches with convolutional neural networks are mentioned and typical problematic situations for segmentation algorithms are examined. A taxonomy of segmentation algorithms is given.

I. INTRODUCTION

Semantic segmentation groups image regions by object class and supports applications from road-sign detection to medical and land-use analysis. The survey organizes segmentation algorithms by their outputs, input data, class handling, and operation mode, while reviewing datasets, measures, traditional and neural approaches, and problematic cases.

  • Semantic segmentation clusters image parts belonging to the same object class, unlike non-semantic segmentation, whose acceptable grouping is not uniquely defined.Applications include road-sign detection, tumor detection, medical-instrument detection, colon-crypt segmentation, and land-use classification.
  • Object detection differs from semantic segmentation because it must distinguish separate instances of the same object class.A tree can visually divide one car into disconnected regions, illustrating why semantic labels alone do not identify instances.
  • The survey covers quality measures and datasets, traditional segmentation methods, neural-network approaches, and typical problematic cases.These topics are presented across the survey’s organization, alongside a taxonomy of segmentation algorithms.
  • Segmentation algorithms can be grouped by the data they process and the kind of segmentation they produce.The survey introduces four classification criteria for organizing these algorithms.
  • II. TAXONOMY OF SEGMENTATION ALGORITHMS: The survey classifies semantic segmentation along fixed-class versus alternative class handling, single-pixel image inputs, and automated, passive operation.Its highlighted taxonomy describes fixed-class, single-class-affiliation algorithms operating on grayscale or colored single-pixel images in a completely automated, passive fashion.
  • C. Input Data: Segmentation inputs vary across grayscale or color, optional depth, single or multiple images, and 2D pixels versus 3D voxels.The survey connects these input choices to medical imaging, robotics, autonomous cars, consumer electronics, stereo imagery, co-segmentation, videos, and volumetric CT.

III. EVALUATION AND DATASETS

The survey presents accuracy measures and datasets for evaluating semantic segmentation, while noting that pixel-wise accuracy and qualitative examples have important limitations. It introduces confusion-matrix-based metrics, void classes, and task-specific measures.

  • A. Quality measures for evaluation: Performance measures are crucial for comparing semantic segmentation systems, with accuracy the most commonly used quality measure.
  • A. Quality measures for evaluation: Qualitative segmentation examples provide an initial impression or illustrate particular problems, but they do not measure overall accuracy.
  • A. Quality measures for evaluation: Pixel-wise accuracy, also called the per-pixel rate, compares predicted labels with reference labels for individual pixels.
  • A. Quality measures for evaluation: Pixel-wise accuracy can be misleading when images contain large regions belonging to a single class, such as sky or road.
  • A. Quality measures for evaluation: Reported alternatives include mean accuracy, mean intersection over union, frequency weighted intersection over union, and the F-measure for binary tasks.The F-measure is defined as the harmonic mean of precision and recall, commonly using β = 1.
  • A. Quality measures for evaluation: Confusion matrices and void classes address evaluation complications, although confusion matrices become infeasible when many classes are present.Void pixels are ignored in quality measures, with the evaluated total defined as width·height−number of void pixels.

2) Speed:

The survey treats inference speed, stability, and memory as important operational characteristics, while noting that timing comparisons depend on hardware and implementation. It also describes commonly used segmentation datasets and the PASCAL VOC evaluation setting.

  • 2) Speed:: Some applications require a maximum upper bound on single-image inference time, such as processing each autonomous-driving image within 20 ms.This time constraint is called latency.
  • 2) Speed:: Execution-time comparisons are difficult because runtime depends on hardware, implementation, and sometimes data.The survey recommends reporting execution time together with hardware so readers can estimate adaptation difficulty.
  • 2) Speed:: Throughput matters more than latency for applications such as automatically describing images to enable text search.
  • 3) Stability:: Segmentation should remain stable under slight input changes, including blur from smoke and small perspective changes.
  • 2) Speed:: Peak memory usage is relevant on smartphones, cameras, and GPU systems, but the survey reports no publications mentioning it.
  • 1) PASCAL VOC:: PASCAL VOC provides annotated photographs, including a 2012 pixel-level segmentation challenge with labels for 20 object classes.The segmentation challenges use the intersection over union criterion, and the 2010–2012 data remain available for evaluation.

2) MSRCv2:

This section surveys medical and surgical image datasets for semantic segmentation, then outlines traditional feature-based and classifier-based pipelines alongside neural-network alternatives.

  • Medical Databases: The survey lists medical datasets covering gland histology, retinal fundus abnormalities and vessels, and laparoscopic surgical instruments.Warwick-QU contains 165 images across five classes; DIARETDB1 contains 89 fundus images with four coarse annotation classes; DRIVE provides annotated retinal vessels; Open-CAS contains laparoscopic procedure images.
  • Medical Databases: Open-CAS combines expert annotations for some images with crowd labels produced by pixel-wise majority vote from ten knowledge workers.The crowd annotation for each image is based on ten segmentations supplied by ten different knowledge workers.
  • Traditional Approaches: Traditional segmentation commonly uses fixed-size feature inputs with a sliding-window classifier that predicts pixels or subsets of each window.The pipeline may include scaling, feature extraction, augmentation, and morphological or Markov Random Field refinement.
  • Traditional Approaches: Applying a patch classifier independently to every pixel is expensive, motivating strides, interpolation, and convolutional processing of complete images.A 512 px×512 px image requires 262 144 patch-classifier applications without acceleration; neural networks can implement the sliding-window approach as a convolution.
  • Traditional Approaches: MRFs and CRFs provide holistic alternatives by using information from the complete image during segmentation.The survey also notes that traditional systems can combine methods, such as SVMs with MRFs, and can use learned features with classifiers.
  • Traditional Approaches: Feature selection remains important in traditional approaches, with color representations including RGB, HSV, YcBcr, HSL, Lab, and YIQ.No single color space is reported as superior in all contexts.

2) Histogram of oriented Gradients:

This section reviews handcrafted visual descriptors and dimensionality-reduction issues relevant to segmentation, emphasizing that feature suitability depends on the image class and task.

  • Histogram of oriented Gradients: HOG converts image gradients into orientation histograms computed over patches, producing feature maps used in segmentation tasks.The image is differentiated along x and y, then gradient directions are histogrammed within patches.
  • Other Features: SIFT describes 16×16 keypoint patches by calculating eight-orientation histograms over sixteen 4×4 parts, yielding a 128-dimensional vector.The survey identifies SIFT as a feature for keypoints and notes its use in segmentation-related work.
  • Other Features: SIFT is characterized as a global feature for a complete image, distinguishing it from descriptors focused on local patches or keypoints.This characterization is stated explicitly in the survey’s discussion of SIFT.
  • Other Features: Bag-of-visual-words represents patch patterns as histograms and is used in combination with SIFT descriptors.The method is based on vector quantization and counts occurrences of visual patterns within image patches.
  • Other Features: Poselets depend on manually selected semantic keypoints, which are easier for humans than for classes such as airplanes, organs, or cells.Keypoints must be chosen for each class, with viewpoint-dependent strategies offered as one way to address the difficulty.
  • Dimensionality Reduction: High-resolution images can create over a million features, so downsampling or PCA may reduce dimensionality, although PCA can destroy class separability.PCA finds a low-loss projection without distinguishing classes, allowing a linearly separable feature set to become non-separable.

1) Clustering Algorithms:

The survey presents clustering, graph-based, contour, watershed, and Random Decision Forest methods as traditional segmentation approaches with distinct inputs, operating principles, and constraints.

  • Clustering Algorithms: Pixel clustering assigns feature vectors to groups using methods such as k-means and mean-shift.K-means requires the number of clusters beforehand, while mean-shift iteratively moves centroids toward local means using ranges or kernels.
  • Graph-based Methods: Graph-based methods model pixels as vertices and dissimilarity measures such as color differences as edge weights.Neighborhoods may use four or eight connected pixels; cutting edges yields connected components as segments.
  • Random Walks: Random walks assign each pixel the class of the seed point it reaches with the highest calculated probability.The method is initially interactive because it requires object-specific seeds, but another segmentation output can provide seeds non-interactively.
  • Active Contours: Active contour models combine edge alignment with smooth boundaries by minimizing an energy function.They can segment images directly or refine an existing segmentation.
  • Watershed: Watershed interprets grayscale intensity as a height map, filling catchment basins until connections define watershed boundaries.The approach can over-segment because of local minima and produce thick watersheds on plateaus.
  • Random Decision Forests: Random Decision Forests use ensembles of classifiers trained through techniques such as random subspaces and bagging.The survey notes arbitrary feature measurement scales and faster training and classification than some SVM and neural-network alternatives.
  • Random Decision Forests: 66.9 % per-pixel accuracy was reported on MSC, with 415 ms segmentation time for 320 px×213 px images; Pascal VOC 2007 accuracy was 42 %.These results came from a Random Decision Forest system using texton features.

D. SVMs

This section introduces SVMs as binary maximum-margin classifiers, explains soft margins and kernels, and summarizes their use in multi-class semantic segmentation.

  • SVM Fundamentals: SVM training represents examples as feature vectors x_i with binary labels y_i ∈ {−1, 1}.The survey frames SVMs as well-studied binary classifiers built around five central ideas.
  • SVM Fundamentals: For linearly separable data, SVMs select a hyperplane maximizing distance to the support vectors.The classifier applies the sign of the resulting decision function.
  • SVM Fundamentals: Slack variables relax linear separability when noise prevents perfect separation, with parameter C weighting errors against model complexity.The survey states that larger C accepts more errors, and identifies C > 0 as a soft-margin SVM.
  • SVM Fundamentals: The kernel trick handles nonlinearly separable data by computing inner products after an implicit mapping into a higher-dimensional space.The survey names polynomial and Gaussian radial basis function kernels among common choices.
  • Multi-class Classification: One-vs-all trains n classifiers for n classes, whereas one-vs-one trains n(n−1)/2 pairwise classifiers.These are the survey’s stated strategies for extending binary SVMs to multi-class classification.
  • Segmentation Results: An SVM-based method achieved seventh rank by mean accuracy in the 2010 PASCAL segmentation challenge and required about 2 s on an 8-core processor.The method used HOG features; another reported SVM study calculated an average rank of 7 across categories.

E. Markov Random Fields

Markov Random Fields model pixel-level segmentation with random variables, graph-based dependencies, and clique potentials. In computer vision, pixel values and class labels are linked through local neighborhoods, commonly using 4- or 8-neighborhoods.

  • MRFs assign random variables to image features and pixels, with conditional dependencies represented by an undirected graph.Nodes represent variables, while edges encode conditional dependencies.
  • A 224 × 224 RGB image with per-pixel labels yields 224 · 224 · 3 input and 224 · 224 output random variables.
  • In segmentation, x variables represent pixel values, y variables represent pixel classes, and edges connect neighboring pixels.Pixel values typically lie in 0–255 or [0, 1], while labels range over the available classes.
  • The model expresses probability through a partition function and energy function, with clique potentials defining local configurations.Clique potentials are functions associated with graph cliques.
  • Maximum A Posteriori estimation is the most common inference method for posterior MRFs in computer vision.

F. Conditional Random Fields

Conditional Random Fields condition clique potentials on input features, directly modeling label distributions given the image. The section then connects CRFs and neural networks to segmentation refinement and recent performance improvements.

  • Conditional Random Fields: CRFs model P(y|x) rather than the joint distribution P(y, x), requiring fewer parameters and no distributional assumption about x.
  • Conditional Random Fields: CRFs define a partition function and joint probability distribution using feature-conditioned clique potentials.
  • Conditional Random Fields: CRF-based methods achieved top performance in PASCAL VOC 2010, and Gonfaus et al. ranked first by mean accuracy in its segmentation task.
  • Post-processing: Post-processing refines segmentations through morphological opening and closing, edge adjustment, or active contour models.Opening removes tiny segments, while closing removes tiny gaps in otherwise filled regions.
  • Neural Networks: CNNs learn image filters while reducing the number of parameters, and rectified linear units are faster to train than sigmoid activations.
  • Neural Networks: AlexNet achieved a 17.0 % error rate on ILSVRC-2010, compared with about 25.7 % for the best other system using SIFT features and Fisher Vectors.Images were downsampled to 256 px×256 px before processing.
  • Neural Networks: Applying a classifying network as an image filter assigns each pixel class probabilities, enabling semantic segmentation at arbitrary image sizes.
  • Neural Networks: Dai et al. showed that much deeper networks are possible for segmentation and achieve better results.

VII. POSSIBLE PROBLEMS IN THE DATA FOR SEGMENTATION ALGORITHMS

Segmentation systems should be tested on uncommon data conditions because production inputs may contain effects absent from training data. Relevant problems include lens artifacts, vignetting, blur, and partial occlusion.

  • Special cases may be rare in training data but can still occur in deployed segmentation systems.
  • The survey reports no systematic work examining the influence of the listed segmentation problems.
  • Lens flare scatters light within the camera lens and appears in some KITTI road benchmark images.
  • Vignetting darkens photograph corners and can result from filters blocking light there.
  • Blur can result from lens mechanics, incorrect focus, rapid movement, smoke, or foam, including cauterization smoke during surgery.
  • Whether these effects cause problems depends heavily on the application domain and the model used.
  • Models that represent target objects may suffer from partial occlusions.

2) Camouflage:

The survey places camouflage and related visual ambiguities among segmentation challenges, alongside viewpoint variation and broader unresolved questions. It presents a taxonomy and overview while identifying comparative evaluation as future work.

  • Camouflage: Camouflaged or hard-to-detect objects create difficulties because systems may share human detection limits or learn from incorrect human labels.
  • Camouflage: Visible objects can leave the objects behind them visible, creating ambiguity over which class should receive the seen pixel.
  • Camouflage: Viewpoint changes can cause problems when those perspectives are absent from training data.
  • Discussion: The survey presents a taxonomy of semantic segmentation types and a brief overview of fully automatic passive algorithms.
  • Discussion: Future work includes comparative studies on public datasets and investigation of the influence of problematic image conditions.
  • Discussion: Combining classifiers in ensembles and combining neural networks with CRFs are identified as directions for improving accuracy.

APPENDIX A TABLES

Table I provides an overview of publicly available image databases containing semantic segmentation ground truth.

  • The table surveys publicly available image databases for semantic segmentation.Its scope is limited to databases with semantic segmentation ground truth.
  • Semantic segmentation ground truth is the defining inclusion criterion for the listed databases.
  • The appendix table functions as a dataset reference for semantic segmentation research.
Loading 1602.06541v2…