PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review on Deep Learning Techniques Applied to Semantic Segmentation

Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, Jose Garcia-Rodriguez

arXiv:1704.06857v1cs.CVcs.AI

TL;DR

Semantic segmentation is an important pixel-level computer-vision problem for applications requiring accurate scene understanding, while existing surveys lacked recent datasets, framework analysis, and deep-learning detail. This paper reviews deep-learning methods, datasets, challenges, and quantitative results, concluding that method strengths vary across data modalities and that important dataset and deployment gaps remain.

  • Problem

    Semantic segmentation requires accurate pixel-level labeling for computer-vision applications, but earlier surveys lacked recent datasets, framework analysis, and detailed coverage of deep-learning techniques.

  • Method

    The paper reviews semantic-segmentation terminology, datasets, challenges, deep-learning methods, quantitative evaluations, results, and future research directions.

  • Results

    DeepLab is reported as the most solid method on almost every RGB-image dataset, while recurrent networks dominate 2.5D or multimodal datasets and Clockwork Convnets are promising for video.

  • Takeaways & Limitations

    The review covers 28 datasets and 27 methods to help researchers select datasets and understand method contributions and raw results.

  • Takeaways & Limitations

    Large-scale real-world 3D semantic-segmentation datasets remain insufficient, and execution-time and memory-footprint reporting is largely absent.

Abstract

from arXiv · show

Image semantic segmentation is more and more being of interest for computer vision and machine learning researchers. Many applications on the rise need accurate and efficient segmentation mechanisms: autonomous driving, indoor navigation, and even virtual or augmented reality systems to name a few. This demand coincides with the rise of deep learning approaches in almost every field or application target related to computer vision, including semantic segmentation or scene understanding. This paper provides a review on deep learning methods for semantic segmentation applied to various application areas. Firstly, we describe the terminology of this field as well as mandatory background concepts. Next, the main datasets and challenges are exposed to help researchers decide which are the ones that best suit their needs and their targets. Then, existing methods are reviewed, highlighting their contributions and their significance in the field. Finally, quantitative results are given for the described methods and the datasets in which they were evaluated, following up with a discussion of the results. At last, we point out a set of promising future works and draw our own conclusions about the state of the art of semantic segmentation using deep learning techniques.

1 INTRODUCTION

Semantic segmentation is a key computer-vision problem supporting scene understanding and applications such as autonomous driving and augmented reality. This review focuses on deep-learning methods, datasets, quantitative evaluation, and future research directions.

  • Semantic segmentation is a key computer-vision problem applied to 2D images, video, 3D data, and volumetric data.
  • The review addresses a gap by focusing explicitly on deep learning for semantic segmentation, including recent datasets and framework details.
  • The paper surveys datasets useful for segmentation projects using deep-learning techniques.
  • It organizes an in-depth review of significant deep-learning methods, their origins, and their contributions.
  • The review gathers quantitative metrics including accuracy, execution time, and memory footprint.
  • It discusses the collected results, identifies possible future research directions, and concludes about the field's state of the art.

2 TERMINOLOGY AND BACKGROUND CONCEPTS

Semantic segmentation assigns labels densely across image or volumetric data, extending computer vision from classification and localization toward fine-grained scene understanding. The review introduces standard architectures, recurrent models, transfer learning, and augmentation as background for deep segmentation.

  • Terminology: Semantic segmentation labels every pixel with the class of its enclosing object or region, extending naturally toward instance and part-based segmentation.
  • Terminology: The task assigns states from a label space to random variables representing image pixels, with formulations extending beyond 2D images to volumetric and hyperspectral data.
  • Convolutional neural networks: AlexNet, VGG-16, GoogLeNet, and ResNet became standard building blocks for segmentation architectures.
  • Recurrent architectures: ReNet extends recurrent modeling to multidimensional tasks by using usual sequence RNNs to model spatial dependencies.
  • Training: Transfer learning and fine-tuning from pretrained classification networks are common because per-pixel segmentation datasets are difficult to gather and smaller than classification datasets.
  • Training: Data augmentation can regularize training and improve generalization, while one portrait-segmentation example increased IoU from 73.09 to 94.20 after augmentation and fine-tuning.

3 DATASETS AND CHALLENGES

The review emphasizes that suitable, sufficiently representative data is central to deep semantic segmentation, while available datasets differ substantially in modality, annotation, scale, and application setting. It surveys 2D, RGB-D, and 3D resources and summarizes their dataset information for project selection.

  • Challenges: Dataset construction is critical because deep segmentation systems require data with sufficient scale and accurate representation of the intended use case.
  • Dataset organization: The surveyed datasets are organized into 2D RGB, 2.5D RGB-D, and volumetric 3D groups, with information on classes, formats, and data splits.
  • 2D datasets: PASCAL VOC provides pixel-wise object-class prediction across 21 segmentation classes, while PASCAL Context expands annotation to 540 classes, with 59 frequent classes emphasized.
  • 2D datasets: PASCAL Part adds per-pixel object-part masks, and SBD adds category-level, instance-level, and boundary annotations to images not labeled in VOC.
  • 2D datasets: COCO provides more than 80 classes and over 82,783 training images, whereas SYNTHIA supplies 13,407 rendered training images with fine-grained labels for 11 classes and varied urban conditions.
  • Driving datasets: Cityscapes targets urban scenes with around 5,000 fine and 20,000 coarse annotated images, while CamVid contains 701 manually annotated driving-scene frames.
  • Driving datasets: KITTI is widely used in robotics and autonomous driving but lacks native semantic-segmentation ground truth, so researchers annotate subsets for particular needs.
  • Video datasets: YouTube-Objects lacks original pixel-wise annotations, but a manually annotated subset contains 10,167 frames at 480 × 360 resolution.

4 METHODS

Deep semantic-segmentation methods grew from CNNs’ success in learning visual feature hierarchies, with FCN-based approaches becoming a central foundation. The review organizes methods by their contributions and the targets they address, including accuracy, efficiency, context, and data modality.

  • Methods: CNNs motivated deep semantic segmentation because they learn visual feature hierarchies for pixel-level labeling.
  • Methods: Fully Convolutional Networks transformed classification CNNs into dense predictors by replacing fully connected layers with convolutional ones and upsampling spatial outputs.
  • Methods: FCN is described as the cornerstone of deep-learning semantic segmentation and as the common forerunner of successful state-of-the-art techniques.
  • Methods: The review summarizes methods by base architecture, principal contribution, and focus areas including accuracy, efficiency, training simplicity, sequence processing, multimodal inputs, and 3D data.
  • Methods: FCN-based methods remain limited by missing global context and default instance awareness, high-resolution efficiency gaps, and limited suitability for unstructured 3D data.

4.1 Decoder Variants

Segmentation architectures commonly separate low-resolution feature extraction from reconstruction, but decoder designs differ in how they upsample and recover pixel-level predictions. SegNet uses pooling indices, whereas FCN variants learn deconvolution filters and fuse encoder features.

  • Decoder variants: Classification networks adapted for segmentation commonly remove fully connected layers, creating an encoder that produces low-resolution representations and a decoder for reconstruction.
  • SegNet: SegNet decodes through upsampling and convolution layers followed by softmax, producing pixel-wise labels at the input resolution.
  • SegNet: Each SegNet decoder upsampling layer corresponds to an encoder max-pooling layer and uses that layer’s pooling indices to upsample feature maps.
  • FCN decoders: FCN-based architectures use learnable deconvolution filters for upsampling, then add the upsampled maps elementwise to corresponding encoder feature maps.

4.2 Integrating Context Knowledge

Semantic segmentation must balance fine local detail with global context, but vanilla CNNs struggle to capture both. Reviewed solutions include CRF refinement, dilated convolutions, multi-scale prediction, and recurrent context modeling.

  • Semantic segmentation requires integrating local detail and global image context to resolve pixel-level ambiguities.
  • Vanilla CNNs lose global context through pooling, while architectures without pooling expand receptive fields only linearly with depth.
  • Conditional Random Fields: Dense CRFs refine segmentation by combining low-level pixel interactions with network outputs and modeling both short- and long-range dependencies.
  • Conditional Random Fields: CRFasRNN incorporates dense-CRF pairwise potentials into an FCN by unrolling mean-field inference as recurrent layers for end-to-end training.
  • Dilated Convolutions: Dilated convolutions enlarge receptive fields without losing resolution by upsampling filters, with dilation rate controlling the expansion.
  • Multi-scale Prediction: Multi-scale architectures combine predictions or features across resolutions, including progressively coarse-to-fine networks and independently trained FCNs fused before fine-tuning.
  • Recurrent Context Modeling: DAG-RNNs combined with CNN feature maps and deconvolution model long-range contextual dependencies while upsampling the feature maps.

4.3 Instance Segmentation

Instance segmentation separates objects belonging to the same class, making it more challenging than semantic segmentation. Reviewed methods combine proposal generation, joint mask prediction, top-down refinement, and detector-based classification, while the problem remains unresolved.

  • Instance segmentation assigns separate masks to objects of the same class, with an initially unknown number of instances and non-pixel-wise evaluation.
  • SDS generates region proposals with hierarchical grouping, then extracts region features using an adapted, fine-tuned R-CNN pipeline.
  • DeepMask jointly predicts an object mask and the likelihood that an input patch contains an object using a single convolutional network.
  • SharpMask refines object encodings top-down by merging low-level spatial features with high-level semantic information through stacked refinement modules.
  • The MultiPath classifier improved performance over baseline Fast R-CNN by 66% on COCO using DeepMask proposals, localization loss, foveal context, and skip connections.
  • Many reviewed methods rely on existing object detectors, and instance segmentation remains an unresolved research problem.

4.4 RGB-D Data

RGB-D methods add geometric depth cues to photometric information for scene and object segmentation. Depth can improve fine-grained labeling, but noise and scene complexity make its use challenging and benefits inconsistent across approaches.

  • Low-cost RGB-D sensors provide geometric cues from depth that complement photometric data for segmentation.
  • Depth information has improved fine-grained labeling precision in several RGB-D segmentation studies, despite illumination variation and occlusion challenges.
  • HHA encodes depth into horizontal disparity, height above ground, and the angle between local surface normals and inferred gravity.
  • Zeng et al. independently processed multi-view RGB-D images with FCNs and found that adding depth produced no major segmentation improvement, possibly because of depth noise.
  • Ma et al. enforced multi-view consistency by warping RGB-D images from moving-camera trajectories into annotated frames using an approach based on FuseNet.

4.5 3D Data

3D point clouds and meshes provide rich spatial information but are irregular inputs for conventional CNNs. Reviewed approaches either regularize them into voxel grids or process raw unordered point sets directly.

  • Point clouds and polygonal meshes offer rich spatial information, but most CNN architectures are not designed for unstructured or irregular inputs.
  • Voxel-based systems transform point clouds into dense occupancy grids, use 3D CNNs to predict per-voxel labels, and map those labels back to points.
  • PointNet directly consumes unordered 3D point sets with a unified architecture supporting both classification and segmentation.
  • PointNet uses fully connected layers, transformation modules, multilayer perceptrons, and max-pooling to derive global features from point clouds.

4.6 Video Sequences

Video segmentation methods address the computational cost and missed temporal cues of processing each frame independently. Reviewed approaches exploit temporal feature updates, spatio-temporal convolutions, or graph-based processing.

  • Frame-by-frame segmentation can be computationally impractical and ignores temporal continuity that could improve accuracy and reduce execution time.
  • Clockwork FCN reduces video inference time by exploiting layer-wise differences in feature velocity across frames.Shallow-layer features change faster than deep-layer features, motivating different recomputation rates.
  • Clockwork FCN supports fixed update schedules and adaptive, data-driven schedules triggered by motion or semantic change.
  • A 3DCNN-based method learns hierarchical spatio-temporal features from video clips, embeds them in a supervoxel graph, and applies graph-cut segmentation.
  • Deep end-to-end voxel-to-voxel prediction extends the C3D network with deconvolutional layers and predicts separately from 16-frame clips.

5 DISCUSSION

The discussion frames evaluation as a multi-dimensional assessment of semantic segmentation systems, covering accuracy, runtime, and memory. It also identifies standard metrics and hardware-dependent limitations in runtime comparisons.

  • Evaluation should assess execution time, memory footprint, and accuracy using standard metrics to support rigorous and fair comparisons.
  • Runtime is valuable because inference commonly faces hard time constraints, but timings depend strongly on hardware and backend implementation.
  • Accuracy metrics: Pixel Accuracy measures the ratio of correctly classified pixels to the total number of pixels.
  • Accuracy metrics: Mean Pixel Accuracy averages per-class ratios of correctly classified pixels.
  • Accuracy metrics: Mean Intersection over Union averages per-class intersection-over-union ratios based on true positives, false negatives, and false positives.
  • Accuracy metrics: Frequency Weighted Intersection over Union weights class contributions according to their appearance frequency.
  • Accuracy metrics: MIoU is the most commonly used segmentation metric because of its representativeness and simplicity.

5.2 Results

The review aggregates reported results across 2D, RGB-D, 3D, and video datasets, but heterogeneous reporting practices limit fair comparisons. Performance leaders vary by dataset and input category.

  • Results are organized by 2D or 2.5D images, volumetric 3D data, and video sequences.
  • Inconsistent dataset, metric, and reporting practices make fair comparison between methods difficult or impossible.
  • Few papers report execution time or memory footprint, and missing reproducibility details can make reported results unusable.
  • 2D RGB: DeepLab leads PASCAL VOC-2012 with 79.70 IoU and PASCAL-Context with 45.70 IoU.
  • 2D RGB: DAG-RNN leads CamVid with 91.60 IoU, while rCNN reaches 80.20 IoU on Stanford Background.
  • 2.5D RGB-D: LSTM-CF reports 48.10 IoU on SUN-RGB-D, 49.40 IoU on NYUDv2, and 58.50 accuracy on SUN-3D.
  • Volumetric 3D: PointNet achieves 83.80 IoU on ShapeNet Part and 47.71 IoU on Stanford-2D-3D-S.
  • Video sequences: Clockwork Convnet reaches 64.40 IoU on CityScapes and 68.50 on YouTube-Objects for video segmentation.

5.3 Summary

The review identifies reproducibility, resource reporting, and dataset limitations as major barriers, while finding different leading approaches across RGB, multimodal, 3D, and video segmentation. It proposes future work spanning larger datasets, contextual modeling, real-time performance, memory efficiency, temporal coherence, and multi-view integration.

  • Comparisons are often impossible because methods use nonstandard datasets, omit experimental details, or fail to release source code and model weights.
  • Execution time and memory footprint are rarely reported, despite likely deployment on computationally and memory-constrained embedded devices.
  • DeepLab outperforms other methods on almost every RGB-image dataset by a significant margin.
  • LSTM-CF dominates 2.5D or multimodal datasets, while PointNet advances segmentation of unordered point clouds without preprocessing or discretization.
  • Clockwork Convnets are the most promising video approach because they combine efficiency and accuracy, while 3D convolutions capture spatial and temporal information from multichannel inputs.
  • Future research directions: Future research should address scarce large-scale 3D and sequence datasets, context modeling, real-time constraints, memory reduction, temporal coherence, and broader multi-view integration.The review specifically highlights graph convolutions for point clouds, CRF-as-RNN context modeling, pruning, and coherent video predictions as directions.

6 CONCLUSION

The paper presents a comprehensive deep-learning-focused review of semantic segmentation datasets and methods, combining descriptive and quantitative comparisons. It concludes that the field has achieved many successes but remains open, with future directions and open problems still requiring attention.

  • This review focuses specifically on deep learning for semantic segmentation and covers 28 datasets and 27 methods.
  • Datasets are described by purpose and characteristics to help researchers select those matching their segmentation needs.
  • Methods are surveyed through their contributions and raw accuracy results, with comparative summaries organized in tables and classifications.
  • The review discusses results, future research directions, and open problems, concluding that semantic segmentation remains an open problem despite many successes.
Loading 1704.06857v1…