PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Semantic Segmentation for Automated Driving: Taxonomy, Roadmap and Challenges

Mennatullah Siam, Sara Elkerdawy, Martin Jagersand, Senthil Yogamani

arXiv:1707.02432v2stat.MLcs.CV

TL;DR

The paper addresses how semantic segmentation should be used in automated driving, where generic algorithms do not incorporate driving-specific structure and end goals. It surveys segmentation methods, deployment challenges, alternatives to an independent module, and architecture trade-offs, then evaluates models on CamVid for accuracy and speed. The review identifies domain-specific design considerations and reports that Dilation8 achieves the highest mean IoU among compared architectures but has the largest running time.

  • Problem

    Generic semantic segmentation algorithms do not incorporate automated-driving structure and end goals, motivating a driving-specific review.

  • Method

    The paper surveys segmentation architectures, driving datasets and challenges, alternative architectures, and comparative accuracy-speed evaluations on CamVid.

  • Results

    Dilation8 achieves the highest mean IoU among previous architectures but has the largest running time.

  • Takeaways & Limitations

    Automated-driving segmentation design must consider efficiency alongside accuracy and can use alternatives such as modular end-to-end learning with auxiliary losses.

Abstract

from arXiv · show

Semantic segmentation was seen as a challenging computer vision problem few years ago. Due to recent advancements in deep learning, relatively accurate solutions are now possible for its use in automated driving. In this paper, the semantic segmentation problem is explored from the perspective of automated driving. Most of the current semantic segmentation algorithms are designed for generic images and do not incorporate prior structure and end goal for automated driving. First, the paper begins with a generic taxonomic survey of semantic segmentation algorithms and then discusses how it fits in the context of automated driving. Second, the particular challenges of deploying it into a safety system which needs high level of accuracy and robustness are listed. Third, different alternatives instead of using an independent semantic segmentation module are explored. Finally, an empirical evaluation of various semantic segmentation architectures was performed on CamVid dataset in terms of accuracy and speed. This paper is a preliminary shorter version of a more detailed survey which is work in progress.

I. INTRODUCTION

Semantic segmentation has advanced with deep learning, but its role in automated driving remains an open question because generic methods do not address driving-specific structure and goals. The paper surveys segmentation methods and examines their relevance to automated driving.

  • Deep learning has made relatively accurate semantic segmentation possible for applications including automated driving.
  • Automated driving research includes mediated perception, end-to-end behavior reflexes, and intermediate affordance-based approaches.
  • The paper asks whether automated driving requires a semantic segmentation module.
  • It addresses a gap in prior surveys by reviewing semantic segmentation specifically in the context of automated driving.
  • The broader literature includes classical approaches based on decision forests and conditional random fields before deep learning became dominant.

B. Fully Convolutional Networks(FCN)

Fully convolutional networks perform end-to-end pixel-wise prediction from full images, while later architectures address multiscale information and resolution loss through architectural modifications.

  • Patch-wise methods classify local regions, whereas fully convolutional networks learn dense pixel-wise predictions from the full image.
  • FCNs upsample learned heatmaps within the network to produce dense segmentation outputs.
  • SegNet provides an example of semantic segmentation output applied to an automated-driving scene.
  • Multiscale architectures combine information from different resolutions, but downsampling can reduce final prediction resolution.

C. Structured Models

Structured models exploit spatial and contextual dependencies that fully convolutional approaches do not explicitly use, with CRFs, recurrent networks, and temporal models addressing increasingly broader structure.

  • Structured-model approaches use prior structure in automotive data to improve segmentation beyond fully convolutional networks.
  • CRFs model contextual consistency either as post-processing or through recurrent mean-field inference integrated into a network.
  • Recurrent neural networks capture long-range dependencies by sweeping images horizontally and vertically.
  • Video segmentation methods incorporate temporal information using clock signals, spatio-temporal LSTM grids, and convolutional recurrent networks.
  • Convolutional recurrent units learn spatial and temporal information with fewer parameters and improved memory efficiency relative to conventional recurrent processing.

III. DEEP SEMANTIC SEGMENTATION IN AUTOMATED DRIVING

Automotive semantic segmentation can exploit spatial, geometric, color, and location priors, while surround-view cameras provide structured multi-view observations across the vehicle’s field of view.

  • Automotive semantic segmentation has a priori constraints that can simplify the problem relative to generic image segmentation.
  • Spatial, geometric, color, and location priors provide domain-specific information for segmentation.
  • Automotive systems commonly use multiple cameras to cover the vehicle’s surrounding field of view.
  • Camera geometry and vehicle motion induce spatio-temporal relationships across surround-view images.

B. Dense High Definition(HD) maps

Dense HD maps provide detailed semantic and geometric priors that can improve object detection and simplify semantic segmentation when localization is reliable.

  • Dense Semantic Point Cloud Maps model entire scenes as semantic 3D point clouds, while landmark-based maps represent selected semantic objects in 2D maps.Google and TomTom use dense point-cloud maps; Mobileye and HERE use landmark-based maps.
  • HD maps improve object detection by supplying static objects such as roads, lanes, curbs, and traffic signs when sensor-map alignment is good.
  • TomTom RoadDNA aligns Lidar, camera, and other sensors with pre-mapped semantic point clouds, achieving average localization error within 10 cm under coarse GPS initialization.
  • With reliable localization, HD maps can make semantic segmentation primarily a refinement of map-derived priors and can support validation or post-processing.Map alignment accurately segments static objects but does not cover distant objects such as sky; localization confidence is therefore needed.
  • Semantic image labels without localization are of limited practical value for automated driving, motivating depth or map-based spatial information.

1) Depth using Structure from Motion(SFM):

Depth sensing and estimation can provide spatial information for semantic segmentation, but current approaches face density, noise, and integration limitations.

  • 2) LIDAR sensors: LIDAR offers highly accurate depth, but its measurements are sparse on the image lattice, complicating dense convolutional feature learning.
  • 2) LIDAR sensors: Semantic fusion maintains per-pixel class probabilities in a map and updates them incrementally in a Bayesian framework.The demonstrated system used RGB-D images but suggests potential use of LIDAR-derived depth.
  • 3) Joint In-the-Network Localization: Jointly estimating depth and semantics within one network is identified as a future research direction that could allow simultaneous end-to-end training.

B. Need of large annotated datasets

Automated-driving semantic segmentation requires substantially more annotated data than image classification, while existing datasets remain limited in scale and generalization.

  • Semantic segmentation requires significantly larger datasets than ImageNet because pixel-level annotation is functionally more complex.
  • Annotating one semantic-segmentation image can take around an hour, although LIDAR, temporal propagation, and classifier bootstrapping can accelerate labeling.
  • Cityscapes contains 5000 annotated frames, but models trained on it do not generalize well across cities or to unseen settings such as tunnels.
  • Synthetic datasets such as Synthia and Virtual KITTI can complement small real datasets, yet available combinations remain limited for commercial automated-driving deployment.
  • Mapillary Vistas provides 25,000 images across 100 classes with variation in weather, camera type, and geographic coverage.
  • Class imbalance underrepresents important objects such as pedestrians relative to sky and buildings, potentially biasing models against small objects.

2) Unobserved Objects:

Semantic segmentation can produce outputs that are too complex or insufficiently instance-specific for downstream driving tasks, motivating alternative representations and architectures.

  • 3) Complexity of Output: Semantic segmentation produces complex contours that can be difficult for mapping or maneuvering modules requiring simpler object representations.
  • 4) Recovering individual objects: Pixel-wise segmentation groups regions by class but does not separate individual objects, limiting direct use for tracking pedestrians independently.
  • 4) Recovering individual objects: Post-processing classifiers can subdivide semantic regions, while instance-level segmentation can separate same-class instances without that post-processing.
  • 4) Recovering individual objects: The paper contrasts classical, shared-encoder multi-task, end-to-end, and modular end-to-end application architectures.
  • 3) Complexity of Output: Generic segmentation may not need precise contours or labels for irrelevant objects such as sky, so driving-oriented losses or modular end-to-end systems are alternatives.

5) Goal Orientation:

The paper explores alternatives to pure semantic segmentation for automated driving, including shared-task features, end-to-end control, and auxiliary-loss architectures. These alternatives trade explicit scene parsing for task-specific or safety-oriented representations.

  • Goal Orientation:: Variable-complexity networks can allocate more capacity to complex objects such as pedestrians than to simple regions such as roads or sky.A cascaded CNN is suggested as a more efficient alternative to applying a small-complexity network uniformly across the image.
  • Goal Orientation:: Tightly coupled object-detection components make hard-negative mining and corner-case analysis difficult, motivating synthetic sequences for scenario design.The paper notes that even known corner cases can be difficult to capture in real video sequences.
  • Goal Orientation:: Multi-task learning reuses common CNN features across tasks such as flow estimation, depth, correspondence, tracking, scene classification, vehicle detection, and semantic segmentation.The cited examples include a joint flow-estimation and semantic-segmentation system and an encoder-decoder for multiple tasks.
  • Goal Orientation:: End-to-end learning maps sensory input directly to driving outputs, avoiding object-level annotation and producing lower-dimensional, temporally smooth signals.The outputs described are braking, steering, and acceleration, whose functional complexity is presented as simpler than semantic-segmentation outputs.
  • Goal Orientation:: Modular end-to-end learning adds auxiliary losses, such as segmentation, to support safety and interpretability while improving over vanilla end-to-end learning.The cited architecture also uses a recurrent gated unit after the CNN to model temporal information.

VI. BENCHMARKING AND DESIGN EXPLORATION

The benchmark evaluates segmentation architectures on CamVid using both mean intersection over union and running time. Results expose a central accuracy–efficiency trade-off: Dilation8 achieves the highest mean IoU but is too slow for practical automated-driving deployment.

  • VI. BENCHMARKING AND DESIGN EXPLORATION: The CamVid comparison measures architectures using mean intersection over union and speed rather than segmentation accuracy alone.The evaluation includes both shallower and state-of-the-art networks.
  • VI. BENCHMARKING AND DESIGN EXPLORATION: The evaluated set includes Unet, Xception, Dilated FCN16s, FCN8s, Segnet Basic, Dilation8, and Enet.The paper describes Dilated FCN16s as computationally and memory efficient, and Enet as the most efficient architecture for semantic segmentation.
  • VI. BENCHMARKING AND DESIGN EXPLORATION: Mean class IoU is computed over all classes, including classes not shown in Table II, while only selected classes are displayed.The table reports only a subset of the 11 CamVid classes.
  • VI. BENCHMARKING AND DESIGN EXPLORATION: Dilation8 achieves higher mean IoU than previous architectures but has the largest running time, making it inefficient for automated-driving semantic segmentation.The paper adapts dilated convolutions into the shallower Dilated FCN16s to reduce network size and runtime while maintaining segmentation resolution.

B. Design Exploration

The paper identifies practical design choices and research bottlenecks for deep semantic segmentation, then situates them within a broader automated-driving survey. Its design discussion emphasizes resolution-preserving convolutions, shallow networks for speed, and normalization for convergence.

  • B. Design Exploration: Deep-learning research is fragmented across architectures, making ideas difficult to formalize and combine for application development.The paper also notes that segmentation research often inherits ideas from image-recognition benchmarks.
  • B. Design Exploration: 3x3 convolutions preserve input resolution better than larger filters and were found experimentally useful for segmentation.The paper relates this choice to VGG-style architectures and the resolution sensitivity of segmentation.
  • B. Design Exploration: Dilated convolutions expand the receptive field without reducing segmentation resolution, but deep implementations such as Dilation8 can be unsuitable for real-time use.The paper presents shallower networks as a way to improve runtime with an accuracy compromise.
  • B. Design Exploration: The paper reviews segmentation architectures, datasets, and automated-driving challenges, then evaluates state-of-the-art networks on an urban-scene dataset using runtime and accuracy.The survey includes architectures not previously tried for automated-driving scenes.
  • B. Design Exploration: The authors frame the survey as a high-level preliminary version intended to orient research and industry toward current bottlenecks.A more detailed survey with deeper analysis and discussion is described as future work.
Loading 1707.02432v2…