PaperlayerView exact revision on arXiv

Source-linked AI summary

RTSeg: Real-time Semantic Segmentation Comparative Study

Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand

arXiv:1803.02758v5cs.CV

TL;DR

Semantic-segmentation research has focused largely on accuracy, with limited attention to computational efficiency and principled comparisons of design choices. RTSeg provides a modular benchmark separating feature extraction from decoding. Its reported results include up to 143x fewer GFLOPs than SegNet, while lower operation counts did not translate into higher inference speed.

  • Problem

    Semantic-segmentation research has given limited attention to computational efficiency and lacks principled comparisons of architectural modules, despite the relevance of efficiency to applications such as autonomous driving.

    Source: §1 · p. 1

  • Method

    RTSeg benchmarks segmentation by decoupling feature-extraction networks from decoding meta-architectures so their design choices can be compared.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    SkipNet-ShuffleNet and SkipNet-MobileNet reduce GFLOPs versus SegNet by up to 143x, while SkipNet-ShuffleNet maintains on-par mIoU with ENet but does not outperform it in inference speed.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    The modular framework gives researchers and practitioners a means to evaluate design choices for their tasks.

    Source: §4 · p. 5

    The authors report that lower computational cost did not yield higher inference speed.

    Source: §1 · p. 2

Abstract

from arXiv · show

Semantic segmentation benefits robotics related applications especially autonomous driving. Most of the research on semantic segmentation is only on increasing the accuracy of segmentation models with little attention to computationally efficient solutions. The few work conducted in this direction does not provide principled methods to evaluate the different design choices for segmentation. In this paper, we address this gap by presenting a real-time semantic segmentation benchmarking framework with a decoupled design for feature extraction and decoding methods. The framework is comprised of different network architectures for feature extraction such as VGG16, Resnet18, MobileNet, and ShuffleNet. It is also comprised of multiple meta-architectures for segmentation that define the decoding methodology. These include SkipNet, UNet, and Dilation Frontend. Experimental results are presented on the Cityscapes dataset for urban scenes. The modular design allows novel architectures to emerge, that lead to 143x GFLOPs reduction in comparison to SegNet. This benchmarking framework is publicly available at "https://github.com/MSiam/TFSegmentation".

1. INTRODUCTION

Semantic-segmentation research has emphasized accuracy, while computational efficiency and principled comparisons of architectural modules have received less attention. RTSeg proposes a modular real-time benchmarking framework to compare feature extractors and decoding methods, including novel architectures with a reported 143x GFLOPs reduction versus SegNet.

1. INTRODUCTION

  • Although segmentation work had advanced accuracy and robustness, computational efficiency received little attention despite its relevance to applications such as autonomous driving.Source: §1 · p. 1
  • RTSeg decouples feature extraction from decoding so their effects on real-time performance can be compared separately.The paper presents this as a principled response to prior comparisons that evaluated networks as wholes rather than comparing modules.Source: §1 · p. 1
  • 143x GFLOPs reduction versus SegNet was reported for novel architectures enabled by the modular framework.The authors also report that lower operation counts did not yield higher inference speed.Source: §1 · p. 2

2. BENCHMARKING FRAMEWORK

The framework separates feature-extraction networks from decoding meta-architectures, allowing design choices to be compared. It includes three decoding approaches and four feature extractors, with a shared downsampling factor for the meta-architecture comparisons.

2.1. Meta-Architectures

  • The framework includes SkipNet, U-Net, and Dilation Frontend, all evaluated with a shared downsampling factor of 32.The common factor is intended to isolate the effect of decoding method.Source: §2.1 · p. 2
  • SkipNet upsamples class heatmaps in label space and combines them with higher-resolution heatmaps, making its decoding more computationally efficient than alternatives.Its design is similar to FCN8s.Source: §2.1 · p. 2
  • U-Net upsamples features at each downsampling stage and fuses them with encoder features at matching resolutions.The framework uses element-wise addition for fusion; concatenation can improve accuracy but increases computational cost.Source: §2.1 · p. 2
  • Dilation Frontend maintains feature-map resolution with dilated convolutions, but its larger-resolution operations increase computational cost.Its encoder uses a downsampling factor of 8 rather than 32, replacing pooling or strided convolutions with dilated convolutions of factors 2 and 4.Source: §2.1 · p. 2

2.2. Feature Extraction Architectures

  • The feature-extraction choices are VGG16, ResNet18, MobileNet, and ShuffleNet.VGG16 serves as a baseline; MobileNet uses depthwise and pointwise convolutions, while ShuffleNet uses grouped convolutions and channel shuffling.Source: §2.2 · p. 3

3. EXPERIMENTS

The experiments evaluate encoder-decoder combinations on Cityscapes and compare their accuracy and computational cost. Results distinguish decoder accuracy and efficiency, coarse-pretraining effects, and comparisons with state-of-the-art networks.

3.1. Experimental Setup

  • Experiments use Cityscapes images at 512x1024, weighted cross-entropy, Adam at 1e-4, L2 weight decay of 5e-4, and ImageNet-pretrained feature extractors.Cityscapes includes 5,000 fine-annotated images and 20,000 coarse-labeled images used for coarse pre-training.Source: §3.1 · p. 3

3.2. Semantic Segmentation Results

  • UNet has the best accuracy, while SkipNet uses 4x fewer GFLOPs; stage-wise UNet upsampling is more accurate than Dilation Frontend's final 8x upsampling.Transposed convolutions operate in feature space in UNet and label space in SkipNet.Source: §3.2 · pp. 3–4
  • Coarse pre-training improves overall mIoU by 1-4%, with underrepresented classes often benefiting.Source: §3.2 · p. 4
  • Up to 143x fewer GFLOPs than SegNet; SkipNet-ShuffleNet maintains on-par mIoU with ENet but does not outperform ENet in inference speed.DeepLab provides the best accuracy among the compared state-of-the-art networks, but is not computationally efficient.Source: §3.2 · p. 4, Table 3 · p. 4
  • SkipNet-MobileNet produces more accurate qualitative segmentation than the shown ResNet18 and ShuffleNet alternatives, correctly labeling the pedestrian and signs on the right.Source: §3.2 · p. 4, Figure 3 · p. 4

4. CONCLUSION

The conclusion presents a principled real-time segmentation benchmark that separates feature extraction from decoding. It includes multiple encoder and decoder architectures so researchers and practitioners can evaluate design choices for their tasks.

4. CONCLUSION

  • The framework decouples feature-extraction networks from decoding meta-architectures for quantitative comparison of their effects.Source: §4 · p. 4
  • It includes ShuffleNet, MobileNet, VGG16, and ResNet-18 feature extractors, alongside Skip, U-Net, and Dilation Frontend decoding methods.Source: §4 · p. 4
  • Researchers and practitioners can use the framework to evaluate design choices for their tasks.Source: §4 · p. 5

Related papers

Loading 1803.02758v5…