PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes

Genshun Dong, Yan Yan, Chunhua Shen, Hanzi Wang

arXiv:2003.08736v2cs.CV

TL;DR

DCNN-based segmentation can be computationally costly, while real-time methods face an accuracy-speed trade-off. The paper combines lightweight feature extraction, multi-scale context, spatial-detail preservation, and feature fusion. It reports results on Cityscapes and CamVid using a single NVIDIA TITAN X.

  • Problem

    The paper addresses the challenge of balancing accuracy and speed in semantic segmentation of urban street scenes.

    Source: §I · p. 2

  • Method

    The method combines LBN-AA, DASPP, SPN, and FFN to extract features, capture multiple scales, preserve spatial detail, and fuse shallow and deep features.

    Source: §I · p. 2

  • Results

    73.6% mIoU at 51.0 fps on Cityscapes and 68.0% mIoU at 39.3 fps on CamVid are reported using one NVIDIA TITAN X.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that the method offers a good accuracy-speed balance for real-time semantic segmentation of urban street scenes.

    Source: §I · p. 2, §VI · pp. 14–15

Abstract

from arXiv · show

Deep Convolutional Neural Networks (DCNNs) have recently shown outstanding performance in semantic image segmentation. However, state-of-the-art DCNN-based semantic segmentation methods usually suffer from high computational complexity due to the use of complex network architectures. This greatly limits their applications in the real-world scenarios that require real-time processing. In this paper, we propose a real-time high-performance DCNN-based method for robust semantic segmentation of urban street scenes, which achieves a good trade-off between accuracy and speed. Specifically, a Lightweight Baseline Network with Atrous convolution and Attention (LBN-AA) is firstly used as our baseline network to efficiently obtain dense feature maps. Then, the Distinctive Atrous Spatial Pyramid Pooling (DASPP), which exploits the different sizes of pooling operations to encode the rich and distinctive semantic information, is developed to detect objects at multiple scales. Meanwhile, a Spatial detail-Preserving Network (SPN) with shallow convolutional layers is designed to generate high-resolution feature maps preserving the detailed spatial information. Finally, a simple but practical Feature Fusion Network (FFN) is used to effectively combine both shallow and deep features from the semantic branch (DASPP) and the spatial branch (SPN), respectively. Extensive experimental results show that the proposed method respectively achieves the accuracy of 73.6% and 68.0% mean Intersection over Union (mIoU) with the inference speed of 51.0 fps and 39.3 fps on the challenging Cityscapes and CamVid test datasets (by only using a single NVIDIA TITAN X card). This demonstrates that the proposed method offers excellent performance at the real-time speed for semantic segmentation of urban street scenes.

I. INTRODUCTION

Semantic segmentation assigns pixel-level labels and supports understanding urban street scenes. The introduction frames the accuracy-speed trade-off and presents a network designed to balance both.

I. INTRODUCTION

  • Semantic segmentation assigns labels pixel by pixel and supports urban-scene understanding in intelligent transportation systems.The introduction names automotive driving, vehicle safety, and video surveillance as applications,,,.Source: §I · p. 1
  • Representative real-time methods achieve full-resolution inference, but their mIoUs are below 60%, leaving accuracy and speed difficult to balance.The introduction also notes that complex, high-accuracy models can take more than one second to predict a 1024×2048 image on a high-end GPU.Source: §I · p. 2
  • LBN-AA provides dense, low-cost features; DASPP captures multiple scales, while SPN preserves spatial detail and FFN fuses shallow and deep features.The components are jointly optimized in an integrated network.Source: §I · p. 2
  • 73.6% mIoU at 51.0 fps on Cityscapes and 68.0% mIoU at 39.3 fps on CamVid are reported using one NVIDIA TITAN X.The authors describe this as a good balance between accuracy and speed for urban street-scene segmentation.Source: §I · p. 2

II. RELATED WORK

Prior work uses DCNN architectures and multi-scale context to improve segmentation, while real-time methods seek lower latency. The section describes continuing computational and accuracy challenges across these approaches.

A. High-performance Semantic Segmentation Methods

  • High-performance methods build on FCN, atrous convolution and ASPP, multi-scale aggregation, encoder-decoder structures, and boundary-refinement methods.FCN replaces the last fully connected layers with convolution layers; atrous convolution expands receptive fields, and ASPP captures objects and context at multiple scales.Source: A. High-performance Semantic… · pp. 2–3
  • High-performance methods can achieve benchmark accuracy but often require many parameters, large-scale calculations, or both.Source: A. High-performance Semantic… · p. 3

B. Real-time Semantic Segmentation Methods

  • Real-time approaches include ENet, ESPNet, ERFNet, and SegNet, but their segmentation accuracy is described as unsatisfactory.ESPNet and ERFNet are reported as faster and more accurate than ENet with a similar number of parameters.Source: B. Real-time Semantic Segmentation… · p. 3
  • ICNet performs well on high-resolution inputs but is not competitive at lower resolutions; LEDNet and BiSeNet pursue efficient designs, with room for further improvement.BiSeNet separates spatial and semantic paths and fuses their features.Source: B. Real-time Semantic Segmentation… · p. 3

C. Lightweight Classification Networks

Lightweight backbones target efficient street-scene segmentation, while atrous convolution and ASPP provide a way to encode multi-scale context. The section also describes how object scale varies within a category and how ASPP combines branch features.

C. Lightweight Classification Networks

  • MobileNetV2 combines depthwise separable convolutions with inverted residuals and linear bottlenecks, and the proposed segmentation method is based on it.The passage describes MobileNetV2 as more efficient and effective than MobileNetV1.Source: C. Lightweight Classification Networks · p. 3

III. PRELIMINARY KNOWLEDGE

  • Atrous convolution expands receptive fields and produces denser feature maps without increasing computation or parameter count.The accompanying illustration shows a 3 × 3 kernel at atrous rates 1, 2, and 3; rate 1 is standard convolution.Source: §III · p. 4, Figure 2 · p. 4
  • Objects of the same category can appear at different scales in street scenes because their distances from the camera differ.The Cityscapes illustration highlights this variation for humans and cars.Source: §III · p. 4, Figure 3 · p. 4
  • Diverse object scales make it challenging to extract and encode multi-scale information correctly.The passage attributes within-category size variation to differing distances from the camera.Source: §III · p. 4
  • ASPP uses parallel atrous-convolution branches with different rates to cover multiple receptive-field scales, while multi-size image inputs require more computation.The example combines outputs from four branches that receive the same input.Source: §III · p. 4, Figure 4 · p. 4
  • ASPP takes input X and forms output Y by concatenating feature maps generated by atrous convolutions at different rates.The passage introduces P_k,d(X) for a branch convolution, with k × k denoting kernel size and d the atrous rate.Source: §III · p. 4

IV. THE PROPOSED METHOD

The proposed segmentation method is organized as four components. Figure 5 presents the overall framework and identifies each component.

IV. THE PROPOSED METHOD

  • The method overview is introduced in Section IV-A, and its components are described in Sections IV-B through IV-E.Source: §IV · p. 4
  • The method comprises LBN-AA, DASPP, SPN, and FFN.Their full names are Lightweight Baseline Network with Atrous Convolution and Attention, Distinctive ASPP, Spatial detail-Preserving Network, and Feature Fusion Network.Source: Figure 5 · p. 5
  • Figure 5 presents the overall framework, with panels (a)–(d) corresponding to LBN-AA, DASPP, SPN, and FFN.The caption identifies each panel by component name.Source: Figure 5 · p. 5

A. Overview

The method combines a semantic path based on LBN-AA and DASPP with a spatial-detail path based on SPN, then fuses them through FFN. LBN-AA modifies MobileNetV2 to retain larger feature maps while using atrous convolution and attention.

A. Overview

  • LBN-AA+DASPP supplies semantic features while SPN supplies spatial details; FFN fuses both paths before bilinear upsampling to the input size.The semantic and spatial paths are computed simultaneously.Source: A. Overview · p. 5

B. Lightweight Baseline Network with Atrous Convolution and Attention (LBN-AA)

  • LBN-AA modifies ImageNet-pretrained MobileNetV2 to retain feature maps at 1/8 of the input resolution for segmentation.The original MobileNetV2 output is only 1/32 of the input size, which would lose spatial detail.Source: B. Lightweight Baseline Network with… · pp. 5–6
  • The simplified MobileNetV2 and atrous convolution combine larger receptive fields with dense feature maps while retaining a balance between accuracy and speed.Atrous convolution increases computational time, while the simplified backbone is extremely lightweight.Source: B. Lightweight Baseline Network with… · pp. 5–6
  • Convolutional Attention Modules weight feature-map channels to emphasize informative information and suppress irrelevant information in the lightweight network.CAM uses global average pooling and a 1 × 1 convolution to encode feature importance before producing channel weights.Source: B. Lightweight Baseline Network with… · p. 6
  • Different atrous rates across LBN-AA blocks 4–7, followed by dense concatenation, provide dense sampling and alleviate gridding while adding little computational burden.The rates are 2, 4, 8, and 16; concatenation combines information from multiple layers.Source: B. Lightweight Baseline Network with… · p. 6

C. Distinctive Atrous Spatial Pyramid Pooling (DASPP)

DASPP adapts parallel atrous convolution branches with differently pooled inputs to produce distinctive, informative features for multi-scale segmentation. Its design also uses a shortcut connection and differs from vortex pooling in how branch outputs are combined.

C. Distinctive Atrous Spatial Pyramid Pooling (DASPP)

  • ASPP branches share identical inputs and sample only distant pixels per atrous operation, motivating DASPP’s use of diverse pooled inputs and neighboring information.The paper identifies less distinctive features and incomplete exploitation of input pixels as shortcomings of ASPP.Source: C. Distinctive Atrous Spatial Pyramid… · p. 7
  • DASPP uses 3 × 3, 5 × 5, and 7 × 7 pooling with atrous rates 12, 24, and 36, respectively, and retains image-level pooling for global context.The original ASPP 1 × 1 branch is replaced with 1 × 1 and 3 × 3 convolution operations.Source: C. Distinctive Atrous Spatial Pyramid… · p. 7
  • The atrous rates must be chosen with input size in mind, because large rates on small images can leave pixels unused and increase receptive fields at the cost of spatial information.The paper uses the atrous rates from DeepLabv2.Source: C. Distinctive Atrous Spatial Pyramid… · p. 7
  • DASPP pools inputs at different scales before parallel atrous convolutions, producing more distinctive and informative features than ASPP by using neighboring information.Different pooling sizes give each branch a different input, while the neighboring pixels of an input are fully used.Source: C. Distinctive Atrous Spatial Pyramid… · p. 7
  • DASPP concatenates outputs from its four atrous branches before adding the shortcut, whereas vortex pooling directly adds branch outputs.The paper also describes the methods as having different motivations and configurations.Source: C. Distinctive Atrous Spatial Pyramid… · pp. 7–8

D. Spatial detail-Preserving Network (SPN)

SPN complements the semantic features from LBN-AA and DASPP with high-resolution features that preserve spatial detail. It uses shallow pretrained convolutional layers to limit computation.

D. Spatial detail-Preserving Network (SPN)

  • SPN generates high-resolution feature maps that preserve spatial detail and refine the semantic output from DASPP.The paper frames this as complementing the semantic branch, whose spatial information is only partially preserved.Source: D. Spatial detail-Preserving Network… · p. 8
  • SPN uses the first two layers of compact, ImageNet-pretrained ResNet-18 and concatenates layer1 with block2 from LBN-AA.Its output is 1/4 of the input resolution with 88 channels, while shallow convolutions keep computational cost small.Source: D. Spatial detail-Preserving Network… · p. 8, Table II · p. 8
  • The paper says SPN’s pretrained weights support low-level feature extraction, easier convergence, and improved accuracy after fine-tuning.This is presented as an advantage over training the network from scratch.Source: D. Spatial detail-Preserving Network… · p. 8

E. Feature Fusion Network (FFN)

FFN fuses deep semantic and shallow spatial features through a sequence of normalization and convolutions. The experiments use Cityscapes and CamVid, with mIoU and fps evaluating accuracy and speed.

E. Feature Fusion Network (FFN)

  • FFN addresses the different representation levels of deep semantic and shallow spatial features, for which simple addition or channel concatenation may decrease accuracy.The two feature sources come from LBN-AA+DASPP and SPN, respectively.Source: E. Feature Fusion Network (FFN) · p. 8
  • FFN concatenates the two branches, applies batch normalization and a 3 × 3 atrous convolution at rate 2, then projects to 19 semantic categories and upsamples bilinearly.The projection reduces channels from 216 to 19; batch normalization also appears between the atrous and projection convolutions.Source: E. Feature Fusion Network (FFN) · p. 8

A. Datasets and Evaluation Metrics

  • The evaluation uses Cityscapes and CamVid; Cityscapes includes 19 common categories, while CamVid testing evaluates 11 common categories.The datasets contain 5,000 high-resolution Cityscapes images and 701 CamVid images, respectively.Source: A. Datasets and Evaluation Metrics · p. 9
  • The experiments use mIoU and fps to evaluate accuracy and speed, with parameters and FLOPs measuring memory consumption and computational complexity.FPS is measured for single forward propagation.Source: A. Datasets and Evaluation Metrics · p. 9

B. Training Protocol

The protocol combines pretrained initialization and standard augmentation with a staged ablation of LBN-AA. LBN-AA improves accuracy and speed over FCN-8s+VGG16 on Cityscapes validation.

B. Training Protocol

  • Cityscapes images are resized to 512 × 1024 for training and 448 × 896 for testing, while CamVid images remain unchanged.Training uses ImageNet-pretrained MobileNetV2, horizontal flips, random scaling and cropping, SGD, and a poly learning-rate schedule; models run for 200 epochs on one NVIDIA TITAN X.Source: B. Training Protocol · p. 9

1) Ablation For LBN-AA:

  • About 4% higher mIoU and roughly 3 times faster than FCN-8s+VGG16, LBN-AA improves both accuracy and inference speed.The comparison uses the Cityscapes validation dataset.Source: Table III · p. 10, 1) Ablation For LBN-AA · p. 10
  • LBN-AA is a modified MobileNetV2 with atrous convolution and attention; its CAM yields about 2% mIoU improvement over LBN-A, while LBN-AA outperforms the SE-block variant.Qualitative examples report more detailed truck, road, and human segmentation than other baseline networks.Source: 1) Ablation For LBN-AA · p. 10, Figure 9 · p. 10

2) Ablation For DASPP:

The ablations show that DASPP’s average-pooling variant improves Cityscapes validation accuracy over ASPP and max pooling, while SPN adds spatial detail and further improves accuracy.

2) Ablation For DASPP:

  • 72.2% mIoU with average pooling, versus 71.3% with max pooling and 70.8% with ASPP, is reported for the DASPP comparison.DASPP slightly decreases computational speed compared with ASPP.Source: Table IV · p. 11, 2) Ablation For DASPP · p. 11

3) Ablation For SPN:

  • 1.4% mIoU improvement with only a slight speed decrease, adding SPN to LBN-AA+DASPP preserves spatial details such as small signs and distant objects.SPN’s spatial information complements the semantic information from the deeper network, which loses spatial detail.Source: Table V · p. 11, 3) Ablation For SPN · p. 12

2) Ablation For DASPP:

  • About 1% higher mIoU at the same inference speed, DASPP with average pooling outperforms the sum variant.DASPP concatenates the four atrous-branch outputs before adding the shortcut; the comparison with vortex pooling also favors DASPP in accuracy and speed.Source: 2) Ablation For DASPP · p. 11

4) Ablation For FFN:

FFN improves the accuracy of feature fusion while retaining real-time inference. On Cityscapes test, the proposed method reports 73.6% mIoU at 51.0 fps and compares favorably with many competing methods.

4) Ablation For FFN:

  • FFN improves accuracy by 0.8% mIoU over element-wise addition, which is more efficient, while the proposed fusion remains real-time.FFN fuses deep semantic and shallow spatial features from different levels.Source: 4) Ablation For FFN · p. 12

D. Comparison with State-of-the-art Methods

  • About 16% higher mIoU than ENet and about 200 times faster and 10% higher mIoU than DeepLabv2, the method offers a speed–accuracy tradeoff across competitors.It is slightly slower than LEDNet and DFANet, whose mIoU is more than 2% lower; it is about eight times smaller than BiSeNet2 by parameter count.Source: D. Comparison with State-of-the-art… · p. 12
  • The qualitative results include over-segmentation within objects and under-segmentation between them, with differing object appearance offered as a possible reason.The authors note that this issue also occurs in other methods, including ICNet.Source: D. Comparison with State-of-the-art… · pp. 12–13
  • 73.6% mIoU at 51.0 fps, with 49.5G FLOPs and 6.2M parameters, is reported for the proposed method on Cityscapes test.Competing methods in the comparison were evaluated using a single NVIDIA TITAN X card.Source: D. Comparison with State-of-the-art… · p. 12, Table VIII · p. 13
  • The proposed method achieves the best accuracy by significant margins on most Cityscapes test classes while remaining faster than the other competing methods.Table IX reports per-class, class, and category IoU results.Source: D. Comparison with State-of-the-art… · p. 12, Table IX · p. 13

E. Comparison on the CamVid Dataset

On CamVid, the proposed method is comparable to BiSeNet in accuracy and improves on DFANet’s accuracy, with small-object segmentation highlighted as a strength. The conclusion describes four jointly optimized components for balancing accuracy and speed.

E. Comparison on the CamVid Dataset

  • On CamVid, the method is comparable to BiSeNet in accuracy and more accurate than DFANet, while segmenting small signs and poles more accurately than other methods.The authors report worse accuracy than competing methods for the tree class, attributing it to appearance similarity with the background and over-segmentation.Source: Table X · p. 13, E. Comparison on the CamVid Dataset · p. 13

VI. CONCLUSION

  • The method combines LBN-AA for dense feature extraction, DASPP for multi-scale context, SPN for spatial detail, and FFN for high-/low-level feature fusion.The conclusion says these components are tightly coupled and jointly optimized; results on Cityscapes and CamVid demonstrate effectiveness and efficiency.Source: §VI · pp. 14–15

Related papers

Loading 2003.08736v2…