Source-linked AI summary
Fast-SCNN: Fast Semantic Segmentation Network
Rudra P K Poudel, Stephan Liwicki, Roberto Cipolla
TL;DR
Real-time semantic segmentation is needed for responsive autonomous systems and remains challenging on resource-constrained devices. Fast-SCNN combines shared shallow feature extraction with low-resolution context learning, achieving 68.0% mIoU at 123.5 fps on full-resolution Cityscapes while showing limited benefit from pre-training and competitive faster processing on subsampled inputs.
Problem
Real-time semantic segmentation remains challenging, particularly for responsive autonomous systems and embedded devices without powerful GPUs.
Method
Fast-SCNN combines a two-branch encoder-decoder with a shallow learning to downsample module that shares low-level computations across resolutions, using efficient convolutions.
Results
68.0% mIoU at 123.5 fps on full-resolution Cityscapes, with only a +0.5% mIoU gain from pre-training or additional coarse labels and competitive results on subsampled inputs without redesign.
Takeaways & Limitations
Fast-SCNN provides above-real-time high-resolution segmentation suited to efficient computation, while long training, augmentation, and additional data can reduce reliance on large-scale pre-training.
Takeaways & Limitations
On the low-capacity Fast-SCNN, the experiments show that ImageNet pre-training is not known to be necessary and provides no significant benefit, with augmentation and longer training producing similar results.
Abstract
from arXiv · showhide
The encoder-decoder framework is state-of-the-art for offline semantic image segmentation. Since the rise in autonomous systems, real-time computation is increasingly desirable. In this paper, we introduce fast segmentation convolutional neural network (Fast-SCNN), an above real-time semantic segmentation model on high resolution image data (1024x2048px) suited to efficient computation on embedded devices with low memory. Building on existing two-branch methods for fast segmentation, we introduce our `learning to downsample' module which computes low-level features for multiple resolution branches simultaneously. Our network combines spatial detail at high resolution with deep features extracted at lower resolution, yielding an accuracy of 68.0% mean intersection over union at 123.5 frames per second on Cityscapes. We also show that large scale pre-training is unnecessary. We thoroughly validate our metric in experiments with ImageNet pre-training and the coarse labeled data of Cityscapes. Finally, we show even faster computation with competitive results on subsampled inputs, without any network modifications.
1. Introduction
Fast-SCNN targets above-real-time semantic segmentation for high-resolution imagery and embedded devices by sharing computations across resolution branches. It combines spatial detail with low-resolution context while maintaining low capacity and competitive Cityscapes performance.
- Motivation: Fast-SCNN targets responsive semantic segmentation for autonomous systems, robotics, and embedded applications where faster-than-real-time preprocessing is valuable.The introduction also motivates applications such as augmented reality for wearables.
- Approach: Fast-SCNN shares initial computations between two branches through a learning to downsample module that extracts low-level features simultaneously.The design merges a two-branch setup with the encoder-decoder framework and uses a single shallow skip connection for efficiency.
- Results: 68.0% mIoU at 123.5 fps on full 1024×2048px Cityscapes images demonstrates above-real-time high-resolution segmentation.The reported speed is twice that of prior art, BiSeNet, which achieves 71.4% mIoU.
- Efficiency: 1.11 million parameters keep Fast-SCNN’s model capacity low to reduce memory requirements and support execution on embedded devices.The authors also expect lower capacity to improve generalisation.
- Training: +0.5% mIoU is the reported improvement from ImageNet pre-training or additional coarse Cityscapes data, indicating limited benefit for this low-capacity network.The paper reports that longer training can be equivalently successful to these additions.
- Results: Subsampled inputs provide even faster computation with competitive results without requiring network redesign.This extends the model’s speed-efficiency trade-off beyond full-resolution inference.
2. Related Work
Prior segmentation systems balance global context, spatial detail, speed, and resource use through encoder-decoder or multi-branch designs. Fast-SCNN builds on these approaches by sharing feature extraction across resolution levels and emphasizing efficient operators.
- Encoder-decoder methods: Encoder-decoder architectures extract sub-resolution features with an encoder and recover spatial detail with a decoder, often using skip connections.FCN uses bilinear upsampling and lower-layer skip connections, while U-Net adds dense skip connections.
- Multi-branch methods: Two-branch and multi-branch systems learn global context at reduced resolution and refine boundaries with a shallow full-resolution branch.ICNet, ContextNet, BiSeNet, and GUN follow this pattern.
- Efficiency challenge: Real-time semantic segmentation remains challenging and typically requires high-end GPUs, motivating Fast-SCNN’s shared shallow path and low-resolution context processing.The architecture is designed around low energy and memory requirements.
- Efficient operators: Depthwise separable convolutions reduce floating-point operations, parameters, computational cost, and memory requirements by factorizing standard convolution.The factorization combines a depthwise convolution with a 1×1 pointwise convolution.
- Fast-SCNN: Fast-SCNN relies on depthwise separable convolutions and residual bottleneck blocks while sharing feature extraction across multiple resolution levels.Its learning to downsample module leverages similarities in early-layer features that common two-branch approaches do not exploit.
- Pre-training: The paper reports that low-capacity networks receive no significant benefit from auxiliary-task pre-training; augmentation and longer training provide similar results.This finding is evaluated against ImageNet pre-training and additional coarse data.
3. Proposed Fast-SCNN
Fast-SCNN combines two-branch and encoder-decoder ideas through a learning to downsample module and a compact sequence of efficient segmentation components.
- Architecture: Table 1 organizes Fast-SCNN into learning to down-sample, global feature extractor, feature fusion, and classifier modules.It lists Conv2D, DSConv, bottleneck blocks, PPM, and FFM components, with parameters t, c, n, and s describing bottleneck structure.
- Building blocks: The model uses standard and depthwise separable convolutions, inverted residual bottleneck blocks, pyramid pooling, and feature fusion to construct the segmentation network.These components are identified as the principal building blocks in the architecture description.
- Architecture: Fast-SCNN reinterprets skip connections as a learning to downsample module that merges low-level detail with the paper’s multi-resolution processing strategy.The architecture layout is presented in Figure 1 and Table 1.
3.1. Motivation
Fast-SCNN is motivated by the complementary strengths of two-branch segmentation: low-resolution processing captures global information, while shallow full-resolution processing preserves precision. Its learning to downsample module shares feature computation between these branches.
- Two-branch motivation: Two-branch networks use low-resolution deep branches for global information and shallow full-resolution branches to refine segmentation precision.Input resolution and network depth are identified as major runtime factors enabling this design’s efficiency.
- Learning to downsample: Learning to downsample shares shallow feature computation between low- and high-resolution branches because early DCNN layers extract common low-level features.The paper gives edges and corners as examples of these features.
3.2. Network Architecture
Fast-SCNN combines shared low-level multi-resolution extraction with efficient global-context and feature-fusion modules. Its architecture uses depthwise separable convolutions, residual bottlenecks, and lightweight classification choices.
- Fast-SCNN comprises learning to downsample, global feature extractor, feature fusion, and classifier modules built with depthwise separable convolutions.
- Learning to Downsample: The learning to downsample module uses three layers: one standard convolution followed by two depthwise separable convolutions.All three layers use stride 2, batch normalization, and ReLU.
- Global Feature Extractor: The global feature extractor captures context from features at 1/8 of the original input resolution using bottleneck residual blocks and pyramid pooling.Depthwise separable convolutions reduce parameters and floating-point operations.
- Feature Fusion Module: The feature fusion module uses simple feature addition to preserve efficiency, while more sophisticated fusion could improve accuracy at a runtime cost.
- Classifier: During inference, Fast-SCNN can use argmax for object labels or softmax for probabilistic outputs instead of always computing softmax.
3.3. Comparison with Prior Art
Fast-SCNN retains the two-branch division between spatial detail and low-resolution context while sharing early computation between branches. It also uses a single early skip connection rather than the multiple or dense skips common in encoder-decoder models.
- Relation with Two-branch Models: Fast-SCNN’s learning to downsample module corresponds to the shallow spatial branch, while its global feature extractor corresponds to the deeper low-resolution branch.
- Relation with Two-branch Models: Sharing the first layers reduces feature-extraction complexity and allows global feature extraction at 1/8 rather than 1/4 resolution.
- Relation with Encoder-Decoder Models: Fast-SCNN uses a single skip connection to reduce computation and memory, unlike the multiple skips in FCN and dense skips in U-Net.
- Relation with Encoder-Decoder Models: The skip connection is positioned early because early-layer features are considered suitable for sharing across resolutions.
4. Experiments
Experiments evaluate Fast-SCNN on Cityscapes across accuracy, runtime, architectural effects, training strategies, and input resolutions. The model delivers high-speed full-resolution segmentation, benefits from its skip connection, and remains competitive on subsampled inputs.
- Cityscapes Evaluation: Fast-SCNN reaches 68.0% mIoU on the Cityscapes test set, compared with 71.5% for BiSeNet, 70.4% for GUN, and 66.1% for ContextNet.
- Ablation and Qualitative Results: Fast-SCNN benefits from its skip connection particularly around object boundaries and for small objects.
- Pre-training and Weakly Labeled Data: ImageNet pre-training yields 69.15% validation mIoU, only 0.53% above Fast-SCNN without pre-training.The authors conclude that ImageNet pre-training provides no significant boost for this low-capacity network.
5. Conclusions
Fast-SCNN provides above-real-time scene understanding by sharing computation across a multi-branch network. Experiments show that its skip connection recovers spatial detail, while extended training can replace large-scale auxiliary pre-training for the low-capacity model.
- Fast-SCNN achieves runtime efficiency by sharing computation across its multi-branch architecture.
- The skip connection benefits recovery of spatial details in segmentation results.
- Training the low-capacity model long enough makes large-scale auxiliary-task pre-training unnecessary.