Source-linked AI summary
Rethinking BiSeNet For Real-time Semantic Segmentation
Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei, Zhenhua Chai, Junfeng Luo, Xiaolin Wei
TL;DR
Real-time segmentation is constrained by the cost of extra spatial paths and the mismatch between classification backbones and segmentation needs. The paper proposes STDC networks with detail-guided single-stream decoding, achieving strong Cityscapes speed-accuracy results.
Problem
BiSeNet’s extra spatial path is time-consuming, while classification backbones may be inefficient for segmentation because they lack task-specific design.
Method
STDC networks remove structural redundancy through gradually reduced feature dimensions and aggregation, while Detail Aggregation learns spatial information in low-level layers without an extra path.
Results
71.9% mIoU is achieved at 250.4 FPS on the Cityscapes test set, and 76.8% mIoU at 97.0 FPS for higher-resolution images.
Takeaways & Limitations
STDC-Seg achieves a state-of-the-art speed-accuracy trade-off for real-time semantic segmentation.
Abstract
from arXiv · showhide
BiSeNet has been proved to be a popular two-stream network for real-time segmentation. However, its principle of adding an extra path to encode spatial information is time-consuming, and the backbones borrowed from pretrained tasks, e.g., image classification, may be inefficient for image segmentation due to the deficiency of task-specific design. To handle these problems, we propose a novel and efficient structure named Short-Term Dense Concatenate network (STDC network) by removing structure redundancy. Specifically, we gradually reduce the dimension of feature maps and use the aggregation of them for image representation, which forms the basic module of STDC network. In the decoder, we propose a Detail Aggregation module by integrating the learning of spatial information into low-level layers in single-stream manner. Finally, the low-level features and deep features are fused to predict the final segmentation results. Extensive experiments on Cityscapes and CamVid dataset demonstrate the effectiveness of our method by achieving promising trade-off between segmentation accuracy and inference speed. On Cityscapes, we achieve 71.9% mIoU on the test set with a speed of 250.4 FPS on NVIDIA GTX 1080Ti, which is 45.2% faster than the latest methods, and achieve 76.8% mIoU with 97.0 FPS while inferring on higher resolution images.
1. Introduction
The paper targets efficient real-time semantic segmentation by replacing redundant two-stream designs and classification-oriented backbones with task-specific STDC networks and detail guidance. Experiments report competitive accuracy-speed trade-offs on Cityscapes.
- Real-time segmentation methods seek low-latency inference with satisfactory accuracy for applications including autonomous driving, surveillance, and robot sensing.
- Lightweight classification backbones may lack task-specific design for segmentation, while extra spatial paths add inference cost.
- The proposed STDC network is designed for faster inference, an explainable structure, and competitive segmentation performance.
- The STDC module concatenates responses from continuous layers to form multi-scale representations while gradually reducing filter sizes with negligible segmentation-performance loss.
- Detail Aggregation guides low-level layers to learn spatial details, replacing an extra path without additional inference-time computation.
- 71.9% mIoU is achieved at 250.4 FPS on the Cityscapes test set, while 76.8% mIoU is achieved at 97.0 FPS with higher-resolution images.
2. Related Work
Real-time segmentation research commonly pursues efficiency through lightweight backbones or multi-branch architectures. The paper situates STDC among these approaches while emphasizing a single-stream architecture designed to reduce structural redundancy.
- The STDC architecture uses a Conv-BN-ReLU-based network structure with STDC modules and a stride-2 variant.
- Lightweight-backbone methods reduce computation and use feature aggregation or pruning to recover segmentation performance.
- Multi-branch methods use strategies such as multi-scale image cascades or spatial paths to obtain speed-accuracy trade-offs.
- Classification-oriented backbones and BiSeNet’s two-stream architecture may be inefficient because of structural redundancy.
3. Proposed Method
The proposed STDC network removes redundancy in real-time segmentation through a multi-scale STDC module and single-stream detail guidance. Its decoder guides low-level features with detail information, then fuses them with deep context features for segmentation.
- Motivation: The method targets redundancy in classification backbones and two-stream designs, which may be inefficient for segmentation and make low-level feature extraction time-consuming.The proposed network is designed for faster inference, an explainable structure, and competitive segmentation performance.
- STDC Module: The first STDC block uses a 1 × 1 kernel, later blocks use 3 × 3 kernels, and filter numbers decrease geometrically across blocks.Only Block2 performs down-sampling; average pooling aligns block outputs before fusion.
- STDC Module: STDC modules concatenate feature maps from multiple blocks to represent scalable receptive fields and multi-scale information.The blocks use progressively reduced filter numbers, and their responses are aligned spatially before concatenation.
- STDC Network: The STDC network uses six stages, with Stages 1–5 progressively down-sampling features and Stage 6 producing prediction logits.Stages 1 and 2 use one convolutional block each, while Stages 3–5 contain repeated STDC modules.
- STDC Network: The encoder uses pretrained STDC features with BiSeNet’s context path, global average pooling, and U-shaped up-sampling to combine multi-scale context.Stages 3, 4, and 5 provide feature maps at down-sample ratios 1/8, 1/16, and 1/32.
- Detail Guidance: Detail Guidance replaces BiSeNet’s extra spatial path by training low-level layers to encode spatial details in a single-stream decoder.Detail maps are generated from segmentation ground truth, optimized with binary cross-entropy and Dice losses, and the auxiliary branch is discarded during inference.
4. Experimental Results
Experiments across ImageNet, Cityscapes, and CamVid evaluate STDC backbones, detail guidance, and segmentation speed–accuracy trade-offs. The ablations and benchmark comparisons report strong accuracy with high inference speed.
- Benchmarks and evaluation: Experiments evaluate the proposed methods on ImageNet, Cityscapes, and CamVid using accuracy and speed comparisons.The study also examines component effectiveness through ablation experiments.
- Ablation study: Ablations show that four STDC2 blocks provide the best performance, while additional blocks offer small benefits and reduce parallel-calculation efficiency.The authors therefore set the block number to four in STDC1 and STDC2.
- Detail guidance: Detail Guidance improves Cityscapes mIoU without harming inference speed, whereas adding Spatial Path improves accuracy but increases computation cost.The Detail Aggregation module achieves the highest mIoU when aggregating 1x, 2x, and 4x detail features.
- ImageNet results: 4.1% higher top-1 classification accuracy is reported for STDC1 than the baseline on the ImageNet validation set.STDC2 also reports 83.7% higher FPS than EfficientNet-B0 with competitive classification results.
- Cityscapes results: 71.9% mIoU at 250.4 FPS is achieved by STDC1-Seg50 on the Cityscapes test set, reported as over 45.2% faster than the runner-up.STDC2-Seg50 reaches 73.4% mIoU at 188.6 FPS, while STDC2-Seg75 reaches 76.8% test mIoU at 97.0 FPS.
- CamVid results: 73.0% mIoU at 197.6 FPS is achieved by STDC1-Seg on CamVid with 720 × 960 input, described as a state-of-the-art speed–accuracy trade-off.
5. Conclusions
The paper revisits BiSeNet for structure optimization and proposes STDC-based, detail-guided segmentation networks. The resulting STDC-Seg achieves a state-of-the-art speed–accuracy trade-off in real-time semantic segmentation.
- Conclusions: STDC replaces redundant BiSeNet structure with scalable-receptive-field, multi-scale feature extraction.
- Conclusions: Detail-guided STDC-Seg combines the STDC backbone with detail guidance to deliver competitive accuracy at high FPS.
- Future work: Future work will validate the backbone on more tasks and further explore spatial boundary utilization in semantic segmentation.