Source-linked AI summary
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation
Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, Nong Sang
TL;DR
Real-time semantic segmentation methods often sacrifice low-level details to accelerate inference, reducing accuracy. BiSeNet V2 separates spatial details from categorical semantics through bilateral branches and guided aggregation, achieving 72.6% Mean IoU at 156 FPS on Cityscapes with a 2,048 × 1,024 input.
Problem
Existing acceleration approaches restrict input size or prune channels, sacrificing low-level details and spatial capacity for faster inference.
Method
BiSeNet V2 uses wide, shallow Detail and narrow, deep Semantic branches, merges them with Guided Aggregation, and applies booster training without extra inference cost.
Results
72.6% Mean IoU at 156 FPS was achieved on the Cityscapes test set for a 2,048 × 1,024 input using one NVIDIA GeForce GTX 1080Ti card.
Takeaways & Limitations
BiSeNet V2 provides a reported trade-off between segmentation accuracy and inference speed for real-time semantic segmentation.
Takeaways & Limitations
The reported inference measurement uses FP32 data precision.
Abstract
from arXiv · showhide
The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048x1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy.
1 Introduction
Real-time semantic segmentation must preserve both spatial details and categorical semantics without sacrificing inference efficiency. BiSeNet V2 addresses this trade-off with separate detail and semantic pathways, feature aggregation, and booster training.
- Existing acceleration methods restrict input resolution or prune channels, sacrificing low-level details and spatial capacity and causing accuracy decreases.
- BiSeNet V2 separates spatial details and categorical semantics into a two-pathway architecture for real-time semantic segmentation.The Detail Branch uses wide channels and shallow layers, while the Semantic Branch uses narrow channels and deep layers.
- A Guided Aggregation Layer merges both feature types, while a booster training strategy improves segmentation performance without increasing inference cost.The booster strategy uses auxiliary prediction heads that can be discarded during inference.
- The Semantic Branch uses depth-wise convolutions, reduced channel capacity, and fast downsampling to capture contextual information with lightweight computation.
- 72.6% Mean IoU and 156 FPS were achieved on the Cityscapes test set for a 2,048 × 1,024 input using one NVIDIA GeForce GTX 1080Ti card.
2 Related Work
Prior semantic-segmentation backbones preserve details and semantics with wide, deep networks but often run slowly. Real-time methods accelerate inference through input restriction or channel pruning, frequently sacrificing low-level detail and accuracy; this paper instead separates details from semantics.
- Generic semantic segmentation: Generic segmentation methods use dilation or encoder-decoder backbones to preserve high-resolution representations while encoding low-level details and high-level semantics.Dilation backbones avoid downsampling, whereas encoder-decoder designs recover resolution through top-down and skip connections.
- Real-time semantic segmentation: Wide and deep backbones achieve state-of-the-art performance but commonly have slow inference, motivating architectures that balance segmentation accuracy and speed.
- Real-time semantic segmentation: Real-time methods mainly restrict input resolution or prune channels to reduce computation cost.
- Real-time semantic segmentation: These acceleration strategies can reach real-time speed but dramatically sacrifice accuracy through loss of low-level details.
- Lightweight architectures: Lightweight architectures use designs such as depth-wise and separable convolutions to trade classification speed against accuracy.
3 Core Concepts of BiSeNetV2
BiSeNet separates spatial details from categorical semantics in parallel branches, then merges their complementary representations through bidirectional aggregation. The design uses lightweight semantic processing and restores spatial alignment before fusion.
- Architecture: BiSeNet uses parallel Detail and Semantic Branches, merging their representations with an Aggregation Layer.
- Detail Branch: The Detail Branch uses wide channels and shallow layers to encode spatial details at relatively high resolution.Its output feature maps are 1/8 of the original input resolution.
- Semantic Branch: The Semantic Branch uses fewer channels than the Detail Branch to capture high-level semantics with a lightweight design.The instantiated channel ratio is λ = 1/4.
- Semantic Branch: Fast downsampling and global average pooling provide the Semantic Branch with efficient computation and global contextual information.
- Aggregation: Because the branches have different spatial dimensions, the Semantic output is upsampled before bidirectional aggregation with the Detail output.
4 Bilateral Segmentation Network
BiSeNet V2 instantiates its bilateral architecture with specialized lightweight components for detail extraction, semantic context, and guided fusion. Auxiliary booster heads improve training representations while being removed during inference.
- Detail Branch: The Detail Branch uses stacked convolutional layers with high channel capacity to encode spatial details while avoiding residual structures’ memory-access cost.Its three stages produce feature maps at 1/8 input resolution.
- Semantic Branch: The Semantic Branch combines a Stem Block, Context Embedding Block, and Gather-and-Expansion layers to obtain efficient high-level representations.The Stem Block downsamples through two branches, while the Context Embedding Block uses global average pooling.
- Semantic Branch: The Gather-and-Expansion Layer gathers features with 3×3 convolution, applies depth-wise convolution, and projects them through a 1×1 convolution.
- Guided Aggregation: Simple summation or concatenation can ignore the different representation levels of the two branches and lead to worse performance and harder optimization.
- Guided Aggregation: Bilateral Guided Aggregation uses Semantic Branch context to guide Detail Branch responses across scales, enabling efficient communication and multi-scale representation.
- Booster: The booster training strategy inserts auxiliary segmentation heads during training and discards them during inference, adding little inference complexity.
5 Experimental Results
Experiments evaluate BiSeNet V2 across datasets, ablations, architectural choices, and lightweight-model combinations. The results show complementary branches, effective semantic-branch designs, gains from booster training, and favourable accuracy–speed performance.
- Experimental Setup: Experiments cover Cityscapes, CamVid, and COCO-Stuff, using dataset-specific resolutions, splits, and evaluation metrics.Cityscapes uses 2,048 × 1,024 images and 19 segmentation classes; CamVid uses 960 × 720 images and 11 classes; COCO-Stuff uses 9K training and 1K testing images with 91 thing and 91 stuff classes.
- Ablative Evaluation: Both branches provide complementary representations: the Semantic Branch alone reaches 64.68% mean IoU, the Detail Branch 62.35%, and their combination improves both pathways.Simple summation lets the Semantic Branch improve the Detail Branch by over 6% and the Detail Branch improve the Semantic Branch by 4%.
- Ablative Evaluation: 6% improvement, from 62.35% to 68.27%, is obtained over the Detail-only baseline even when the Semantic Branch channel ratio is λ = 1/16.The experiment varies λ while keeping the last two Semantic Branch stages at 64 and 128 channels.
- Ablative Evaluation: 4% improvement, from 62.35% to 67.48% mean IoU, is achieved with expansion ratio ϵ = 1, supporting the efficiency of the lightweight Semantic Branch.The Gather-and-Expansion Layer enlarges the receptive field to capture high-level semantics efficiently.
- Ablative Evaluation: Over 3% mean IoU improvement, from 69.67% to 73.19%, is obtained through booster training without sacrificing inference speed.Auxiliary segmentation heads are inserted during training and discarded during inference; online bootstrapping is then applied for further improvement.
- Generalization Capability: 75.8% mIoU is achieved by BiSeNetV2-Large with α = 2.0 and d = 3.0, while the architecture remains compatible with other lightweight convolutional models.The compatibility experiments combine the bilateral architecture with different general lightweight models.
6 Concluding Remarks
BiSeNet V2 separates spatial details from categorical semantics through a generic bilateral architecture. Its instantiations achieve a favourable trade-off between segmentation accuracy and inference speed.
- Conclusion: BiSeNet V2 treats spatial details and categorical semantics separately in a generic architecture for semantic segmentation.The framework can be implemented with most convolutional models.
- Conclusion: The architecture achieves a good trade-off between segmentation accuracy and inference speed.This conclusion applies to the reported BiSeNet V2 instantiations.
- Conclusion: The authors present BiSeNet V2 as a basis for further research in semantic segmentation.The concluding passage states that the generic architecture is intended to foster further research.