Source-linked AI summary
BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang
TL;DR
Real-time semantic segmentation must retain spatial detail while obtaining a large receptive field, but common acceleration strategies compromise these demands. BiSeNet separates them into Spatial and Context Paths, then fuses and refines their features; it reports 68.4% Mean IOU at 105 FPS on Cityscapes.
Problem
Real-time segmentation methods speed inference by reducing input size, pruning channels, or dropping stages, which compromises spatial information and makes simultaneous spatial-detail preservation and large receptive fields difficult.
Method
BiSeNet uses a Spatial Path for rich spatial information and a Context Path with lightweight downsampling and global average pooling for a large receptive field, with feature fusion and refinement modules.
Results
68.4% Mean IOU is achieved on the Cityscapes test dataset at 105 FPS.
Takeaways & Limitations
BiSeNet balances spatial detail, receptive field, inference speed, and segmentation accuracy across Cityscapes, CamVid, and COCO-Stuff benchmarks.
Takeaways & Limitations
The U-shape structure cannot easily recover spatial information lost through pruning or cropping and adds computation on high-resolution feature maps.
Abstract
from arXiv · showhide
Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new Feature Fusion Module to combine features efficiently. The proposed architecture makes a right balance between the speed and segmentation performance on Cityscapes, CamVid, and COCO-Stuff datasets. Specifically, for a 2048x1024 input, we achieve 68.4% Mean IOU on the Cityscapes test dataset with speed of 105 FPS on one NVIDIA Titan XP card, which is significantly faster than the existing methods with comparable performance.
1 Introduction
BiSeNet addresses the speed–accuracy dilemma in real-time semantic segmentation by separating spatial-information preservation from receptive-field enlargement into two paths. It combines these paths with additional refinement modules and reports 68.4% Mean IOU at 105 FPS on Cityscapes.
- Motivation: Real-time segmentation methods accelerate inference through input cropping or resizing, channel pruning, or dropping stages, but these choices damage spatial detail or weaken spatial capacity.Input resizing particularly harms predictions around boundaries, while dropping stages pursues an extremely tight framework.
- Motivation: The U-shape structure recovers some missing spatial information but adds computation on high-resolution feature maps and cannot easily restore details lost through pruning or cropping.The paper characterizes U-shape as a relief rather than an essential solution.
- Proposed approach: Feature Fusion and Attention Refinement Modules further improve accuracy while maintaining an acceptable computational cost.The paper evaluates these components on Cityscapes, CamVid, and COCO-Stuff benchmarks.
- Results: 68.4% Mean IOU is achieved on the Cityscapes test dataset at 105 FPS.This result is reported for the real-time segmentation setting described by the paper.
2 Related Work
Related work targets semantic segmentation’s need for spatial detail and contextual receptive fields through dilated convolutions, pyramid or large-kernel modules, U-shape structures, and lightweight real-time designs. These approaches motivate BiSeNet’s focus on balancing spatial information, receptive field, and inference speed.
- Spatial and context information: Dilated convolution and large kernels are used to preserve feature-map resolution or enlarge the receptive field.Examples include DUC, PSPNet, DeepLab, and Global Convolution Network.
- U-Shape method: U-shape methods recover some spatial information by combining features from different levels through skip connections or refinement structures.FCN, U-net, deconvolution-based methods, and related architectures use this design pattern.
- Context information: Context-aware segmentation methods capture diverse receptive fields using dilation, image pyramids, ASPP, or pyramid pooling.These methods fuse contextual information across scales.
- Attention mechanism: Attention mechanisms use high-level information or global context to guide feature learning and revise representations.Prior work applies attention across input scales, channels, or learned global context.
- Real-time segmentation: Real-time methods pursue fast prediction with small networks, skip connections, image cascades, or computation reduction in easy regions.SegNet, E-Net, ICNet, and cascade-based approaches exemplify these strategies.
3 Bilateral Segmentation Network
BiSeNet addresses the tension between spatial detail and receptive field with separate Spatial and Context Paths, then combines them for real-time prediction. Its architecture adds ARM refinement, FFM fusion, and auxiliary training supervision.
- Spatial Path: The Spatial Path stacks three stride-2 convolutional layers and produces feature maps at 1/8 of the original resolution.Its relatively large feature-map size is intended to retain rich spatial information.
- Context Path: The Context Path combines a lightweight, rapidly downsampling model with global average pooling to obtain a large receptive field efficiently.It also combines upsampled global-pooling output with lightweight-model features and uses an incomplete U-shape across the last two stages.
- Attention Refinement Module: The Attention Refinement Module uses global average pooling and an attention vector to refine Context Path stages with negligible computation.The module integrates global context without an up-sampling operation.
- Bilateral Segmentation Network: BiSeNet uses a Spatial Path and Context Path to address spatial-information loss and limited receptive field, respectively.The Spatial Path preserves detail, while the Context Path supplies broader semantic context.
- Network architecture: BiSeNet fuses the two path outputs for final prediction, while auxiliary Softmax losses supervise Context Path outputs during training.The principal loss supervises the whole network, and the auxiliary losses are weighted through α.
- Feature Fusion Module: The Feature Fusion Module concatenates Spatial and Context Path outputs, balances their scales, and re-weights features for selection and combination.This addresses their different representation levels: low-level detail from the Spatial Path and high-level context from the Context Path.
4 Experimental Results
Experiments evaluate BiSeNet’s implementation, component effects, speed, and accuracy across Cityscapes, CamVid, and COCO-Stuff. The results examine how its paths and modules balance segmentation quality with efficient inference.
- Implementation: BiSeNet uses three convolutions for the Spatial Path, Xception39 for the Context Path, and Feature Fusion Module to produce predictions at 1/8 resolution.The Spatial Path and final prediction both have 1/8 of the original image resolution.
- Ablation study: BiSeNet compares U-shape variants, component combinations, and baseline configurations using accuracy, parameters, FLOPS, and speed measurements.Table 3 evaluates CP, SP, global average pooling, ARM, and FFM; speed tables use an NVIDIA Titan XP card and varying image sizes.
- Ablation study: 66.01% to 67.42%: adding the Spatial Path improves Cityscapes validation performance while encoding richer spatial details.The Spatial Path contains three stride-2 convolutions followed by batch normalization and ReLU.
- Speed and Accuracy Analysis: The experiments also report results on Cityscapes, CamVid, and COCO-Stuff, using dataset-specific training, validation, testing, and cropping protocols.Cityscapes uses fine annotations, CamVid uses 960×720 resolution, and COCO-Stuff results are reported on its validation dataset.
5 Conclusions
BiSeNet targets simultaneous improvements in real-time semantic-segmentation speed and accuracy through separate spatial and contextual processing paths. Its reported Cityscapes result combines detailed spatial features with a large receptive field.
- Conclusions: BiSeNet separates spatial-information preservation from receptive-field enlargement using Spatial and Context Paths.The Spatial Path preserves information from the original images, while the Context Path uses a lightweight model and global average pooling.
- Conclusions: The architecture combines affluent spatial details with a large receptive field for real-time semantic segmentation.The conclusion attributes these properties to the two-path design and global average pooling.