PaperlayerView exact revision on arXiv

Source-linked AI summary

FBSNet: A Fast Bilateral Symmetrical Network for Real-Time Semantic Segmentation

Guangwei Gao, Guoan Xu, Juncheng Li, Yi Yu, Huimin Lu, Jian Yang

arXiv:2109.00699v3cs.CV

TL;DR

Real-time semantic segmentation faces calculation and parameter burdens. FBSNet combines semantic and spatial branches in a bilateral encoder-decoder with lightweight feature modules. The paper concludes that its design balances accuracy, model size, and inference speed.

  • Problem

    Real-time segmentation faces substantial calculation burdens and redundant parameters, while practical edge equipment calls for small models and fast interaction.

    Source: §I · p. 1

  • Method

    FBSNet uses a bilateral encoder-decoder with semantic and spatial branches, lightweight Bottleneck Residual Units, detail-preserving modules, and feature aggregation.

    Source: §I · p. 2

  • Results

    The authors report that FBSNet achieves a relatively good balance between accuracy, model size, and inference speed.

    Source: §V · p. 10

  • Takeaways & Limitations

    The authors identify spatial information as critical to recovering resolution during decoding.

    Source: §V · p. 10

Abstract

from arXiv · show

Real-time semantic segmentation, which can be visually understood as the pixel-level classification task on the input image, currently has broad application prospects, especially in the fast-developing fields of autonomous driving and drone navigation. However, the huge burden of calculation together with redundant parameters are still the obstacles to its technological development. In this paper, we propose a Fast Bilateral Symmetrical Network (FBSNet) to alleviate the above challenges. Specifically, FBSNet employs a symmetrical encoder-decoder structure with two branches, semantic information branch and spatial detail branch. The Semantic Information Branch (SIB) is the main branch with semantic architecture to acquire the contextual information of the input image and meanwhile acquire sufficient receptive field. While the Spatial Detail Branch (SDB) is a shallow and simple network used to establish local dependencies of each pixel for preserving details, which is essential for restoring the original resolution during the decoding phase. Meanwhile, a Feature Aggregation Module (FAM) is designed to effectively combine the output of these two branches. Experimental results of Cityscapes and CamVid show that the proposed FBSNet can strike a good balance between accuracy and efficiency. Specifically, it obtains 70.9\% and 68.9\% mIoU along with the inference speed of 90 fps and 120 fps on these two test datasets, respectively, with only 0.62 million parameters on a single RTX 2080Ti GPU. The code is available at https://github.com/IVIPLab/FBSNet.

I. INTRODUCTION

Real-time segmentation must balance accuracy and speed while keeping model size and computation low. FBSNet uses a bilateral encoder-decoder design with distinct semantic and spatial branches and modules for efficient feature extraction and fusion.

I. INTRODUCTION

  • Real-world edge applications such as autonomous-driving chips require small model size and computational cost alongside fast interaction speed.The paper frames real-time segmentation as a response to the parameter burden of larger networks.Source: §I · p. 1
  • Prior lightweight segmentation networks seek an accuracy-speed balance, but cited approaches have limitations in receptive field, low-resolution input suitability, or computational cost.ENet's receptive field is described as too small for large objects; ICNet is described as unsuitable for low-resolution inputs, and BiSeNet-v2's increased parameters and computation slow inference.Source: §I · pp. 1–2
  • The lightweight Bottleneck Residual Unit extracts rich features with different receptive fields using few parameters and little computation.It uses dilated factorized depth-wise separable convolutions to deepen the network and enlarge receptive fields.Source: §I · p. 2
  • A Detail Residual Module preserves shallow spatial features, while a Feature Aggregation Module fuses features from the two branches.The modules aim to compensate for details lost in the semantic branch and increase global and local dependencies.Source: §I · p. 2
  • FBSNet pairs a Semantic Information Branch for deep semantics with a Spatial Detail Branch for preserving shallow boundary details in a symmetrical encoder-decoder.The branches use different structures for their different tasks.Source: §I · p. 2

II. RELATED WORKS

Related work addresses contextual information and spatial detail, both relevant to segmentation quality, while real-time models must manage their computational costs.

A. Contextual Information

  • Increasing receptive fields and capturing broader global context are recurring aims in segmentation methods, including pooling and dilated-convolution approaches.The cited examples include ParseNet, PSPNet, and DFN.Source: A. Contextual Information · p. 3

B. Spatial Information

  • Spatial-detail recovery is computationally costly, and existing approaches include shallow-feature concatenation, encoder-stage feature supplementation, cascades, and dense connections.The passage notes that UNet-style concatenation adds parameters and computation, while some dense-connection models do not meet real-time requirements.Source: B. Spatial Information · p. 3
  • The related methods' decoder can lack semantic guidance, resulting in unsatisfactory performance.Source: B. Spatial Information · p. 3

C. General Structure

FBSNet combines a symmetrical semantic encoder-decoder with a spatial detail branch and aggregates their features at semantic and spatial levels. Its BRU uses three branches to preserve input information while extracting local and long-distance features.

C. General Structure

  • FBSNet pairs a symmetrical semantic information branch with a spatial detail branch that preserves shallow boundary details without downsampling.The branches extract deep semantic information and shallow boundary details, respectively.Source: C. General Structure · p. 3
  • Successive aggregation at semantic and spatial levels promotes fused-feature representation with negligible parameter and computational overhead.This differs from simple element-wise feature fusion.Source: C. General Structure · p. 3

A. Bottleneck Residual Unit (BRU)

  • Fig. 4 compares BRU with LEDNet’s SS-nbt [24], MSCFNet’s EAR [2], and ESPNet’s pyramid cascade [6].The caption identifies R as dilation rate, C as output channels, and D as depth-wise separable convolution.Source: Figure 4 · p. 4

III. PROPOSED METHOD

  • FBSNet’s method comprises BRU, SIB with channel attention, SDB with spatial attention, and an initial block with FAM.Source: §III · p. 4

A. Bottleneck Residual Unit (BRU)

  • The BRU uses local-feature, receptive-field-expansion, and input-preserving branches, then merges their outputs with channel attention and channel shuffle.Its operations include a 1×1 convolution, a residual connection to the input, and channel shuffle.Source: A. Bottleneck Residual Unit (BRU) · p. 4

B. Semantic Information Branch (SIB)

The SIB uses BRU-based stages and channel attention to capture contextual information with large receptive fields while keeping parameters and computation low. It limits downsampling to retain detail.

B. Semantic Information Branch (SIB)

  • The SIB uses BRU and stage-specific dilation rates to capture semantic information and larger receptive fields while keeping parameters and computation low.The passage contrasts this design with complex networks and a ResNet-backed dual-branch approach.Source: B. Semantic Information Branch (SIB) · p. 5
  • CAM uses global average pooling to obtain global context and guide feature extraction while emphasizing useful channels and suppressing interference noise.Its channel-attention map is generated from the input features.Source: B. Semantic Information Branch (SIB) · p. 5
  • The SIB downsamples twice, reaching 1/8 of the input image size to balance receptive-field growth against boundary-detail loss.The downsampling block is inspired by ENet’s initial block.Source: B. Semantic Information Branch (SIB) · p. 5

C. Spatial Detail Branch (SDB)

The SDB supplements spatial detail lost in the semantic branch through a simple detail residual module and spatial attention. Its attention emphasizes where useful features occur.

C. Spatial Detail Branch (SDB)

  • The SDB supplements details lost in the semantic branch using a simple DRM and SAM.The branch is intended to help the model predict more accurately.Source: C. Spatial Detail Branch (SDB) · p. 5
  • DRM extracts shallow spatial features using three 3×3 convolutions and a 1×1 point-wise convolution, with depth-wise separable convolutions replacing the latter two 3×3 layers.The second and third convolutional layers expand channels to 4C before the 1×1 layer reduces them to C.Source: C. Spatial Detail Branch (SDB) · p. 5
  • SAM preserves shallow spatial information by pooling across channels, concatenating the max- and average-pooled maps, and convolving them into a spatial attention map.The map has dimensions 1×H×W.Source: C. Spatial Detail Branch (SDB) · pp. 5–6

D. Fast Bilateral Symmetrical Network (FBSNet)

FBSNet combines an initial block, a dual-branch backbone, and a feature aggregation module, with symmetry reflected in the SIB encoding and decoding resolutions. Its design preserves spatial information and integrates semantic and spatial features using attention.

D. Fast Bilateral Symmetrical Network (FBSNet)

  • The initial block downsamples once and divides the branches after three 3×3 convolutions, preserving spatial information and partially correlating semantic and spatial features for later merging.The first convolution uses stride 2.Source: D. Fast Bilateral Symmetrical Network… · p. 6
  • FAM addresses differences between branch features that element-wise addition or concatenation alone ignore, using horizontal and vertical pooling to encode long-range dependencies and precise positional information.The two spatial pooling kernels have sizes (H, 1) and (1, W).Source: D. Fast Bilateral Symmetrical Network… · p. 6
  • FBSNet did not achieve the best Cityscapes mIoU, but the comparison describes it as competitive in parameters, FLOPs, and FPS and balanced across performance, size, and speed.The supplied caption provides no metric values.Source: Table III · p. 7
  • FAM applies sigmoid-derived attention weights to the branch features, integrating them and adaptively highlighting information along channel and spatial directions.The module's output is Y.Source: D. Fast Bilateral Symmetrical Network… · p. 7

IV. EXPERIMENTS

The experiments cover benchmark datasets, training parameter settings, ablations, and comparisons with state-of-the-art semantic segmentation methods.

IV. EXPERIMENTS

  • The experiments introduce the benchmark datasets used to evaluate FBSNet.Source: §IV · p. 7
  • The experiments report the parameter settings used during training.Source: §IV · p. 7
  • The experiments include ablations and comparisons with state-of-the-art image semantic segmentation methods to illustrate FBSNet's advantages.Source: §IV · p. 7

A. Datasets

The evaluation uses Cityscapes and CamVid, with dataset descriptions and Cityscapes test-set qualitative and per-class result visuals.

A. Datasets

  • Cityscapes uses 19 fine-annotated categories from 2,975 training, 500 validation, and 1,525 test images.The images are 2048 × 1024; the dataset also contains 19,998 coarsely annotated images.Source: A. Datasets · pp. 7–8
  • FBSNet achieves the best mIoU results in the Cityscapes test-set per-class comparison.Source: Table IV · p. 8
  • The Cityscapes validation figure compares FBSNet's segmentation outputs with ground truth and five other methods, with yellow dotted boxes highlighting regions the caption says show FBSNet's superiority.Source: Figure 6 · p. 8
  • CamVid has 11 categories and 701 fine-annotated images split into 367 training, 101 validation, and 233 test samples.The images are 960 × 720.Source: A. Datasets · p. 8

B. Implementation details

The implementation uses PyTorch on one RTX 2080 Ti GPU, with different optimizers and initial learning rates for Cityscapes and CamVid.

B. Implementation details

  • The model is implemented in PyTorch, and all experiments run on a single RTX 2080 Ti GPU.Source: B. Implementation details · p. 8
  • Cityscapes training uses cross-entropy with OHEM and SGD, whereas CamVid uses Adam because of its different input resolution.Both optimizers use momentum 0.9; weight decay is 1×10−4 for Cityscapes and 2 × 10−4 for CamVid.Source: B. Implementation details · p. 8

C. Ablation Studies

The ablations assess the contribution of modules in the full network. CAM, FAM, SAM, and the SDB each improve reported performance, with differing computational costs.

C. Ablation Studies

  • CAM provides a reported 0.6% accuracy improvement with almost no increase in parameters or computation.The ablations add modules step by step on CamVid to assess effects on prediction results, parameters, and computation.Source: C. Ablation Studies · p. 8
  • FAM achieves the best performance of 68.58%, 0.71% and 0.32% better than Add and Concatenate, respectively.Compared with Add, FAM increases parameters by 0.000556M and FLOPs by 0.0003G; compared with Concatenate, it achieves better results with fewer parameters and FLOPs.Source: C. Ablation Studies · p. 9
  • Adding SDB improves performance from 66.72% to 67.87% while increasing whole-network computation by nearly half.The authors attribute the feasibility of combining SIB and SDB to the lightweight structure of SIB.Source: C. Ablation Studies · p. 9
  • 68.58% to 68.86% is the reported performance change with SAM, which uses 0.01G computation and very few parameters.The authors state that CAM, FAM, and SAM each play a unique role in the network.Source: C. Ablation Studies · p. 9

D. Comparison with SOTA Methods

FBSNet is evaluated against real-time segmentation methods on Cityscapes and CamVid, with testing on original dataset sizes and no post-processing. The authors report a balance of accuracy, model size, and speed, including 70.9% mIoU at 90 FPS on Cityscapes with 0.62M parameters.

D. Comparison with SOTA Methods

  • Testing uses original dataset sizes without post-processing such as CRF; evaluation considers mIoU, parameters, FLOPs, and FPS.Training crops are 512×1024 for Cityscapes and 480×360 for CamVid to reduce computation and GPU memory use.Source: D. Comparison with SOTA Methods · p. 9
  • On Cityscapes, FBSNet runs at 90 FPS and uses 0.62M parameters, while DFANet reports 100 FPS and 7.8M parameters.The authors report that attention mechanisms add computational overhead; FBSNet’s mIoU is slightly below DFANet’s, while several methods, including DFANet and BiSeNet-v2, have higher mIoU.Source: D. Comparison with SOTA Methods · p. 9
  • On CamVid, FBSNet has the second-best segmentation results, slightly below MSCFNet, while using half its parameters and FLOPs and running twice as fast.The authors describe the overall accuracy-efficiency balance as favorable; Fig. 7 presents visual comparisons of segmentation outputs.Source: Figure 7 · p. 9, D. Comparison with SOTA Methods · p. 10

V. CONCLUSIONS

  • 70.9% mIoU at 90 FPS is reported for FBSNet on Cityscapes, with 0.62M parameters and a 512 × 1024 input on one RTX 2080Ti GPU.The authors characterize the model as balancing accuracy and efficiency.Source: §V · p. 10

Related papers

Loading 2109.00699v3…