PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Real-time Semantic Segmentation Network with Efficient Transformer and CNN

Guoan Xu, Juncheng Li, Guangwei Gao, Huimin Lu, Jian Yang, Dong Yue

arXiv:2302.10484v1cs.CV

TL;DR

CNNs provide insufficient global representation, while Transformers can lose local spatial details and incur high computational costs in high-resolution segmentation. LETNet combines a U-shaped CNN with an efficient Transformer capsule, LDB modules, and Feature Enhancement in skip connections. LETNet achieved 72.8% mIoU on the Cityscapes test set and 70.5% on CamVid using only 0.95M parameters on an RTX3090.

  • Problem

    CNNs provide insufficient global representation, while Transformers can lose local spatial details and incur high computational costs in high-resolution segmentation.

  • Method

    LETNet combines a U-shaped CNN with an efficient Transformer capsule, LDB modules, and Feature Enhancement in skip connections.

  • Results

    LETNet achieved 72.8% mIoU on the Cityscapes test set and 70.5% on CamVid using only 0.95M parameters on an RTX3090.

  • Takeaways & Limitations

    The reported experiments show an accuracy–efficiency balance for lightweight real-time semantic segmentation.

Abstract

from arXiv · show

In the past decade, convolutional neural networks (CNNs) have shown prominence for semantic segmentation. Although CNN models have very impressive performance, the ability to capture global representation is still insufficient, which results in suboptimal results. Recently, Transformer achieved huge success in NLP tasks, demonstrating its advantages in modeling long-range dependency. Recently, Transformer has also attracted tremendous attention from computer vision researchers who reformulate the image processing tasks as a sequence-to-sequence prediction but resulted in deteriorating local feature details. In this work, we propose a lightweight real-time semantic segmentation network called LETNet. LETNet combines a U-shaped CNN with Transformer effectively in a capsule embedding style to compensate for respective deficiencies. Meanwhile, the elaborately designed Lightweight Dilated Bottleneck (LDB) module and Feature Enhancement (FE) module cultivate a positive impact on training from scratch simultaneously. Extensive experiments performed on challenging datasets demonstrate that LETNet achieves superior performances in accuracy and efficiency balance. Specifically, It only contains 0.95M parameters and 13.6G FLOPs but yields 72.8\% mIoU at 120 FPS on the Cityscapes test set and 70.5\% mIoU at 250 FPS on the CamVid test dataset using a single RTX 3090 GPU. The source code will be available at https://github.com/IVIPLab/LETNet.

I. INTRODUCTION

LETNet addresses the tension between CNNs' limited global context and Transformers' weak local-detail preservation with a lightweight hybrid design. Its LDB and feature-enhancement components support an accuracy–efficiency balance on Cityscapes and CamVid.

  • I. INTRODUCTION: CNN-based models have limited receptive fields, while Transformer-based image processing can impose high cost and damage local spatial connections.The introduction identifies insufficient remote-dependency modeling in CNNs and quadratic self-attention complexity plus disrupted local structure for Transformers.
  • I. INTRODUCTION: The Lightweight Dilated Bottleneck uses dilated and depth-wise separable convolutions to reduce parameters and computational quantities.LDB is designed to extract important semantic information while reducing model weights and computation.
  • I. INTRODUCTION: LETNet combines a CNN encoder-decoder with an efficient Transformer capsule to model local features and global information.The hybrid network uses CNN structures for local feature extraction and Transformer-based modeling for global dependencies.
  • I. INTRODUCTION: A Feature Enhancement module in the skip connections supplements boundary details during resolution recovery.The module is added to help restore boundary information in the hybrid segmentation architecture.
  • I. INTRODUCTION: 72.8% mIoU on Cityscapes and 70.5% on CamVid demonstrate LETNet's reported segmentation performance with a lightweight design.The Cityscapes result uses a single RTX3090 platform; the introduction also states that performance exceeds most existing models.

II. RELATED WORK

CNN-based semantic segmentation has progressed through architectures that improve contextual representation and feature dependencies, while lightweight designs address their computational demands.

  • ASPP-based methods introduce descriptors for contextual information at various scales through parallel atrous convolutions.
  • Attention-based methods model feature dependencies through local cross-channel interaction and adaptive one-dimensional convolution kernels.
  • Lightweight segmentation networks use multi-scale inputs or two-path architectures to balance high-level semantic information with low-level spatial details.

B. Transformer-based Semantic Segmentation Methods

Transformer-based segmentation methods model global context and long-range dependencies, but their computational cost and sequence-based inputs can weaken local structural detail.

  • Transformer-based vision methods apply sequence modeling to recognition, detection, and segmentation, including hierarchical encoders that produce and fuse multi-scale features.
  • Transformer computation grows quadratically with image size, increasing computational burden for image-domain semantic segmentation.
  • Treating image patches as one-dimensional sequences disrupts local structural connections and leaves detailed localization information insufficiently represented.
  • LETNet combines CNN encoder-decoder structures for local feature extraction with an efficient Transformer for global representation and long-distance dependencies.

B. Lightweight Dilated Bottleneck (LDB)

The Lightweight Dilated Bottleneck uses residual processing, channel reduction, attention, feature fusion, and channel shuffle to collect features while limiting network complexity.

  • LDB adopts a ResNet-inspired residual structure designed to collect more feature information with as few network layers as possible.
  • Reducing bottleneck channels by half with a 1×1 convolution lowers parameters and computation, with a stated trade-off of some accuracy loss.
  • Channel attention enhances feature expression and suppresses noise interference by modeling information along the channel dimension.
  • The module fuses two low-dimensional branches and middle features, restores channels with a 1×1 convolution, and applies channel shuffle to avoid its drawback.
  • The LDB mapping is expressed as y = fCS(f1×1(F1 + F21 + F22) + x), where x and y are input and output feature maps and f denotes convolution.

C. Efficient Transformer (ET)

The Efficient Transformer reduces the computational burden of Transformer-based vision processing while modeling long-distance dependencies, and the Feature Enhancement module jointly models channel and spatial dependencies to transmit low-level information more effectively.

  • C. Efficient Transformer (ET): Efficient Transformer reduces computing resources while learning long-range dependencies through self-attention and multilayer perceptron structures.It is introduced because CNNs have limited global representation ability and conventional image Transformers are costly at high resolution.
  • C. Efficient Transformer (ET): Feature Enhancement models feature dependencies in both channel and spatial dimensions before fusing them to improve low-level information transfer.The module is used in UNet-style long connections to combine high-resolution spatial information with high-level semantic information.
  • C. Efficient Transformer (ET): The Feature Enhancement operation defines separate channel and spatial outputs using input features, pooling, convolution, normalization, activation, and sigmoid functions.MC denotes the channel-dimension output and MS denotes the spatial-dimension output; average and max pooling provide spatial attention inputs.

E. Pixel Attention (PA)

Pixel Attention assigns position-dependent weights to feature pixels while sharing weights across channels, emphasizing edges and textures during segmentation recovery.

  • E. Pixel Attention (PA): Pixel Attention assigns different weights to pixel positions while using the same weight across channels at each position.The mechanism learns these weights from feature importance at different pixel locations.
  • E. Pixel Attention (PA): Pixel Attention emphasizes object edges and textures, facilitating edge-detail recovery and improving segmentation performance.Its implementation uses the sigmoid function and a 1×1 convolutional layer.

IV. EXPERIMENTS

The experiments use Cityscapes and CamVid, two urban-road semantic-segmentation datasets with different image resolutions, category counts, and train-validation-test splits.

  • IV. EXPERIMENTS: Cityscapes contains 5000 finely annotated 2048×1024 images across 19 categories, split into 2075 training, 500 validation, and 1525 testing images.The images come from German and French urban road scenes across 50 cities.
  • IV. EXPERIMENTS: CamVid contains 701 urban-road 960×720 images across 11 categories, split into 367 training, 101 validation, and 233 testing images.The dataset consists of finely annotated images for semantic segmentation.

B. Model Settings

Model settings are reported for LETNet training with PyTorch on an RTX3090 GPU, while learning rates vary with iterations and dataset-specific parameters account for differing resolutions.

  • B. Model Settings: LETNet is built with PyTorch and trained on an RTX3090 GPU, with detailed settings reported for Cityscapes and CamVid.The two datasets are treated as separate training configurations.
  • B. Model Settings: Cityscapes and CamVid use different parameter settings because their image resolutions differ.The learning rate varies with iterations, and lr_initial denotes the initial learning rate.
  • B. Model Settings: The baseline architecture is an encoder-decoder composed of Lightweight Dilated Bottlenecks without the proposed modules.Its validation performance is 69.85% mIoU.

C. Ablation Study

Ablations show that long connections, Feature Enhancement, and Efficient Transformer components each improve LETNet, with the complete configuration gaining 5.15% mIoU over baseline.

  • C. Ablation Study: The baseline is an encoder-decoder built from LDBs and achieves 69.85% mIoU on the validation set without the proposed modules.
  • C. Ablation Study: Adding three long skip connections increases performance by 1.16% mIoU, with the first connection alone improving it by 0.58%.The ablation attributes this benefit to shallow information aiding recovery of deep semantic information.
  • C. Ablation Study: 0.2% mIoU improvement is obtained from Feature Enhancement while adding only 101K parameters and 0.0032G FLOPs.
  • C. Ablation Study: 5.15% mIoU improvement over baseline is achieved by the final LETNet configuration C3*.C3* is the final version of the proposed LETNet.

D. Comparisons with Advanced Models

Across Cityscapes and CamVid, LETNet targets a balance between segmentation accuracy and efficiency, achieving strong results in the small-model setting and more accurate visual segmentations.

  • D. Comparisons with Advanced Models: HSBNet and LBN-AA slightly outperform LETNet in the medium-size category, but use 12 and 6 times more parameters, respectively.
  • D. Comparisons with Advanced Models: LETNet achieves the best results in the small-size Cityscapes category with fewer parameters, balancing model size and performance.Its results are similar to BiseNet-v2 while using one-quarter as many parameters.
  • D. Comparisons with Advanced Models: LETNet achieves the best results in almost every Cityscapes class according to the per-class comparison.
  • D. Comparisons with Advanced Models: LETNet achieves the best CamVid result with only 0.95M parameters.
  • D. Comparisons with Advanced Models: Visual comparisons on Cityscapes and CamVid report more accurate segmentation results for LETNet than the compared methods.The comparisons include LETNet alongside multiple lightweight segmentation models on both datasets.

V. CONCLUSION

The conclusion presents LETNet as a lightweight hybrid that combines CNN local feature extraction with Transformer long-range dependency modeling for efficient semantic segmentation.

  • V. CONCLUSION: LETNet combines CNN local feature extraction with Transformer long-range dependency modeling in a lightweight real-time segmentation network.An efficient Transformer is introduced as a capsule network with lightweight multi-head attention.
  • V. CONCLUSION: The LDB module learns more features while maintaining extreme simplicity and lightweight computation.
  • V. CONCLUSION: A U-shaped connection restores shallow detail information, while Feature Enhancement improves feature expression and suppresses noise across levels.
  • V. CONCLUSION: Experiments show that LETNet balances model size and performance.
Loading 2302.10484v1…