Source-linked AI summary
LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation
Yu Wang, Quan Zhou, Jia Liu, Jian Xiong, Guangwei Gao, Xiaofu Wu, Longin Jan Latecki
TL;DR
CNN computational demands limit dense estimation on mobile devices, motivating a lightweight real-time semantic segmentation model. LEDNet uses an asymmetric encoder-decoder with split-shuffle residual encoding and an attention pyramid decoder, achieving the best reported accuracy-efficiency trade-off on CityScapes.
Problem
Computationally heavy CNNs are poorly suited to mobile platforms with limited energy, memory, and computational capabilities, especially for semantic segmentation.
Method
LEDNet combines an asymmetric encoder-decoder architecture, channel split and shuffle in residual modules, and an attention pyramid network decoder.
Results
LEDNet achieves the best reported accuracy-efficiency trade-off on CityScapes, including 70.6% class mIoU and 87.1% category mIoU.
Takeaways & Limitations
The results support LEDNet as a lightweight real-time semantic segmentation network with reduced model complexity and competitive segmentation accuracy.
Abstract
from arXiv · showhide
The extensive computational burden limits the usage of CNNs in mobile devices for dense estimation tasks. In this paper, we present a lightweight network to address this problem,namely LEDNet, which employs an asymmetric encoder-decoder architecture for the task of real-time semantic segmentation.More specifically, the encoder adopts a ResNet as backbone network, where two new operations, channel split and shuffle, are utilized in each residual block to greatly reduce computation cost while maintaining higher segmentation accuracy. On the other hand, an attention pyramid network (APN) is employed in the decoder to further lighten the entire network complexity. Our model has less than 1M parameters,and is able to run at over 71 FPS in a single GTX 1080Ti GPU. The comprehensive experiments demonstrate that our approach achieves state-of-the-art results in terms of speed and accuracy trade-off on CityScapes dataset.
1. INTRODUCTION
LEDNet addresses the tension between CNN accuracy and computational demands for real-time semantic segmentation on resource-constrained platforms. It uses an asymmetric encoder-decoder design with split-shuffle residual modules and an attention pyramid decoder.
- Motivation: Resource-constrained mobile platforms require smaller, computationally cheaper networks for timely semantic segmentation.The task assigns a semantic category label to every image pixel, making computational limitations particularly consequential.
- Motivation: Existing lightweight approaches balance efficiency and accuracy through network compression or convolution factorization.Compression includes hashing, pruning, and quantization, while factorization directly trains networks using grouped or depthwise separable convolutions.
- LEDNet: LEDNet introduces an asymmetric encoder-decoder architecture for real-time semantic segmentation to address the accuracy-efficiency trade-off.Its encoder uses a novel residual module, while its decoder uses an attention pyramid network.
- LEDNet: The encoder’s channel split and shuffle operations support information exchange and feature reuse, while the decoder’s attention pyramid extracts dense features with lower complexity.Skip connections facilitate residual learning, and attention estimates a semantic label for each pixel.
2. OUR APPROACH
LEDNet combines split-shuffle residual encoding with asymmetric downsampling and an attention pyramid decoder. These components reduce computation while preserving feature exchange, receptive field, and output resolution.
- Residual Module with Split and Shuffle Operations: The SS-nbt residual module splits input channels into two branches, transforms them with specialized 1D filters, merges them, and shuffles channels.Identity mapping supports residual learning, while shuffle enables communication between the split branches.
- Residual Module with Split and Shuffle Operations: Channel shuffle enables feature reuse that enlarges network capacity without significantly increasing complexity.The shuffled merged channels are passed into the next SS-nbt unit.
- LEDNet Architecture Designment: LEDNet uses an asymmetric sequential encoder-decoder in which the encoder downsamples feature maps and the APN decoder upsamples them to input resolution.The architecture is described in Table 1 and contrasts with the approach in.
- LEDNet Architecture Designment: Encoder downsampling combines stride-2 convolution with max-pooling, while dilated convolutions provide a larger receptive field with reduced computation compared with larger kernels.Downsampling also enables deeper context gathering and lowers computation.
- LEDNet Architecture Designment: The APN integrates three pyramid scales, applies spatial attention and global context, and upsamples features to the input resolution.It uses 3 × 3, 5 × 5, and 7 × 7 convolutions to form the feature pyramid before attention-weighted fusion.
3. EXPERIMENTS
LEDNet is evaluated on CityScapes using segmentation accuracy and implementation-efficiency metrics against six lightweight baselines. It achieves a strong accuracy-efficiency trade-off and produces consistent qualitative outputs.
- Implementation Details: The CityScapes evaluation uses class and category mIoU for accuracy, alongside running time, FPS, and parameter count for efficiency.The dataset includes fine and coarse annotations, and six lightweight networks serve as baselines.
- Evaluation Results: 70.6% class mIoU and 87.1% category mIoU are reported for LEDNet, with best scores in 13 of 19 categories.These results are reported in comparison with lightweight segmentation baselines.
- Evaluation Results: Nearly 5× faster and 30× smaller than SegNet, LEDNet improves efficiency while retaining higher segmentation accuracy than ENet.ENet is reported as 1.5× more efficient and having 3× fewer parameters, but with a 10% accuracy drop relative to LEDNet.
- Evaluation Results: LEDNet correctly classifies objects at different scales and produces consistent qualitative results across classes on the CityScapes validation set.Figure 3 compares outputs from SegNet, ENet, ERFNet, ESPNet, ICNet, CGNet, and LEDNet.
4. CONCLUSION AND FUTURE WORK
LEDNet combines an asymmetric encoder-decoder design with channel split and shuffle in the encoder and an attention pyramid network in the decoder for real-time semantic segmentation. On CityScapes, it achieves the best trade-off between segmentation accuracy and implementation efficiency, while future work targets further APN lightweighting.
- LEDNet uses an asymmetric encoder-decoder architecture for real-time semantic segmentation.
- Channel split and shuffle operations in residual layers enhance feature communication through feature reuse.
- The decoder employs an attention pyramid network whose spatial pyramid structure enlarges receptive fields without significant computational budgets.
- The entire network is trained end-to-end.
- LEDNet achieves the best trade-off between segmentation accuracy and implementation efficiency on the CityScapes dataset.
- Future work will decompose standard convolutions in APN into 1D convolutions to further lighten the network while retaining segmentation accuracy.