Source-linked AI summary
PP-LiteSeg: A Superior Real-Time Semantic Segmentation Model
Juncai Peng, Yi Liu, Shiyu Tang, Yuying Hao, Lutao Chu, Guowei Chen, Zewu Wu, Zeyu Chen, Zhiliang Yu, Yuning Du, Qingqing Dang, Baohua Lai, Qiwen Liu, Xiaoguang Hu, Dianhai Yu, Yanjun Ma
TL;DR
Real-time semantic segmentation methods do not achieve a satisfactory accuracy–speed trade-off. PP-LiteSeg addresses this with three lightweight modules and achieves a state-of-the-art trade-off, including 72.0% mIoU/273.6 FPS and 77.5% mIoU/102.6 FPS on Cityscapes.
Problem
Existing real-time semantic segmentation models have high computation costs and do not achieve a satisfactory trade-off between accuracy and speed.
Method
PP-LiteSeg combines a Flexible and Lightweight Decoder, Unified Attention Fusion Module, and Simple Pyramid Pooling Module to improve efficiency, feature representation, and global-context aggregation.
Results
72.0% mIoU/273.6 FPS and 77.5% mIoU/102.6 FPS are achieved on the Cityscapes test set.
Takeaways & Limitations
PP-LiteSeg achieves a state-of-the-art trade-off between segmentation accuracy and inference speed for real-time semantic segmentation.
Takeaways & Limitations
The paper focuses on real-time semantic segmentation and identifies applying the methods to matting and interactive segmentation as future work.
Abstract
from arXiv · showhide
Real-world applications have high demands for semantic segmentation methods. Although semantic segmentation has made remarkable leap-forwards with deep learning, the performance of real-time methods is not satisfactory. In this work, we propose PP-LiteSeg, a novel lightweight model for the real-time semantic segmentation task. Specifically, we present a Flexible and Lightweight Decoder (FLD) to reduce computation overhead of previous decoder. To strengthen feature representations, we propose a Unified Attention Fusion Module (UAFM), which takes advantage of spatial and channel attention to produce a weight and then fuses the input features with the weight. Moreover, a Simple Pyramid Pooling Module (SPPM) is proposed to aggregate global context with low computation cost. Extensive evaluations demonstrate that PP-LiteSeg achieves a superior trade-off between accuracy and speed compared to other methods. On the Cityscapes test set, PP-LiteSeg achieves 72.0% mIoU/273.6 FPS and 77.5% mIoU/102.6 FPS on NVIDIA GTX 1080Ti. Source code and models are available at PaddleSeg: https://github.com/PaddlePaddle/PaddleSeg.
1. Introduction
PP-LiteSeg addresses the unsatisfactory accuracy–speed trade-off of real-time semantic segmentation with three lightweight modules: FLD, UAFM, and SPPM. Evaluations on Cityscapes and CamVid report a superior trade-off between segmentation accuracy and inference speed.
- Motivation: Real-time segmentation methods have not achieved a satisfactory trade-off between accuracy and speed despite efforts to reduce computation cost.Prior approaches include lightweight convolutions, bilateral architectures, and efficient backbones, but the cited work identifies the trade-off as unresolved.
- Approach: PP-LiteSeg combines a lightweight encoder, global-context aggregation, and a decoder that fuses detail and semantic features.Its architecture comprises encoder, aggregation, and decoder stages.
- Approach: FLD gradually reduces decoder channels while increasing spatial size, balancing encoder and decoder computation and allowing adjustment to the encoder.This design targets redundancy in previous decoders and improves overall model efficiency.
- Approach: UAFM uses spatial and channel attention to produce fusion weights that strengthen feature representations while fusing input features efficiently.The module exploits inter-spatial and inter-channel relationships of the input features.
- Approach: SPPM reduces channels, removes the shortcut, and replaces concatenation with addition to aggregate global context at low computation cost.The reported design contributes to segmentation accuracy with minor extra inference time.
2. Related Work
Semantic segmentation research has advanced through encoder-decoder, pyramid-pooling, feature-fusion, and lightweight real-time designs, but existing methods still struggle to balance accuracy and speed.
- Encoder-decoder architectures established end-to-end pixel-level segmentation, while later methods enhanced features through pooling and alignment modules.
- Real-time methods reduce computation using lightweight modules, dual branches, early downsampling, or multiresolution image cascades.
- The proposed FLD departs from conventional decoders by gradually reducing feature channels from high level to low level and matching decoder volume to the encoder.
3. Proposed Method
PP-LiteSeg combines a flexible lightweight decoder, attention-based feature fusion, and simplified pyramid pooling within an encoder-aggregation-decoder architecture for efficient segmentation.
- Flexible and Lightweight Decoder: FLD gradually reduces decoder channels while increasing spatial resolution, allowing computation to be adjusted for better balance with the encoder.
- Unified Attention Fusion Module: UAFM upsamples high-level features, derives attention weights from spatial or channel relationships, and fuses them with low-level features using multiplication and addition.
- Simple Pyramid Pooling Module: SPPM aggregates global context with 1 × 1, 2 × 2, and 4 × 4 global-average-pooling bins, reduced channels, and additive fusion.
- Network Architecture: The architecture uses STDCNet encoders, with PP-LiteSeg-T prioritizing inference speed and PP-LiteSeg-B prioritizing segmentation accuracy.
- Network Architecture: The decoder contains two UAFMs and a segmentation head that reduces channels, upsamples predictions, and applies argmax to label pixels.
4. Experiments
Experiments on Cityscapes and CamVid evaluate PP-LiteSeg against real-time methods and assess its modules through ablation. The results show a strong accuracy–speed trade-off, while FLD, SPPM, and UAFM improve segmentation accuracy with a slight speed decrease.
- Experimental setup: Experiments compare accuracy and inference speed on Cityscapes and CamVid, including state-of-the-art comparisons and module ablations.Cityscapes uses 19 classes, while CamVid uses 11 categories; the ablation uses PP-LiteSeg-B2 under identical training and inference settings.
- Cityscapes comparison: 72.0% mIoU and 273.6 FPS make PP-LiteSeg-T1 the fastest Cityscapes configuration with competitive accuracy.The result is reported for the 512 × 1024 evaluation setting.
- Cityscapes comparison: 77.5% mIoU and 102.6 FPS are achieved by PP-LiteSeg-B2 on the Cityscapes test set at 768 × 1536 resolution.The corresponding validation result is 78.2% mIoU.
- Ablation study: 0.71% mIoU is gained over the baseline when FLD, SPPM, and UAFM are combined in PP-LiteSeg-B2.FLD alone improves mIoU by 0.17%; adding SPPM and UAFM further improves accuracy while slightly reducing inference speed.
- Ablation study: Qualitative predictions become more consistent with ground truth as FLD, SPPM, and UAFM are added sequentially.Figure 6 compares the baseline and each cumulative module configuration on the Cityscapes validation set.
- CamVid comparison: 222.3 FPS makes PP-LiteSeg-T over 12.5% faster than other methods on CamVid, while PP-LiteSeg-B reaches 75.0% mIoU at 154.8 FPS.The comparisons use 960 × 720 input resolution.
5. Conclusions
The paper proposes PP-LiteSeg, a real-time semantic segmentation network built from FLD, UAFM, and SPPM. Extensive experiments demonstrate a state-of-the-art trade-off between segmentation accuracy and inference speed; future work will apply the methods to additional tasks.
- 5. Conclusions: PP-LiteSeg combines FLD for decoder efficiency, UAFM for feature representation strengthening, and SPPM for low-cost global-context aggregation.These three modules form the paper’s proposed real-time semantic segmentation network.
- 5. Conclusions: Extensive experiments demonstrate that PP-LiteSeg achieves a state-of-the-art trade-off between segmentation accuracy and inference speed.The conclusion summarizes the model’s supported performance across the reported experiments.
- 5. Conclusions: Future work plans to apply the proposed methods to matting and interactive segmentation.These tasks are identified as directions beyond the present semantic segmentation focus.