Source-linked AI summary
PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers
Jiacong Xu, Zixiang Xiong, Shankar P. Bhattacharyya
TL;DR
Real-time semantic segmentation must preserve detailed features while incorporating context without the overshoot associated with existing two-branch designs. PIDNet addresses this by linking CNN branches to PID control and adding a boundary-aware third branch; it achieves the best reported speed–accuracy trade-off, including 78.6% mIOU at 93.2 FPS for PIDNet-S on Cityscapes.
Problem
Existing real-time segmentation models face a trade-off between computational efficiency and segmentation accuracy, while two-branch fusion can suffer from overshoot.
Method
PIDNet connects CNN architectures with PID controllers and uses a three-branch network with efficient feature-fusion modules.
Results
PIDNet achieves the best trade-off between inference speed and accuracy among existing models; PIDNet-S reaches 78.6% mIOU at 93.2 FPS on Cityscapes.
Takeaways & Limitations
The proposed architecture surpasses previous works in reported accuracy while maintaining real-time inference speed across the evaluated segmentation benchmarks.
Abstract
from arXiv · showhide
Two-branch network architecture has shown its efficiency and effectiveness in real-time semantic segmentation tasks. However, direct fusion of high-resolution details and low-frequency context has the drawback of detailed features being easily overwhelmed by surrounding contextual information. This overshoot phenomenon limits the improvement of the segmentation accuracy of existing two-branch models. In this paper, we make a connection between Convolutional Neural Networks (CNN) and Proportional-Integral-Derivative (PID) controllers and reveal that a two-branch network is equivalent to a Proportional-Integral (PI) controller, which inherently suffers from similar overshoot issues. To alleviate this problem, we propose a novel three-branch network architecture: PIDNet, which contains three branches to parse detailed, context and boundary information, respectively, and employs boundary attention to guide the fusion of detailed and context branches. Our family of PIDNets achieve the best trade-off between inference speed and accuracy and their accuracy surpasses all the existing models with similar inference speed on the Cityscapes and CamVid datasets. Specifically, PIDNet-S achieves 78.6% mIOU with inference speed of 93.2 FPS on Cityscapes and 80.1% mIOU with speed of 153.7 FPS on CamVid.
1. Introduction
Real-time semantic segmentation seeks to balance accurate pixel labeling with low computational cost. PIDNet connects CNN architectures to PID controllers and proposes a three-branch design that improves this speed–accuracy trade-off.
- Semantic segmentation assigns each input pixel a class label for visual scene parsing.
- High segmentation accuracy often requires substantial computation, hindering real-time applications such as autonomous driving and robot surgery.
- Earlier efficient models reduced latency and memory usage, but their low accuracy limited real-world application.
- PIDNet interprets two-branch networks as PI controllers and introduces a three-branch architecture based on PID controller principles.
- Efficient Bag fusion modules balance detailed and context features to improve PIDNet performance.
- 78.6% mIOU at 93.2 FPS is achieved by PIDNet-S on Cityscapes, while PIDNet-L reaches 80.6% mIOU in real time without acceleration tools.
2. Related Work
Related work addresses semantic segmentation through encoder–decoder, lightweight, and two-branch architectures. These approaches trade computational efficiency against spatial detail, contextual representation, latency, or accuracy.
- 2.1. High-accuracy Semantic Segmentation: Encoder–decoder models recover details from high-level semantics, but downsampling can discard spatial information.
- 2.1. High-accuracy Semantic Segmentation: Figure 2 contrasts PI and PID controller step responses with segmentation outputs from DDRNet-23 and ADB-Bag-DDRNet-23.
- 2.1. High-accuracy Semantic Segmentation: Dilated convolutions enlarge the field of view without reducing spatial resolution, supporting high-accuracy segmentation.
- 2.2. Real-time Semantic Segmentation: Lightweight models reduce computational cost through smaller inputs, efficient backbones, depth-wise separable convolutions, or channel shuffling.
- 2.2. Real-time Semantic Segmentation: Some lightweight architectures still incur latency through deep encoder–decoder paths, while GPU optimization for depth-wise separable convolutions may be immature.
- 2.2. Real-time Semantic Segmentation: Two-branch networks combine a context branch, a detail branch, and a feature fusion module to capture contextual dependency and spatial detail.
3. Method
PIDNet interprets two-branch networks as PI controllers whose low-frequency context accumulation can cause overshoot, then adds a boundary-focused derivative branch and guided fusion. Its design assigns complementary roles to detail, context, and boundary processing while retaining efficient context aggregation and auxiliary training supervision.
- PIDNet: A Novel Three-branch Network: Two-branch networks resemble PI controllers: detail processing captures current local information, while context processing accumulates surrounding information and can produce overshoot.The analogy is supported spatially and in the Fourier domain, where detail and context branches behave like proportional and integral components.
- PIDNet: A Novel Three-branch Network: PIDNet adds a derivative branch that extracts high-frequency features for boundary detection, complementing detailed and contextual representations.The auxiliary derivative branch targets object boundaries because semantic differences are concentrated there.
- PIDNet: A Novel Three-branch Network: The P branch preserves high-resolution details, the I branch aggregates local and global context, and the D branch predicts boundary regions.These complementary responsibilities implement the proportional-integral-derivative analogy in the network architecture.
- 3.2. Pag: Learning High-level Semantics Selectively: Pag selectively transfers semantic information from the I branch to the P branch using pixel-level attention, reducing the risk that detailed features are overwhelmed.The attention weight reflects whether corresponding pixels likely belong to the same object and determines how much semantic information to trust.
- 3.4. Bag: Balancing the Details and Contexts: Boundary-attention-guided Bag fusion selects detailed features near high-frequency regions and context features in low-frequency regions.The module addresses contextual features' loss of spatial detail around boundaries and small objects; when σ > 0.5, it favors detailed features, otherwise context information.
4. Experiment
Experiments on Cityscapes, CamVid, and PASCAL Context evaluate PIDNet's modules, losses, efficiency, and speed–accuracy trade-offs. Ablations and visualizations support the roles of Pag, Bag, PAPPM, boundary losses, and OHEM.
- Experimental setup: PIDNet is evaluated on Cityscapes, CamVid, and PASCAL Context benchmarks.Cityscapes contains 5,000 images, while the experiments also use CamVid and PASCAL Context.
- Ablation study: Pag and Bag collaboration outperforms alternative lateral-connection and fusion combinations in the reported ablations.Pag preserves detail before fusion, while Bag guides the fusion of detailed and context features.
- Ablation study: Bag outputs greatly enhance regions and small objects, supporting the use of coarse boundary detection.Feature visualizations compare predictions and the P, I, and D inputs with Light-Pag output.
- Ablation study: PAPPM matches DAPPM accuracy while providing a 9.5 FPS speed-up for the lightweight model.PAPPM uses a parallel structure and few parameters to limit the cost of context aggregation.
- Ablation study: Boundary loss improves accuracy by 1.1% mIOU, while boundary-awareness loss is also necessary and OHEM further improves accuracy.These extra losses emphasize the functionality of PIDNet's components, and the boundary loss supports the necessity of the D branch.
- Comparison: On PASCAL Context, PIDNet remains competitive with existing heavy networks despite less detailed annotations than the other datasets.Multi-scale and flip inference are used for comparison on this benchmark, and one PAPPM path is removed because the images are small.
5. Conclusion
PIDNet is presented as a three-branch architecture for real-time semantic segmentation that balances inference time and accuracy. Its reliance on boundary prediction makes precise boundary annotations preferable for better performance.
- PIDNet achieves the best trade-off between inference time and accuracy for real-time semantic segmentation.
- Precise boundary annotations are preferred because PIDNet uses boundary prediction to balance detailed and context information.