Source-linked AI summary
RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer
Jian Wang, Chenhui Gou, Qiman Wu, Haocheng Feng, Junyu Han, Errui Ding, Jingdong Wang
TL;DR
Real-time semantic segmentation needs transformer-style global context without the costly computation that has kept CNNs dominant. RTFormer addresses this with an efficient dual-resolution transformer using GPU-Friendly Attention and cross-resolution attention, achieving state-of-the-art results on three benchmarks and promising ADE20K performance.
Problem
Transformer attention offers useful global context for semantic segmentation, but its costly computation, quadratic complexity, and multi-head mechanism hinder real-time GPU-like inference.
Method
RTFormer uses a dual-resolution transformer with GPU-Friendly Attention in the low-resolution branch and cross-resolution attention to spread low-resolution context to high-resolution features.
Results
RTFormer achieves state-of-the-art results on Cityscapes, CamVid, and COCOStuff, and shows promising performance on ADE20K.
Takeaways & Limitations
RTFormer provides a transformer-based approach for real-time semantic segmentation that targets a better performance–efficiency trade-off on GPU-like devices.
Abstract
from arXiv · showhide
Recently, transformer-based networks have shown impressive results in semantic segmentation. Yet for real-time semantic segmentation, pure CNN-based approaches still dominate in this field, due to the time-consuming computation mechanism of transformer. We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency on GPU-like devices, our RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, we find that cross-resolution attention is more efficient to gather global context information for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments on mainstream benchmarks demonstrate the effectiveness of our proposed RTFormer, it achieves state-of-the-art on Cityscapes, CamVid and COCOStuff, and shows promising results on ADE20K. Code is available at PaddleSeg: https://github.com/PaddlePaddle/PaddleSeg.
1 Introduction
Real-time semantic segmentation is dominated by CNNs because transformer attention is costly for GPU-like inference, motivating RTFormer’s efficiency-focused design. RTFormer combines GPU-Friendly Attention and cross-resolution attention to improve the performance–efficiency trade-off and reports strong benchmark results.
- Real-time semantic segmentation supports perception in autonomous driving, mobile applications, and robot sensing, increasing demand for fast execution.
- Transformer architectures capture long-range context useful for semantic segmentation, but attention has seen limited adoption in real-time systems.
- Quadratic attention complexity burdens high-resolution dense prediction, while multi-head mechanisms are also unfriendly to GPU-like inference.
- RTFormer introduces GPU-Friendly Attention for the low-resolution branch and cross-resolution attention for the high-resolution branch.The design targets a better performance–efficiency trade-off on GPU-like devices.
- RTFormer uses attention deeply to exploit global context without losing efficiency in semantic segmentation.
- RTFormer achieves state-of-the-art results on Cityscapes, CamVid, and COCOStuff, with promising performance on ADE20K.
2 Related Work
Related work spans CNN-based semantic segmentation, real-time architectures, attention mechanisms, and transformer-based segmentation. These studies establish the value of multi-resolution processing and global context while highlighting inference challenges for attention.
- Deep-learning segmentation methods improve fully convolutional networks through architectures such as Deeplabv3, PSPNet, SegNet, and HRNet.
- RTFormer’s block combines dual-resolution processing with GPU-Friendly Attention in the low-resolution branch and cross-resolution attention in the high-resolution branch.
- Real-time segmentation methods use multi-resolution cascades, neural architecture search, and two-stream networks to balance accuracy and latency.
- Attention mechanisms model channel information, object relations, and long-range dependencies, while self-attention is limited by quadratic complexity.
- Some methods reform self-attention for linear complexity, but the paper identifies GPU inference friendliness as an unresolved concern.
- Transformer segmentation methods show promising performance, but heavy backbones and high spatial resolution make real-time use difficult.
3 Methodology
RTFormer combines dual-resolution attention with convolutional stages to balance semantic-segmentation performance and GPU-like-device efficiency. Its GPU-Friendly Attention retains linear complexity while cross-resolution attention transfers low-resolution global context to high-resolution features.
- RTFormer block: RTFormer block uses a stepped dual-resolution layout with GPU-Friendly Attention in the low-resolution branch and cross-resolution attention in the high-resolution branch.The block includes attention types alongside feed-forward networks and follows a multi-resolution fusion design.
- GPU-Friendly Attention: GPU-Friendly Attention derives from external attention's linear complexity and removes channel splitting within matrix multiplications to improve GPU-like inference efficiency.The method enlarges external parameters from (M, d′) to (Mg, d) and uses grouped double normalization.
- GPU-Friendly Attention: GFA integrates matrix multiplication while grouped double normalization preserves multiple attention maps, allowing more feature elements to contribute to the final output.It can capture different token relations without retaining the split matrix multiplication of multi-head external attention.
- RTFormer architecture: RTFormer combines convolutional blocks in earlier stages with dual-resolution modules in later stages, maintaining stride 8 for high-resolution branches and adding DAPPM before prediction.The segmentation head fuses DAPPM output with high-resolution features before pixel-level classification.
- Cross-resolution Attention: Cross-resolution attention gathers global context in the high-resolution branch by broadcasting high-level semantic information learned from the low-resolution branch.The cross-feature is computed from the low-resolution feature map through pooling and convolution, with its spatial size determining the number of generated tokens.
- RTFormer architecture: The architecture is instantiated as RTFormer-Slim and RTFormer-Base with configurations specified across five stages.Table 1 records the channel and block configurations for the two variants.
4 Experiments
RTFormer is evaluated across Cityscapes, CamVid, ADE20K, and COCOStuff using mIoU and FPS, with comparisons, qualitative results, and ablations covering attention, FFN, and cross-resolution settings.
- Evaluation setup: Experiments evaluate RTFormer on Cityscapes, CamVid, ADE20K, and COCOStuff, using mIoU for performance and FPS for efficiency.FPS is measured on an RTX 2080Ti without TensorRT acceleration by default.
- Cityscapes and CamVid: 76.3% mIoU at 110.0 FPS makes RTFormer-Slim faster and more accurate than STDC2-Seg75 and DDRNet-23-Slim on Cityscapes.RTFormer-Base reaches 79.3% mIoU at 39.1 FPS, establishing a new state-of-the-art result.
- Cityscapes and CamVid: 82.5% mIoU at 94.0 FPS on CamVid significantly outperforms other real-time methods, including STDC2-Seg with additional Cityscapes pre-training.RTFormer-Slim reaches 81.4 mIoU at 190.7 FPS with 4.8M parameters.
- Generalization capability: RTFormer-Base achieves 42.1% mIoU at 71.4 FPS on ADE20K and 35.3 mIoU at 143.3 FPS on COCOStuff, outperforming compared methods.Qualitative results report better details and context information on ADE20K.
- Attention ablation: GPU-Friendly Attention outperforms tested multi-head external-attention settings and is more efficient than multi-head self-attention with comparable performance.Adding cross-resolution attention improves performance while reducing FPS by less than 2.
- FFN ablation: The proposed FFN outperforms the typical FFN on both mIoU and FPS, making it more suitable when GPU-like-device latency matters.The typical FFN uses two MLP layers and a 3 × 3 depth-wise convolution; the proposed design uses two 3 × 3 convolutions.
- Hyperparameter ablation: An 8 × 8 cross-feature spatial size provides the best RTFormer-Slim trade-off between FPS and mIoU among the tested settings.The study compares 6 × 6, 8 × 8, and 12 × 12 cross-feature sizes.
5 Conclusion
The paper presents RTFormer as an efficient approach for capturing global context in real-time semantic segmentation. It reports state-of-the-art or promising results, while noting that greater parameter efficiency may still be needed for edge-device chips.
- RTFormer efficiently captures global context to improve real-time semantic segmentation performance.
- The method achieves new state-of-the-art results on common real-time segmentation datasets and promising performance on a challenging general-segmentation dataset.
- RTFormer-Slim has 4.8M parameters, but greater parameter efficiency may be needed for edge-device chips.The paper leaves this improvement for future work.
A ImageNet Pre-training
RTFormer variants are pretrained on ImageNet-1K before semantic-segmentation evaluation, and both outperform corresponding DDRNet variants in ImageNet classification.
- RTFormer is composed of convolution blocks and RTFormer blocks containing different attention types, so it is pretrained on ImageNet-1K.
- Both RTFormer-Slim and RTFormer-Base outperform their corresponding DDRNet variants on ImageNet classification.RTFormer-Base achieves the best performance among existing backbones adopted for real-time semantic segmentation.
B More Experiments
The paper extends the attention ablation with additional experimental details and variants, then analyzes the resulting attention combinations in detail.
- The extended ablation supplements experimental details and introduces more attention variants for analysis.
B.1 Experimental Details.
The experiments compare several efficient attention mechanisms and specify branch-specific settings for resolution reduction, heads, groups, and external-attention dimensions. Comparisons also include Linformer and Nyströmformer variants.
- Self-attention shrinks key and value spatial sizes to reduce computation from large input resolutions.The reduction factor is σ = 4 for the high-resolution branch and σ = 1 for the low-resolution branch.
- Multi-head self-attention and external attention use 2 heads in the high-resolution branch and 8 in the low-resolution branch.GPU-Friendly attention uses the same branch-specific counts as groups.
- Multi-head external attention is evaluated with r = [0.125, 0.25, 1] to compare performance and inference-speed trade-offs.At r = 0.25, the low-resolution parameter dimension M is 64, matching the setting in [15].
- The study also compares Linformer attention and Nyströmformer attention with 32 or 64 landmark points.Linformer is reported without hyperparameter modification, while NA(32) and NA(64) vary the number of landmark points.
B.2 Analyses.
The attention ablations show a trade-off between accuracy and efficiency: self-attention reaches 32.7 mIoU but is slowed by quadratic complexity and multiple heads, while GPU-Friendly and cross-resolution attention provide a stronger balance. The combined design improves both accuracy and efficiency, including a reported 41% speed advantage over EA+EA at matched performance.
- 32.7 mIoU is achieved by multi-head self-attention, but its inference speed is not competitive because of quadratic complexity and the multi-head mechanism.
- Multi-head external attention improves inference speed through linear complexity, shared external parameters, and a low parameter dimension M, but its network capacity is limited.
- Linformer and Nyströmformer attain linear complexity, but multi-head computation remains in Linformer and Nyströmformer’s split softmax reconstruction reduces inference efficiency.
- GPU-Friendly attention combines linear complexity and GPU-friendly matrix multiplication with grouped double normalization that preserves diverse-information capacity.
- 41% faster than EA+EA is reported for GFA+GFA at the same performance, while enlarging EA+EA’s base feature dimension raises mIoU to 32.8 and lowers FPS to 134.8.The comparison is based on EA+EA with r = 1, whose FPS drops from 180.8 when C increases from 32 to 36.
- GPU-Friendly attention combined with cross-resolution attention outperforms other attention types and combinations in both accuracy and efficiency.